Lack of Instruction & Different System Prompt in Qwen3 Reranker

[wdhwg001]

EmbedAnything/rust/src/reranker/model.rs

Line 158 in 95f56eb

let prefix = "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be yes or no.<|im_end|>\n<|im_start|>user\n";

According to the page: https://huggingface.co/Qwen/Qwen3-Reranker-0.6B

Our approach:

        let prefix = "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be yes or no.<|im_end|>\n<|im_start|>user\n";
        let suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n";

is slightly different from the reference implementation:

prefix = "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n"
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"

the yes and no are not quoted, which may cause slightly lower probability scores. Given that the reranker is trained with exactly the same system prompt, it's best to keep it perfectly aligned with the official approach.

Also, there's a larger issue:

        let pairs = queries
            .iter()
            .flat_map(|query| documents.iter().map(move |doc| (*query, *doc)))
            .collect::<Vec<_>>();

        let pairs = if is_qwen3 {
            pairs.iter().map(|(query, doc)| (format!("{}{}", prefix, query), format!("{}{}", doc, suffix))).collect::<Vec<_>>()
        } else {
            pairs.iter().map(|(query, doc)| (query.to_string(), doc.to_string())).collect::<Vec<_>>()
        };

And correct me if I'm wrong, the pair then get sent to be tokenized straightaway, without further formatting.

However, in the official reference implementation:

def format_instruction(instruction, query, doc):
    if instruction is None:
        instruction = 'Given a web search query, retrieve relevant passages that answer the query'
    output = "<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}".format(instruction=instruction,query=query, doc=doc)
    return output

If it's not formatted in the way how it was trained, the LLM can only try to split the query and document based on its semantic understanding, since there's no structural format to split the query and doc.

A workaround is to do the formatting before passing into EmbedAnything (by combining instruction & query as a new query, and also formatting the doc), but I would really love to see if there could be an official support.

[sonam-pankaj95]

Hey @wdhwg001

First of all, thank you so much for bringing our attention to this.

For the first issue, it could have been updated by Hugging Face later.
For the second issue: The idea was for the user themselves to format it. It's hard to adapt for the user to provide the instruction.

It would be great if you could raise a pr in the dev branch to fix this!!

We would quickly merge it.

CC: @akshayballal95

Thanks.

[wdhwg001]

#177