Skip to content

Qualify E5-Mistral-7B, and support multiple reference sets - #108

Merged
bsbodden merged 1 commit into
mainfrom
feat/e5-mistral-embedding-qualification
Aug 7, 2026
Merged

Qualify E5-Mistral-7B, and support multiple reference sets#108
bsbodden merged 1 commit into
mainfrom
feat/e5-mistral-embedding-qualification

Conversation

@bsbodden

@bsbodden bsbodden commented Aug 7, 2026

Copy link
Copy Markdown
Member

Extends the gate past a single hardcoded reference, qualifies one more model, and reports a real divergence the gate caught.

Multiple references

The gate pinned reference-qwen3-embedding-0.6b-q8_0.json, so it could only test one model. References are now listed in references.txt and selected by artifact SHA-256, so a model can only be compared against vectors generated from its own exact bytes. Pointing it at an unknown artifact exits 2 rather than silently comparing against the wrong model.

Qualified

E5-Mistral-7B-Instruct Q4_K_M reproduces llama.cpp 6ea215d17: min cosine 0.9995117, mean 0.9996140, max norm deviation 5.54e-09.

Not qualified, and why

The three Qwen3-Embedding Q4_K_M variants fail. The cause is narrower than "Q4_K is lossy" — holding the model fixed and changing only quantization:

model arch quant min cosine
Qwen3-Embedding-0.6B qwen3 Q8_0 0.99950 pass
Qwen3-Embedding-0.6B qwen3 Q4_K_M 0.99700 fail
Qwen3-Embedding-4B qwen3 Q4_K_M 0.99849 fail
Qwen3-Embedding-8B qwen3 Q4_K_M 0.99797 fail
E5-Mistral-7B llama Q4_K_M 0.99951 pass

Q4_K passes on llama and fails on every qwen3, so neither quantization nor architecture explains it alone.

Ruled out:

  • Tensor type mix. Qwen3-Embedding-4B is {F32:145, Q6_K:37, Q4_K:216}; E5-Mistral is {F32:65, Q6_K:33, Q4_K:193} — same three types.
  • mappedKQuantLongOffsets at true/false/default gave results identical to 7 decimal places.

Remaining difference: qwen3 carries per-layer attn_q_norm/attn_k_norm weights that llama does not — those extra ~80 F32 tensors. Tracked for investigation; it may also affect generative qwen3 Q4_K_M artifacts, which the RAG policy does not test numerically.

Their references are deliberately not committed, since the gate would report a failure for artifacts nobody is publishing.

Verification

:models-bench:check and complianceCheck pass; new tests cover the reference index, digest-based selection, rejection of an unknown artifact, and that every committed reference covers the same probe set.

🤖 Generated with Claude Code

The gate pinned one reference file, so it could only ever test
Qwen3-Embedding-0.6B. References are now listed in an index and selected
by artifact SHA-256, which also means a model can only be compared
against vectors generated from its own exact bytes.

E5-Mistral-7B-Instruct Q4_K_M reproduces llama.cpp 6ea215d17: minimum
cosine 0.9995117, mean 0.9996140, max norm deviation 5.54e-09.

The three Qwen3-Embedding Q4_K_M variants do not, and the cause is
narrower than quantization. Holding the model fixed and changing only
quantization:

  Qwen3-Embedding-0.6B  Q8_0    0.99950  pass
  Qwen3-Embedding-0.6B  Q4_K_M  0.99700  fail
  Qwen3-Embedding-4B    Q4_K_M  0.99849  fail
  Qwen3-Embedding-8B    Q4_K_M  0.99797  fail
  E5-Mistral-7B         Q4_K_M  0.99951  pass

Q4_K passes on llama and fails on every qwen3, so neither quantization
nor architecture explains it alone. Both files carry the same tensor
type mix; qwen3 additionally carries per-layer QK-norm weights that
llama does not, which is where the interaction most likely lives.

Their references are deliberately not committed: the gate would report a
failure for artifacts that are not being published.
@bsbodden
bsbodden merged commit 4c1b27f into main Aug 7, 2026
5 checks passed
@bsbodden
bsbodden deleted the feat/e5-mistral-embedding-qualification branch August 7, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant