Qualify E5-Mistral-7B, and support multiple reference sets - #108
Merged
Conversation
The gate pinned one reference file, so it could only ever test Qwen3-Embedding-0.6B. References are now listed in an index and selected by artifact SHA-256, which also means a model can only be compared against vectors generated from its own exact bytes. E5-Mistral-7B-Instruct Q4_K_M reproduces llama.cpp 6ea215d17: minimum cosine 0.9995117, mean 0.9996140, max norm deviation 5.54e-09. The three Qwen3-Embedding Q4_K_M variants do not, and the cause is narrower than quantization. Holding the model fixed and changing only quantization: Qwen3-Embedding-0.6B Q8_0 0.99950 pass Qwen3-Embedding-0.6B Q4_K_M 0.99700 fail Qwen3-Embedding-4B Q4_K_M 0.99849 fail Qwen3-Embedding-8B Q4_K_M 0.99797 fail E5-Mistral-7B Q4_K_M 0.99951 pass Q4_K passes on llama and fails on every qwen3, so neither quantization nor architecture explains it alone. Both files carry the same tensor type mix; qwen3 additionally carries per-layer QK-norm weights that llama does not, which is where the interaction most likely lives. Their references are deliberately not committed: the gate would report a failure for artifacts that are not being published.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the gate past a single hardcoded reference, qualifies one more model, and reports a real divergence the gate caught.
Multiple references
The gate pinned
reference-qwen3-embedding-0.6b-q8_0.json, so it could only test one model. References are now listed inreferences.txtand selected by artifact SHA-256, so a model can only be compared against vectors generated from its own exact bytes. Pointing it at an unknown artifact exits2rather than silently comparing against the wrong model.Qualified
E5-Mistral-7B-Instruct Q4_K_M reproduces llama.cpp
6ea215d17: min cosine 0.9995117, mean 0.9996140, max norm deviation 5.54e-09.Not qualified, and why
The three Qwen3-Embedding Q4_K_M variants fail. The cause is narrower than "Q4_K is lossy" — holding the model fixed and changing only quantization:
Q4_K passes on
llamaand fails on everyqwen3, so neither quantization nor architecture explains it alone.Ruled out:
{F32:145, Q6_K:37, Q4_K:216}; E5-Mistral is{F32:65, Q6_K:33, Q4_K:193}— same three types.mappedKQuantLongOffsetsattrue/false/default gave results identical to 7 decimal places.Remaining difference: qwen3 carries per-layer
attn_q_norm/attn_k_normweights that llama does not — those extra ~80 F32 tensors. Tracked for investigation; it may also affect generative qwen3 Q4_K_M artifacts, which the RAG policy does not test numerically.Their references are deliberately not committed, since the gate would report a failure for artifacts nobody is publishing.
Verification
:models-bench:checkandcomplianceCheckpass; new tests cover the reference index, digest-based selection, rejection of an unknown artifact, and that every committed reference covers the same probe set.🤖 Generated with Claude Code