Gate releases on embedding equivalence with llama.cpp

[bsbodden]

Certifies that this runtime reproduces an embedding model, rather than grading the model.

Retrieval quality is a published property of the weights — Qwen3-Embedding-0.6B scores what it scores whoever runs it. What a runtime can get wrong is reproducing the model: pooling, rotary embeddings, dequantization, normalization. That has an unambiguous correct answer, so it is what this gates on. It also sidesteps inventing a retrieval threshold, which would be dataset-dependent and arbitrary.

Shape

./gradlew :models-bench:run --args="embedding-equivalence --model <artifact.gguf> [--report out.json]"

Eight probes chosen to exercise the paths a runtime gets wrong — single token, long input, non-Latin script, code, rare tokens, ordinary prose. No corpus, no retrieval metric, no scoring. Reference vectors are committed rather than recomputed, so only this side runs: 5.6s end to end, and no machine running the gate needs a built llama.cpp to re-derive a constant.

Exits 0 reproduced, 1 not reproduced, 2 usage or integrity problem.

Two floors, because cosine alone is not enough

Run	Min cosine	Max ‖v‖ − 1
Correct (this runtime vs the reference)	0.99950	2.7e-09
Mean pooling instead of last-token	0.66156	—
L2 normalization skipped	1.00000	~11
floor	0.999	1e-3

Wrong pooling lands nowhere near the floor, so there is no threshold-tuning judgment call to get wrong.

But cosine is scale-invariant: a runtime that skips L2 normalization agrees with a normalized reference at exactly 1.0. That number is measured via llama-embedding --embd-normalize -1, not assumed. Callers that use a bare dot product as a cosine shortcut — as vectors does — would be silently wrong while the gate reported perfection. Hence the separate unit-length check.

The gate takes the worst probe, not the mean, because averaging lets one broken case hide behind seven good ones.

Integrity

The CLI refuses to run when the probe-set digest or the artifact digest has moved, so a stale reference fails loudly rather than comparing against vectors from different inputs. Verified: pointing it at a different model exits 2 with both digests printed.

Notes

• MINIMUM_COSINE mirrors ModelEmbeddingQualification.MINIMUM_ORACLE_COSINE in ModelJars. Duplicated rather than imported because ModelJars depends on this project, not the reverse; every report writes the value it used, so the two cannot silently diverge.
• Latency is recorded as diagnostics only. Eight probes cannot establish a throughput floor and this gate does not claim to.
• Regeneration path documented in embedding-equivalence/README.md, including why each llama.cpp flag matters.

Unblocks qualifying and publishing the Qwen3-Embedding-0.6B ModelJar.

🤖 Generated with Claude Code