Offer a half-precision KV cache behind a flag (default off)

[bsbodden]

llama.cpp holds cache_k/cache_v as F16, so attention there reads back rounded keys and values. This runtime keeps F32 — more precise than the reference rather than equivalent to it.

How it was localized

Replayed layer 0 of Qwen3-Embedding-0.6B Q6_K op by op against llama-eval-callback. Every matmul already matched — Q6_K dequant is bit-exact, q8_K activation quantization matches -127.f/max with upper-only clamp, and the Q6_K×Q8_K dot matches to 3e-8 at both cols=1024 and cols=2048.

The first divergence was node_27, the attention output projection. Its only differing input was V:

	F32 V	F16 V	llama.cpp
node_27	0.635219	0.619453	0.619455
l_out-0	−7.648265	−7.676750	−7.676746

Why it is off by default

Two llama.cpp equivalence references disagree about this change.

Rounding reproduces Qwen3 layer output exactly and raises embedding agreement everywhere measured:

model	quant	default	flag on
Qwen3-Embedding-0.6B	Q8_0	0.9995014	0.9996100
Qwen3-Embedding-0.6B	Q4_K_M	0.9969991	0.9977295
Qwen3-Embedding-0.6B	Q6_K	0.9969038	0.9970416

But it flips one greedy token on the pinned SQLCoder Q5_K_M fixture, which asserts token equality against llama.cpp b9960:

actual   [13, 917, 29901, 4576]
expected [13, 917, 29901,  274]

Since one llama.cpp reference says the rounding is right and another says it is wrong, it cannot be the default until that is resolved. Default behaviour is unchanged, so no recorded evidence moves and the full slow-model suite is unaffected. Select it with -Dmodels.purejava.halfPrecisionKvCache=true.

Scope

This does not on its own lift the qwen3 K-quant variants over the 0.999 floor — 0.99773 is closer but still short. A second source remains, tracked in the follow-up task.

Separately, this work found that :models-bench:run does not forward arbitrary -D system properties (only models.native.kernels.library), so any tuning-flag experiment run through Gradle run silently had no effect. The toggle sweep was re-done with direct java -cp invocation; the conclusions held, but the earlier method was invalid.

🤖 Generated with Claude Code