perf(qwen36): Q8_0→Q4_K requant of GDN ssm_out projections (+2.8% @128) by blinkeye-lcm · Pull Request #355 · gittensor-ai-lab/sparkinfer

blinkeye-lcm · 2026-07-12T23:04:16Z

Proof of speedup

Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, qwen3_gguf_bench, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, 128 decode tokens), this PR default vs SPARKINFER_ATTN_REQUANT_Q4K=attn_q,attn_output,qkv,attn_gate (current main), same binary:

	decode tok/s (128)
before (`main`: q/o + GDN-input requant, `ssm_out` native Q8_0)	483.3
after (this PR, default)	496.2

# ctx=128:  main 483.29 -> this PR 496.18  (+2.67%)
# ctx=512:  main 477.36 -> this PR 489.88  (+2.62%)

ssm_out lives on all 30 GDN layers, whose O(1)-state matvecs are a fixed per-token cost independent of KV depth, so the requant helps across contexts. A byte-reducing requant cannot slow decode. Qwythos-9B is unchanged (no-op).

Relation to open PRs

ssm_out is the GDN output projection — complementary to the merged #267 (GDN input projections attn_qkv/attn_gate) and to #353 (full-attention q/o); no PR requantizes it. This PR touches only runtime/src/models/qwen35.cpp (+11/−1): it adds an ssm_out token to the existing requant mode, a per-layer floor env, and ssm_out to the Qwen3.6 default, reusing the merged proj_q4k_lloyd.cu fitter with no kernel changes. Added-line containment against every requant PR on that file is 0.0% (block ≥ 85%, warn ≥ 75%); per-function max 0.0% (warn ≥ 92%).

@128

…attn_gate) (+11.5% @128) Qwen3.6-35B-A3B UD ships the Gated-DeltaNet input projections — attn_qkv (wqkv) and attn_gate (the z gate) — as Q8_0 on all 30 GDN layers. These are the single largest per-token weight read in decode, larger than the full-attention q/o of gittensor-ai-lab#353 or the ssm_out of gittensor-ai-lab#355. Requantizing both to Q4_K once at model load reads ~47% fewer bytes on those matvecs. No decode-kernel change: after the stored type flips Q8_0→Q4_K, the GDN qkv+z projection already branches on tensor type and routes through the existing Q4_K fused kernel (launch_mmvq_gdn_qkv_z_pack2) automatically. The fit reuses the merged Lloyd-max coordinate-descent Q4_K quantizer (proj_q4k_lloyd.cu) — high enough quality to keep all 30 GDN layers inside the accuracy gate (an affine fit does not). Gated to the Qwen3.6 fingerprint; a strict no-op on every other model. Extends the existing SPARKINFER_ATTN_REQUANT_Q4K mode — set it to "attn_q,attn_output" to restore the gittensor-ai-lab#353-only behavior. Decode tok/s (RTX 5090, same-binary A/B vs gittensor-ai-lab#353-only baseline): 128: 437.57 -> 488.08 +11.54% 4096: 415.79 -> 461.19 +10.92% 16384: 410.65 -> 455.78 +10.99% 32768: 394.31 -> 433.94 +10.05% Correctness (natural text, 1500 teacher-forced positions, Q4_K vs Q8_0 baseline): top-1 96.3% (gate >= 0.90), KL 0.021 (gate <= 0.20), PPL 2.820 vs 2.788.

@128

…attn_gate) (+11.5% @128) Qwen3.6-35B-A3B UD ships the Gated-DeltaNet input projections — attn_qkv (wqkv) and attn_gate (the z gate) — as Q8_0 on all 30 GDN layers. These are the single largest per-token weight read in decode, larger than the full-attention q/o of gittensor-ai-lab#353 or the ssm_out of gittensor-ai-lab#355. Requantizing both to Q4_K once at model load reads ~47% fewer bytes on those matvecs. No decode-kernel change: after the stored type flips Q8_0→Q4_K, the GDN qkv+z projection already branches on tensor type and routes through the existing Q4_K fused kernel (launch_mmvq_gdn_qkv_z_pack2) automatically. The fit reuses the merged Lloyd-max coordinate-descent Q4_K quantizer (proj_q4k_lloyd.cu) — high enough quality to keep all 30 GDN layers inside the accuracy gate (an affine fit does not). Gated to the Qwen3.6 fingerprint; a strict no-op on every other model. Extends the existing SPARKINFER_ATTN_REQUANT_Q4K mode — set it to "attn_q,attn_output" to restore the gittensor-ai-lab#353-only behavior. Decode tok/s (RTX 5090, same-binary A/B vs gittensor-ai-lab#353-only baseline): 128: 437.57 -> 488.08 +11.54% 4096: 415.79 -> 461.19 +10.92% 16384: 410.65 -> 455.78 +10.99% 32768: 394.31 -> 433.94 +10.05% Correctness (natural text, 1500 teacher-forced positions, Q4_K vs Q8_0 baseline): top-1 96.3% (gate >= 0.90), KL 0.021 (gate <= 0.20), PPL 2.820 vs 2.788.

skyrocket2026 · 2026-07-13T04:14:10Z

✅ sparkinfer auto-eval — `e05a38a`

metric	value
label	`eval:M`
Qwen3.5 score	`eval-qwen35:none` (pass)
Qwen3.6 score	`eval-qwen36:M` (pass)
Qwen3.5 vs same-box main	294.23 tok/s → +0.1% (+0.3)
Qwen3.5 scored decode (128 ctx · 128-context)	294.55 tok/s
Qwen3.5 correctness	top-1 92.7% · KL 0.0395
Qwen3.5 128-token no-regression gate	294.55 tok/s vs main 294.23 tok/s · pass
Qwen3.5 512-context no-regression gate	291.74 tok/s vs main 291.66 tok/s · pass
Qwen3.5 4k-context no-regression gate	282.8 tok/s vs main 282.68 tok/s · pass
Qwen3.6 vs same-box main	411.16 tok/s → +3.0% (+12.2)
Qwen3.6 scored decode (512 ctx · 512-context)	423.38 tok/s
Qwen3.6 correctness	top-1 92.0% · KL 0.068
Qwen3.6 128-token no-regression gate	429.13 tok/s vs main 417.52 tok/s · pass
Qwen3.6 512-context no-regression gate	423.38 tok/s vs main 411.16 tok/s · pass
Qwen3.6 4k-context no-regression gate	404.99 tok/s vs main 394.87 tok/s · pass
Qwen3.6 16k-context no-regression gate	402.54 tok/s vs main 392.58 tok/s · pass
Qwen3.6 32k-context no-regression gate	385.69 tok/s vs main 376.21 tok/s · pass
Qwen3.5 optimize	`eval:none` · 294.55 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy	top-1 92.0% · KL 0.068 · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 128	429.09 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 512	423.44 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 4k	404.85 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 16k	402.43 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 32k	385.45 tok/s · pass
Qwen3.6 optimize	`eval:M` · 423.38 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy	top-1 92.7% · KL 0.0395 · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128	294.43 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512	291.68 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k	282.7 tok/s · pass

Verified speedup over same-box origin/main — 423.38 tok/s (main was 411.16 tok/s).

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

@128

…attn_gate) (+11.5% @128) (#267) * perf(qwen36): Q8_0→Q4_K requant of GDN input projections (attn_qkv + attn_gate) (+11.5% @128) Qwen3.6-35B-A3B UD ships the Gated-DeltaNet input projections — attn_qkv (wqkv) and attn_gate (the z gate) — as Q8_0 on all 30 GDN layers. These are the single largest per-token weight read in decode, larger than the full-attention q/o of #353 or the ssm_out of #355. Requantizing both to Q4_K once at model load reads ~47% fewer bytes on those matvecs. No decode-kernel change: after the stored type flips Q8_0→Q4_K, the GDN qkv+z projection already branches on tensor type and routes through the existing Q4_K fused kernel (launch_mmvq_gdn_qkv_z_pack2) automatically. The fit reuses the merged Lloyd-max coordinate-descent Q4_K quantizer (proj_q4k_lloyd.cu) — high enough quality to keep all 30 GDN layers inside the accuracy gate (an affine fit does not). Gated to the Qwen3.6 fingerprint; a strict no-op on every other model. Extends the existing SPARKINFER_ATTN_REQUANT_Q4K mode — set it to "attn_q,attn_output" to restore the #353-only behavior. Decode tok/s (RTX 5090, same-binary A/B vs #353-only baseline): 128: 437.57 -> 488.08 +11.54% 4096: 415.79 -> 461.19 +10.92% 16384: 410.65 -> 455.78 +10.99% 32768: 394.31 -> 433.94 +10.05% Correctness (natural text, 1500 teacher-forced positions, Q4_K vs Q8_0 baseline): top-1 96.3% (gate >= 0.90), KL 0.021 (gate <= 0.20), PPL 2.820 vs 2.788. * re-eval: verify Qwen3.6 accuracy passes after rebase to ce33e7f Re-benchmarked on RTX 5090 (75.152.195.46). vs #353-only main on same box: 128: 438.9 -> 489.3 tok/s (+11.5%) 512: 432.2 -> 481.7 tok/s (+11.4%) 4k: 415.7 -> 461.2 tok/s (+10.9%) 16k: 411.0 -> 455.6 tok/s (+10.9%) 32k: 394.2 -> 433.6 tok/s (+10.0%) Correctness (fuzzed prompt, teacher-forced vs llama.cpp): top-1 93.6%, KL 0.026 (gate >= 90%, <= 0.20). Full evaluate.sh + bidir harness pass locally. Prior eval:REJECT (top1=0, kl=99) was accuracy.sh infra failure, not model regression.

skyrocket2026 · 2026-07-13T04:14:33Z

The round's merge-first PR was just merged. Please rebase this branch onto main — the bot re-evaluates it on push (crediting your marginal gain on top of what merged).

blinkeye-lcm · 2026-07-13T04:35:50Z

Rebased onto main (now includes #267's GDN input requant). ssm_out is the GDN output projection — complementary to #267 (input attn_qkv/attn_gate) and #353 (full-attn q/o), so it stacks additively: +2.67% @128 / +2.62% @512 marginal over the current frontier, correctness holds at top-1 92.5% / KL 0.07. qwen35.cpp-only (+11/−1), no kernel changes — reuses the merged proj_q4k_lloyd.cu fitter. Ready for re-eval.

skyrocket2026 · 2026-07-13T07:33:54Z

⚠️ sparkinfer auto-eval errored for e7b37b6 — re-run manually.

log tail

k-baseline P_LLAMA_32K_BASELINE] [--bidir]
                    [--p35-guard-128-baseline P35_GUARD_128_BASELINE]
                    [--p35-guard-512-baseline P35_GUARD_512_BASELINE]
                    [--p35-guard-4k-baseline P35_GUARD_4K_BASELINE]
                    [--g35-guard-128-baseline G35_GUARD_128_BASELINE]
                    [--g35-guard-512-baseline G35_GUARD_512_BASELINE]
                    [--g35-guard-4k-baseline G35_GUARD_4K_BASELINE] [--triple]
                    [--primary-quant {Q4_K_M,Q8_0,BF16}]
                    [--g36-guard-128-baseline G36_GUARD_128_BASELINE]
                    [--g36-guard-512-baseline G36_GUARD_512_BASELINE]
                    [--g36-guard-4k-baseline G36_GUARD_4K_BASELINE]
                    [--g36-guard-16k-baseline G36_GUARD_16K_BASELINE]
                    [--g36-guard-32k-baseline G36_GUARD_32K_BASELINE]
                    [--eval-mode {longctx,short}] [--reuse REUSE]
                    [--ssh HOST:PORT] [--keep] [--destroy] [--gpu GPU]
                    [--image IMAGE] [--reuse-timeout REUSE_TIMEOUT]
                    [--new-timeout NEW_TIMEOUT] [--no-recreate] [--pinned]
                    [--destroy-on-error] [--polaris] [--no-polaris]
                    [--baseline-only]
vast_eval.py: error: unrecognized arguments: --p35-guard-32k-baseline 241.17 --p35-guard-64k-baseline 204.26 --p35-guard-128k-baseline 159.13 --g35-guard-32k-baseline 241.17 --g35-guard-64k-baseline 204.26 --g35-guard-128k-baseline 159.13

skyrocket2026 · 2026-07-13T07:34:16Z