perf(qwen36): Q8_0→Q4_K requant of GDN input projections (attn_qkv + attn_gate) (+11.5% @128) by inference2026 · Pull Request #267 · gittensor-ai-lab/sparkinfer

inference2026 · 2026-07-06T09:14:18Z

Summary

Load-time Q8_0 → Q4_K requant of the Qwen3.6 Gated-DeltaNet input projections — attn_qkv (wqkv) and attn_gate (the z gate), shipped Q8_0 on all 30 GDN layers. These are the single largest per-token weight read in decode — larger than the full-attention q/o requantized by #353 or the ssm_out of #355. This extends the merged #353 requant path to these two tensors, reading ~47% fewer bytes on those matvecs. Gated to the Qwen3.6 fingerprint; a strict no-op on every other model.

Proof of speedup

Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, qwen3_gguf_bench, Qwen3.6-35B-A3B UD-Q4_K_M, same-box A/B):

	decode tok/s
before (main)	437.70
after (this PR)	487.96

# Qwen3.6-35B-A3B-UD-Q4_K_M — RTX 5090, same-binary A/B via env toggle
#   before = SPARKINFER_ATTN_REQUANT_Q4K=attn_q,attn_output   (main / #353-only, GDN input stays Q8_0)
#   after  = default                                          (adds qkv,attn_gate)
ctx=128    before 437.70   after 487.96   +11.48%
ctx=4096   before 415.79   after 461.19   +10.92%
ctx=16384  before 410.65   after 455.78   +10.99%
ctx=32768  before 394.31   after 433.94   +10.05%

Correctness

The GDN input projections feed the recurrent state, so the lossy Q4_K fit was validated carefully. Natural text (Project-Gutenberg-style corpus), 1500 teacher-forced positions, this PR (Q4_K) vs main (Q8_0):

top-1 agreement : 1444/1499 = 96.3%   (gate >= 0.90)
KL(Q8_0 || Q4_K): 0.021                (gate <= 0.20)
PPL             : 2.820  vs  2.788 (Q8_0)   (+1.15%)

All 30 GDN layers stay inside the accuracy gate with ~10x margin because the fit reuses the merged Lloyd-max coordinate-descent Q4_K quantizer (proj_q4k_lloyd.cu, from #353). A plain affine fit is not accuracy-safe at full layer coverage on this tensor. SPARKINFER_ATTN_REQUANT_Q4K=attn_q,attn_output restores the #353-only behavior for A/B.

Changes

runtime/src/models/qwen35.cpp only (+6/-1): extends the Qwen3.6 default SPARKINFER_ATTN_REQUANT_Q4K mode to attn_q,attn_output,qkv,attn_gate and adds the attn_gate handler to req_attn_q4. No new kernel — after the stored type flips to Q4_K, the GDN qkv+z projection site already branches on tensor type and routes through the existing launch_mmvq_gdn_qkv_z_pack2 (Q4_K fused) path automatically.

Test plan

Build qwen3_gguf_bench + qwen3_gguf_score on RTX 5090 (sm_120)
Same-box A/B at 128/4096/16384/32768 ctx (default vs attn_q,attn_output)
Natural-text top-1/KL vs the Q8_0 baseline over 1500 teacher-forced positions
No-op confirmed on non-matching models (gated to the Qwen3.6 fingerprint)

ai-hpc · 2026-07-06T10:37:09Z

⚪ sparkinfer auto-eval — `20df85f`

metric	value
label	`eval:none`
scored decode (512 ctx · 512-context · Qwen3.6)	252.23 tok/s
vs same-box main	251.87 tok/s → +0.1% (+0.4)
correctness (Qwen3.6 vs llama.cpp)	top-1 97.1% · KL 0.0161
Qwen3.6 128-token no-regression gate	255.02 tok/s vs main 254.72 tok/s · pass
Qwen3.6 512-context no-regression gate	252.23 tok/s vs main 251.87 tok/s · pass
Qwen3.6 4k-context no-regression gate	239.27 tok/s vs main 239.0 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 95.3% · KL 0.0196 · pass
Qwen3-30B-A3B guard — 128-token	498.01 tok/s · pass
Qwen3-30B-A3B guard — 512-context	473.04 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	395.27 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	331.54 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	262.01 tok/s · pass

Within the significance gate — no verified speedup over same-box main.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

ai-hpc · 2026-07-06T12:56:47Z

✅ sparkinfer auto-eval — `c42844c`

metric	value
label	`eval:M`
scored decode (128 ctx · 128-context · Qwen3.6)	278.76 tok/s
vs same-box main	254.72 tok/s → +9.4% (+24.0)
correctness (Qwen3.6 vs llama.cpp)	top-1 97.7% · KL 0.0165
Qwen3.6 128-token no-regression gate	278.76 tok/s vs main 254.72 tok/s · pass
Qwen3.6 512-context no-regression gate	275.13 tok/s vs main 251.87 tok/s · pass
Qwen3.6 4k-context no-regression gate	259.22 tok/s vs main 239.0 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 96.3% · KL 0.0337 · pass
Qwen3-30B-A3B guard — 128-token	481.04 tok/s · pass
Qwen3-30B-A3B guard — 512-context	458.7 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	385.49 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	326.91 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	254.88 tok/s · pass

Verified speedup over same-box origin/main — 278.76 tok/s (main was 254.72 tok/s).

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

ai-hpc · 2026-07-06T13:06:21Z

The round's merge-first PR was just merged. Please rebase this branch onto main — the bot re-evaluates it on push (crediting your marginal gain on top of what merged).

ai-hpc · 2026-07-06T14:07:16Z

The round's merge-first PR was just merged. Please rebase this branch onto main — once you push the rebase the bot re-evaluates it against the new frontier (crediting your marginal gain on top of what merged).

skyrocket2026 · 2026-07-13T02:21:35Z

❌ sparkinfer auto-eval — `e36ea94`

metric	value
label	`eval:REJECT`
Qwen3.5 score	`eval-qwen35:REJECT` (fail)
Qwen3.6 score	`eval-qwen36:REJECT` (fail)
Qwen3.5 vs same-box main	291.58 tok/s → +0.1% (+0.2)
Qwen3.5 scored decode (512 ctx · 512-context)	291.74 tok/s
Qwen3.5 correctness	top-1 95.5% · KL 0.0269
Qwen3.5 128-token no-regression gate	294.53 tok/s vs main 294.41 tok/s · pass
Qwen3.5 512-context no-regression gate	291.74 tok/s vs main 291.58 tok/s · pass
Qwen3.5 4k-context no-regression gate	282.76 tok/s vs main 282.76 tok/s · pass
Qwen3.6 vs same-box main	411.07 tok/s → +0.0% (+0.0)
Qwen3.6 scored decode (512 ctx · 512-context)	456.42 tok/s
Qwen3.6 correctness	top-1 0.0% · KL 99.0
Qwen3.6 128-token no-regression gate	463.37 tok/s vs main 417.33 tok/s · pass
Qwen3.6 512-context no-regression gate	456.42 tok/s vs main 411.07 tok/s · pass
Qwen3.6 4k-context no-regression gate	435.23 tok/s vs main 394.76 tok/s · pass
Qwen3.6 16k-context no-regression gate	432.01 tok/s vs main 392.56 tok/s · pass
Qwen3.6 32k-context no-regression gate	413.18 tok/s vs main 376.4 tok/s · pass
Qwen3.5 optimize	`eval:REJECT` · 291.74 tok/s · fail
Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy	top-1 0.0% · KL 99.0 · FAIL
Qwen3.5 optimize — Qwen3.6-35B-A3B 128	463.39 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 512	456.46 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 4k	435.11 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 16k	432.03 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 32k	413.24 tok/s · pass
Qwen3.6 optimize	`eval:REJECT` · 456.42 tok/s · fail
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy	top-1 95.5% · KL 0.0269 · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128	294.48 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512	291.66 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k	282.72 tok/s · pass

Rejected — correctness gate: top1=0.0 (need >= 0.9), kl=99.0 (need <= 0.2).

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

Re-benchmarked on RTX 5090 (75.152.195.46). vs gittensor-ai-lab#353-only main on same box: 128: 438.9 -> 489.3 tok/s (+11.5%) 512: 432.2 -> 481.7 tok/s (+11.4%) 4k: 415.7 -> 461.2 tok/s (+10.9%) 16k: 411.0 -> 455.6 tok/s (+10.9%) 32k: 394.2 -> 433.6 tok/s (+10.0%) Correctness (fuzzed prompt, teacher-forced vs llama.cpp): top-1 93.6%, KL 0.026 (gate >= 90%, <= 0.20). Full evaluate.sh + bidir harness pass locally. Prior eval:REJECT (top1=0, kl=99) was accuracy.sh infra failure, not model regression.

skyrocket2026 · 2026-07-13T03:53:32Z

✅ sparkinfer auto-eval — `ca7028c`

metric	value
label	`eval:XL`
Qwen3.5 score	`eval-qwen35:none` (pass)
Qwen3.6 score	`eval-qwen36:XL` (pass)
Qwen3.5 vs same-box main	294.23 tok/s → +0.1% (+0.2)
Qwen3.5 scored decode (128 ctx · 128-context)	294.39 tok/s
Qwen3.5 correctness	top-1 93.5% · KL 0.0353
Qwen3.5 128-token no-regression gate	294.39 tok/s vs main 294.23 tok/s · pass
Qwen3.5 512-context no-regression gate	291.62 tok/s vs main 291.66 tok/s · pass
Qwen3.5 4k-context no-regression gate	282.74 tok/s vs main 282.68 tok/s · pass
Qwen3.6 vs same-box main	411.16 tok/s → +11.0% (+45.3)
Qwen3.6 scored decode (512 ctx · 512-context)	456.42 tok/s
Qwen3.6 correctness	top-1 95.3% · KL 0.031
Qwen3.6 128-token no-regression gate	463.27 tok/s vs main 417.52 tok/s · pass
Qwen3.6 512-context no-regression gate	456.42 tok/s vs main 411.16 tok/s · pass
Qwen3.6 4k-context no-regression gate	435.14 tok/s vs main 394.87 tok/s · pass
Qwen3.6 16k-context no-regression gate	432.1 tok/s vs main 392.58 tok/s · pass
Qwen3.6 32k-context no-regression gate	413.14 tok/s vs main 376.21 tok/s · pass
Qwen3.5 optimize	`eval:none` · 294.39 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy	top-1 95.3% · KL 0.031 · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 128	463.25 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 512	456.36 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 4k	435.14 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 16k	432.18 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 32k	413.21 tok/s · pass
Qwen3.6 optimize	`eval:XL` · 456.42 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy	top-1 93.5% · KL 0.0353 · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128	294.36 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512	291.64 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k	282.75 tok/s · pass

Verified speedup over same-box origin/main — 456.42 tok/s (main was 411.16 tok/s).

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

skyrocket2026 · 2026-07-13T04:14:29Z

✅ Auto-merged as the round's merge-first winner — verified same-box speedup over main, all checks green. Thanks for the contribution!

PR #267 was rewritten but data.json kept the July 6 eval, so record_merge never advanced landed_qwen36. Backfill the XL bidir eval, restore baseline_tps, and use ctx_128_tps for Qwen3.6 journey steps.

Backfill left polaris_receipt_url unset even though 0267-ca7028c has a TDX receipt; the proof column showed clipboard instead of shield. Teach backfill to pull polaris fields from sparkinfer-log receipt.json.

ai-hpc added area:kernels subsystem (emission weight 0.42) test-on-5090 Maintainer-approved to evaluate on RTX 5090 (greenlight) eval:none sparkinfer auto-eval verdict: none 512-context UI-only: strongest measured context in sparkinfer eval labels Jul 6, 2026

ai-hpc added a commit that referenced this pull request Jul 6, 2026

dashboard: PR #267 -> eval:none (252.23 tok/s)

a47a5d1

inference2026 force-pushed the opt-f512-gateup branch 4 times, most recently from c42844c to d771b0e Compare July 6, 2026 12:20

ai-hpc added the area:runtime subsystem (emission weight 0.26) label Jul 6, 2026

inference2026 marked this pull request as draft July 6, 2026 12:29

inference2026 force-pushed the opt-f512-gateup branch 4 times, most recently from 99f613b to 0af2761 Compare July 6, 2026 12:54

ai-hpc added eval:M sparkinfer auto-eval verdict: M and removed eval:none sparkinfer auto-eval verdict: none 512-context UI-only: strongest measured context in sparkinfer eval labels Jul 6, 2026

ai-hpc added the 128-context UI-only: strongest measured context in sparkinfer eval label Jul 6, 2026

ai-hpc added a commit that referenced this pull request Jul 6, 2026

dashboard: PR #267 -> eval:M (278.76 tok/s)

a324750

ai-hpc added the needs-rebase Verified speedup but not the round winner — rebase after merge-first lands label Jul 6, 2026

inference2026 force-pushed the opt-f512-gateup branch 4 times, most recently from 19b2c7f to 374d05e Compare July 6, 2026 13:56

fansilas mentioned this pull request Jul 6, 2026

perf(qwen3.6): fused router GEMV+top-k + folded residual-norm (decode) #273

Closed

1 task

inference2026 force-pushed the opt-f512-gateup branch from e0e6d65 to e36ea94 Compare July 13, 2026 01:41

skyrocket2026 added eval:XL sparkinfer auto-eval verdict: XL eval-qwen35:none eval-qwen36:XL and removed eval:REJECT sparkinfer auto-eval verdict: REJECT eval-qwen35:REJECT eval-qwen36:REJECT labels Jul 13, 2026

skyrocket2026 added merge-first Round's biggest verified speedup — merge this first and removed re-evaluate Winner merged — rebase onto main; bot re-evaluates on push labels Jul 13, 2026

skyrocket2026 merged commit 3294d9a into gittensor-ai-lab:main Jul 13, 2026
1 check passed

blinkeye-lcm mentioned this pull request Jul 13, 2026

perf(qwen36): Q8_0→Q4_K requant of GDN ssm_out projections (+2.8% @128) #355

Merged

1 task

skyrocket2026 mentioned this pull request Jul 13, 2026

fix(dashboard): backfill PR #267 Qwen3.6 journey after merge #367

Merged

2 tasks

skyrocket2026 mentioned this pull request Jul 13, 2026

fix(dashboard): show TDX shield for PR #267 eval proof #368

Merged

2 tasks

skyrocket2026 removed the merge-first Round's biggest verified speedup — merge this first label Jul 13, 2026

blinkeye-lcm pushed a commit to blinkeye-lcm/sparkinfer that referenced this pull request Jul 13, 2026

dashboard: PR gittensor-ai-lab#267 merged -> bidir frontier update

91b0ed5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(qwen36): Q8_0→Q4_K requant of GDN input projections (attn_qkv + attn_gate) (+11.5% @128)#267

perf(qwen36): Q8_0→Q4_K requant of GDN input projections (attn_qkv + attn_gate) (+11.5% @128)#267
skyrocket2026 merged 2 commits into
gittensor-ai-lab:mainfrom
inference2026:opt-f512-gateup

inference2026 commented Jul 6, 2026 •

edited

Loading

Uh oh!

ai-hpc commented Jul 6, 2026

Uh oh!

ai-hpc commented Jul 6, 2026

Uh oh!

ai-hpc commented Jul 6, 2026

Uh oh!

ai-hpc commented Jul 6, 2026

Uh oh!

skyrocket2026 commented Jul 13, 2026

Uh oh!

skyrocket2026 commented Jul 13, 2026

Uh oh!

Uh oh!

skyrocket2026 commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

inference2026 commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of speedup

Correctness

Changes

Test plan

Uh oh!

ai-hpc commented Jul 6, 2026

⚪ sparkinfer auto-eval — 20df85f

Uh oh!

ai-hpc commented Jul 6, 2026

✅ sparkinfer auto-eval — c42844c

Uh oh!

ai-hpc commented Jul 6, 2026

Uh oh!

ai-hpc commented Jul 6, 2026

Uh oh!

skyrocket2026 commented Jul 13, 2026

❌ sparkinfer auto-eval — e36ea94

Uh oh!

skyrocket2026 commented Jul 13, 2026

✅ sparkinfer auto-eval — ca7028c

Uh oh!

Uh oh!

skyrocket2026 commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

inference2026 commented Jul 6, 2026 •

edited

Loading

⚪ sparkinfer auto-eval — `20df85f`

✅ sparkinfer auto-eval — `c42844c`

❌ sparkinfer auto-eval — `e36ea94`

✅ sparkinfer auto-eval — `ca7028c`