perf(attn): grid-target KV-split cap for int8-MMA long-context decode#345
perf(attn): grid-target KV-split cap for int8-MMA long-context decode#345fansilas wants to merge 1 commit into
Conversation
skyrocket2026
added
area:runtime
⚪ sparkinfer auto-eval —
|
| metric | value |
|---|---|
| label | eval:none |
| Qwen3.5 score | eval-qwen35:none (pass) |
| Qwen3.6 score | eval-qwen36:none (pass) |
| Qwen3.5 vs same-box main | 282.88 tok/s → +0.6% (+1.8) |
| Qwen3.5 scored decode (128 ctx · 128-context) | 284.66 tok/s |
| Qwen3.5 correctness | top-1 96.8% · KL 0.0195 |
| Qwen3.5 128-token no-regression gate | 284.66 tok/s vs main 282.88 tok/s · pass |
| Qwen3.5 512-context no-regression gate | 281.53 tok/s vs main 280.04 tok/s · pass |
| Qwen3.5 4k-context no-regression gate | 272.64 tok/s vs main 271.88 tok/s · pass |
| Qwen3.6 vs same-box main | 427.46 tok/s → -0.5% (-2.2) |
| Qwen3.6 scored decode (128 ctx · 128-context) | 425.22 tok/s |
| Qwen3.6 correctness | top-1 96.1% · KL 0.0174 |
| Qwen3.6 128-token no-regression gate | 425.22 tok/s vs main 427.46 tok/s · pass |
| Qwen3.6 512-context no-regression gate | 420.41 tok/s vs main 422.8 tok/s · pass |
| Qwen3.6 4k-context no-regression gate | 403.36 tok/s vs main 405.75 tok/s · pass |
| Qwen3.6 16k-context no-regression gate | 379.27 tok/s vs main 381.9 tok/s · pass |
| Qwen3.6 32k-context no-regression gate | 351.92 tok/s vs main 354.31 tok/s · pass |
| Qwen3.5 optimize | eval:none · 284.66 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy | top-1 96.1% · KL 0.0174 · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 128 | 425.17 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 512 | 420.35 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 4k | 403.29 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 16k | 379.78 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 32k | 352.01 tok/s · pass |
| Qwen3.6 optimize | eval:none · 425.22 tok/s · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy | top-1 96.8% · KL 0.0195 · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128 | 284.36 tok/s · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512 | 281.0 tok/s · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k | 272.81 tok/s · pass |
Within the significance gate — no verified speedup over same-box main.
RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.
0954339 to
66650c4
Compare
|
The speedup here is entirely long-context (≥8k) — the split cap is a deliberate no-op below 8k, so the 128-context scored decode reads it as Rebased onto current
Would it be possible to run a longctx eval on this one so the 32k result is scored rather than only guarded? The change is correctness-neutral — the log-sum-exp combine is exact for any split count, and the 128/512/4k guards are unchanged — and |
|
log tail |
skyrocket2026
added
the
needs-rebase
⏸ Merge conflict — rebase before evalThis branch conflicts with Please rebase onto |
skyrocket2026
removed
the
test-on-5090
|
The round's |
… (+11.5% 32k, +8.1% 16k)
|
The round's |
66650c4 to
911e280
Compare
|
Rebased onto current Re-verified same-box on RTX 5090 (170 SM);
The win is entirely ≥8k, so 128-context bidir scoring reads it as |
skyrocket2026
added
test-on-5090
skyrocket2026
added
the
re-evaluate
⚪ sparkinfer auto-eval —
|
| metric | value |
|---|---|
| label | eval:none |
| Qwen3.5 score | eval-qwen35:none (pass) |
| Qwen3.6 score | eval-qwen36:none (pass) |
| Qwen3.5 vs same-box main | 293.34 tok/s → +0.4% (+1.2) |
| Qwen3.5 scored decode (512 ctx · 512-context) | 294.55 tok/s |
| Qwen3.5 correctness | top-1 96.3% · KL 0.0152 |
| Qwen3.5 128-token no-regression gate | 298.85 tok/s vs main 298.46 tok/s · pass |
| Qwen3.5 512-context no-regression gate | 294.55 tok/s vs main 293.34 tok/s · pass |
| Qwen3.5 4k-context no-regression gate | 283.55 tok/s vs main 283.3 tok/s · pass |
| Qwen3.6 vs same-box main | 412.09 tok/s → +0.1% (+0.2) |
| Qwen3.6 scored decode (128 ctx · 128-context) | 412.32 tok/s |
| Qwen3.6 correctness | top-1 97.3% · KL 0.0102 |
| Qwen3.6 128-token no-regression gate | 412.32 tok/s vs main 412.09 tok/s · pass |
| Qwen3.6 512-context no-regression gate | 418.02 tok/s vs main 418.61 tok/s · pass |
| Qwen3.6 4k-context no-regression gate | 402.53 tok/s vs main 402.5 tok/s · pass |
| Qwen3.6 16k-context no-regression gate | 398.8 tok/s vs main 399.25 tok/s · pass |
| Qwen3.6 32k-context no-regression gate | 382.33 tok/s vs main 382.52 tok/s · pass |
| Qwen3.5 optimize | eval:none · 294.55 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy | top-1 97.3% · KL 0.0102 · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 128 | 412.31 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 512 | 418.05 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 4k | 402.54 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 16k | 398.76 tok/s · pass |
| Qwen3.5 optimize — Qwen3.6-35B-A3B 32k | 382.34 tok/s · pass |
| Qwen3.6 optimize | eval:none · 412.32 tok/s · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy | top-1 96.3% · KL 0.0152 · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128 | 298.4 tok/s · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512 | 294.21 tok/s · pass |
| Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k | 283.33 tok/s · pass |
Within the significance gate — no verified speedup over same-box main.
RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.
* dashboard: PR #345 -> eval:none (412.32 tok/s) * dashboard: PR #353 -> eval:XL (427.54 tok/s) * dashboard: PR #353 merged -> bidir frontier update * dashboard: use Hugging Face logo for Qwen3.5 model link Replace the org avatar with the official yellow HF icon so the Qwythos repo link is recognizable on file:// and GitHub Pages.
Closed: draft open for 4+ daysThis PR has remained a draft for 5 days (threshold: 4 days). Recent commits or comments do not reset this timer. Mark ready for review or reopen when you're ready to continue. |
2 participants
Summary
One decode scheduling fix for the int8 tensor-core flash-decode at long context, output-neutral to main (only the KV-work partition changes).
Grid-target n_splits cap. The depth-adaptive KV-split count jumps to 256 at >8k context — a value tuned for the older bf16 tile kernel. For the int8-MMA flash-decode (
fa_split_gqa_mma_i8_kernel, 5 CTAs/SM), 256 over-splits: thenum_kv_heads*256grid spills a partial 2nd occupancy wave (tail), and the per-split chunk shrinks until the QK warps idle (gblk<8). On an nsys node trace the split kernel is ~27% of the 32k-context token and already runs at ~89% of memory bandwidth — so the win is the schedule, not the kernel.This caps
n_splitsto a grid that fills ~3/4 of one occupancy wave:(num_sms*5*3/4)/num_kv_heads. The optimal grid is independent of seqlen — on the RTX 5090 (170 SMs) both 16k and 32k decode peak at grid ~640 (n_splits 159 at 4 KV heads). The cap only bites for higher KV-head counts: hd128 Qwen3-MoE (4 KV) -> 159; hd256 (2 KV -> cap 318, above the 256 tier) and short context (<8k, n_splits 32/128) are unchanged. The online-softmax combine is exact for any split count, so this is correctness-neutral.A/B toggle:
SPARKINFER_NSPLITS=256restores the pre-cap fixed split count for the comparison below.Proof of speedup
sm_120)Decode tok/s (end-to-end,
bench/scripts/bench.sh,Qwen3-30B-A3B-Q4_K_M.gguf, 128 decode tokens):Same-box A/B, both built on current main (v0.4.0), 2 reps each:
128 / 512 / 4k are unchanged — the cap does not engage below 8k (int8 KV + the MMA path start at ctx>=8192). Correctness: the split count is a KV-work partition; the log-sum-exp combine is exact for any n_splits (main already ships 128 at 4k, 256 at 16k), so the attention output is unchanged up to float reduction order.
Relation to open PRs
The n_splits grid-target cap is original — no open PR does it. The only file touched is
runtime/src/models/qwen35.cpp, shared with several open PRs (#329, #308, #294, #285, #283, #272, #267), but the change is a small self-contained addition to the depth-adaptive split block. Added-line containment against every one of those (and every merged/closed PR on this file) is <=14.3% — a single shared}. No kernel changes.