perf(qwen35): use in-place GDN state write in pure AR#473
Open
dusterbloom wants to merge 9 commits into
Open
Conversation
This was referenced Jun 30, 2026
In build_delta_net_block, replace the two-op silu(z_4d) + mul(output, z_silu) pattern with ggml_swiglu_split(z_4d, output_n). swiglu_split(a,b)=silu(a)*b, mathematically identical. Removes 30 kernel launches per AR decode step (UNARY-30, MUL-30, GLU+30; graph 2744->2714). Token-identical at temp=0 confirmed. Part 1 of launch-count gap closure (30/162 = 18.5% of gap).
…re AR FWHT gate: DFLASH_NO_WHT=1 disables the K/Q rotation that costs ~16 extra kernels/token (~3% decode). q4_0 KV doesn't need the outlier spreading on Ampere. Default off for TQ3_0 (WHT baked into quant). Repeat_back skip: in pure AR (no tree/capture), the SSM kernel's broadcast handles the head mismatch natively — skip the repeat_back copy per DeltaNet layer per step (~1% decode).
The DeltaNet conv_input concat (conv_states_r, qkv_T) hit the slow concat_f32_dim0 kernel (15us) because both inputs were non-contiguous (reshape + transpose). Wrapping in ggml_cont routes to the fast concat_cont path (3.3us) — ~0.5ms/token saved in the 30-layer serial DeltaNet recurrence.
…n FA bucket build_target_step rebuilds the 2744-node cgraph every decode step, costing ~0.38ms/token and resetting ggml-cuda's CUDA-graph warmup counter. The only per-step topology change is win_len_padded = round_up(committed+1, 256), which only advances every 256 steps. Gate the rebuild on committed/256; within a bucket reuse the cached graph and update only mutable inputs. DFLASH_AR_NO_REUSE=1 restores the per-step rebuild. Measured: +6% decode tok/s (106→113) on Q3_K_XL, q4_0 KV, 16K ctx.
Move ggml_cont from the concat call to the qkv_T transpose definition. This makes qkv_T contiguous once (1 cont per DeltaNet layer) instead of wrapping both concat inputs (2 conts per layer). conv_states_r is already contiguous (reshape of contiguous cache tensor). Net: -30 graph nodes (2704→2674), compute -24µs/step.
dusterbloom
force-pushed
the
perf/qwen35-gdn-inplace-consumer
branch
from
6309f28 to
9d45062
Compare
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…se; K-basis in disk-cache salt; CPU GDN inplace guard
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 8 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dense (27B) qwen35 AR-decode performance stack: in-place GDN state write, 256-token FA-bucket CUDA-graph reuse (with a props-check skip), FWHT K-rotation gate for q4_0 KV, swiglu fuse, and concat fast-paths. Rebased clean onto current
main.Supersedes #476, whose micro-opts are a subset of this stack.
Dependency (resolved)
The in-place GDN op itself (
ggml_gated_delta_net_inplace, originally Lucebox ggml #24/#27) is now vendored intomainvia #499 (5e302cbb, "port(ggml): in-place GDN state write + qwen dense decode recovery").mainships the op but still calls the non-inplace variant at the qwen call site (qwen35_target_graph.cpp:941); this PR wires the qwen35 dense pure-AR consumer to use it (ce2325aa). After the ggml-only vendorize (#486), there is no external submodule bump left — the earlier "draft until the ggml submodule commit is available" blocker no longer applies. This branch was rebased onto currentmain, dropping the obsolete submodule-gitlink bumps.Commits
ce2325aause in-place GDN state write in pure ARe1ce9c73fuse silu+mul → swiglu_split in DeltaNet gated output96fd3ea9gate FWHT rotation for q4_0 KV + skip repeat_back in pure AR5fd7c3fbcont concat inputs for fast concat_cont patha5915e29graph reuse in AR decode — skip rebuild within 256-token FA bucketa8253730reduce concat cont overhead — materialize qkv_T once9d450624recover dense AR replay fast pathCorrectness (verified)
DFLASH_AR_NO_REUSEON vs OFF → byte-identical greedy output at 4K and 16K (16K crosses two bucket boundaries). The 256-bucket rebuild guard re-captures exactly when the FA window advances; the props-check skip is a launch-overhead optimization only, not a numerics change.ce2325aaA/B with WHT held constant → byte-identical. Output-inert when live.Performance
27B dense, q4_0 KV, pure AR, temp 0, HTTP serving path, median of N post-warmup, same binary both arms (
accept_rate=0,prefix_len=0verified — no caching asymmetry). AR graph-reuse, replay ON vs OFF:The graph-reuse decode gap is a flat ~1.6% band across a 37× context range (3K→114K) and never clears each arm's own run-to-run spread (e.g. at 71K, ON's own min–max is 2.7% vs a 1.57% gap). It does not grow with depth, contradicting the "launch overhead scales with KV length → win grows with context" rationale. prefill throughput and wall are statistically identical between arms (the reuse only touches per-token decode launch).
The optimization's originally reported ~+6% (106→113 tok/s) was measured on Q3_K_XL at 16K; it does not reproduce on Q4_K_M at any depth tested here (3K/16K/32K/71K/114K). Read this as: correctness-neutral and safe on Q4_K_M, but no HTTP-measurable decode win on this quant — the speedup appears quant-specific.
Notes
llama.cppsubmodule-gitlink vs the ggml vendorize onmain(dropped the obsolete gitlink bumps;rererecombined main'ssrc[7]persist-wiring with thepure_arin-place op call — verified bit-identical).