glm-dsa : NextN/MTP speculative decoding for GLM-5.2 - #1
Open
auroter wants to merge 2 commits into
Open
Conversation
The qwen35, qwen35moe and cohere2moe MTP graphs carried identical copies of the llm_graph_input_embd_h construction, each annotated with a TODO asking for this extraction. The resolved token embedding is exposed as llm_graph_input_embd_h::tok_embd. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enables the LLM_GRAPH_TYPE_DECODER_MTP draft path for GLM_DSA (GLM-5.2), following the qwen35moe/step35 pattern: - load the NextN/MTP block tensors (previously TENSOR_SKIP), including support for standalone MTP-only draft GGUFs - add graph_mtp: eh_proj glue around the NextN block, whose MLA (absorption path) and MoE mirror the main deepseek2 graph so the draft distribution tracks the target - export t_h_nextn from the shared deepseek2 graph, gated on n_layer_nextn > 0 so existing DeepSeek models are unaffected - add KV cache layer filters so the main context caches only trunk layers and the MTP context caches only the NextN layer Usage: llama-server --spec-type draft-mtp (no draft model required; the MTP context shares the target model weights). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
auroter
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables
--spec-type draft-mtpfor the GLM_DSA architecture: self-speculative decoding through the model's own NextN head (blk.78) — no separate draft model, the draft context shares the target weights.Performance
Hardware: 4× RTX PRO 6000 Blackwell (no NVLink, layer-split). Model: GLM-5.2 753B Unsloth UD-IQ4_XS (336 GiB), q8_0 K cache. Baseline is the same binary with MTP off. All numbers greedy (temperature 0) unless noted, 128–256 tokens generated.
Production sampling (temperature 1.0, top-p 0.95, min-p 0.01): 66–73 tok/s at 68–79% acceptance — the speedup survives stochastic verification.
Memory cost: the NextN block (~4.3 GiB weights) plus a one-layer draft context (~0.6 GiB KV at q8_0 + compute buffer) on the card holding the last layer. On this 4×96 GiB box that traded the context ceiling from 512K down to 384K.
Draft depth
The default
--spec-draft-n-max 3is optimal for this head. Chaining deeper collapses acceptance (74% → 46% at depth 6 ungated); confidence gating (--spec-draft-p-min 0.75) restores ~90% acceptance but shortens chains below the default's yield. The single-layer head is intrinsically depth-3 quality.Correctness
Temperature-0 output is byte-identical with speculation on and off, verified across runs with different draft/accept trajectories. The
build_inp_embd_hrefactor commit reproduces pre-refactor draft counts exactly (238 drafted / 175 accepted on the code-generation benchmark).Notes