[SpecDecode][EXL3] deepseek_mtp: normalize rank-sliced weight names (fixes MTP trellis overlay loading) - #11
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
- entrypoint: download malaiwah/GLM-5.2-EXL3-TR3-MTP78 3bpw-keep0, apply the deepseek_mtp loader patch (voipmonitor/vllm#11), graft in place (revertible, MTP78_TRELLIS=0 reverts); BF16 fallback on any failure - scripts/graft_mtp78.py: in-place revertible graft (shard merge, index rebuild, moe_layers [3,78] + ignore-list config edits, .orig backups) - scripts/patch_deepseek_mtp.py: idempotent one-hunk vLLM fix - README: MTP78 section (BF16-parity validation, patch requirement) + the July 2026 OpenAI/HF incident as the project's origin story Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WnCy3BTrTvoeqfRGu8s97o
…_weights
DeepSeekMTP.load_weights does not apply normalize_rank_sliced_weight_name(),
which DeepseekV2ForCausalLM.load_weights already does. Quant configs that
serialize per-TP-rank payloads (EXL3 rank-sliced trellis) therefore fail to load
an MTP layer stored in that schema:
KeyError: 'model.layers.78.mtp_block.mlp.experts.routed_experts.w2_rank0.mcg'
The expert mapping rewrites the projection but the '.rank{r}.{suffix}' tail is
never normalized. This mirrors the deepseek_v2 hunk and is a no-op (getattr
returns None) for quant configs without rank slicing.
Validated on GLM-5.2 753B, TP4, 4x RTX PRO 6000 Blackwell: with this fix a
3.0bpw EXL3 trellis MTP-layer overlay loads and serves at BF16-parity draft
acceptance (GSM8K MAL 3.517 vs 3.528, acceptance 83.9% vs 84.3%) while the
draft's footprint drops from 19.3 GB to 3.7 GB, freeing ~3.65 GiB/GPU of KV.
Signed-off-by: Michel Belleau <michel.belleau@malaiwah.com>
6f3c874 to
4e52eb8
Compare
|
Note on the base, since I offered to retarget to Happy to rebase onto |
1 participant
Problem
DeepSeekMTP.load_weightsdoes not applynormalize_rank_sliced_weight_name(), whichDeepseekV2ForCausalLM.load_weightsalready does. Quant configs that serialize per-TP-rank payloads (EXL3 rank-sliced trellis) therefore fail to load an MTP layer stored in that schema:The expert mapping rewrites the projection, but the
.rank{r}.{suffix}tail is never normalized.Change
One hunk in
deepseek_mtp.py, mirroring the existing deepseek_v2 logic. No-op for any quant config without rank slicing (getattrreturnsNone), so it is safe on this base even though rank-sliced EXL3 is not merged here yet — it is the prerequisite that lets an MTP layer load once it is.Validation
GLM-5.2 753B, TP4, 4x RTX PRO 6000 Blackwell, RC2-lineage image. With this fix a 3.0 bpw EXL3 trellis overlay for the MTP layer loads and serves at BF16-parity draft acceptance:
Free KV memory goes 5.27 → 8.92 GiB/GPU (+69%). The overlay also loads through
--speculative-config {"model": ...}without modifying the base checkpoint.Artifacts if useful for review: weights malaiwah/GLM-5.2-EXL3-TR3-MTP78, method + full measurements gist.
Happy to retarget to
mainif that is the preferred base — the change is base-agnostic.🤖 Generated with Claude Code
https://claude.ai/code/session_01WnCy3BTrTvoeqfRGu8s97o