[MTP][Runtime] Reuse draft attention metadata across MTP steps - #1
[MTP][Runtime] Reuse draft attention metadata across MTP steps#1voipmonitor wants to merge 2 commits into
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
49816b9 to
f4ba2a2
Compare
Summary
This stacks on top of the model-side GLM patch in upstream PR vllm-project#39550.
It avoids rebuilding draft attention metadata on every multi-step MTP iteration by reusing and updating metadata in-place for builders that explicitly support drafting updates.
Files:
vllm/v1/spec_decode/eagle.pyvllm/model_executor/layers/attention/mla_attention.pyvllm/v1/attention/backends/mla/indexer.pyTracking issue:
Dependency:
glm5-nextn-model-parityin the fork so the diff stays runtime-only.Validation
Long single-request benchmark (
ignore_eos,max_tokens=4096,global_warmup_requests=2,temperature=0):MTP=4+probabilistic:114.75 tok/sMTP=5+probabilistic:139.76 tok/smedian156.67 tok/smedian138.82 tok/smedianNotes: