Skip to content

[MTP][Runtime] Reuse draft attention metadata across MTP steps - #1

Open
voipmonitor wants to merge 2 commits into
glm5-nextn-model-parityfrom
glm51-runtime-reusemeta-pr
Open

[MTP][Runtime] Reuse draft attention metadata across MTP steps#1
voipmonitor wants to merge 2 commits into
glm5-nextn-model-parityfrom
glm51-runtime-reusemeta-pr

Conversation

@voipmonitor

Copy link
Copy Markdown
Owner

Summary

This stacks on top of the model-side GLM patch in upstream PR vllm-project#39550.

It avoids rebuilding draft attention metadata on every multi-step MTP iteration by reusing and updating metadata in-place for builders that explicitly support drafting updates.

Files:

  • vllm/v1/spec_decode/eagle.py
  • vllm/model_executor/layers/attention/mla_attention.py
  • vllm/v1/attention/backends/mla/indexer.py

Tracking issue:

Dependency:

Validation

Long single-request benchmark (ignore_eos, max_tokens=4096, global_warmup_requests=2, temperature=0):

  • Unpatched vLLM GLM-5.1 MTP=4 + probabilistic: 114.75 tok/s
  • Hotfix runtime image GLM-5.1 MTP=5 + probabilistic: 139.76 tok/s median
  • Git-backed rebuilt image from this branch, after full warm state: 156.67 tok/s median
  • SGLang GLM-5.1 reference on the same long benchmark: 138.82 tok/s median

Notes:

  • The first full benchmark run immediately after server startup is not a stable throughput reference for this path; after startup, one long warm run should be discarded before comparing medians.
  • This patch is specific to the MLA/indexer drafting path. It is not yet evidence of the same gain for Qwen3.5 MTP.

@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant