Skip to content

Add GPU Mooncake layerwise connector - #2

Draft
Pz1116 wants to merge 1 commit into
luoxiaolin712:mainfrom
Pz1116:mooncake-layerwise-gpu
Draft

Add GPU Mooncake layerwise connector#2
Pz1116 wants to merge 1 commit into
luoxiaolin712:mainfrom
Pz1116:mooncake-layerwise-gpu

Conversation

@Pz1116

@Pz1116 Pz1116 commented May 21, 2026

Copy link
Copy Markdown

Summary

  • add a GPU-safe MooncakeLayerwiseConnector without torch_npu or torch.npu dependencies
  • support layerwise KV push with PP-aware side-channel ports and one prefill rank pushing token/head slices to multiple decode ranks for Prefill TP < Decode TP
  • register the connector as MooncakeLayerwiseConnector in the v1 KV connector factory

Validation

  • python -m py_compile vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_layerwise_connector.py vllm/distributed/kv_transfer/kv_connector/factory.py
  • env XDG_CACHE_HOME=/tmp PRE_COMMIT_HOME=/tmp/pre-commit pre-commit run --files vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_layerwise_connector.py vllm/distributed/kv_transfer/kv_connector/factory.py
  • git diff --check

Signed-off-by: Pz1116 <zpbzpb123123@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant