Community post: Frontier MoE sleep/wake at TP=4 on consumer Blackwell — 2s wake, 4.5s swap - #3383
Open
DoradusResearch wants to merge 7 commits into
Open
Conversation
added 7 commits
May 16, 2026 20:08
DeepSeek-V4-Flash + MiMo-V2.5-Flash rotation at TP=4 on 4x RTX PRO 6000. ~2s wake, ~4.5s cross-peer swap (down from ~50s). Recipe, live config, known limits. Bundle image on GHCR. Companion canonical post: https://doradusresearch.ai/blog/vllm-blackwell-sleep-mode/ Companion upstream PR: vllm-project/vllm#42856 Signed-off-by: R. Ogden <robert@doradusresearch.ai>
Surfaces honest expectation: 46 safetensor shards + CUDA graph capture take ~12 min from docker run to first request. Bundle image is ~29 GB, so first pull on a fresh host adds image-pull time on top. Signed-off-by: R. Ogden <robert@doradusresearch.ai>
Initial draft recommended 0.70 based on vLLM's residue assumption (~4 GiB/GPU). Live measurement after a startup OOM showed MiMo's real sleeping-peer residue is ~9 GiB/GPU on consumer Blackwell. 0.60 is the honest correct value; pool boots cleanly with ~7 GiB margin. Adds decode-TPS sweep table across 7 input sizes (500 - 100K tokens), ~70 - 42 tok/s decode showing the PCIe-sync degradation curve at TP=4. Signed-off-by: R. Ogden <robert@doradusresearch.ai>
MiMo is currently running on stock cu129-nightly without PR #35489 — first-cycle rotation is clean but multi-cycle stress hits the same EINVAL race. Unification onto a single bundle:v5 (cu129-nightly base + cumem cherry-picks + V-pad overlay) is in build/test. Signed-off-by: R. Ogden <robert@doradusresearch.ai>
…DSv4 DeepGEMM gap explained, hybrid config working
…43 tok/s post-rotation
…v4 both public on GHCR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Community post submission
Posting a recipe + operator-perspective writeup that I think fits the HF Blog audience well — getting vLLM
--enable-sleep-modemodel rotation working at TP=4 on 4× NVIDIA RTX PRO 6000 Blackwell (consumer-Blackwell SM_120). DeepSeek-V4-Flash + MiMo-V2.5-Flash hot-swap on the same 4 GPUs in ~4.5 s cross-peer swap (down from ~50 s on the prior image stack).Why it matters for HF readers
--gpu-memory-utilizationknob plus two cumem-allocator cherry-picks plus one upstream PR (#41834)What ships
ghcr.io/doradusresearch/vllm-blackwell-sm12x-bundle:v4(Apache-2.0)Happy to make any edits the team prefers — style, scope, additional context. Thumbnail is 1200×630 PNG generated from the canonical post's OG image.
Signed-off-by: R. Ogden robert@doradusresearch.ai