Skip to content

Commit 4e9b8a0

Browse files
authored
[Doc][Misc]Add sleep mode documentation for unquantized MoE models (vllm-project#11110)
### What this PR does / why we need it? Adds a "MoE vs. dense (non-MoE) model differences" section to the sleep mode guide, documenting the expert weight layout restoration: process_weights_after_loading() transposes w13_weight/w2_weight into the torch_npu.npu_grouped_matmul layout at load time (assuming standard checkpoint layout, no validation), and wake_up() re-applies the same transpose after the allocator restores memory; dense/quantized models skip it. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@967c5c3 --------- Co-authored-by: yjyang62 <yjyang62@users.noreply.github.com>
1 parent cadefd2 commit 4e9b8a0

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

docs/source/user_guide/feature_guide/sleep_mode.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,19 @@ llm.wake_up(tags=["kv_cache"])
6767

6868
With extra cleanup enabled, ACL graphs are recaptured only when `tags` is `None` or contains `"kv_cache"`. This avoids recapturing graphs before externally reloaded weights and KV-cache state are ready.
6969

70+
### Expert weight layout restoration
71+
72+
For dense models, `wake_up()` simply restores the model weights to NPU memory; the tensor layout is unchanged.
73+
74+
For **unquantized MoE models** (`quant_config is None`), the fused expert weights are stored in a transposed layout for NPU matmul efficiency. This layout is produced once at model load time by `process_weights_after_loading()`: after the weights are loaded, the method transposes the second and third dimensions (`transpose(1, 2)`) of `w13_weight` and `w2_weight` to convert the standard checkpoint layout into the format required by the `torch_npu.npu_grouped_matmul` operator.
75+
76+
After the sleep-mode allocator restores the original (untransposed) memory, `wake_up()` re-applies the same transpose to the affected expert weights when the `"weights"` tag is being restored:
77+
78+
- `w13_weight` (gate/up projection): transposed back to the runtime layout when its second dimension matches `hidden_size`;
79+
- `w2_weight` (down projection): transposed back to the runtime layout when its third dimension matches `hidden_size`.
80+
81+
This step is skipped entirely for dense models (which have no expert weights) and for quantized models (whose weights are handled by the quantization method).
82+
7083
## Usage
7184

7285
The following is a simple example of how to use sleep mode.

0 commit comments

Comments
 (0)