Skip to content

Commit 1119bb6

Browse files
authored
[Doc]fix internvl3.5 doc (vllm-project#11305)
### What this PR does / why we need it? fix internvl3.5 doc, update deployment - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc Signed-off-by: nakairika <35144242+nakairika@users.noreply.github.com>
1 parent a0d0671 commit 1119bb6

1 file changed

Lines changed: 6 additions & 8 deletions

File tree

docs/source/tutorials/models/InternVL3.5.md

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ sysctl -w vm.swappiness=0
101101
sysctl -w kernel.numa_balancing=0
102102
sysctl -w kernel.sched_migration_cost_ns=50000
103103
104-
export VLLM_ASCEND_ENABLE_FLASHCOMM1=0
104+
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
105105
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
106106
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
107107
export TASK_QUEUE_ENABLE=1
@@ -127,10 +127,8 @@ vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/InternVL3_5-38B-w8a8/
127127
--additional-config '{"enable_weight_nz_layout": true, "enable_cpu_binding": true}' \
128128
--mm-processor-cache-gb 0 \
129129
--enable-chunked-prefill \
130-
--enable-prefix-caching \
131130
--safetensors-load-strategy 'prefetch' \
132131
--allowed-local-media-path "/
133-
134132
```
135133

136134
::::
@@ -167,16 +165,16 @@ vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/InternVL3_5-241B-A28B-
167165
--trust-remote-code \
168166
--async-scheduling \
169167
--max-model-len 40960 \
170-
--max-num-batched-tokens 16384 \
171-
--tensor-parallel-size 16 \
172-
--max-num-seqs 16 \
168+
--max-num-batched-tokens 4096 \
169+
--tensor-parallel-size 4 \
170+
--data-parallel-size 2 \
171+
--max-num-seqs 70 \
173172
--gpu-memory-utilization 0.9 \
174173
--async-scheduling \
175-
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY", "cudagraph_capture_sizes":[4,32,64,128,192,256,512]}' \
174+
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
176175
--additional-config '{"enable_weight_nz_layout": true, "enable_cpu_binding": true}' \
177176
--mm-processor-cache-gb 0 \
178177
--enable-chunked-prefill \
179-
--enable-prefix-caching \
180178
--enable-expert-parallel \
181179
--safetensors-load-strategy 'prefetch' \
182180
--allowed-local-media-path "/"

0 commit comments

Comments
 (0)