1- # Atlas 300I DUO
1+ # Atlas 300I DUO / Atlas 200I Pro
2+
3+ This tutorial covers vLLM deployment on Ascend 310P inference hardware, including
4+ Atlas 300I DUO and Atlas 200I Pro acceleration modules.
25
36``` {note}
47Atlas 300I DUO does not support `triton` or `triton-ascend`.
@@ -27,6 +30,10 @@ The current release supports `FULL_DECODE_ONLY` graph mode on Atlas 300I DUO dev
2730
2831### Deployment
2932
33+ Choose the startup command according to your hardware form factor.
34+
35+ #### Atlas 300I DUO
36+
3037Run docker container:
3138
3239``` {code-block} bash
@@ -59,6 +66,93 @@ docker run --rm \
5966-it $IMAGE bash
6067```
6168
69+ #### Atlas 200I Pro Acceleration Module
70+
71+ When starting a container on Atlas 200I Pro, mount additional driver libraries and
72+ configuration files required by ` npu-smi ` . Without these mounts, ` npu-smi ` commands
73+ may fail inside the container. Use the command below for your container OS.
74+
75+ ``` {note}
76+ Atlas 200I Pro also uses Ascend 310P. Adjust `--device=/dev/davinci0` according to
77+ the NPU ID you want to use. You can query available devices with
78+ `ll /dev/ | grep davinci`.
79+ ```
80+
81+ :::::{tab-set}
82+
83+ ::::{tab-item} Ubuntu 24.04
84+ :selected:
85+
86+ ``` {code-block} bash
87+ :substitutions:
88+
89+ export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p
90+
91+ docker run --rm \
92+ --privileged \
93+ --name vllm-ascend \
94+ --shm-size=10g \
95+ --device=/dev/davinci0:/dev/davinci0 \
96+ --device=/dev/davinci_manager \
97+ --device=/dev/ascend_manager \
98+ --device=/dev/user_config \
99+ -v /etc/sys_version.conf:/etc/sys_version.conf \
100+ -v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
101+ -v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
102+ -v /var/dmp_daemon:/var/dmp_daemon \
103+ -v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
104+ -v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
105+ -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
106+ -v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
107+ -v /usr/lib/aarch64-linux-gnu/libyaml-0.so.2:/usr/lib64/libyaml-0.so.2 \
108+ -v /etc/slog.conf:/etc/slog.conf \
109+ -v /var/slogd:/var/slogd \
110+ -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
111+ -v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
112+ -v /root/.cache:/root/.cache \
113+ -p 8080:8080 \
114+ -it $IMAGE bash
115+ ```
116+
117+ ::::
118+
119+ ::::{tab-item} openEuler 24.03
120+
121+ ``` {code-block} bash
122+ :substitutions:
123+
124+ export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p-openeuler
125+
126+ docker run --rm \
127+ --privileged \
128+ --name vllm-ascend \
129+ --shm-size=10g \
130+ --device=/dev/davinci0:/dev/davinci0 \
131+ --device=/dev/davinci_manager \
132+ --device=/dev/ascend_manager \
133+ --device=/dev/user_config \
134+ -v /etc/sys_version.conf:/etc/sys_version.conf \
135+ -v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
136+ -v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
137+ -v /var/dmp_daemon:/var/dmp_daemon \
138+ -v /usr/lib64/libsemanage.so.2:/usr/lib64/libsemanage.so.2 \
139+ -v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
140+ -v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
141+ -v /usr/lib64/libyaml-0.so.2.0.9:/usr/lib64/libyaml-0.so.2 \
142+ -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
143+ -v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
144+ -v /etc/slog.conf:/etc/slog.conf \
145+ -v /var/slogd:/var/slogd \
146+ -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
147+ -v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
148+ -v /root/.cache:/root/.cache \
149+ -p 8080:8080 \
150+ -it $IMAGE bash
151+ ```
152+
153+ ::::
154+ :::::
155+
62156Set up environment variables:
63157
64158``` bash
@@ -106,7 +200,10 @@ Use the W8A8SC quantized weights from the Eco-Tech official ModelScope repositor
106200 - [Eco-Tech/Qwen3-32B-w8a8sc-310-vllm](https://www.modelscope.cn/models/Eco-Tech/Qwen3-32B-w8a8sc-310-vllm)
107201```
108202
109- #### Qwen3-8B-W8A8SC
203+ :::::{tab-set}
204+
205+ ::::{tab-item} Qwen3-8B-W8A8SC
206+ :selected:
110207
111208``` {code-block} bash
112209 :substitutions:
@@ -127,7 +224,9 @@ vllm serve Eco-Tech/Qwen3-8B-w8a8sc-310-vllm/TP1/Qwen3-8B-w8a8sc-310-vllm-tp1 \
127224 --load_format sharded_state
128225```
129226
130- #### Qwen3-14B-W8A8SC
227+ ::::
228+
229+ ::::{tab-item} Qwen3-14B-W8A8SC
131230
132231``` {code-block} bash
133232 :substitutions:
@@ -148,7 +247,9 @@ vllm serve Eco-Tech/Qwen3-14B-w8a8sc-310-vllm/TP1/Qwen3-14B-w8a8sc-310-vllm-tp1
148247 --load_format sharded_state
149248```
150249
151- #### Qwen3-32B-W8A8SC
250+ ::::
251+
252+ ::::{tab-item} Qwen3-32B-W8A8SC
152253
153254``` {code-block} bash
154255 :substitutions:
@@ -171,6 +272,9 @@ vllm serve Eco-Tech/Qwen3-32B-w8a8sc-310-vllm/TP4/Qwen3-32B-w8a8sc-310-vllm-tp4
171272 --load_format sharded_state
172273```
173274
275+ ::::
276+ :::::
277+
174278Once the server is started, you can query the model with input prompts:
175279
176280``` bash
@@ -190,11 +294,9 @@ If the script runs successfully, you can see the generated result.
190294Run the following script, ` example.py ` , to execute offline inference on NPU.
191295
192296:::::{tab-set}
193- :sync-group: inference
194297
195298::::{tab-item} Qwen3-8B-W8A8SC
196299:selected:
197- :sync: qwen3-8b
198300
199301``` {code-block} python
200302 :substitutions:
@@ -259,7 +361,6 @@ clean_up()
259361::::
260362
261363::::{tab-item} Qwen3-14B-W8A8SC
262- :sync: qwen3-14b
263364
264365``` {code-block} python
265366 :substitutions:
@@ -324,7 +425,6 @@ clean_up()
324425::::
325426
326427::::{tab-item} Qwen3-32B-W8A8SC
327- :sync: qwen3-32b
328428
329429``` {code-block} python
330430 :substitutions:
0 commit comments