如何部署

[lzl-hello]

所以源码在哪里？我看mini版本的文件夹下啥都没有呀，哪里可以找到webdemo.py
如果有大佬可以说一下mini版本的部署步骤就更好了
谢谢

[lzl-hello]

我自己试着摸索了一下，部署了mini的docker，但是启动失败，报错说models里缺少文件：
lzl@HIT-lzl:~$ docker logs gallant_cray

Traceback (most recent call last):
File "/secgpt-mini/webdemo.py", line 207, in
main(args.base_model, args.share_gradio)
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/secgpt-mini/webdemo.py", line 76, in main
model = AutoModelForCausalLM.from_pretrained(
base_model,
device_map=device,
trust_remote_code=True,
)
File "/usr/local/lib/python3.13/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
~~~~~~~~~~~~~~~~~~~~~~~~~~~^
pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.13/site-packages/transformers/modeling_utils.py", line 4260, in from_pretrained
checkpoint_files, sharded_metadata = _get_resolved_checkpoint_files(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
pretrained_model_name_or_path=pretrained_model_name_or_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<13 lines>...
commit_hash=commit_hash,
^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/transformers/modeling_utils.py", line 952, in _get_resolved_checkpoint_files
raise EnvironmentError(
...<3 lines>...
)
OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /secgpt-mini/models.
Traceback (most recent call last):
File "/secgpt-mini/webdemo.py", line 207, in
main(args.base_model, args.share_gradio)
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/secgpt-mini/webdemo.py", line 76, in main
model = AutoModelForCausalLM.from_pretrained(
base_model,
device_map=device,
trust_remote_code=True,
)
File "/usr/local/lib/python3.13/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
~~~~~~~~~~~~~~~~~~~~~~~~~~~^
pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.13/site-packages/transformers/modeling_utils.py", line 4260, in from_pretrained
checkpoint_files, sharded_metadata = _get_resolved_checkpoint_files(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
pretrained_model_name_or_path=pretrained_model_name_or_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<13 lines>...
commit_hash=commit_hash,
^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/transformers/modeling_utils.py", line 952, in _get_resolved_checkpoint_files
raise EnvironmentError(
...<3 lines>...
)
OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /secgpt-mini/models.
lzl@HIT-lzl:~$ ls

my_docker my_tmux_learn secgpt-mini
lzl@HIT-lzl:$ cd secgpt-mini/
lzl@HIT-lzl:/secgpt-mini$ ls

Dockerfile README.md models requirements.txt webdemo.py 大模型回答面试问题-cot.txt
lzl@HIT-lzl:/secgpt-mini$ cd models/
lzl@HIT-lzl:/secgpt-mini/models$ ls

added_tokens.json config.json generation_config.json merges.txt special_tokens_map.json tokenizer.json tokenizer_config.json vocab.json

[snoopy7713]

run_secgpt.sh

#!/bin/bash

服务配置参数

MODEL_DIR="/home/ubuntu/lp/model/secgpt" # 模型实际存放路径
GPU_INDICES="0,1" # 使用的GPU索引
CONDA_ENV="secgpt-vllm" # Conda环境名称
LOG_FILE="secgpt_server.log" # 日志文件
PORT=9012 # 新增端口配置

rm -rf secgpt_server.log

检查模型目录是否存在

if [ ! -d "$MODEL_DIR" ]; then
echo "[ERROR] 模型目录不存在: $MODEL_DIR"
echo "请确认："
echo "1. 是否已从 HuggingFace 下载模型到正确位置"
echo "2. 是否包含 tokenizer.model 和 pytorch_model.bin 等文件"
exit 1
fi

检查Conda环境是否存在

if ! conda env list | grep -q $CONDA_ENV; then
echo "[ERROR] Conda环境 '$CONDA_ENV' 不存在"
echo "请先运行: conda create -n $CONDA_ENV python=3.10 -y"
exit 1
fi

激活Conda环境并打印输出

echo "[INFO] 正在激活 Conda 环境 '$CONDA_ENV'..."
eval "$(conda shell.bash hook)" # 初始化 Conda 环境
if ! conda activate $CONDA_ENV; then
echo "[ERROR] 无法激活 Conda 环境 '$CONDA_ENV'"
exit 1
fi
echo "[INFO] Conda 环境 '$CONDA_ENV' 已成功激活"

确保CUDA_VISIBLE_DEVICES环境变量正确设置

export CUDA_VISIBLE_DEVICES=$GPU_INDICES

启动服务（后台运行）

echo "=================================================="
echo "模型路径 : $MODEL_DIR"
echo "使用GPU : $GPU_INDICES"
echo "监听端口 : $PORT" # 新增端口显示
echo "日志文件 : $LOG_FILE"
echo "=================================================="

输出启动命令

echo "nohup vllm serve
--model $MODEL_DIR
--port=$PORT
--tokenizer=$MODEL_DIR
--tensor-parallel-size=2
--max-model-len=32768
--gpu-memory-utilization=0.9
--dtype=bfloat16 >> $LOG_FILE 2>&1 &"
echo

nohup vllm serve $MODEL_DIR
--port=$PORT
--tokenizer=$MODEL_DIR
--tensor-parallel-size=2
--max-model-len=32768
--gpu-memory-utilization=0.8
--dtype=bfloat16 >> $LOG_FILE 2>&1 &

等待几秒钟，给服务一些时间启动

sleep 5

显示进程信息

echo "[INFO] 检查 vllm serve 是否启动..."

1. 使用 pgrep 检查 vllm serve 进程是否存在

PGREP_RESULT=$(pgrep -f "vllm serve $MODEL_DIR")
if [ -z "$PGREP_RESULT" ]; then
echo "[ERROR] vllm serve 未启动"
else
echo "[INFO] vllm serve 正在运行，进程ID：$PGREP_RESULT"
fi

2. 检查端口是否被监听

PORT_LISTEN=$(netstat -tuln | grep ":$PORT")
if [ -z "$PORT_LISTEN" ]; then
echo "[ERROR] 端口 $PORT 没有被监听，vllm serve 可能未启动"
else
echo "[INFO] 端口 $PORT 正在监听，服务已启动"
fi

3. 检查日志文件中是否有任何启动错误

if tail -n 20 $LOG_FILE | grep -q "ERROR"; then
echo "[ERROR] 检测到日志文件中有错误，请检查 $LOG_FILE"
else
echo "[INFO] 日志文件检查通过，无明显错误"
fi

提示测试接口命令

echo "[提示] 测试接口命令："
echo "curl http://192.168.30.12:$PORT/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "secgpt", "messages": [{"role": "user", "content": "什么是 XSS 攻击？"}], "temperature": 0.7}'"

[snoopy7713]

修改一下扩展名txt 为 sh就可以了

run_secgpt.txt

[snoopy7713]

给dify配置secgpt，模型名称如何填写？

[lzl-hello]

修改一下扩展名txt 为 sh就可以了

run_secgpt.txt

大佬好！我按照您的脚本在服务器上部署了一下14B的模型，日志如下：
INFO 04-17 02:06:11 [init.py:239] Automatically detected platform cuda.
INFO 04-17 02:06:12 [api_server.py:121] vLLM API server version 0.8.4
INFO 04-17 02:06:12 [api_server.py:122] args: Namespace(host='0.0.0.0', port=20000, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, log_level='debug', model='/root/hzc2/lzl/SecGPT/SecGPT-14B', task='auto', tokenizer='/root/hzc2/lzl/SecGPT/SecGPT-14B', hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config=None, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='bfloat16', kv_cache_dtype='auto', max_model_len=32768, guided_decoding_backend='auto', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.8, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_chunked_mm_input=False, disable_log_requests=False)
WARNING 04-17 02:06:12 [config.py:2836] Casting torch.float16 to torch.bfloat16.
INFO 04-17 02:06:19 [config.py:689] This model supports multiple tasks: {'score', 'reward', 'generate', 'classify', 'embed'}. Defaulting to 'generate'.
INFO 04-17 02:06:19 [config.py:1713] Defaulting to use mp for distributed inference
INFO 04-17 02:06:19 [config.py:1901] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 04-17 02:06:21 [core.py:61] Initializing a V1 LLM engine (v0.8.4) with config: model='/root/hzc2/lzl/SecGPT/SecGPT-14B', speculative_config=None, tokenizer='/root/hzc2/lzl/SecGPT/SecGPT-14B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/root/hzc2/lzl/SecGPT/SecGPT-14B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 04-17 02:06:21 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 32 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 04-17 02:06:21 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 10485760, 10, 'psm_b9280669'), local_subscribe_addr='ipc:///tmp/d14de930-84c9-43a8-aeb9-be667baabf74', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 04-17 02:06:22 [utils.py:2444] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x74348db9d2a0>
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:22 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_d4406788'), local_subscribe_addr='ipc:///tmp/baeb3756-cdff-464b-8c41-1be06687be54', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 04-17 02:06:22 [utils.py:2444] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x74348db9d510>
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:22 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_3a58686b'), local_subscribe_addr='ipc:///tmp/8e6eea2a-cff7-454e-8a69-3127e1501a42', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [utils.py:993] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [utils.py:993] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,2.json
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,2.json
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_408474c1'), local_subscribe_addr='ipc:///tmp/1341dcea-3ee4-477c-9f2e-1f3476fd6dab', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [parallel_state.py:959] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [parallel_state.py:959] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [gpu_model_runner.py:1276] Starting to load model /root/hzc2/lzl/SecGPT/SecGPT-14B...
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [gpu_model_runner.py:1276] Starting to load model /root/hzc2/lzl/SecGPT/SecGPT-14B...
(VllmWorker rank=1 pid=159016) WARNING 04-17 02:06:27 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=0 pid=158999) WARNING 04-17 02:06:27 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 17% Completed | 1/6 [00:04<00:20, 4.02s/it]
Loading safetensors checkpoint shards: 33% Completed | 2/6 [00:08<00:16, 4.08s/it]
Loading safetensors checkpoint shards: 50% Completed | 3/6 [00:12<00:12, 4.09s/it]
Loading safetensors checkpoint shards: 67% Completed | 4/6 [00:16<00:08, 4.11s/it]
Loading safetensors checkpoint shards: 83% Completed | 5/6 [00:20<00:04, 4.08s/it]
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:50 [loader.py:458] Loading weights took 23.23 seconds
(VllmWorker rank=1 pid=159016) INFO 04-17 02:06:50 [gpu_model_runner.py:1291] Model loading took 13.9281 GiB and 24.878455 seconds
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:24<00:00, 4.04s/it]
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:24<00:00, 4.06s/it]
(VllmWorker rank=0 pid=158999)
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:51 [loader.py:458] Loading weights took 24.46 seconds
(VllmWorker rank=0 pid=158999) INFO 04-17 02:06:51 [gpu_model_runner.py:1291] Model loading took 13.9281 GiB and 26.129283 seconds
(VllmWorker rank=1 pid=159016) INFO 04-17 02:07:03 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/17f5bfb989/rank_1_0 for vLLM's torch.compile
(VllmWorker rank=1 pid=159016) INFO 04-17 02:07:03 [backends.py:426] Dynamo bytecode transform time: 11.58 s
(VllmWorker rank=0 pid=158999) INFO 04-17 02:07:03 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/17f5bfb989/rank_0_0 for vLLM's torch.compile
(VllmWorker rank=0 pid=158999) INFO 04-17 02:07:03 [backends.py:426] Dynamo bytecode transform time: 12.01 s
(VllmWorker rank=1 pid=159016) INFO 04-17 02:07:04 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=0 pid=158999) INFO 04-17 02:07:04 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=1 pid=159016) INFO 04-17 02:07:16 [monitor.py:33] torch.compile takes 11.58 s in total
(VllmWorker rank=0 pid=158999) INFO 04-17 02:07:16 [monitor.py:33] torch.compile takes 12.01 s in total
INFO 04-17 02:07:18 [kv_cache_utils.py:634] GPU KV cache size: 168,944 tokens
INFO 04-17 02:07:18 [kv_cache_utils.py:637] Maximum concurrency for 32,768 tokens per request: 5.16x
INFO 04-17 02:07:18 [kv_cache_utils.py:634] GPU KV cache size: 168,944 tokens
INFO 04-17 02:07:18 [kv_cache_utils.py:637] Maximum concurrency for 32,768 tokens per request: 5.16x
(VllmWorker rank=1 pid=159016) INFO 04-17 02:08:10 [custom_all_reduce.py:195] Registering 6499 cuda graph addresses
(VllmWorker rank=0 pid=158999) INFO 04-17 02:08:13 [custom_all_reduce.py:195] Registering 6499 cuda graph addresses
(VllmWorker rank=0 pid=158999) INFO 04-17 02:08:13 [gpu_model_runner.py:1626] Graph capturing finished in 55 secs, took 0.72 GiB
(VllmWorker rank=1 pid=159016) INFO 04-17 02:08:13 [gpu_model_runner.py:1626] Graph capturing finished in 55 secs, took 0.72 GiB
INFO 04-17 02:11:49 [core.py:163] init engine (profile, create kv cache, warmup model) took 297.90 seconds
INFO 04-17 02:11:49 [core_client.py:435] Core engine process 0 ready.
INFO 04-17 02:11:49 [launcher.py:26] Available routes are:
INFO 04-17 02:11:49 [launcher.py:34] Route: /openapi.json, Methods: HEAD, GET
INFO 04-17 02:11:49 [launcher.py:34] Route: /docs, Methods: HEAD, GET
INFO 04-17 02:11:49 [launcher.py:34] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 04-17 02:11:49 [launcher.py:34] Route: /redoc, Methods: HEAD, GET
INFO 04-17 02:11:49 [launcher.py:34] Route: /health, Methods: GET
INFO 04-17 02:11:49 [launcher.py:34] Route: /generate, Methods: POST
INFO: Started server process [158788]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:20000 (Press CTRL+C to quit)

我没有看到error，应该是正常启动了，但是我看给的路由没有[/v1/chat/completions]
于是我使用如下测试命令：curl http://localhost:20000/generate
-H "Content-Type: application/json"
-d '{
"prompt": "什么是 XSS 攻击？",
"temperature": 0.7
}'
得到了无用回答：{"text":["什么是 XSS 攻击？!!!!!!!!!!!!!!!!"]}
服务端的日志：
INFO 04-17 02:49:32 [async_llm.py:228] Added request 23b0d47c34b941998962a38016c52314.
INFO: 127.0.0.1:58018 - "GET / HTTP/1.1" 404 Not Found
INFO: ::1:39670 - "POST /generate HTTP/1.1" 200 OK
INFO 04-17 02:55:18 [async_llm.py:228] Added request cc2c7afebd4049978a7d38e603f90712.

第二次request是我又尝试问了一次，十几分钟还没有得到回答。
请问您部署的模型有遇到这个问题吗？如果没有，我是哪里出了问题

感激不尽！

[snoopy7713]

这是我最终使用的脚本，置顶了模型名称

run_secgpt.zip

[snoopy7713]

我制定了了http端口 9012，通过nvidia-smi看到模型被加载了2份，等一会儿vllm开启端口监听。

netstat -antlpo|grep 9012 可以看到。里面需要修改一下具体部署机器的ip地址

[lzl-hello]

我制定了了http端口 9012，通过nvidia-smi看到模型被加载了2份，等一会儿vllm开启端口监听。

netstat -antlpo|grep 9012 可以看到。里面需要修改一下具体部署机器的ip地址

感谢，按照您的方法成功运行了

[Keep-SY]

修改一下扩展名txt 为 sh就可以了
run_secgpt.txt

大佬好！我按照您的脚本在服务器上部署了一下14B的模型，日志如下： INFO 04-17 02:06:11 [init.py:239] Automatically detected platform cuda. INFO 04-17 02:06:12 [api_server.py:121] vLLM API server version 0.8.4 INFO 04-17 02:06:12 [api_server.py:122] args: Namespace(host='0.0.0.0', port=20000, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, log_level='debug', model='/root/hzc2/lzl/SecGPT/SecGPT-14B', task='auto', tokenizer='/root/hzc2/lzl/SecGPT/SecGPT-14B', hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config=None, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='bfloat16', kv_cache_dtype='auto', max_model_len=32768, guided_decoding_backend='auto', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.8, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_chunked_mm_input=False, disable_log_requests=False) WARNING 04-17 02:06:12 [config.py:2836] Casting torch.float16 to torch.bfloat16. INFO 04-17 02:06:19 [config.py:689] This model supports multiple tasks: {'score', 'reward', 'generate', 'classify', 'embed'}. Defaulting to 'generate'. INFO 04-17 02:06:19 [config.py:1713] Defaulting to use mp for distributed inference INFO 04-17 02:06:19 [config.py:1901] Chunked prefill is enabled with max_num_batched_tokens=2048. INFO 04-17 02:06:21 [core.py:61] Initializing a V1 LLM engine (v0.8.4) with config: model='/root/hzc2/lzl/SecGPT/SecGPT-14B', speculative_config=None, tokenizer='/root/hzc2/lzl/SecGPT/SecGPT-14B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/root/hzc2/lzl/SecGPT/SecGPT-14B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512} WARNING 04-17 02:06:21 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 32 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. INFO 04-17 02:06:21 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 10485760, 10, 'psm_b9280669'), local_subscribe_addr='ipc:///tmp/d14de930-84c9-43a8-aeb9-be667baabf74', remote_subscribe_addr=None, remote_addr_ipv6=False) WARNING 04-17 02:06:22 [utils.py:2444] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x74348db9d2a0> (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:22 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_d4406788'), local_subscribe_addr='ipc:///tmp/baeb3756-cdff-464b-8c41-1be06687be54', remote_subscribe_addr=None, remote_addr_ipv6=False) WARNING 04-17 02:06:22 [utils.py:2444] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x74348db9d510> (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:22 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_3a58686b'), local_subscribe_addr='ipc:///tmp/8e6eea2a-cff7-454e-8a69-3127e1501a42', remote_subscribe_addr=None, remote_addr_ipv6=False) (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [utils.py:993] Found nccl from library libnccl.so.2 (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [utils.py:993] Found nccl from library libnccl.so.2 (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [pynccl.py:69] vLLM is using nccl==2.21.5 (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [pynccl.py:69] vLLM is using nccl==2.21.5 (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,2.json (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,2.json (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [shm_broadcast.py:264] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_408474c1'), local_subscribe_addr='ipc:///tmp/1341dcea-3ee4-477c-9f2e-1f3476fd6dab', remote_subscribe_addr=None, remote_addr_ipv6=False) (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [parallel_state.py:959] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1 (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [parallel_state.py:959] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0 (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [cuda.py:221] Using Flash Attention backend on V1 engine. (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [cuda.py:221] Using Flash Attention backend on V1 engine. (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:25 [gpu_model_runner.py:1276] Starting to load model /root/hzc2/lzl/SecGPT/SecGPT-14B... (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:25 [gpu_model_runner.py:1276] Starting to load model /root/hzc2/lzl/SecGPT/SecGPT-14B... (VllmWorker rank=1 pid=159016) WARNING 04-17 02:06:27 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer. (VllmWorker rank=0 pid=158999) WARNING 04-17 02:06:27 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer. Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 17% Completed | 1/6 [00:04<00:20, 4.02s/it] Loading safetensors checkpoint shards: 33% Completed | 2/6 [00:08<00:16, 4.08s/it] Loading safetensors checkpoint shards: 50% Completed | 3/6 [00:12<00:12, 4.09s/it] Loading safetensors checkpoint shards: 67% Completed | 4/6 [00:16<00:08, 4.11s/it] Loading safetensors checkpoint shards: 83% Completed | 5/6 [00:20<00:04, 4.08s/it] (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:50 [loader.py:458] Loading weights took 23.23 seconds (VllmWorker rank=1 pid=159016) INFO 04-17 02:06:50 [gpu_model_runner.py:1291] Model loading took 13.9281 GiB and 24.878455 seconds Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:24<00:00, 4.04s/it] Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:24<00:00, 4.06s/it] (VllmWorker rank=0 pid=158999) (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:51 [loader.py:458] Loading weights took 24.46 seconds (VllmWorker rank=0 pid=158999) INFO 04-17 02:06:51 [gpu_model_runner.py:1291] Model loading took 13.9281 GiB and 26.129283 seconds (VllmWorker rank=1 pid=159016) INFO 04-17 02:07:03 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/17f5bfb989/rank_1_0 for vLLM's torch.compile (VllmWorker rank=1 pid=159016) INFO 04-17 02:07:03 [backends.py:426] Dynamo bytecode transform time: 11.58 s (VllmWorker rank=0 pid=158999) INFO 04-17 02:07:03 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/17f5bfb989/rank_0_0 for vLLM's torch.compile (VllmWorker rank=0 pid=158999) INFO 04-17 02:07:03 [backends.py:426] Dynamo bytecode transform time: 12.01 s (VllmWorker rank=1 pid=159016) INFO 04-17 02:07:04 [backends.py:115] Directly load the compiled graph for shape None from the cache (VllmWorker rank=0 pid=158999) INFO 04-17 02:07:04 [backends.py:115] Directly load the compiled graph for shape None from the cache (VllmWorker rank=1 pid=159016) INFO 04-17 02:07:16 [monitor.py:33] torch.compile takes 11.58 s in total (VllmWorker rank=0 pid=158999) INFO 04-17 02:07:16 [monitor.py:33] torch.compile takes 12.01 s in total INFO 04-17 02:07:18 [kv_cache_utils.py:634] GPU KV cache size: 168,944 tokens INFO 04-17 02:07:18 [kv_cache_utils.py:637] Maximum concurrency for 32,768 tokens per request: 5.16x INFO 04-17 02:07:18 [kv_cache_utils.py:634] GPU KV cache size: 168,944 tokens INFO 04-17 02:07:18 [kv_cache_utils.py:637] Maximum concurrency for 32,768 tokens per request: 5.16x (VllmWorker rank=1 pid=159016) INFO 04-17 02:08:10 [custom_all_reduce.py:195] Registering 6499 cuda graph addresses (VllmWorker rank=0 pid=158999) INFO 04-17 02:08:13 [custom_all_reduce.py:195] Registering 6499 cuda graph addresses (VllmWorker rank=0 pid=158999) INFO 04-17 02:08:13 [gpu_model_runner.py:1626] Graph capturing finished in 55 secs, took 0.72 GiB (VllmWorker rank=1 pid=159016) INFO 04-17 02:08:13 [gpu_model_runner.py:1626] Graph capturing finished in 55 secs, took 0.72 GiB INFO 04-17 02:11:49 [core.py:163] init engine (profile, create kv cache, warmup model) took 297.90 seconds INFO 04-17 02:11:49 [core_client.py:435] Core engine process 0 ready. INFO 04-17 02:11:49 [launcher.py:26] Available routes are: INFO 04-17 02:11:49 [launcher.py:34] Route: /openapi.json, Methods: HEAD, GET INFO 04-17 02:11:49 [launcher.py:34] Route: /docs, Methods: HEAD, GET INFO 04-17 02:11:49 [launcher.py:34] Route: /docs/oauth2-redirect, Methods: HEAD, GET INFO 04-17 02:11:49 [launcher.py:34] Route: /redoc, Methods: HEAD, GET INFO 04-17 02:11:49 [launcher.py:34] Route: /health, Methods: GET INFO 04-17 02:11:49 [launcher.py:34] Route: /generate, Methods: POST INFO: Started server process [158788] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:20000 (Press CTRL+C to quit)

我没有看到error，应该是正常启动了，但是我看给的路由没有[/v1/chat/completions] 于是我使用如下测试命令：curl http://localhost:20000/generate -H "Content-Type: application/json" -d '{ "prompt": "什么是 XSS 攻击？", "temperature": 0.7 }' 得到了无用回答：{"text":["什么是 XSS 攻击？!!!!!!!!!!!!!!!!"]} 服务端的日志： INFO 04-17 02:49:32 [async_llm.py:228] Added request 23b0d47c34b941998962a38016c52314. INFO: 127.0.0.1:58018 - "GET / HTTP/1.1" 404 Not Found INFO: ::1:39670 - "POST /generate HTTP/1.1" 200 OK INFO 04-17 02:55:18 [async_llm.py:228] Added request cc2c7afebd4049978a7d38e603f90712.

第二次request是我又尝试问了一次，十几分钟还没有得到回答。 请问您部署的模型有遇到这个问题吗？如果没有，我是哪里出了问题

感激不尽！

请问找到这个问题的解决方法了吗，现在我也是这种情况，大佬解决了可以分享一下经验嘛

[black6816]

这个curl http://$HOST_NAME:$PORT/v1/chat/completions 是不是只能在vllm 0.7上用？在vllm0.8上怎么用呢？ @snoopy7713