Skip to content

Commit a08abea

Browse files
authored
[Test] Limit DeepSeek R1 longseq accuracy prompts (vllm-project#10886)
## What this PR does / why we need it? This PR adds num_prompts: 360 to the DeepSeek-R1-W8A8 longseq nightly accuracy benchmark config. The original benchmark may timeout because it runs too many samples from the dataset. This change limits the number of evaluated prompts to 360, reducing the nightly benchmark runtime while still keeping enough samples for accuracy validation. Changes: Add num_prompts: 360 to the acc benchmark case. increase `batch_size` from 16 to 32 to lower benchmark pressure. Keep dataset_path, request_conf, dataset_conf, max_out_len, baseline, and threshold unchanged. ## Does this PR introduce any user-facing change? No. This PR only updates an internal nightly benchmark configuration and does not introduce any user-facing API, interface, or behavior change. ## How was this patch tested? Not run locally. This change only updates the nightly benchmark configuration. The updated benchmark is expected to be validated by the nightly CI. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@967c5c3 --------- Signed-off-by: Madilyn537 <zhexuanwu12@gmail.com>
1 parent 77de9e5 commit a08abea

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

tests/e2e/nightly/multi_node/internal_dp/config/DeepSeek-R1-W8A8-longseq.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,8 @@ benchmarks:
106106
dataset_path: vllm-ascend/gsm8k
107107
request_conf: vllm_api_general_chat
108108
dataset_conf: gsm8k/gsm8k_gen_0_shot_cot_chat_prompt
109+
num_prompts: 360
109110
max_out_len: 4096
110-
batch_size: 16
111+
batch_size: 32
111112
baseline: 95
112113
threshold: 5

0 commit comments

Comments
 (0)