Commit a08abea
authored
[Test] Limit DeepSeek R1 longseq accuracy prompts (vllm-project#10886)
## What this PR does / why we need it?
This PR adds num_prompts: 360 to the DeepSeek-R1-W8A8 longseq nightly
accuracy benchmark config.
The original benchmark may timeout because it runs too many samples from
the dataset. This change limits the number of evaluated prompts to 360,
reducing the nightly benchmark runtime while still keeping enough
samples for accuracy validation.
Changes:
Add num_prompts: 360 to the acc benchmark case.
increase `batch_size` from 16 to 32 to lower benchmark pressure.
Keep dataset_path, request_conf, dataset_conf, max_out_len, baseline,
and threshold unchanged.
## Does this PR introduce any user-facing change?
No. This PR only updates an internal nightly benchmark configuration and
does not introduce any user-facing API, interface, or behavior change.
## How was this patch tested?
Not run locally. This change only updates the nightly benchmark
configuration.
The updated benchmark is expected to be validated by the nightly CI.
- vLLM version: v0.23.0
- vLLM main:
vllm-project/vllm@967c5c3
---------
Signed-off-by: Madilyn537 <zhexuanwu12@gmail.com>1 parent 77de9e5 commit a08abea
1 file changed
Lines changed: 2 additions & 1 deletion
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| 109 | + | |
109 | 110 | | |
110 | | - | |
| 111 | + | |
111 | 112 | | |
112 | 113 | | |
0 commit comments