Skip to content

Commit 52da9d9

Browse files
authored
[CI] Modify the threshold of the gsm8k lite dataset to 3 questions (vllm-project#11247)
### What this PR does / why we need it? This PR updates the accuracy threshold for the `gsm8k-lite` benchmark across various nightly and weekly configuration files (both single-node and multi-node setups). The threshold is relaxed to `10` to prevent transient test failures or strict accuracy checks from blocking the CI pipeline. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Tested via the existing E2E CI/nightly/weekly test pipelines. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@dc68bd8 Signed-off-by: chen-commits <1636718796@qq.com> Signed-off-by: chen <1636718796@qq.com>
1 parent 04ad252 commit 52da9d9

18 files changed

Lines changed: 18 additions & 18 deletions

tests/e2e/nightly/multi_node/internal_dp/config/DeepSeek-V3.1-BF16.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,4 +82,4 @@ benchmarks:
8282
max_out_len: 4096
8383
batch_size: 512
8484
baseline: 95
85-
threshold: 5
85+
threshold: 10

tests/e2e/nightly/multi_node/internal_dp/config/DeepSeek-V3_2-W8A8-A3-dual-nodes.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ benchmarks:
113113
max_out_len: 4096
114114
batch_size: 128
115115
baseline: 95
116-
threshold: 5
116+
threshold: 10
117117

118118
acc_aime2025:
119119
case_type: accuracy

tests/e2e/nightly/multi_node/internal_dp/config/DeepSeek-V3_2-W8A8-EP.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -265,4 +265,4 @@ benchmarks:
265265
max_out_len: 4096
266266
batch_size: 64
267267
baseline: 96.88
268-
threshold: 5
268+
threshold: 10

tests/e2e/nightly/multi_node/internal_dp/config/Qwen3-235B-A22B-A2.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,4 +71,4 @@ benchmarks:
7171
max_out_len: 7680
7272
batch_size: 256
7373
baseline: 96
74-
threshold: 5
74+
threshold: 10

tests/e2e/nightly/multi_node/internal_dp/config/Qwen3-235B-A22B-Mooncake-Layerwise.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,4 +112,4 @@ benchmarks:
112112
max_out_len: 7680
113113
batch_size: 512
114114
baseline: 97
115-
threshold: 3
115+
threshold: 10

tests/e2e/nightly/multi_node/internal_dp/config/Qwen3-235B-disagg-pd.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,4 +115,4 @@ benchmarks:
115115
max_out_len: 7680
116116
batch_size: 512
117117
baseline: 97
118-
threshold: 3
118+
threshold: 10

tests/e2e/nightly/single_node/models/configs/DeepSeek-R1-0528-W8A8.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ _benchmarks_acc: &benchmarks_acc
4545
max_out_len: 32768
4646
batch_size: 32
4747
baseline: 95
48-
threshold: 5
48+
threshold: 10
4949

5050
_benchmarks_perf: &benchmarks_perf
5151
perf:

tests/e2e/nightly/single_node/models/configs/GLM-4.7.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ _benchmarks: &benchmarks
4747
max_out_len: 4096
4848
batch_size: 8
4949
baseline: 95
50-
threshold: 5
50+
threshold: 10
5151

5252
# ==========================================
5353
# ACTUAL TEST CASES

tests/e2e/nightly/single_node/models/configs/Hy3-preview.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ _benchmarks: &benchmarks
4242
max_out_len: 4096
4343
batch_size: 8
4444
baseline: 93.07
45-
threshold: 5
45+
threshold: 10
4646

4747
# ==========================================
4848
# ACTUAL TEST CASES

tests/e2e/nightly/single_node/models/configs/Kimi-K2-Thinking.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ test_cases:
3737
max_out_len: 4096
3838
batch_size: 32
3939
baseline: 95
40-
threshold: 5
40+
threshold: 10
4141
perf:
4242
case_type: performance
4343
dataset_path: vllm-ascend/GSM8K-in3500-bs400

0 commit comments

Comments
 (0)