Skip to content

Commit ee8bbbd

Browse files
czydyyzhangxinyuehfad
czydyy
and
zhangxinyuehfad
authored
[Test] add AOP failure interception and bisect pipeline for single/multi-node testsFeature bridge bisect (vllm-project#10937)
### What this PR does / why we need it? Adds an AOP (Aspect-Oriented Programming) failure interception and bisect pipeline for single-node and multi-node nightly E2E tests. When a test fails, the pipeline: 1. Classify: Scans pod logs for environment error patterns (rules-env.txt) to distinguish environment issues from real code bugs 2. Check commit age: Looks up the last successful run in good_table.csv; skips bisect for failures older than 3 days 3. Trigger bisect: Calls auto_bisect for recent, non-environment failures to identify the first bad commit 4. Update good_table: On success, updates the shared good_table.csv with the current commit info Both single-node (GHA workflow steps) and multi-node (in-pod aop_pipeline()) share the same decision logic. ### Does this PR introduce _any_ user-facing change? No. All AOP features are disabled by default (aop_single_enabled: false, aop_multi_enabled: 'false'). Enable by passing aop_single_enabled: true or aop_multi_enabled: 'true' from the calling workflow. ### How was this patch tested? - Multi-node AOP pipeline tested on A2 cluster with Kimi-K2_5-W4A8-A2-dual-nodes: verified classify (env/not_env), commit age lookup, worker-leader synchronization - Single-node AOP pipeline tested with Qwen3.5-27B-w8a8-A2: verified capture → classify → check age → trigger bisect flow - CSV table lookup tested with good_table.csv: name matching, success row filtering, date parsing - Worker pod log collection verified via PVC shared storage - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: chenzeyu <2978509328@qq.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com> Co-authored-by: hfadzxy <starmoon_zhang@163.com>
1 parent 271d251 commit ee8bbbd

11 files changed

Lines changed: 901 additions & 7 deletions

File tree

‎.github/workflows/_e2e_nightly_multi_node.yaml‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,11 @@ on:
7373
default: 120
7474
type: string
7575
description: test case execution timeout period,default:120 minutes
76+
aop_multi_enabled:
77+
required: false
78+
type: boolean
79+
default: false
80+
description: enable Pod AOP pipeline on failure
7681
secrets:
7782
KUBECONFIG_B64:
7883
required: true
@@ -225,6 +230,8 @@ jobs:
225230
-D pvc_name="$pvc_name" \
226231
-D benchmark_job_name="$benchmark_job_name" \
227232
-D runner="$runner" \
233+
-D aop_multi_enabled="${{ inputs.aop_multi_enabled }}" \
234+
-D good_table="/root/.cache/vllm-ascend/${{ inputs.vllm_ascend_branch }}/nightly/good_table.csv" \
228235
--outfile lws.yaml
229236
230237
kubectl apply -f ./lws.yaml
@@ -301,13 +308,46 @@ jobs:
301308
pids+=($!)
302309
done
303310
304-
kubectl logs -f "$LEADER_POD" -n "$NAMESPACE" | while IFS= read -r line; do
311+
LEADER_TMP=$(mktemp)
312+
kubectl logs -f "$LEADER_POD" -n "$NAMESPACE" | tee "$LEADER_TMP" | while IFS= read -r line; do
305313
echo "$line"
306314
if echo "$line" | grep -q "$FAIL_TAG"; then
307315
exit 1
308316
fi
309317
done
310318
319+
eval "$(awk '
320+
/^vLLM Git information$/ { sec="vllm"; next }
321+
/^vLLM-Ascend Git information$/ { sec="vllm_ascend"; next }
322+
sec && /^Commit hash: / {
323+
print "HASH_" sec "=" $NF
324+
sec=""
325+
}
326+
' "$LEADER_TMP")"
327+
echo "vllm_hash=${HASH_vllm:-N/A}" >> "$GITHUB_OUTPUT"
328+
echo "vllm_ascend_hash=${HASH_vllm_ascend:-N/A}" >> "$GITHUB_OUTPUT"
329+
rm -f "$LEADER_TMP"
330+
331+
- name: Update good_table.csv on success
332+
# if: ${{ steps.stream_logs.outcome == 'success' && inputs.request_id == '' }}
333+
if: ${{ steps.stream_logs.outcome == 'success'}}
334+
env:
335+
TEST_NAME: ${{ inputs.name || inputs.config_file_path }}
336+
TEST_PATH: ${{ inputs.config_file_path }}
337+
CONFIG_BASE_PATH: ${{ inputs.config_base_path }}
338+
RUN_LINK: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
339+
run: |
340+
python3 tests/e2e/nightly/scripts/update_good_table.py \
341+
--cache-csv "/root/.cache/vllm-ascend/${{ inputs.vllm_ascend_branch || 'main' }}/nightly/good_table.csv" \
342+
--test-name "$TEST_NAME" \
343+
--test-path "$TEST_PATH" \
344+
--config-base-path "$CONFIG_BASE_PATH" \
345+
--scene "multi_node" \
346+
--run-link "$RUN_LINK" \
347+
--vllm-ascend-dir "." \
348+
--vllm-ascend-version "${{ steps.stream_logs.outputs.vllm_ascend_hash }}" \
349+
--vllm-version "${{ steps.stream_logs.outputs.vllm_hash }}"
350+
311351
- name: Fetch benchmark results from PVC
312352
if: always()
313353
run: |

‎.github/workflows/_e2e_nightly_single_node.yaml‎

Lines changed: 130 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,11 @@ on:
6464
default: 120
6565
type: string
6666
description: test case execution timeout period,default:120 minutes
67+
aop_single_enabled:
68+
required: false
69+
type: boolean
70+
default: false
71+
description: enable AOP hooks (capture / classify / skip-or-process)
6772
secrets:
6873
OBS_ACCESS_KEY_ID:
6974
required: false
@@ -220,6 +225,7 @@ jobs:
220225
fi
221226
222227
- name: Run Pytest (py-driven)
228+
id: pytest-driven
223229
if: ${{ inputs.tests != '' }}
224230
env:
225231
VLLM_WORKER_MULTIPROC_METHOD: spawn
@@ -229,10 +235,14 @@ jobs:
229235
run: |
230236
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
231237
echo "Running pytest with tests path: ${{ inputs.tests }}"
238+
mkdir -p /tmp/test-logs
239+
set -o pipefail
232240
pytest -sv "${{ inputs.tests }}" \
233-
--ignore=tests/e2e/nightly/single_node/ops/singlecard_ops/test_fused_moe.py
241+
--ignore=tests/e2e/nightly/single_node/ops/singlecard_ops/test_fused_moe.py \
242+
2>&1 | tee /tmp/test-logs/pytest-driven.log
234243
235244
- name: Run Pytest (YAML-driven)
245+
id: yaml-test
236246
if: ${{ always() && inputs.config_file_path != '' }}
237247
env:
238248
VLLM_WORKER_MULTIPROC_METHOD: spawn
@@ -246,7 +256,125 @@ jobs:
246256
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
247257
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc
248258
echo "Running YAML-driven test with config: ${{ inputs.config_file_path }}"
249-
pytest -sv tests/e2e/nightly/single_node/models/scripts/test_single_node.py
259+
mkdir -p /tmp/test-logs
260+
set -o pipefail
261+
pytest -sv tests/e2e/nightly/single_node/models/scripts/test_single_node.py \
262+
2>&1 | tee /tmp/test-logs/yaml-test.log
263+
264+
- name: Update good_table.csv on success
265+
if: >-
266+
${{ always() &&
267+
(steps.pytest-driven.outcome == 'success' || steps.pytest-driven.outcome == 'skipped') &&
268+
(steps.yaml-test.outcome == 'success' || steps.yaml-test.outcome == 'skipped') &&
269+
(steps.pytest-driven.outcome == 'success' || steps.yaml-test.outcome == 'success') }}
270+
working-directory: /vllm-workspace/vllm-ascend
271+
env:
272+
TEST_NAME: ${{ inputs.name || inputs.config_file_path || inputs.tests }}
273+
TEST_PATH: ${{ inputs.config_file_path || inputs.tests }}
274+
CONFIG_BASE_PATH: ${{ inputs.config_base_path }}
275+
RUN_LINK: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
276+
VLLM_ASCEND_VERSION: ${{ env.VLLM_ASCEND_VERSION }}
277+
run: |
278+
python3 tests/e2e/nightly/scripts/update_good_table.py \
279+
--cache-csv "/root/.cache/vllm-ascend/${{ inputs.vllm_ascend_branch }}/nightly/good_table.csv" \
280+
--test-name "$TEST_NAME" \
281+
--test-path "$TEST_PATH" \
282+
--config-base-path "$CONFIG_BASE_PATH" \
283+
--run-link "$RUN_LINK" \
284+
--vllm-dir "/vllm-workspace/vllm" \
285+
--vllm-ascend-dir "/vllm-workspace/vllm-ascend" \
286+
--vllm-ascend-version "$VLLM_ASCEND_VERSION"
287+
288+
# ============================================
289+
# AOP hooks: intercept test results uniformly
290+
# ============================================
291+
292+
- name: Capture test result
293+
id: test-result
294+
if: ${{ always() && inputs.aop_single_enabled && (inputs.config_file_path != '' || inputs.tests != '') }}
295+
working-directory: /vllm-workspace/vllm-ascend
296+
run: |
297+
bash tests/e2e/nightly/scripts/aop_capture.sh \
298+
"${{ steps.yaml-test.outcome }}" \
299+
"${{ steps.pytest-driven.outcome }}"
300+
301+
- name: Classify failure
302+
id: classify
303+
if: ${{ always() && inputs.aop_single_enabled && steps.test-result.outputs.result == 'failure' }}
304+
working-directory: /vllm-workspace/vllm-ascend
305+
run: |
306+
bash tests/e2e/nightly/scripts/aop_classify.sh \
307+
"${{ steps.test-result.outputs.failed_test }}"
308+
309+
# ============================================================
310+
# Decision gate:
311+
# env_failure → Skip directly (no need to check age)
312+
# not_env_failure → Check commit age → old? Skip : Process
313+
# ============================================================
314+
315+
- name: Skip - env issue
316+
if: >-
317+
${{ always() && inputs.aop_single_enabled &&
318+
steps.classify.outputs.failure_type == 'env_failure' }}
319+
working-directory: /vllm-workspace/vllm-ascend
320+
run: |
321+
bash tests/e2e/nightly/scripts/aop_skip.sh \
322+
"env_failure" "" "" "" \
323+
"${{ steps.test-result.outputs.pytest_summary }}" \
324+
"${{ steps.test-result.outputs.yaml_summary }}"
325+
326+
- name: Check commit age
327+
id: commit-age
328+
if: ${{ always() && inputs.aop_single_enabled && steps.classify.outputs.failure_type == 'not_env_failure' }}
329+
working-directory: /vllm-workspace/vllm-ascend
330+
run: |
331+
bash tests/e2e/nightly/scripts/aop_commit_age.sh \
332+
"${{ inputs.name || inputs.config_file_path || inputs.tests }}" \
333+
"/root/.cache/vllm-ascend/${{ inputs.vllm_ascend_branch }}/nightly/good_table.csv"
334+
335+
- name: Skip - old commit
336+
if: >-
337+
${{ always() && inputs.aop_single_enabled &&
338+
steps.classify.outputs.failure_type == 'not_env_failure' &&
339+
steps.commit-age.outputs.is_old == 'true' }}
340+
working-directory: /vllm-workspace/vllm-ascend
341+
run: |
342+
bash tests/e2e/nightly/scripts/aop_skip.sh \
343+
"not_env_failure" \
344+
"${{ steps.commit-age.outputs.last_status }}" \
345+
"${{ steps.commit-age.outputs.last_date }}" \
346+
"${{ steps.commit-age.outputs.commit_age_days }}" \
347+
"${{ steps.test-result.outputs.pytest_summary }}" \
348+
"${{ steps.test-result.outputs.yaml_summary }}"
349+
350+
- name: Process - recent real failure
351+
if: >-
352+
${{ always() && inputs.aop_single_enabled &&
353+
steps.classify.outputs.failure_type == 'not_env_failure' &&
354+
steps.commit-age.outputs.is_old == 'false' }}
355+
working-directory: /vllm-workspace/vllm-ascend
356+
env:
357+
GOOD_TABLE: /root/.cache/vllm-ascend/${{ inputs.vllm_ascend_branch }}/nightly/good_table.csv
358+
run: |
359+
bash tests/e2e/nightly/scripts/aop_process.sh \
360+
"${{ steps.classify.outputs.failure_type }}" \
361+
"${{ steps.commit-age.outputs.commit_age_days }}" \
362+
"${{ inputs.runner }}" \
363+
"${{ inputs.tests }}" \
364+
"${{ inputs.config_file_path }}" \
365+
"${{ steps.test-result.outputs.pytest_summary }}" \
366+
"${{ steps.test-result.outputs.yaml_summary }}" \
367+
"single_node" \
368+
"HEAD" \
369+
"" \
370+
"${{ inputs.name }}"
371+
372+
- name: On success
373+
if: ${{ always() && inputs.aop_single_enabled && steps.test-result.outputs.result == 'success' }}
374+
run: |
375+
echo ">>> All tests passed!"
376+
377+
# ============================================
250378
251379
- name: Set benchmark artifact timestamp
252380
if: ${{ always() && inputs.config_file_path != '' }}

‎tests/e2e/nightly/multi_node/scripts/lws.yaml.jinja2‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,10 @@ spec:
4545
value: {{ runner | default("linux-aarch64-a3-0") }}
4646
- name: VLLM_ASCEND_VERSION
4747
value: {{ vllm_ascend_ref | default("main") }}
48+
- name: AOP_MULTI_ENABLED
49+
value: "{{ aop_multi_enabled }}"
50+
- name: GOOD_TABLE
51+
value: "{{ good_table | default("/root/.cache/vllm-ascend/nightly/good_table.csv") }}"
4852
command:
4953
- sh
5054
- -c
@@ -116,6 +120,10 @@ spec:
116120
value: {{ benchmark_job_name | default("") }}
117121
- name: VLLM_CI_RUNNER
118122
value: {{ runner | default("linux-aarch64-a3-0") }}
123+
- name: AOP_MULTI_ENABLED
124+
value: "{{ aop_multi_enabled }}"
125+
- name: GOOD_TABLE
126+
value: "{{ good_table | default("/root/.cache/vllm-ascend/nightly/good_table.csv") }}"
119127
command:
120128
- sh
121129
- -c

0 commit comments

Comments
 (0)