Skip to content

fix(ci): don't cancel healthy jobs on sibling infra failures - #267

Merged
lvyufeng merged 1 commit into
candle-org:mainfrom
lvyufeng:fix-failfast-watcher
Mar 25, 2026
Merged

fix(ci): don't cancel healthy jobs on sibling infra failures#267
lvyufeng merged 1 commit into
candle-org:mainfrom
lvyufeng:fix-failfast-watcher

Conversation

@lvyufeng

Copy link
Copy Markdown
Contributor

Summary

  • ci-failfast-watcher.sh: Now checks failed step names against test/lint patterns. Infrastructure failures (checkout, install, setup) from sibling jobs are ignored — only real test failures trigger kill.
  • ci.yaml cancel steps: Changed from if: failure() to if: steps.<test-step>.outcome == 'failure', so checkout/install failures don't cascade-cancel the entire workflow.
  • ci-auto-retry.yaml: Changed from "don't retry if any real test failure exists" to "retry if any infra failure exists". Mixed failures (some real, some infra) now trigger retry — real failures will fail again but infra failures get another chance.

Problem

PR #222 had 5 failed jobs: 3 real test failures + 2 infra failures (checkout). The old logic:

  1. Watcher killed healthy jobs when a sibling failed at checkout
  2. Cancel steps fired on any step failure, not just test steps
  3. Auto-retry refused to retry because real test failures existed alongside infra failures

Test plan

  • Verify infra-only failures trigger auto-retry
  • Verify mixed failures (real + infra) trigger auto-retry
  • Verify real-only failures do NOT trigger auto-retry
  • Verify watcher ignores sibling infra failures
  • Verify watcher still kills on sibling real test failures

🤖 Generated with Claude Code

The failfast watcher and cancel-on-failure steps were treating all job
failures equally — a checkout timeout on one runner would cancel tests
running normally on other runners. Now:

1. ci-failfast-watcher.sh: only kills the current job when a sibling
   has a real test/lint failure, not infra failures (checkout, install)
2. ci.yaml cancel steps: use step outcome IDs so only test step failures
   trigger workflow cancellation, not checkout/install failures
3. ci-auto-retry.yaml: retries when any infra failure exists, even if
   mixed with real test failures (real failures will just fail again)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Candle AI Review

Scope

  • Files changed: 3 | Lines: +73 / -23
  • Areas: ci

Findings

No invariant violations detected.

PR Completeness

  • Linked issue: no
  • Test plan: filled
  • Template: correct

Risk: Low

CI/CD-only changes with no source modifications; no candle invariants are in scope.


Full review at commit 10a72a4 (2026-03-25 04:55 UTC)

@lvyufeng
lvyufeng merged commit 305ec9a into candle-org:main Mar 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant