test(cli): retry flaky subprocess tests in CI#39
Merged
Conversation
The evaluations/high-impact/* tests spawn CLI subprocesses and occasionally exceed the 15s timeout on a loaded runner (observed intermittently on unrelated PRs). Add `retry: 2` in CI to vitest's config for this package so a transient timeout is absorbed; a genuinely broken test still fails all attempts. Local runs stay strict (retry: 0). Verified the retry applies from packages/cli/vite.config.ts (the config vitest actually loads for this package — the root vitest.config.ts does not reach it). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vianmora
added a commit
that referenced
this pull request
Jul 20, 2026
The evaluations/high-impact/* tests spawn CLI subprocesses and occasionally exceed the 15s timeout on a loaded runner (observed intermittently on unrelated PRs). Add `retry: 2` in CI to vitest's config for this package so a transient timeout is absorbed; a genuinely broken test still fails all attempts. Local runs stay strict (retry: 0). Verified the retry applies from packages/cli/vite.config.ts (the config vitest actually loads for this package — the root vitest.config.ts does not reach it).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
packages/cli/evaluations/high-impact/fixture-api.test.tsintermittently fails withTest timed out in 15000ms. It spawns two CLI subprocesses (cold-start via tsx) against a local fixture API; on a loaded runner the two cold starts occasionally exceed the default 15s timeout. Seen on unrelated PRs (#36, #38), each time green on re-run — i.e. flaky, not a real regression.Fix
Add
test.retry: process.env.CI ? 2 : 0topackages/cli/vite.config.ts. In CI a transient timeout gets up to 3 attempts; a genuinely broken test still fails all three. Local runs stay strict.defineConfigfromvitest/config(superset of vite's — thebuildblock is unaffected) so thetestfield type-checks.test.retryhere does apply to this package's tests, whereas the rootvitest.config.tsdoes not (vitest loads the package-localvite.config.ts).Note
This is upstream's flaky test; the fix is generic and worth contributing upstream so it stops being fork divergence.
🤖 Generated with Claude Code