Skip to content

Add stable,flaky syntax to --flaky_test_attempts - #18

Merged
alexeagle merged 1 commit into
8.7.0-figmafrom
aeagle/flaky-test-attempts-stable-flaky
Aug 7, 2026
Merged

Add stable,flaky syntax to --flaky_test_attempts#18
alexeagle merged 1 commit into
8.7.0-figmafrom
aeagle/flaky-test-attempts-stable-flaky

Conversation

@alexeagle

@alexeagle alexeagle commented Aug 7, 2026

Copy link
Copy Markdown

Summary

  • Extend --flaky_test_attempts to accept two comma-separated integers (<stable>,<flaky>) so CI can retry only flaky-marked targets without changing BUILD files.
  • Preserve backward compatibility: a bare integer (e.g. 3) still applies to all tests; default still means stable=1 / flaky=3.
  • Per-target regex@N overrides remain supported with a single integer only (regex@1,3 is rejected).

Upstream PR: bazelbuild#30638
Tracking issue: bazelbuild#30108

Semantics

Flag value flaky=False flaky=True
(unset) / default 1 3
3 3 3
1,3 1 3
1,5 1 5
//pkg/@3 (matched targets) 3 3

If upstream never merges, are we stuck carrying this patch forever?

No — but we would need to keep carrying it as long as we want --flaky_test_attempts=1,3 in .bazelrc.

This patch is small and isolated (~300 lines, mostly tests; core logic is FlakyTestAttempts + converter + TestStrategy lookup). It does not fork broad Bazel subsystems, and it rebases cleanly onto upstream releases (the upstream PR is the same diff against master).

If upstream merges: drop the fork patch on the next Bazel version bump / rebase from bazelbuild/bazel. No migration needed — behavior and flag syntax are identical.

If upstream does not merge: we have three options, none of which block upgrading Bazel:

  1. Keep cherry-picking the patch on each Bazel bump. Cost is low: touch-conflict surface is 3–4 files, and the change is backward-compatible so it does not perturb unrelated behavior.
  2. Revert the patch and change .bazelrc back to stock semantics. --flaky_test_attempts=3 still works on vanilla Bazel (retries all tests). We lose the stable/flaky split until we find another approach (e.g. integer flaky attribute, or regex@N lists).
  3. Stop using the new syntax without reverting the patch. Unused syntax in the binary is harmless; if CI never passes 1,3, behavior matches stock Bazel.

The patch is a capability opt-in, not a lock-in. Stock Bazel continues to work; we only depend on this fork delta for the specific .bazelrc value 1,3 (or other <stable>,<flaky> pairs).

Test plan

  • FlakyTestAttemptsTest — parsing and validation
  • TestAttemptsConverterTest — global, regex, and rejection of regex@1,3
  • test_flaky_test_attempts_stable_flaky_split shell test — verifies 1,2 split and bare integer backward compat
  • Full CI

Allow CI to set different retry counts for non-flaky and flaky-marked
tests via comma-separated values while preserving backward compatibility
for bare integer and regex@N overrides.
@alexeagle
alexeagle merged commit c58ec28 into 8.7.0-figma Aug 7, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants