Retry Unity tests on license contention with jittered backoff#4703
Open
allister-beamable wants to merge 5 commits into
Open
Retry Unity tests on license contention with jittered backoff#4703allister-beamable wants to merge 5 commits into
allister-beamable wants to merge 5 commits into
Conversation
allister-beamable
force-pushed
the
ci/unity-license-retry-with-jitter
branch
2 times, most recently
from
ed4d38b to
bd56d9b
Compare
The PR Build Unity matrix fired all 14 jobs (10 tests + 4 device builds)
at once against 3 Unity Pro serials of 2 seats each. With demand (14) far
above the 6-seat ceiling, a random subset of jobs lost the activation race
each run and failed with '404 / found 0 entitlement groups', producing red
checks that are pure license contention rather than real defects. Over the
last ~4 weeks this hit ~half of all PR Build runs.
Shard the Unity work into one job per serial, each capped at
max-parallel: 2, with serials assigned so no two jobs share a serial:
- testGroupDefault -> UNITY_SERIAL (2021.3.29, 2022.3.7, 6000.0.37)
- testGroupBuild -> UNITY_SERIAL_BUILD (6000.2.8, 6000.3.0)
- build -> UNITY_SERIAL_DEVOPS (all 4 device platforms)
Concurrent activations on any serial can now never exceed its 2 seats.
Per-cell check names ('Tests {ver} {mode}', 'Build for {platform}') are
unchanged, so branch-protection required checks keep matching.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run 27972570615 showed that capping each per-serial shard at max-parallel 2 still tripped "No valid Unity Editor license found": the license return of a finishing matrix leg overlaps the activation of the next, momentarily demanding a third seat against a 2-seat serial. Dropping to 1 leaves a full seat of headroom for that return/activate transition. Costs ~2x wall clock on the Unity test stage (~30m -> ~60m) at identical runner-minutes and halved peak runner concurrency. Does not address cross-run contention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a Unity test shard fails without producing any result XML, the editor exited before tests ran -- the signature of a license/seat activation failure. Emit a GitHub ::error annotation naming license contention so the failed job is self-explanatory at a glance instead of presenting only a generic "docker failed with exit code 1". Uses artifact presence rather than parsing the editor stdout, which a later step cannot read back; all Unity jobs run on uniform ubuntu-latest hosts, so the signal is consistent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Register a problem matcher before each Unity test shard's game-ci step so the
literal license-contention lines ("No valid Unity Editor license found",
"Serial number unavailable for ULF return") surface as error annotations live,
during the run. This is the belt to the XML-check step's suspenders: the matcher
gives an immediate mid-run tidbit quoting Unity verbatim, while the post-failure
XML check classifies whether the job died before tests ran. Problem matchers are
a native runner feature (no dependency). Whether they scan container-action
stdout is unconfirmed and should be verified on a real run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a composite action wrapping game-ci/unity-test-runner that retries up to 3 times with exponential backoff plus jitter, but only when a run produced no result XML (the signature of a license/seat activation failure, which aborts before tests execute). Runs that produced results -- pass or fail -- surface immediately, so genuine test failures are never masked or slowed. Wired into the two unity-test-runner shards (testGroupDefault, testGroupBuild); device builds use unity-builder and are out of scope. Shard timeouts raised 30 -> 75 to cover the extra attempts. Stacked on the max-parallel:1 change (#4702), which targets the intra-run transition race; this targets residual cross-run contention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
allister-beamable
force-pushed
the
ci/unity-license-serial-throttle
branch
from
7ec0e37 to
21d5bf5
Compare
allister-beamable
force-pushed
the
ci/unity-license-retry-with-jitter
branch
from
bd56d9b to
018ed66
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
May be unnecessary — read first
This PR is provisional and may not be worth merging. It is a safety net for cross-run Unity license-seat contention that stacks on top of #4702. If #4702 (the
max-parallel: 1per-serial cap) prevents 90% or more of the license-contention failures on its own — which is plausible, since the failure we diagnosed was an intra-run transition race that #4702 deterministically closes — then this retry harness is not worth the ~190 lines and the maintenance of a failure-classification heuristic. Recommended: merge #4702, watch PR CI for a week or two, and only merge this if cross-runNo valid Unity Editor license foundfailures actually keep appearing.What this does
Adds
.github/actions/unity-test-with-retry, a composite action wrappinggame-ci/unity-test-runner@v4that retries up to 3 times with exponential backoff plus jitter — but only on license-activation failures, never on genuine test failures.backoff * 2^(N-1) + rand(0..jitter)seconds (defaults 30s base, 45s jitter), so concurrent contenders desynchronize instead of retrying in lockstep.unity-test-runnershards (testGroupDefault,testGroupBuild). Device builds useunity-builder(different action, build output not test XML) and are out of scope.timeout-minutesraised 30 → 75 to cover the extra attempts (composite-action steps cannot set per-steptimeout-minutes, so the bound is at the job level).Relationship to #4702
Base branch is
ci/unity-license-serial-throttle(#4702) — this is stacked on it and should merge after it. #4702 caps each serial at one concurrent activation within a run, fixing the transition race that caused the original failure (run 27972570615). This PR only addresses the residual case of a second concurrent run contending for the same serial, whichmax-parallelcannot see.Caveats / not yet validated
checkNamecheck; duplicate check-run annotations are possible.🤖 Generated with Claude Code