Skip to content

Stabilize release UI validation and Xcode selection - #31

Merged
RNT56 merged 2 commits into
mainfrom
codex/retry-ui-smoke-shards
Jul 18, 2026
Merged

Stabilize release UI validation and Xcode selection#31
RNT56 merged 2 commits into
mainfrom
codex/retry-ui-smoke-shards

Conversation

@RNT56

@RNT56 RNT56 commented Jul 18, 2026

Copy link
Copy Markdown
Owner

What changed

  • prefer the GitHub runner image maintained stable /Applications/Xcode.app instead of the first lexicographic versioned install
  • keep versioned stable Xcodes as fallbacks and prerelease installs last, with an explicit PINES_XCODE_PATH override
  • give each hosted UI smoke shard one bounded retry on a newly cloned simulator while keeping the underlying Xcode invocation at one attempt
  • expand only the outer Xcode job budgets to 120 minutes in CI and 150 minutes for release validation; the per-process watchdog stays at 12 minutes
  • enforce the retry environment and outer budgets with workflow contract checks
  • add regression tests for Xcode candidate ordering and UI shard retry/exhaustion
  • document the seven-shard release policy and v0.1.1 preview procedure

Root cause

The initial release workflow ran the full UI target because it omitted the CI smoke-mode environment; #30 fixed that drift.

The first run of this PR then exposed two more infrastructure defects. Six smoke shards passed, while testArtifactsResearchComposerFlow hit Failed to get matching snapshots: Timed out while evaluating UI query on both fresh clones. The outer 90-minute job limit canceled the step just after the second failure, before the script could emit its final retry-exhausted error.

The job log also showed that select-xcode.sh chose Xcode 26.0.1. GitHub macOS images install several Xcode 26 versions and maintain the current stable release at /Applications/Xcode.app, but the old script accepted the first Xcode*.app glob entry. That silently pinned CI, CodeQL, and release validation to the oldest available toolchain.

No smoke journey was removed or weakened. A deterministic product failure must still fail independently on both clones, and each Xcode test process keeps its 720-second guard.

Validation

  • bash -n scripts/ci/*.sh
  • ShellCheck on all CI scripts
  • actionlint on all workflows
  • ruby -c scripts/ci/check-xcode-workflow-contracts.rb
  • bash scripts/ci/test-select-xcode.sh
  • bash scripts/ci/test-xcode-ui-shard-retry.sh
  • bash scripts/ci/check-xcode-workflow-contracts.sh
  • bash scripts/ci/check-public-hygiene.sh
  • local stable selector probe: Xcode 26.6 with the iOS 26.5 SDK and Metal tools
  • git diff --check

RNT56 added 2 commits July 18, 2026 12:58
Give each hosted UI smoke shard one bounded retry on a newly cloned simulator so transient XCUITest accessibility-service stalls can recover without masking deterministic failures. Keep CI and release settings aligned, document the seven-shard policy, and add a shell regression test for recovery, exhaustion, and invalid configuration.
Prefer the runner image's maintained stable Xcode instead of the oldest versioned install, keep prereleases as last-resort fallbacks, and regression-test candidate ordering. Expand the outer CI and release job budgets while retaining the strict per-shard watchdog and bounded retry policy.
@RNT56 RNT56 changed the title Retry stalled UI smoke shards on fresh clones Stabilize release UI validation and Xcode selection Jul 18, 2026
@RNT56
RNT56 merged commit 7d42ad9 into main Jul 18, 2026
13 checks passed
@RNT56
RNT56 deleted the codex/retry-ui-smoke-shards branch July 21, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant