[PM-33025] ci: Speed up Test workflows

[vvolkgang]

🎟️ Tracking

PM-33025

📔 Objective

Test workflows tune-up inbound! While at it, updated the branch name patterns too. In my testing, some of these changes didn't net obvious runtime gains but they may require merging to main first and analysing future runs.

Performance improvements I confirmed had immediate gains:

1. Large runners - better hardware, everything runs faster
2. Remove postCompileScripts (SwiftLint, SwiftFormat lint) and postBuildScripts (Settings.bundle version bump) so they no longer run on every build, these should run as git hooks instead. Found them by looking at the .xcresult files where we can confirm these tasks were taking a big chunk and weren't parallelised

3. Caching DerivedData
4. Split cache into separate restore/save steps so caches are only saved on push to main, similar to the android setup-gradle action approach - PRs cache were reducing cache effectiveness by evicting main branch cache.

Improvements that need further testing:

1. Xcode parallelisation flags suggested by @fedemkr - looking at .xcresult we can see tasks are already being parallelised but, we'll have to test these flags in future PR runs
2. Enabling Xcode build cache - based on https://bitrise.io/blog/post/lifting-the-hood-on-build-cache-for-xcode
3. Disabling Compiler Index - responsible for Autocomplete and rich navigation in Xcode, which we don't need in our CI environment https://stackoverflow.com/questions/71109718/what-is-the-compiler-index-store-enable-xcode-build-settings-default-value
4. set IgnoreFileSystemDeviceInodeChanges so Xcode treats restored DerivedData files as up-to-date - source: https://stackoverflow.com/questions/53753511/is-it-possible-to-copy-an-xcode-derived-data-cache

Changes that were tested but removed:

1. Restore file timestamps via chetan/git-restore-mtime-action - didn't see an immediate runtime impact and I'm hoping IgnoreFileSystemDeviceInodeChanges is enough to deal with this. If not, we should look into using the original python script instead of introducing an action: https://github.com/MestreLion/git-tools/blob/main/git-restore-mtime
2. Caching homebrew dependencies - didn't save us much, we may introduce it later if needed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.54%. Comparing base (acacec0) to head (be08f45).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2739      +/-   ##
==========================================
- Coverage   80.79%   78.54%   -2.25%     
==========================================
  Files        1019     1142     +123     
  Lines       64854    72208    +7354     
==========================================
+ Hits        52400    56718    +4318     
- Misses      12454    15490    +3036     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[withinfocus]

Early review, since I was analyzing this for Actions approvals:

• Keep actions/cache (combined) for Mint and SPM. They're stable, lockfile-keyed, small, and the combined action already avoids re-saving them. Converting them adds two extra steps each and abandons the familiar, post-phase-managed pattern for no benefit. The save steps the PR adds for them are gated cache-hit != 'true' and ref == main which is functionally close to what the combined action already did on its own, minus the PR-branch save (and PR-branch saves of a lockfile-hashed cache are cheap and rare anyway).
• Use the restore / save split only for DerivedData, where it's load-bearing.

Does DerivedData need to be keyed by github.sha at all though? The per-SHA key is what forces the manual save. If a more stable key were viable (it usually isn't for DerivedData, since it depends on all sources -- this is why incremental-build caches universally use prefix+save-control), the whole manual apparatus would dissolve.

[vvolkgang]

• (and PR-branch saves of a lockfile-hashed cache are cheap and rare anyway).

Only changed those too because I found that wasn't the case, between renovate PRs, SDK updates and team PRs that modified these files a big chunk of those caches came from branches other than main, when the file hash didn't match and it used the fallback macOS-spm- key most often than not that would be a cache that didn't came from main too, and while it had a cache-hit it wasn't actually useful. Took me a while to figure why the profiler was still showing time spent there.

Does DerivedData need to be keyed by github.sha at all though?

Any ideas? Currently it's trying to avoid the same issue we had with the other caches, save step if: ensures we only save in main, commit sha ensures it's unique per merge to main and when commit sha doesn't match it falls back to a relevant cache. When I tested caching incremental changes it didn't impact build times much, and with each DerivedData cache being >700MB, their upload consistently increased runtime. Not caching PR updates saved 3-4min per run overall, IIRC.

[withinfocus]

Only changed those too because I found that wasn't the case, between renovate PRs, SDK updates and team PRs that modified these files a big chunk of those caches came from branches other than main, when the file hash didn't match and it used the fallback macOS-spm- key most often than not that would be a cache that didn't came from main too, and while it had a cache-hit it wasn't actually useful. Took me a while to figure why the profiler was still showing time spent there.

Makes sense and is justified then.

Any ideas? Currently it's trying to avoid the same issue we had with the other caches, save step if: ensures we only save in main, commit sha ensures it's unique per merge to main and when commit sha doesn't match it falls back to a relevant cache. When I tested caching incremental changes it didn't impact build times much, and with each DerivedData cache being >700MB, their upload consistently increased runtime. Not caching PR updates saved 3-4min per run overall, IIRC.

I vibed a bit on this one and alas, no, so you're on the right track most likely. Said vibe:

Having thought about it more: yes, your scheme is the right idiom, and I'd argue it's actually required, not just convenient: GitHub Actions caches are immutable once written, so you can't update a fixed key like …-deriveddata-main; the second save just no-ops with "cache already exists". A rolling unique suffix is the only way to push a fresh snapshot on each merge, and the restore-keys prefix then always pulls the newest one. github.sha is a fine token for that (github.run_id would behave identically); the sha has the minor edge of tying the entry to exact commit content.

One refinement worth considering on the key prefix: fold the Xcode/toolchain version (and maybe Package.resolved) in, e.g. ${{ runner.os }}-deriveddata-testbwpm-${xcode}-${{ github.sha }}. DerivedData restored across an Xcode bump or a major dependency change can carry stale incremental state, and since you're also setting IgnoreFileSystemDeviceInodeChanges (telling Xcode to trust restored files as up-to-date), the usual safety net is off. Pinning the prefix to the toolchain means a runner-image upgrade cleanly invalidates instead of silently restoring mismatched module caches.

Last thing, genuinely curious rather than blocking: once PR incremental saves are gone, what's the measured win from the DerivedData restore by itself, separate from the xlarge runner and the postCompileScript removal? You mentioned incremental caching didn't move build times much, so I want to make sure the 700MB entry is still earning its keep on the restore side.