ci: PR fast tier — 6-leg sharded matrix + duration-balanced e2e groups - #124
Draft
CSSFrancis wants to merge 5 commits into
Draft
ci: PR fast tier — 6-leg sharded matrix + duration-balanced e2e groups#124CSSFrancis wants to merge 5 commits into
CSSFrancis wants to merge 5 commits into
Conversation
PR-tier CI legs split the suite into 2 duration-balanced shards (--splits/--group against spyde/tests/.test_durations); pytest-split falls back to count-based splitting when the durations file is absent. Lock delta is pytest-split 0.11.0 only (pure Python, depends on pytest).
Python matrix becomes two tiers behind one job definition: a matrix-prep job emits the matrix JSON on github.event_name. PRs run 6 legs (ubuntu 3.10-3.13, windows 3.12, macos 3.13) x 2 pytest-split shards; pushes to main keep the full 3 OS x 4 python matrix unsharded (group 0 = falsy = no --splits). Sharded legs time out at 20 min, main-tier legs keep 40. The extras job shards x2 on PRs the same way. Coverage stays on ubuntu-3.12 (both PR shards upload; Codecov merges server-side). e2e drops Playwright --shard: it balances by file count and parked one shard at 25.8 min next to an 8.0 one (two green main runs measured). SLOW_SPECS now lists every >=45s spec slowest-first and CI takes round-robin groups of each slice via SPYDE_E2E_GROUP (fast x4 + slow x5, all groups ~6-9 min of tests + ~1.5 min setup). Verified locally: the 9 groups partition the 137-file suite exactly, no overlap. Playwright retries are 0 under CI (a retry of a timed-out app boot doubles the damage); e2e job timeout drops 45 -> 25 min.
Spec discovery in playwright.config.ts is now recursive, so a spec file
added in a subdirectory of electron/tests/ lands in a CI slice instead
of silently never running ('**/'+name matches at any depth). A
SLOW_SPECS entry with no file on disk now throws at config load naming
the stale glob — a renamed slow spec previously self-healed into the
fast slice without a trace.
The e2e slow slice goes x5 -> x6 (worst group 8.9 -> 7.7 min of tests,
~9.3 min wall) toward the closer-to-5 wall-clock preference. Verified:
the 10 groups still partition the 137-file suite exactly, no overlap.
2657 entries from a full green run of spyde/tests/migrated on a Windows dev box (811 s of measured test time). Relative durations transfer well enough across OSes for 2-way shard balance; pytest-split treats unknown/new tests as average-cost, so drift degrades balance, never correctness.
The validation run showed round-robin parking progressive_signal_preview's ~7-min slot (2.3x its in-file time -- slow batch finalize on CI) with four other files in one 12.3-min group. SLOW_SPECS now carries a measured slot-weight per file and the slow slice greedy-bin-packs by weight, so that spec rides alone and every bin levels near it; editing a weight rebalances. Verified: the 10 groups partition all 137 spec files, 0 dupes / 0 missing. npm + Playwright-browser caches trim the ~2-min per-job e2e setup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets PR wall clock < 10 min (projected worst leg ~9.3 min, conservative). Pushes to main keep the full sweep unchanged.
Python matrix, two tiers, one job definition (a matrix-prep job emits the matrix JSON on
event_name;group: 0= unsharded, falsy in every expression, so main-tier job names/timeouts are byte-identical to today):spyde/tests/.test_durationsseeded from a full green local run (2657 entries); pytest-split falls back to count-based splitting if the file is missing/stale — degrades balance, never the run.e2e: replaces
--shard(count-balanced; reliably parked one shard at 2-3x the others — 25.8 vs 8.0 min measured) with duration-ordered round-robin groups: SLOW_SPECS now lists all 28 specs >= 45s slowest-first (the order IS the balancing),SPYDE_E2E_GROUP=k/ntakes every n-th file. fast x4 + slow x6 = 10 groups;playwright --listverified they partition the 137-file suite exactly (0 missing / 0 duplicated). Spec discovery is recursive with a loud config-load error on any stale SLOW_SPECS entry.retries: 0in CI (a retry of a timed-out boot doubles the damage; locally still 1), job timeout 45 -> 25. Local runs are unaffected (no env vars = whole suite).Durations measured from the two most recent green main runs; parser validated by reproducing the observed shard walls. PR-tier job count is 30 vs the 20-concurrent public cap — typecheck + unit legs finish inside a minute, so a few python shards queue briefly without moving the wall.
pytest-splitadded to the tests extra (lock diff: that package alone).