feat(ci): fill mainnet fixtures nightly, draft cached tests@ releases - #3100
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #3100 +/- ##
===================================================
+ Coverage 93.23% 93.37% +0.13%
===================================================
Files 624 624
Lines 36924 36990 +66
Branches 3373 3386 +13
===================================================
+ Hits 34427 34539 +112
+ Misses 1704 1677 -27
+ Partials 793 774 -19
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4750499 to
3feae97
Compare
spencer-tb
added
C-feat
danceratopz
left a comment
danceratopz
left a comment
There was a problem hiding this comment.
Could we add a cron trigger to the release workflow and avoid triggering the release stage/tagging if trigged by cron? Or are you trying to be considerate of runner usage? Being kind to the gigachungi is valid; just asking!
Otherwise, couple of comment below.
|
Using rotating artifacts with FIFO of size 5. Ready for re-review :) |
danceratopz
left a comment
danceratopz
left a comment
There was a problem hiding this comment.
Thanks for fixing these up! It still didn't sit right with me to add a new workflow that almost duplicates the workflow that we already had. I understand that the priority is to ensure that tests fill correctly, but by re-using the existing workflow, we can test that much more of the release machinery is working in the nightly. Also, if/when we move the release flow to pytest-split, this will also be present in the nightly, etc. I PR'd this suggestion so that .github/workflows/release_fixtures.yaml is used for nightly builds, too.
As a follow-up we could consider how we can create tests@.. from the nightly. Then we can basically have mainnet releases ready/on-demand.
Move the fill flags shared by all fixture releases (`-n`, `--no-html`, `--durations=100`, `--log-level=DEBUG`) out of the inline fill step in `.github/actions/build-fixtures` into a new `just fill-release` recipe. Callers append the feature's fill params, fork range and output via pass-through args; for repeated flags, the last occurrence wins. The xdist worker count is plumbed via `PYTEST_XDIST_AUTO_NUM_WORKERS` (`xdist: auto` in `evm.yaml` for all EVM types, unchanged). Reimplement `fill-nightly` as a thin wrapper around `fill-release` so the nightly fill shares the release flags. Deliberate changes to the nightly invocation: it gains `--no-html` and `--log-level=DEBUG`, and it drops flags that defaults already provide: `--dist=loadgroup` and the trailing `tests` path (both defaults in `pytest-fill.ini`), and the explicit `-n` (now supplied by `fill-release`, same value).
Replace the standalone nightly-fill workflow with a schedule trigger on `release_fixtures.yaml`: scheduled runs fill the new `nightly` feature (`--until=Amsterdam --generate-all-formats`, slow tests included) through the exact release pipeline (matrix generation, `build-fixtures`, index merge and tarball) and skip only the `release` job, so no tag or draft release is created. The nightly is thereby a daily rehearsal of the release flow itself. Scheduled runs skip when there are no new commits since the last successful scheduled run (query filtered with `event=schedule` so manual releases never advance the nightly baseline) and list the new commits in the step summary. `fail-fast` is disabled on schedule so every fork range reports. `run-name` distinguishes nightly runs from manual releases in the run history. Artifact retention replaces the FIFO rotation of the standalone workflow: the combined `fixtures_nightly.tar.gz` is kept for 5 days (roughly the last five nightly builds at a daily cadence) and split intermediates for 1 day, via a new `split_retention_days` input on the `build-fixtures` action. Manual releases keep the repo defaults. Remove the `fill-nightly` recipe: the nightly fill now goes through `fill-release` inside `build-fixtures`, gaining the fixture index and release tarball packaging that the standalone workflow lacked.
The `benchmark` key appeared twice with identical values. The two readers of this file disagree on which duplicate wins (the awk snippet in `build-evm-base` matches the first, `yaml.safe_load` keeps the last), so a future edit to only one copy would apply inconsistently. Keep the first occurrence.
Move the `evm` override check from inline bash in the release workflow's setup step into `validate_inputs`, where the rest of the dispatch input validation already lives and is unit-tested. The check becomes a YAML key lookup instead of a line-anchored grep.
Move the inline bash of the "Check for new commits" step into `.github/scripts/check_new_commits.py`, following the pattern of the other release helper scripts (`uv run`-able, unit-tested). The last-success baseline semantics, the dispatch short-circuit, the no-baseline bootstrap and the empty-commit-list handling now have unit tests (via a fake `gh` on `PATH`); the workflow step shrinks to a single command.
With the nightly skipping itself when there are no new commits, a quiet stretch longer than the five-day artifact retention would leave no live nightly tarball. Re-run the fill once the last successful scheduled run is four days old: a release-shaped artifact then always exists within the retention window, and the release pipeline keeps getting exercised during quiet periods.
spencer-tb
changed the title
danceratopz
left a comment
danceratopz
left a comment
There was a problem hiding this comment.
I think we're super close! Thanks a lot for the bug fix regarding "successful" empty runs (as no new commits)!
I think your last commit that removes the nightly features and simply uses tests@ as the nightly build makes a bunch of sense, let's do that!
There's just one thing missing imo to make this a killer feature that I think is worth shipping with this PR: The ability to basically one-click create a release from the last successful nightly artifact. I added a "Promote nightly" workflow here:
Using tests reduces coverage slightly (over the previous version), but I think that getting that back will require a follow-up. We could run multiple nightlies of relevant branches. But this is fine for now, especially at the late stage of Amsterdam fork dev and the fact that we'll probably bump the tests release feature to Amsterdam pretty soon Follow-up issue:
I also had this idea to make the nightly flow triggerable for easier testing but this is less important, just close if it's too much noise.
|
@danceratopz I pushed to 31 as discussed! |
Co-authored-by: spencer-tb <spencer.tb@ethereum.org>
The cached-release resolver read only the first page of git/matching-refs/tags/tests@ (30 refs). The listing is ordered by ref name, not version (tests@v9... sorts after tests@v20...), so once more than 30 tags exist the version-monotonicity and no-regression checks could baseline on a stale tag. Fetch every page with --paginate --slurp and flatten before taking the maximum. The canned tag response in the unit tests now spans two pages so every test exercises the flattening.
just bakes PYTEST_XDIST_AUTO_NUM_WORKERS into the -n value at parse
time, but pytest-xdist also reads the same env var as a numeric
worker-count override in -n auto mode, so transporting "auto"
through it made every fill and test run warn:
UserWarning: PYTEST_XDIST_AUTO_NUM_WORKERS is not a number:
{env_var!r}. Ignoring it.
The warning is truthful (xdist falls back to psutil CPU detection
either way), so export the env var empty to child processes: xdist
skips empty values silently, and numeric values never consult the
env var because they produce a plain -n N.
There was a problem hiding this comment.
Thanks @spencer-tb!! Enjoyed working on this one with you!
Please excuse me sneaking in the last commit. It was totally unrelated, but it does silence an irrelevant error.
|
Looks good thx! Just updating the PR desc/title to reflect everything then will merge :) |
spencer-tb
changed the title
tests@ releases
2 participants
🗒️ Description
TLDR: Fill the fixtures for mainnet forks nightly through the release workflow, and draft a
tests@release on demand from any nightly artifact in minutes.Run the fixture release workflow on a nightly cron (02:00 UTC). It fills the mainnet
testsfeature (all tests, slow included, all fixture formats) through the exact release pipeline but stops aftercombine: No tag and no release, just a rotatingfixtures_<commit>workflow artifact with 5 day retention that is always available.A unit-tested gate skips quiet nights with no new commits since the last nightly that actually filled. A failed or skipped nightly never advances that baseline, so no commit slips through unfilled, and a quiet stretch still refills after 4 days (or once the artifact has expired) so a live artifact always exists.
The nightly doubles as a
tests@release candidate on demand: Dispatching the workflow withcached=true(orcommit=<sha>to pick a specific nightly) skipsbuildandcombineand drafts the release from the nightly's tarball in minutes, tagged at the commit the nightly built. A unit-tested resolver guards the shortcut:testson the default branch only, the version must exceed the newesttests@tag, the nightly must contain the previous release and be an ancestor of the current head. The step summary lists any commits the release will not contain.Also extracts a shared
just fill-releaserecipe, moves the evm-override validation intogenerate_build_matrix.py, removes a duplicatebenchmarkentry fromevm.yaml, and exportsPYTEST_XDIST_AUTO_NUM_WORKERSempty from the Justfile so pytest-xdist stops warning on non-numeric values such asauto.Follow-ups
Deliberately not included: Dev-fork nightly fills (#3178) and failure notifications for red nightlies.
🔗 Related Issues or PRs
Complements the per-PR fill trim in #3096.
✅ Checklist
just static<type>(<area>):, where<type>and<area>come from an approrpriateC-<type>, respectivelyA-<area>, label. The title should match the a target squash commit message.Cute Animal Picture