Skip to content

feat(ci): run the fixture release flow nightly - #28

Merged
spencer-tb merged 6 commits into
spencer-tb:ci/nightly-fillfrom
danceratopz:ci/nightly-fill-suggestions
Jul 14, 2026
Merged

feat(ci): run the fixture release flow nightly#28
spencer-tb merged 6 commits into
spencer-tb:ci/nightly-fillfrom
danceratopz:ci/nightly-fill-suggestions

Conversation

@danceratopz

Copy link
Copy Markdown

🗒️ Description

This makes the cron-trigger idea from the ethereum#3100 review thread concrete: instead of adding a parallel nightly workflow, the existing release_fixtures.yaml flow runs on a nightly schedule and stops after combine, so no tag or draft release is created. The nightly thereby becomes a daily rehearsal of the actual release pipeline: generate_build_matrix.py, build-fixtures, the index merge and the release tarball all get exercised every night, and the resulting fixtures_nightly.tar.gz artifact is release-shaped (merged .meta/index.json included), so it can be consumed directly.

Scheduled runs fill a new nightly feature in feature.yaml (--until=Amsterdam --generate-all-formats, slow tests included) and skip themselves when there are no new commits since the last successful scheduled run: the baseline query filters on event=schedule so manual releases never advance it, and a failed nightly keeps re-running until it goes green. fail-fast is disabled on schedule so every fork range reports, run-name distinguishes "Nightly Fill" runs from manual releases in the run history, and artifact retention (5 days for the combined tarball, 1 day for the per-range splits) replaces the FIFO rotation of the standalone workflow.

Commits, in order:

  • refactor(ci): Extract a shared fill-release just recipe holding the flags common to all fixture releases (previously inline in build-fixtures); callers append feature params, fork range and output via pass-through args (last flag wins). Fork splitting needs no special handling: --dist loadgroup and testpaths = tests/ are already defaults in pytest-fill.ini.
  • feat(ci): The workflow merge described above; deletes nightly-fill.yaml and the fill-nightly recipe.
  • chore(ci): Remove a duplicate benchmark entry from evm.yaml (the awk reader matches the first occurrence, yaml.safe_load keeps the last; identical today, a landmine tomorrow).
  • refactor(ci): Move the evm override validation from inline bash into generate_build_matrix.py, where the rest of the input validation lives and is unit-tested.
  • refactor(ci): Extract the "Check for new commits" inline bash into .github/scripts/check_new_commits.py with unit tests (dispatch short-circuit, baseline bootstrap, new/no commits, gh failure), following the other release helper scripts.

Deliberate trades versus the standalone nightly workflow, flagging for visibility:

  • Per-range fixture artifacts are no longer uploaded when a fill leg fails (the composite action only uploads on success); debugging a regressed fill falls back to the job logs.
  • A nightly with one failed range publishes nothing (release semantics: combine requires a fully green build); the baseline logic refills everything the next night.
  • The cron path cannot run in PR CI, so before merging feat(ci): fill mainnet fixtures nightly, draft cached tests@ releases ethereum/execution-specs#3100 it is worth one workflow_dispatch validation run on a fork, ideally also confirming that the empty-string retention-days on dispatch releases falls back to the repo default as expected.

Net effect on the workflow YAML: the 163-line nightly-fill.yaml is deleted and release_fixtures.yaml grows by ~45 lines of schedule-gated conditionals; the overall diff is dominated by the new script and its tests.

🔗 Related Issues or PRs

Targets the head branch of ethereum#3100; see the review discussion there for context.

✅ Checklist

  • All: Ran fast static checks to avoid unnecessary CI fails, see also Code Standards and Verifying Changes:
    just static
  • All: PR title have the form <type>(<area>):, where <type> and <area> come from an approrpriate C-<type>, respectively A-<area>, label. The title should match the a target squash commit message.

Move the fill flags shared by all fixture releases (`-n`, `--no-html`,
`--durations=100`, `--log-level=DEBUG`) out of the inline fill step in
`.github/actions/build-fixtures` into a new `just fill-release` recipe.
Callers append the feature's fill params, fork range and output via
pass-through args; for repeated flags, the last occurrence wins. The
xdist worker count is plumbed via `PYTEST_XDIST_AUTO_NUM_WORKERS`
(`xdist: auto` in `evm.yaml` for all EVM types, unchanged).

Reimplement `fill-nightly` as a thin wrapper around `fill-release` so
the nightly fill shares the release flags. Deliberate changes to the
nightly invocation: it gains `--no-html` and `--log-level=DEBUG`, and
it drops flags that defaults already provide: `--dist=loadgroup` and
the trailing `tests` path (both defaults in `pytest-fill.ini`), and
the explicit `-n` (now supplied by `fill-release`, same value).
Replace the standalone nightly-fill workflow with a schedule trigger on
`release_fixtures.yaml`: scheduled runs fill the new `nightly` feature
(`--until=Amsterdam --generate-all-formats`, slow tests included)
through the exact release pipeline (matrix generation, `build-fixtures`,
index merge and tarball) and skip only the `release` job, so no tag or
draft release is created. The nightly is thereby a daily rehearsal of
the release flow itself.

Scheduled runs skip when there are no new commits since the last
successful scheduled run (query filtered with `event=schedule` so
manual releases never advance the nightly baseline) and list the new
commits in the step summary. `fail-fast` is disabled on schedule so
every fork range reports. `run-name` distinguishes nightly runs from
manual releases in the run history.

Artifact retention replaces the FIFO rotation of the standalone
workflow: the combined `fixtures_nightly.tar.gz` is kept for 5 days
(roughly the last five nightly builds at a daily cadence) and split
intermediates for 1 day, via a new `split_retention_days` input on the
`build-fixtures` action. Manual releases keep the repo defaults.

Remove the `fill-nightly` recipe: the nightly fill now goes through
`fill-release` inside `build-fixtures`, gaining the fixture index and
release tarball packaging that the standalone workflow lacked.
The `benchmark` key appeared twice with identical values. The two
readers of this file disagree on which duplicate wins (the awk snippet
in `build-evm-base` matches the first, `yaml.safe_load` keeps the
last), so a future edit to only one copy would apply inconsistently.
Keep the first occurrence.
Move the `evm` override check from inline bash in the release
workflow's setup step into `validate_inputs`, where the rest of the
dispatch input validation already lives and is unit-tested. The check
becomes a YAML key lookup instead of a line-anchored grep.
Move the inline bash of the "Check for new commits" step into
`.github/scripts/check_new_commits.py`, following the pattern of the
other release helper scripts (`uv run`-able, unit-tested). The
last-success baseline semantics, the dispatch short-circuit, the
no-baseline bootstrap and the empty-commit-list handling now have unit
tests (via a fake `gh` on `PATH`); the workflow step shrinks to a
single command.
@danceratopz
danceratopz marked this pull request as draft July 13, 2026 10:39
With the nightly skipping itself when there are no new commits, a quiet
stretch longer than the five-day artifact retention would leave no live
nightly tarball. Re-run the fill once the last successful scheduled run
is four days old: a release-shaped artifact then always exists within
the retention window, and the release pipeline keeps getting exercised
during quiet periods.
@danceratopz
danceratopz marked this pull request as ready for review July 13, 2026 11:09
@spencer-tb
spencer-tb merged commit 78e25fb into spencer-tb:ci/nightly-fill Jul 14, 2026
9 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants