feat(ci): nightly fills of active dev-fork branches

[danceratopz]

Context

#3100 pointed the scheduled nightly at the mainnet tests feature (--until=BPO4) so the rotating fixtures_tests artifact is a release-ready tests@ candidate. That deliberately dropped the dev-fork coverage the interim nightly feature had (--until=Amsterdam): no scheduled run fills the active dev fork any more (all tests, slow included, all fixture formats). The dev fork is where fill regressions are most likely to land, and per-PR CI only runs a trimmed fill without slow tests.

This is acceptable right now: Amsterdam fork development is late-stage and the tests feature's --until will bump to Amsterdam soon, restoring coverage. The gap re-opens whenever the next dev-fork branch (e.g. forks/hegota) starts: this issue tracks closing it structurally.

Proposal

Keep the tests nightly exactly as is and add scheduled dev-fork fills parametrized by branch, e.g. a nightly-dev-forks entry in feature.yaml that lists the active dev-fork branches:

# Filled nightly per active dev-fork branch; never tagged or released.
nightly-dev-forks:
  evm-type: eels
  branches:
    forks/amsterdam: --until=Amsterdam --generate-all-formats
    # forks/hegota: --until=Hegota --generate-all-formats

The per-branch --until could alternatively be derived from the branch suffix, but an explicit map keeps generate_build_matrix.py trivial and makes fork rotation a one-line config edit.

Design considerations

• Trigger shape: the lightest option is a second cron line, dispatched on github.event.schedule (e.g. 02:00 for tests, 04:00 for the dev leg), keeping each scheduled run single-feature and staggering runner load. Fanning both features out of one scheduled run is cleaner conceptually but needs per-entry checkout refs in build and a per-feature combine (its feature_name/combine_labels outputs are scalars today).
• Commit gate: check_new_commits.py anchors on the checked-out branch's GITHUB_SHA against the last scheduled run that uploaded artifacts, workflow-global. With two legs, each would advance the baseline the other sees. Either discriminate legs by artifact name in last_real_nightly() (this interacts with the artifact-liveness anchoring from feat(ci): fill mainnet fixtures nightly, draft cached tests@ releases #3100), or let the dev leg skip the gate and fill unconditionally: the active fork changes most days anyway.
• Combine is optional for the dev leg: if the goal is "ensure all tests fill", the per-range job results are the signal and the leg could stop after build. A combined Amsterdam artifact has consumption value too (validating clients against incoming dev-fork tests), so leaning towards keeping it.
• Fork-order lookup: scheduled runs execute the default branch's scripts, so an --until fork missing from FORK_ORDER in the default branch's generate_build_matrix.py would KeyError in applicable_ranges. The follow-up must handle unknown forks explicitly (fail with a clear message, or fill unsplit).
• Promotion safety: promote_nightly.py selects candidates by the fixtures_tests artifact name, so dev-fork artifacts (e.g. fixtures_nightly-dev-forks) can never be promoted to a mainnet release. Keep that name-based discrimination when naming the dev-leg artifacts.
• Runner load: a full dev-fork fill roughly doubles nightly gigachungus usage; stagger the cron and reuse the existing skip/refresh logic where possible.

Related

• feat(ci): fill mainnet fixtures nightly, draft cached tests@ releases #3100: nightly mainnet fill via the release workflow (dropped dev-fork coverage, flagged as a deliberate follow-up).

[spencer-tb]

Thanks! All LGTM, only comment I had is we should use the devnet: feature instead of adding a new nightly-dev-forks feature!

The assumption here for me is that this CI will only run in our main branch, so only tests that are in our main branch will run.