fix(nightly): fail on empty filters, correct the coverage claim, detect upstream drift - #349
Merged
Conversation
…ct upstream drift The nightlies' job changed when PRs started testing what changed: they are no longer the safety net for changed code, they are the only thing that can catch what changes when the repo does NOT. Three fixes: 1. Silent green. nightly-replay and nightly-agents-smoke exited 0 when a filter matched no tests, so a typo'd or stale dispatch filter reported success having run nothing. Now a loud failure. 2. Stale claim. nightly-oracle advertised '~78 checks' in two places; the suite actually has 27 (11 auto-covered exact-match benchmarks + 16 pinned SPECIAL). Comment now states the real number and how to list it (ORACLE_LIST=1). 3. Upstream drift — the gap no PR can close. The input hash sees the repository, so an upstream rebuild (a new python:3.12-slim, a moved :latest) changes what our images contain while every hash and every freshness verdict stays put; mmmu's dataset 404'd for months exactly that way. containers/scripts/external-drift.sh resolves the digest of all 18 external bases (from fleet-hash's own externals column, so it cannot drift from what we build FROM) and warns per moved digest; the Fleet status workflow now also runs nightly, comparing against the previous run's digests via the Actions cache and publishing the table to the job summary. Report-only: the response to drift is a force_rebuild dispatch, which stays a human decision. Deliberately NOT done: filtering the nightlies by the affected set. On a quiet day that set is empty, so a filtered nightly would test nothing — exactly when drift detection matters most. Verified live: 18 bases resolved; seeding a previous-digest file with one altered digest produces the warning and exit 1, and an unchanged file exits 0. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightlies' role changed when PRs started testing what changed (#327, #333): they are no longer the safety net for changed code — they are the only thing that can catch what changes when the repo does not.
Three fixes
1. Silent green.
nightly-replayandnightly-agents-smokeexited 0 when a dispatch filter matched no tests, so a typo'd or stale filter reported success having run nothing. Now a loud failure.2. Stale claim.
nightly-oracleadvertised "~78 checks" in two comments; the suite has 27 (11 auto-covered exact-match benchmarks + 16 pinnedSPECIAL). Corrected, with how to list them (ORACLE_LIST=1).3. Upstream drift — the gap no PR can close. The input hash sees the repository, so an upstream rebuild (a new
python:3.12-slim, a moved:latest) changes what our images contain while every hash and every freshness verdict stays put. That is exactly how mmmu's dataset could 404 for months. Newcontainers/scripts/external-drift.shresolves the digest of all 18 external bases — taken fromfleet-hash's own externals column, so it cannot drift from what the fleet actually buildsFROM— and warns per moved digest. The Fleet status workflow now also runs nightly, comparing against the previous run via the Actions cache and publishing the digest table to the job summary.Report-only: the response to drift is a
force_rebuilddispatch, which stays a human decision.Deliberately not done
Filtering the nightlies by the affected set. I recommended this earlier and it is wrong: on a quiet day the affected set is empty, so a filtered nightly would test nothing — precisely when drift detection matters most. The affected set belongs on PRs, where it already is.
Verified
18 bases resolved live. Seeding a previous-digest file with one altered digest produces
::warning::upstream base moved: alpine:3.20 …and exit 1; an unchanged file exits 0. actionlint/shellcheck clean.Rules checked against
.agents/contributing/RULES.md1–3 (code-only; this section)..agents/verification/RULES.md2 (nightly scope), 8 (fail loud — the whole point of fix 1)..agents/delivery/RULES.md11 (external digests are inputs; this makes their movement observable ahead of the release-time resolution the rule defers).