test(ci): replay what changed — per-PR affected-fixture gate - #327
Merged
Conversation
The PR lane was change-blind: one hardcoded fixture regardless of what a PR touched. New affected-replay job in test.yml: - affected = diff of two fleet-hash runs (PR base SHA vs HEAD) — fully offline as the PR gate must be (verification RULES rule 1, #208 precedent); base-image cascades arrive free via the hash's bases component, so a core/ change selects fixtures for dependent leaves - selection: one recorded replay fixture per affected benchmark/agent, deduped, capped at 3 (a core change affects dozens of leaves; three fixtures exercise the cascade without an hour-long PR) - loud, never silent-green (the rung-6 audit's hard requirement): affected components with NO fixture produce a ::warning naming them (25 benchmarks have no fixtures — a visible gap, not a quiet pass); a machine-generated selection matching 0 tests FAILS the job - PRs touching nothing under containers/ exit in ~30s (selection runs before the Rust toolchain is even installed) Validated through real bash across four history scenarios: agent+bench change -> 3 correct fixtures; per-task-only change -> loud uncovered warning, no tests; new agent without fixture -> warns and tests the covered sibling; identical refs -> clean fast path. Validation caught two real bugs pre-merge: head-masked grep status (empty pick counted as covered) and pipefail aborting the step on a fixtureless component. Resolves #326. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
elronbandel
added a commit
that referenced
this pull request
Aug 10, 2026
…racle (#333) * test(ci): run the changed agents' smoke and the changed benchmarks' oracle #327 selected replay fixtures for affected components; a changed agent still waited for the nightly to run its own smoke suite, and a changed benchmark's gold-solution (oracle) check never ran on a PR at all. Both now run in the affected job, from the same offline hash diff: - agent smoke: --exact idents (agent_claude_code never matches _rtk); AGENTS_SMOKE_BUILD=1 so the suite builds the PR's agent image from source instead of pulling one that does not exist yet; agents with no agent_smoke! entry warn by name (bob is deliberately absent — it is IBM-backend-tied, see tests/run/agents/test.rs) - oracle: ORACLE_ONLY per changed benchmark, so a broken grader or a moved dataset fails the PR that broke it — the mmmu-class rot the convergence run surfaced. Per-task benchmarks warn and stay nightly (their upstream bases are GB-scale) Harness fixes this needed (both audit findings): - ORACLE_ONLY becomes a comma/space-separated LIST (it was a single substring, unable to express a set) - an ORACLE_ONLY that matches nothing now ASSERTS instead of running zero checks and passing green — a machine-generated filter must never pass vacuously - every new step fails loudly on 'running 0 tests' Both suites are capped at 2 components per PR with a ::warning naming what the cap dropped (no silent truncation): a core-base change affects dozens of leaves and these builds are minutes each. Selection validated through real bash on three history points: agent+ benchmark change -> 2 smoke idents + 1 oracle; per-task-only change -> two warnings, nothing selected; new agent -> smoke for both agents. Oracle list semantics verified including the fail-loud empty case. Signed-off-by: Elron Bandel <elron.bandel@ibm.com> * fix: oracle one pinned task for per-task benchmarks; ask the harness for coverage Per-task benchmarks were excluded from the PR oracle on the assumption they cost a ~600-task build. Wrong: the suite already covers each with ONE pinned task (swe-bench sympy__sympy-24066, skills-bench two tasks, swe-lancer 12155_1, swe-bench-pro one instance), so a changed per-task benchmark can and should be oracled on the PR that changes it. Removing the exclusion naively would have failed those PRs, though: terminal-bench has NO oracle check by design (its upstream gold drifted past deterministic scoring) and the strict ORACLE_ONLY now asserts on a no-match. So coverage is no longer guessed in YAML — the harness declares it: ORACLE_LIST=1 cargo test --test oracle -- --ignored --nocapture -> "oracle-label: <benchmark>[ (task <id>)]" The oracle step intersects the changed benchmarks with that list, runs what is covered, and warns by name for what is not. Asking the harness immediately paid: gsm8k and aider-polyglot turn out to have no oracle check either (only 11 benchmarks meet the auto-cover conditions, plus 16 SPECIAL), which a per-task heuristic would have silently mistaken for coverage. Also: the replay cap now announces what it drops like the other two (it was the only silent truncation left), cap() sits beside pick(), the three setup steps share one "any" output instead of repeating a three-clause OR, and the job id is "affected" — it stopped being replay-only when smoke and oracle joined it. Signed-off-by: Elron Bandel <elron.bandel@ibm.com> * refactor: selection logic moves to tests/select-affected.sh 66 lines of shell inline in a workflow, when this repo's pattern is a script the workflow calls in one line (six precedents in tests/static/). The tell was that validating it meant reconstructing the logic in a scratch copy — testing a lookalike, not the artifact. The affected job drops 161 -> 91 lines and the script is runnable locally: tests/select-affected.sh <base-ref> # what a change selects tests/select-affected.sh oracle-filter <b>... # what the oracle covers Extraction immediately exposed two real bugs the inline version hid: 1. cap()'s ::warning:: went to stdout INSIDE command substitution, so the warning text was captured into the value: tests=::warning::replay: 5 affected... — a garbage --exact argument would have shipped. Warnings cannot go to stderr either (GitHub parses workflow commands from stdout), so they now accumulate in a file that survives the subshell and flush once at the end. 2. head on empty input emits a newline, so an empty list became ' ' and the any flag read true: a PR that selected nothing would still have installed the Rust toolchain and logged into GHCR. Values are normalized before the flag is computed. Both are now covered by running the real script: no-change -> any=false with empty values; agent+benchmark change -> 3 replay idents, 2 smoke idents, 5 oracle candidates, warnings listing what the caps dropped and which components have no fixture. Signed-off-by: Elron Bandel <elron.bandel@ibm.com> * proof: exercise the affected gate end to end (reverted next commit) Touches ai2d (replay fixture + oracle check) and openclaw (replay fixture + agent_smoke) so CI runs the smoke and oracle steps for real — they have only ever run the empty path. Also fixes a selection flaw the proof exposed: the diff used GitHub's pull_request.base.sha, which is main's TIP, so anything main merged after the branch point counted as affected by this PR (the preview selected open-interpreter/appworld/mmmu — the chip session's fixes on main, nothing to do with this branch). Now diffs the merge base, so a PR is judged on what it changed. Signed-off-by: Elron Bandel <elron.bandel@ibm.com> * revert the proof commit's Dockerfile edits (keep the merge-base fix) The ai2d/openclaw comments existed only to make CI exercise the smoke and oracle steps for real. Proof run: job 'Test affected' ran Select -> Replay (replay_ai2d_1852_openclaw) -> Smoke (agent_openclaw, built from source) -> Oracle (ai2d gold = 1.0), every step success. That was the first end-to-end execution of the smoke and oracle paths. The merge-base selection fix from that commit stays — it is a real fix, not scaffolding. Signed-off-by: Elron Bandel <elron.bandel@ibm.com> --------- Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #326. The per-PR half of "test what changed" (nightly filter injection is follow-up, tracked in the rung-6 audit notes).
What
A new
affected-replayjob intest.yml: every PR replays a recorded fixture for each benchmark/agent it actually changed.diffof twofleet-hashruns (PR base SHA vs HEAD) — fully offline, as the PR gate must be (.agents/verification/RULES.mdrule 1; the replay tests: bake path pulls from ghcr.io registry, violates MUST-run-offline rule #208 precedent forbids registry reads). Base-image cascades come free: acore/change moves dependent leaves' bases components, so their fixtures get selected without any hand-written path mapping — the exact objection that killed feat(release): affected-set — build only what changed since the last release #241.::warningnaming them — 25 benchmarks have no fixtures, and that gap is now visible on every PR that touches one instead of quietly passing. A machine-generated selection that matches 0 tests fails the job.containers/exit in ~30s. This PR is itself the live demo — it touches onlytest.yml, so its own CI run exercises the empty fast path.Validation
Run through real bash (not the dev-shell approximation, which masked two bugs) across four history scenarios:
09e5ae10)d841ce10)118d9aa8)pi, tests the covered siblingValidation caught two real bugs before merge:
head -1masking a failedgrep(a fixtureless component counted as covered), andpipefailaborting the whole step on the same case. Both fixed; actionlint/shellcheck clean.Rules checked against
.agents/contributing/RULES.md1–3 (resolves bug: the per-PR gate is change-blind — one hardcoded fixture regardless of what the PR touches #326; code-only; this section)..agents/verification/RULES.md1 (offline PR gate — git only), 8 (fail loud: zero-match fails, gaps warn)..agents/RULES.md7–8 (bash + git + grep on the existing hash primitive; no new machinery).