test(ci): run the changed agents' smoke and the changed benchmarks' oracle - #333
Merged
Conversation
…racle #327 selected replay fixtures for affected components; a changed agent still waited for the nightly to run its own smoke suite, and a changed benchmark's gold-solution (oracle) check never ran on a PR at all. Both now run in the affected job, from the same offline hash diff: - agent smoke: --exact idents (agent_claude_code never matches _rtk); AGENTS_SMOKE_BUILD=1 so the suite builds the PR's agent image from source instead of pulling one that does not exist yet; agents with no agent_smoke! entry warn by name (bob is deliberately absent — it is IBM-backend-tied, see tests/run/agents/test.rs) - oracle: ORACLE_ONLY per changed benchmark, so a broken grader or a moved dataset fails the PR that broke it — the mmmu-class rot the convergence run surfaced. Per-task benchmarks warn and stay nightly (their upstream bases are GB-scale) Harness fixes this needed (both audit findings): - ORACLE_ONLY becomes a comma/space-separated LIST (it was a single substring, unable to express a set) - an ORACLE_ONLY that matches nothing now ASSERTS instead of running zero checks and passing green — a machine-generated filter must never pass vacuously - every new step fails loudly on 'running 0 tests' Both suites are capped at 2 components per PR with a ::warning naming what the cap dropped (no silent truncation): a core-base change affects dozens of leaves and these builds are minutes each. Selection validated through real bash on three history points: agent+ benchmark change -> 2 smoke idents + 1 oracle; per-task-only change -> two warnings, nothing selected; new agent -> smoke for both agents. Oracle list semantics verified including the fail-loud empty case. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
…for coverage
Per-task benchmarks were excluded from the PR oracle on the assumption
they cost a ~600-task build. Wrong: the suite already covers each with
ONE pinned task (swe-bench sympy__sympy-24066, skills-bench two tasks,
swe-lancer 12155_1, swe-bench-pro one instance), so a changed per-task
benchmark can and should be oracled on the PR that changes it.
Removing the exclusion naively would have failed those PRs, though:
terminal-bench has NO oracle check by design (its upstream gold drifted
past deterministic scoring) and the strict ORACLE_ONLY now asserts on a
no-match. So coverage is no longer guessed in YAML — the harness
declares it:
ORACLE_LIST=1 cargo test --test oracle -- --ignored --nocapture
-> "oracle-label: <benchmark>[ (task <id>)]"
The oracle step intersects the changed benchmarks with that list, runs
what is covered, and warns by name for what is not. Asking the harness
immediately paid: gsm8k and aider-polyglot turn out to have no oracle
check either (only 11 benchmarks meet the auto-cover conditions, plus
16 SPECIAL), which a per-task heuristic would have silently mistaken
for coverage.
Also: the replay cap now announces what it drops like the other two
(it was the only silent truncation left), cap() sits beside pick(), the
three setup steps share one "any" output instead of repeating a
three-clause OR, and the job id is "affected" — it stopped being
replay-only when smoke and oracle joined it.
Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
elronbandel
force-pushed
the
elron/affected-agents-oracle
branch
from
August 10, 2026 15:15
1ada9cf to
a72c733
Compare
66 lines of shell inline in a workflow, when this repo's pattern is a script the workflow calls in one line (six precedents in tests/static/). The tell was that validating it meant reconstructing the logic in a scratch copy — testing a lookalike, not the artifact. The affected job drops 161 -> 91 lines and the script is runnable locally: tests/select-affected.sh <base-ref> # what a change selects tests/select-affected.sh oracle-filter <b>... # what the oracle covers Extraction immediately exposed two real bugs the inline version hid: 1. cap()'s ::warning:: went to stdout INSIDE command substitution, so the warning text was captured into the value: tests=::warning::replay: 5 affected... — a garbage --exact argument would have shipped. Warnings cannot go to stderr either (GitHub parses workflow commands from stdout), so they now accumulate in a file that survives the subshell and flush once at the end. 2. head on empty input emits a newline, so an empty list became ' ' and the any flag read true: a PR that selected nothing would still have installed the Rust toolchain and logged into GHCR. Values are normalized before the flag is computed. Both are now covered by running the real script: no-change -> any=false with empty values; agent+benchmark change -> 3 replay idents, 2 smoke idents, 5 oracle candidates, warnings listing what the caps dropped and which components have no fixture. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
Touches ai2d (replay fixture + oracle check) and openclaw (replay fixture + agent_smoke) so CI runs the smoke and oracle steps for real — they have only ever run the empty path. Also fixes a selection flaw the proof exposed: the diff used GitHub's pull_request.base.sha, which is main's TIP, so anything main merged after the branch point counted as affected by this PR (the preview selected open-interpreter/appworld/mmmu — the chip session's fixes on main, nothing to do with this branch). Now diffs the merge base, so a PR is judged on what it changed. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
The ai2d/openclaw comments existed only to make CI exercise the smoke and oracle steps for real. Proof run: job 'Test affected' ran Select -> Replay (replay_ai2d_1852_openclaw) -> Smoke (agent_openclaw, built from source) -> Oracle (ai2d gold = 1.0), every step success. That was the first end-to-end execution of the smoke and oracle paths. The merge-base selection fix from that commit stays — it is a real fix, not scaffolding. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #327 (which landed the affected-fixture selection). Answers "are agent tests running when agents change?" — they weren't, and neither were benchmark oracle checks.
What now runs on a PR
From the same offline hash diff #327 introduced:
agent_smoketest (new)oraclegold-solution check (new)--exactidents soagent_claude_codenever matchesagent_claude_code_rtk;AGENTS_SMOKE_BUILD=1so the suite builds the PR's agent image from source rather than pulling one that doesn't exist yet. Agents with noagent_smoke!entry warn by name —bobis deliberately absent (IBM-backend-tied, documented in the suite).Harness fixes this required (both from the rung-6 audit)
ORACLE_ONLYbecomes a comma/space-separated list — it was a single substring and literally could not express a set.ORACLE_ONLYthat matches nothing now asserts instead of running zero checks and reporting success. A machine-generated filter must never pass vacuously.running 0 tests.Cost control
Both suites cap at 2 components per PR, with a
::warningnaming what the cap dropped (no silent truncation — the audit's rule). A core-base change affects dozens of leaves and each of these is a minutes-long build.Validation
Selection run through real bash at three history points: agent+benchmark change → 2 smoke idents + 1 oracle benchmark; per-task-only change → two warnings, nothing selected; new-agent commit → smoke for both changed agents. Oracle list semantics verified separately, including the fail-loud empty case (
"nope"→ assert, not green).Rules checked against
.agents/contributing/RULES.md1–3 (extends bug: the per-PR gate is change-blind — one hardcoded fixture regardless of what the PR touches #326's fix; code-only; this section)..agents/verification/RULES.md1 (selection stays offline — git only), 8 (fail loud: zero-match errors, gaps and caps warn).tests/run/oracle+tests/run/agentsharness contracts preserved (unset env ⇒ unchanged full-suite behavior for the nightlies).