From 89a7175786e156dceefe55b482ad6e5c35201d35 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Mon, 10 Aug 2026 18:09:56 +0300 Subject: [PATCH 1/5] test(ci): run the changed agents' smoke and the changed benchmarks' oracle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #327 selected replay fixtures for affected components; a changed agent still waited for the nightly to run its own smoke suite, and a changed benchmark's gold-solution (oracle) check never ran on a PR at all. Both now run in the affected job, from the same offline hash diff: - agent smoke: --exact idents (agent_claude_code never matches _rtk); AGENTS_SMOKE_BUILD=1 so the suite builds the PR's agent image from source instead of pulling one that does not exist yet; agents with no agent_smoke! entry warn by name (bob is deliberately absent — it is IBM-backend-tied, see tests/run/agents/test.rs) - oracle: ORACLE_ONLY per changed benchmark, so a broken grader or a moved dataset fails the PR that broke it — the mmmu-class rot the convergence run surfaced. Per-task benchmarks warn and stay nightly (their upstream bases are GB-scale) Harness fixes this needed (both audit findings): - ORACLE_ONLY becomes a comma/space-separated LIST (it was a single substring, unable to express a set) - an ORACLE_ONLY that matches nothing now ASSERTS instead of running zero checks and passing green — a machine-generated filter must never pass vacuously - every new step fails loudly on 'running 0 tests' Both suites are capped at 2 components per PR with a ::warning naming what the cap dropped (no silent truncation): a core-base change affects dozens of leaves and these builds are minutes each. Selection validated through real bash on three history points: agent+ benchmark change -> 2 smoke idents + 1 oracle; per-task-only change -> two warnings, nothing selected; new agent -> smoke for both agents. Oracle list semantics verified including the fail-loud empty case. Signed-off-by: Elron Bandel --- .github/workflows/test.yml | 81 +++++++++++++++++++++++++++++++++++--- tests/run/oracle/test.rs | 16 ++++++-- 2 files changed, 87 insertions(+), 10 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 0741ad1c..36cf9b2a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -160,10 +160,10 @@ jobs: # fixture are named in a ::warning (a visible coverage gap, never a quiet # pass). PRs touching nothing under containers/ exit in ~30s. affected-replay: - name: Replay affected (changed components) + name: Test affected (changed components) runs-on: ubuntu-latest if: github.event_name == 'pull_request' - timeout-minutes: 40 + timeout-minutes: 60 steps: - uses: actions/checkout@v6.0.3 with: @@ -200,20 +200,53 @@ jobs: tests=$(tr ' ' '\n' <<< "$picks" | grep . | sed 's/\.traces\.jsonl$//' | tr '-' '_' | sed 's/^/replay_/' \ | LC_ALL=C sort -u | head -3 | tr '\n' ' ' || true) echo "tests=${tests% }" >> "$GITHUB_OUTPUT" - echo "selected: ${tests:-}" + echo "selected replay: ${tests:-}" + + # Agent smoke: the changed agents' own suite (bob is deliberately + # absent from it — IBM-backend-tied, see tests/run/agents/test.rs). + # --exact idents, so agent_claude_code never matches _rtk. + cap() { # $1 = cap; announce what the cap drops (no silent truncation) + local all; all=$(cat); local n; n=$(grep -c . <<< "$all" || true) + if [ "$n" -gt "$1" ]; then + echo "::warning::$2: $n affected, testing the first $1 ($(tr '\n' ' ' <<< "$all" | cut -c1-200))" >&2 + fi + head -"$1" <<< "$all" + } + smoke="" + for a in $agents; do + id="agent_$(tr '-' '_' <<< "$a")" + if grep -q "agent_smoke!($id," tests/run/agents/test.rs; then smoke="$smoke$id"$'\n' + else echo "::warning::agent $a has no agent_smoke! test (untested by this gate)"; fi + done + smoke=$(printf '%s' "$smoke" | grep . | LC_ALL=C sort -u | cap 2 "agent smoke" | tr '\n' ' ' || true) + echo "agents=${smoke% }" >> "$GITHUB_OUTPUT" + echo "selected agent smoke: ${smoke:-}" + + # Oracle: the changed benchmarks' gold-solution check. ORACLE_ONLY is + # a list and fails loudly if it matches nothing (the harness asserts). + # Per-task benchmarks build ~GB upstream images — out of the PR lane. + orc="" + for b in $benches; do + grep -q 'eval.benchmark.env="per-task"' "containers/benchmarks/$b/Dockerfile" 2>/dev/null \ + && { echo "::warning::benchmark $b is per-task — oracle stays nightly for it"; continue; } + orc="$orc$b"$'\n' + done + orc=$(printf '%s' "$orc" | grep . | LC_ALL=C sort -u | cap 2 "oracle" | tr '\n' ',' || true) + echo "oracle=${orc%,}" >> "$GITHUB_OUTPUT" + echo "selected oracle: ${orc:-}" - name: Install Rust toolchain - if: steps.sel.outputs.tests != '' + if: steps.sel.outputs.tests != '' || steps.sel.outputs.agents != '' || steps.sel.outputs.oracle != '' uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable - name: Cache cargo - if: steps.sel.outputs.tests != '' + if: steps.sel.outputs.tests != '' || steps.sel.outputs.agents != '' || steps.sel.outputs.oracle != '' uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: shared-key: replay - name: Log in to GHCR - if: steps.sel.outputs.tests != '' + if: steps.sel.outputs.tests != '' || steps.sel.outputs.agents != '' || steps.sel.outputs.oracle != '' uses: docker/login-action@v4 with: registry: ghcr.io @@ -235,6 +268,42 @@ jobs: echo "::error::machine-selected fixtures matched 0 tests: $TESTS"; exit 1 fi + # The changed agents' own suite. AGENTS_SMOKE_BUILD makes it build from + # local source (the nightly path) — a PR's agent image does not exist in + # the registry yet, which is exactly what we need to test. + - name: Smoke the affected agents + if: ${{ !cancelled() && steps.sel.outputs.agents != '' }} + env: + HF_TOKEN: ${{ secrets.HF_TOKEN }} + AGENTS_SMOKE_BUILD: "1" + IDS: ${{ steps.sel.outputs.agents }} + run: | + set -eo pipefail + # shellcheck disable=SC2086 # IDS is a space-separated ident list + cargo test -p eval-containers-tests-run --test agents \ + -- --ignored --nocapture --test-threads=1 --exact $IDS \ + | tee agents.log + if grep -qE "running 0 tests" agents.log; then + echo "::error::machine-selected agent smoke matched 0 tests: $IDS"; exit 1 + fi + + # The changed benchmarks' gold-solution check: the oracle must score 1.0, + # so a broken grader or dataset fails the PR that broke it. ORACLE_ONLY + # is a list and the harness asserts it matched something. + - name: Oracle the affected benchmarks + if: ${{ !cancelled() && steps.sel.outputs.oracle != '' }} + env: + HF_TOKEN: ${{ secrets.HF_TOKEN }} + ORACLE_ONLY: ${{ steps.sel.outputs.oracle }} + run: | + set -eo pipefail + cargo test -p eval-containers-tests-run --test oracle \ + -- --ignored --nocapture --test-threads=1 \ + | tee oracle.log + if grep -qE "running 0 tests" oracle.log; then + echo "::error::oracle suite matched 0 tests for: $ORACLE_ONLY"; exit 1 + fi + # Per-PR gate: cheapest full-stack fixture (bigcodebench + zerostack) on the real # stack (otelcol → bifrost → runner), so a gateway/otelcol regression fails the PR. replay-e2e: diff --git a/tests/run/oracle/test.rs b/tests/run/oracle/test.rs index 35e32cbf..3b20fdcb 100644 --- a/tests/run/oracle/test.rs +++ b/tests/run/oracle/test.rs @@ -84,8 +84,11 @@ async fn benchmarks_are_oracle_solvable() { // Collect every oracle check as (label, argv) up front so the heavy part — // each check `--local`-builds the benchmark image — can be fanned across a // matrix. ORACLE_SHARDS = matrix size, ORACLE_SHARD = this job's index; each - // job runs the round-robin slice `i % SHARDS == SHARD`. ORACLE_ONLY= - // narrows to matching labels (single-benchmark validation). All three unset + // job runs the round-robin slice `i % SHARDS == SHARD`. ORACLE_ONLY is a + // comma/space-separated list of substrings and narrows to labels matching + // ANY of them (the per-PR affected gate passes the changed benchmarks); a + // list that matches nothing is a hard error, never a green no-op — a + // machine-generated filter must not pass vacuously. All three unset // (the default) => SHARDS=1, SHARD=0, no filter => every check runs, so a // plain `cargo test --test oracle -- --ignored` is unchanged. let mut checks: Vec<(String, Vec)> = Vec::new(); @@ -125,8 +128,13 @@ async fn benchmarks_are_oracle_solvable() { } let only = std::env::var("ORACLE_ONLY").unwrap_or_default(); - if !only.is_empty() { - checks.retain(|(label, _)| label.contains(&only)); + let wanted: Vec<&str> = only.split([',', ' ']).filter(|s| !s.is_empty()).collect(); + if !wanted.is_empty() { + checks.retain(|(label, _)| wanted.iter().any(|w| label.contains(w))); + assert!( + !checks.is_empty(), + "ORACLE_ONLY={only:?} matched no benchmark — refusing to pass vacuously" + ); } checks.sort(); // deterministic order so each shard's slice is stable let shard: usize = std::env::var("ORACLE_SHARD") From a72c73382f7cb07473bbfa4ab07c453fcb3c65d0 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Mon, 10 Aug 2026 18:14:51 +0300 Subject: [PATCH 2/5] fix: oracle one pinned task for per-task benchmarks; ask the harness for coverage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per-task benchmarks were excluded from the PR oracle on the assumption they cost a ~600-task build. Wrong: the suite already covers each with ONE pinned task (swe-bench sympy__sympy-24066, skills-bench two tasks, swe-lancer 12155_1, swe-bench-pro one instance), so a changed per-task benchmark can and should be oracled on the PR that changes it. Removing the exclusion naively would have failed those PRs, though: terminal-bench has NO oracle check by design (its upstream gold drifted past deterministic scoring) and the strict ORACLE_ONLY now asserts on a no-match. So coverage is no longer guessed in YAML — the harness declares it: ORACLE_LIST=1 cargo test --test oracle -- --ignored --nocapture -> "oracle-label: [ (task )]" The oracle step intersects the changed benchmarks with that list, runs what is covered, and warns by name for what is not. Asking the harness immediately paid: gsm8k and aider-polyglot turn out to have no oracle check either (only 11 benchmarks meet the auto-cover conditions, plus 16 SPECIAL), which a per-task heuristic would have silently mistaken for coverage. Also: the replay cap now announces what it drops like the other two (it was the only silent truncation left), cap() sits beside pick(), the three setup steps share one "any" output instead of repeating a three-clause OR, and the job id is "affected" — it stopped being replay-only when smoke and oracle joined it. Signed-off-by: Elron Bandel --- .github/workflows/test.yml | 79 +++++++++++++++++++++++--------------- tests/run/oracle/test.rs | 13 ++++++- 2 files changed, 59 insertions(+), 33 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 36cf9b2a..ab40cf8d 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -159,7 +159,7 @@ jobs: # zero matched tests is a FAILURE, and affected components with no recorded # fixture are named in a ::warning (a visible coverage gap, never a quiet # pass). PRs touching nothing under containers/ exit in ~30s. - affected-replay: + affected: name: Test affected (changed components) runs-on: ubuntu-latest if: github.event_name == 'pull_request' @@ -186,6 +186,13 @@ jobs: pick() { # $1 = fixture-name regex; no match = "" (|| true absorbs the basename -a "$FIX"/*.traces.jsonl | grep -E "$1" | LC_ALL=C sort | head -1 || true } # grep miss, which pipefail would otherwise turn into an abort) + cap() { # $1 = cap; announce what the cap drops (no silent truncation) + local all; all=$(cat); local n; n=$(grep -c . <<< "$all" || true) + if [ "$n" -gt "$1" ]; then + echo "::warning::$2: $n affected, testing the first $1 ($(tr '\n' ' ' <<< "$all" | cut -c1-200))" >&2 + fi + head -"$1" <<< "$all" + } for b in $benches; do f=$(pick "^${b}-[0-9]+-[a-z0-9-]+\.traces\.jsonl$") if [ -n "$f" ]; then picks="$picks $f"; else uncovered="$uncovered benchmark/$b"; fi @@ -198,20 +205,13 @@ jobs: # Fixture name -> test ident; dedup; cap (a core change affects dozens # of leaves — three fixtures exercise the cascade without an hour-long PR). tests=$(tr ' ' '\n' <<< "$picks" | grep . | sed 's/\.traces\.jsonl$//' | tr '-' '_' | sed 's/^/replay_/' \ - | LC_ALL=C sort -u | head -3 | tr '\n' ' ' || true) + | LC_ALL=C sort -u | cap 3 "replay" | tr '\n' ' ' || true) echo "tests=${tests% }" >> "$GITHUB_OUTPUT" echo "selected replay: ${tests:-}" # Agent smoke: the changed agents' own suite (bob is deliberately # absent from it — IBM-backend-tied, see tests/run/agents/test.rs). # --exact idents, so agent_claude_code never matches _rtk. - cap() { # $1 = cap; announce what the cap drops (no silent truncation) - local all; all=$(cat); local n; n=$(grep -c . <<< "$all" || true) - if [ "$n" -gt "$1" ]; then - echo "::warning::$2: $n affected, testing the first $1 ($(tr '\n' ' ' <<< "$all" | cut -c1-200))" >&2 - fi - head -"$1" <<< "$all" - } smoke="" for a in $agents; do id="agent_$(tr '-' '_' <<< "$a")" @@ -222,31 +222,30 @@ jobs: echo "agents=${smoke% }" >> "$GITHUB_OUTPUT" echo "selected agent smoke: ${smoke:-}" - # Oracle: the changed benchmarks' gold-solution check. ORACLE_ONLY is - # a list and fails loudly if it matches nothing (the harness asserts). - # Per-task benchmarks build ~GB upstream images — out of the PR lane. - orc="" - for b in $benches; do - grep -q 'eval.benchmark.env="per-task"' "containers/benchmarks/$b/Dockerfile" 2>/dev/null \ - && { echo "::warning::benchmark $b is per-task — oracle stays nightly for it"; continue; } - orc="$orc$b"$'\n' - done - orc=$(printf '%s' "$orc" | grep . | LC_ALL=C sort -u | cap 2 "oracle" | tr '\n' ',' || true) - echo "oracle=${orc%,}" >> "$GITHUB_OUTPUT" - echo "selected oracle: ${orc:-}" + # Oracle: every changed benchmark is a candidate — including per-task + # ones, which the suite covers with ONE pinned task each. Which + # candidates it can actually oracle is the harness's business, so the + # oracle step below intersects this list with `ORACLE_LIST` coverage. + orc=$(grep . <<< "$benches" | LC_ALL=C sort -u | tr '\n' ' ' || true) + echo "oracle=${orc% }" >> "$GITHUB_OUTPUT" + echo "oracle candidates: ${orc:-}" + + # One flag for the setup steps: is there anything at all to run? + any=false; [ -z "${tests}${smoke}${orc}" ] || any=true + echo "any=$any" >> "$GITHUB_OUTPUT" - name: Install Rust toolchain - if: steps.sel.outputs.tests != '' || steps.sel.outputs.agents != '' || steps.sel.outputs.oracle != '' + if: steps.sel.outputs.any == 'true' uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable - name: Cache cargo - if: steps.sel.outputs.tests != '' || steps.sel.outputs.agents != '' || steps.sel.outputs.oracle != '' + if: steps.sel.outputs.any == 'true' uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 with: shared-key: replay - name: Log in to GHCR - if: steps.sel.outputs.tests != '' || steps.sel.outputs.agents != '' || steps.sel.outputs.oracle != '' + if: steps.sel.outputs.any == 'true' uses: docker/login-action@v4 with: registry: ghcr.io @@ -287,21 +286,37 @@ jobs: echo "::error::machine-selected agent smoke matched 0 tests: $IDS"; exit 1 fi - # The changed benchmarks' gold-solution check: the oracle must score 1.0, - # so a broken grader or dataset fails the PR that broke it. ORACLE_ONLY - # is a list and the harness asserts it matched something. + # The changed benchmarks' gold-solution check: the oracle must still score + # 1.0, so a broken grader or a moved dataset fails the PR that broke it. + # Per-task benchmarks are included — the suite covers each with one pinned + # task. Coverage comes from the harness (ORACLE_LIST), never re-derived + # here: a benchmark it deliberately cannot oracle (terminal-bench, whose + # upstream gold drifted) warns instead of failing the PR. - name: Oracle the affected benchmarks if: ${{ !cancelled() && steps.sel.outputs.oracle != '' }} env: HF_TOKEN: ${{ secrets.HF_TOKEN }} - ORACLE_ONLY: ${{ steps.sel.outputs.oracle }} + CAND: ${{ steps.sel.outputs.oracle }} run: | set -eo pipefail - cargo test -p eval-containers-tests-run --test oracle \ - -- --ignored --nocapture --test-threads=1 \ - | tee oracle.log + labels=$(ORACLE_LIST=1 cargo test -p eval-containers-tests-run --test oracle \ + -- --ignored --nocapture 2>/dev/null | sed -n 's/^oracle-label: //p') + only=""; uncovered="" + for b in $CAND; do + if grep -qE "^${b}( \(task |\$)" <<< "$labels"; then only="$only$b"$'\n' + else uncovered="$uncovered $b"; fi + done + [ -z "$uncovered" ] || echo "::warning::no oracle check exists for:$uncovered (unverified by this gate)" + # Cap AFTER coverage filtering, announcing what it drops (each check + # --local-builds a benchmark image; per-task ones pull a GB base). + n=$(grep -c . <<< "$only" || true) + [ "$n" -le 2 ] || echo "::warning::oracle: $n covered, checking the first 2" + only=$(grep . <<< "$only" | head -2 | tr '\n' ',' || true) + [ -n "$only" ] || { echo "no covered benchmark to oracle"; exit 0; } + ORACLE_ONLY="${only%,}" cargo test -p eval-containers-tests-run --test oracle \ + -- --ignored --nocapture --test-threads=1 | tee oracle.log if grep -qE "running 0 tests" oracle.log; then - echo "::error::oracle suite matched 0 tests for: $ORACLE_ONLY"; exit 1 + echo "::error::oracle suite matched 0 tests for: ${only%,}"; exit 1 fi # Per-PR gate: cheapest full-stack fixture (bigcodebench + zerostack) on the real diff --git a/tests/run/oracle/test.rs b/tests/run/oracle/test.rs index 3b20fdcb..46d9b55b 100644 --- a/tests/run/oracle/test.rs +++ b/tests/run/oracle/test.rs @@ -127,6 +127,18 @@ async fn benchmarks_are_oracle_solvable() { )); } + checks.sort(); + // ORACLE_LIST=1 prints the coverage this suite has and returns — the + // per-PR gate asks the harness what it can oracle instead of + // re-deriving it (per-task benchmarks are covered by one pinned task + // each; a few, like terminal-bench, are deliberately uncovered). + if std::env::var_os("ORACLE_LIST").is_some() { + for (label, _) in &checks { + println!("oracle-label: {label}"); + } + return; + } + let only = std::env::var("ORACLE_ONLY").unwrap_or_default(); let wanted: Vec<&str> = only.split([',', ' ']).filter(|s| !s.is_empty()).collect(); if !wanted.is_empty() { @@ -136,7 +148,6 @@ async fn benchmarks_are_oracle_solvable() { "ORACLE_ONLY={only:?} matched no benchmark — refusing to pass vacuously" ); } - checks.sort(); // deterministic order so each shard's slice is stable let shard: usize = std::env::var("ORACLE_SHARD") .ok() .and_then(|s| s.parse().ok()) From f614836854af1b6c951ca19ff62b1375d679dec7 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Mon, 10 Aug 2026 20:12:16 +0300 Subject: [PATCH 3/5] refactor: selection logic moves to tests/select-affected.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 66 lines of shell inline in a workflow, when this repo's pattern is a script the workflow calls in one line (six precedents in tests/static/). The tell was that validating it meant reconstructing the logic in a scratch copy — testing a lookalike, not the artifact. The affected job drops 161 -> 91 lines and the script is runnable locally: tests/select-affected.sh # what a change selects tests/select-affected.sh oracle-filter ... # what the oracle covers Extraction immediately exposed two real bugs the inline version hid: 1. cap()'s ::warning:: went to stdout INSIDE command substitution, so the warning text was captured into the value: tests=::warning::replay: 5 affected... — a garbage --exact argument would have shipped. Warnings cannot go to stderr either (GitHub parses workflow commands from stdout), so they now accumulate in a file that survives the subshell and flush once at the end. 2. head on empty input emits a newline, so an empty list became ' ' and the any flag read true: a PR that selected nothing would still have installed the Rust toolchain and logged into GHCR. Values are normalized before the flag is computed. Both are now covered by running the real script: no-change -> any=false with empty values; agent+benchmark change -> 3 replay idents, 2 smoke idents, 5 oracle candidates, warnings listing what the caps dropped and which components have no fixture. Signed-off-by: Elron Bandel --- .github/workflows/test.yml | 82 ++------------------------- tests/select-affected.sh | 112 +++++++++++++++++++++++++++++++++++++ 2 files changed, 118 insertions(+), 76 deletions(-) create mode 100755 tests/select-affected.sh diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ab40cf8d..d6585099 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -169,70 +169,11 @@ jobs: with: fetch-depth: 0 # the base SHA must be resolvable for REF= fleet-hash - - name: Select fixtures for affected components + - name: Select affected components id: sel env: BASE: ${{ github.event.pull_request.base.sha }} - run: | - set -eo pipefail - changed=$(comm -13 \ - <(REF="$BASE" bash containers/scripts/fleet-hash.sh | LC_ALL=C sort) \ - <(bash containers/scripts/fleet-hash.sh | LC_ALL=C sort) | cut -f1) - benches=$(sed -n 's/^benchmark-//p' <<< "$changed") - agents=$(sed -n 's/^agent-//p' <<< "$changed") - echo "affected: $(tr '\n' ' ' <<< "$changed")" - FIX=tests/run/replay/fixtures - picks=""; uncovered="" - pick() { # $1 = fixture-name regex; no match = "" (|| true absorbs the - basename -a "$FIX"/*.traces.jsonl | grep -E "$1" | LC_ALL=C sort | head -1 || true - } # grep miss, which pipefail would otherwise turn into an abort) - cap() { # $1 = cap; announce what the cap drops (no silent truncation) - local all; all=$(cat); local n; n=$(grep -c . <<< "$all" || true) - if [ "$n" -gt "$1" ]; then - echo "::warning::$2: $n affected, testing the first $1 ($(tr '\n' ' ' <<< "$all" | cut -c1-200))" >&2 - fi - head -"$1" <<< "$all" - } - for b in $benches; do - f=$(pick "^${b}-[0-9]+-[a-z0-9-]+\.traces\.jsonl$") - if [ -n "$f" ]; then picks="$picks $f"; else uncovered="$uncovered benchmark/$b"; fi - done - for a in $agents; do - f=$(pick "^[a-z0-9-]+-[0-9]+-${a}\.traces\.jsonl$") - if [ -n "$f" ]; then picks="$picks $f"; else uncovered="$uncovered agent/$a"; fi - done - [ -z "$uncovered" ] || echo "::warning::affected components with no replay fixture (untested by this gate):$uncovered" - # Fixture name -> test ident; dedup; cap (a core change affects dozens - # of leaves — three fixtures exercise the cascade without an hour-long PR). - tests=$(tr ' ' '\n' <<< "$picks" | grep . | sed 's/\.traces\.jsonl$//' | tr '-' '_' | sed 's/^/replay_/' \ - | LC_ALL=C sort -u | cap 3 "replay" | tr '\n' ' ' || true) - echo "tests=${tests% }" >> "$GITHUB_OUTPUT" - echo "selected replay: ${tests:-}" - - # Agent smoke: the changed agents' own suite (bob is deliberately - # absent from it — IBM-backend-tied, see tests/run/agents/test.rs). - # --exact idents, so agent_claude_code never matches _rtk. - smoke="" - for a in $agents; do - id="agent_$(tr '-' '_' <<< "$a")" - if grep -q "agent_smoke!($id," tests/run/agents/test.rs; then smoke="$smoke$id"$'\n' - else echo "::warning::agent $a has no agent_smoke! test (untested by this gate)"; fi - done - smoke=$(printf '%s' "$smoke" | grep . | LC_ALL=C sort -u | cap 2 "agent smoke" | tr '\n' ' ' || true) - echo "agents=${smoke% }" >> "$GITHUB_OUTPUT" - echo "selected agent smoke: ${smoke:-}" - - # Oracle: every changed benchmark is a candidate — including per-task - # ones, which the suite covers with ONE pinned task each. Which - # candidates it can actually oracle is the harness's business, so the - # oracle step below intersects this list with `ORACLE_LIST` coverage. - orc=$(grep . <<< "$benches" | LC_ALL=C sort -u | tr '\n' ' ' || true) - echo "oracle=${orc% }" >> "$GITHUB_OUTPUT" - echo "oracle candidates: ${orc:-}" - - # One flag for the setup steps: is there anything at all to run? - any=false; [ -z "${tests}${smoke}${orc}" ] || any=true - echo "any=$any" >> "$GITHUB_OUTPUT" + run: tests/select-affected.sh - name: Install Rust toolchain if: steps.sel.outputs.any == 'true' @@ -299,24 +240,13 @@ jobs: CAND: ${{ steps.sel.outputs.oracle }} run: | set -eo pipefail - labels=$(ORACLE_LIST=1 cargo test -p eval-containers-tests-run --test oracle \ - -- --ignored --nocapture 2>/dev/null | sed -n 's/^oracle-label: //p') - only=""; uncovered="" - for b in $CAND; do - if grep -qE "^${b}( \(task |\$)" <<< "$labels"; then only="$only$b"$'\n' - else uncovered="$uncovered $b"; fi - done - [ -z "$uncovered" ] || echo "::warning::no oracle check exists for:$uncovered (unverified by this gate)" - # Cap AFTER coverage filtering, announcing what it drops (each check - # --local-builds a benchmark image; per-task ones pull a GB base). - n=$(grep -c . <<< "$only" || true) - [ "$n" -le 2 ] || echo "::warning::oracle: $n covered, checking the first 2" - only=$(grep . <<< "$only" | head -2 | tr '\n' ',' || true) + # shellcheck disable=SC2086 # CAND is a space-separated name list + only=$(tests/select-affected.sh oracle-filter $CAND | tee /dev/stderr | tail -1) [ -n "$only" ] || { echo "no covered benchmark to oracle"; exit 0; } - ORACLE_ONLY="${only%,}" cargo test -p eval-containers-tests-run --test oracle \ + ORACLE_ONLY="$only" cargo test -p eval-containers-tests-run --test oracle \ -- --ignored --nocapture --test-threads=1 | tee oracle.log if grep -qE "running 0 tests" oracle.log; then - echo "::error::oracle suite matched 0 tests for: ${only%,}"; exit 1 + echo "::error::oracle suite matched 0 tests for: $only"; exit 1 fi # Per-PR gate: cheapest full-stack fixture (bigcodebench + zerostack) on the real diff --git a/tests/select-affected.sh b/tests/select-affected.sh new file mode 100755 index 00000000..fb7ad65c --- /dev/null +++ b/tests/select-affected.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# tests/select-affected.sh — pick the tests a change actually needs. +# +# Affected components come from the build-input hashes: the targets whose hash +# differs between BASE and HEAD (containers/scripts/fleet-hash.sh). That is a +# pure git computation — no registry, no daemon — which is what lets the per-PR +# gate use it (.agents/verification/RULES.md rule 1). Base-image cascades are +# free: a core/ change moves every dependent leaf's bases component. +# +# Selection is machine-generated, so it never passes vacuously: a component +# with no test of its own is named in a ::warning, and a cap that drops +# candidates says what it dropped. The callers fail on "running 0 tests". +# +# select-affected.sh [base-ref] emit tests/agents/oracle/any (below) +# select-affected.sh oracle-filter … keep only benchmarks the oracle covers +# +# Outputs (to $GITHUB_OUTPUT when set, else stdout): +# tests replay test idents agents agent_smoke idents +# oracle benchmark candidates any 'true' if anything was selected +# Env: BASE (base ref; default origin/main), REPLAY_CAP=3, SMOKE_CAP=2 +set -eo pipefail +cd "$(dirname "$0")/.." + +FIX=tests/run/replay/fixtures +# Warnings are collected, not printed inline: cap() runs inside command +# substitution (a subshell), so a warning echoed there would be captured into +# the value being computed, and one sent to stderr would not be rendered as a +# workflow command. A file survives the subshell; the flush at the end puts +# every warning on stdout, where GitHub parses it. +WARNF=$(mktemp); trap 'rm -f "$WARNF"' EXIT +warn() { echo "::warning::$1" >> "$WARNF"; } +out() { if [ -n "${GITHUB_OUTPUT:-}" ]; then echo "$1=$2" >> "$GITHUB_OUTPUT"; else echo "$1=$2"; fi; } +# Keep the first $1 entries of stdin, announcing what that drops — a cap that +# hides candidates would read as "covered" when it is not. +cap() { + local all n; all=$(cat); n=$(grep -c . <<< "$all" || true) + [ "$n" -le "$1" ] || warn "$2: $n affected, testing the first $1 ($(tr '\n' ' ' <<< "$all" | cut -c1-200))" + head -"$1" <<< "$all" +} + +# ── oracle-filter: intersect candidates with the suite's declared coverage ── +# Coverage is the harness's business (per-task benchmarks are covered by one +# pinned task each; a few, like terminal-bench, are deliberately uncovered), so +# ask it rather than re-deriving the rules here. +if [ "${1:-}" = "oracle-filter" ]; then + shift + labels=$(ORACLE_LIST=1 cargo test -p eval-containers-tests-run --test oracle \ + -- --ignored --nocapture 2>/dev/null | sed -n 's/^oracle-label: //p') + [ -n "$labels" ] || { echo "::error::ORACLE_LIST produced no labels"; exit 1; } + covered=""; uncovered="" + for b in "$@"; do + if grep -qE "^${b}( \(task |$)" <<< "$labels"; then covered="$covered$b"$'\n' + else uncovered="$uncovered $b"; fi + done + [ -z "$uncovered" ] || warn "no oracle check exists for:$uncovered (unverified by this gate)" + # Each check --local-builds a benchmark image (per-task ones pull a GB base). + filtered=$(grep . <<< "$covered" | cap "${ORACLE_CAP:-2}" oracle | tr '\n' ',' | sed 's/,$//') + cat "$WARNF" # flush before the value, so stdout's last line is it + printf '%s\n' "$filtered" + exit 0 +fi + +# ── affected set ──────────────────────────────────────────────────────────── +BASE="${1:-${BASE:-origin/main}}" +changed=$(comm -13 \ + <(REF="$BASE" containers/scripts/fleet-hash.sh | LC_ALL=C sort) \ + <(containers/scripts/fleet-hash.sh | LC_ALL=C sort) | cut -f1) +benches=$(sed -n 's/^benchmark-//p' <<< "$changed") +agents=$(sed -n 's/^agent-//p' <<< "$changed") +echo "affected: $(tr '\n' ' ' <<< "$changed")" + +# ── replay: one recorded fixture per affected component ───────────────────── +pick() { basename -a "$FIX"/*.traces.jsonl | grep -E "$1" | LC_ALL=C sort | head -1 || true; } +picks=""; uncovered="" +for b in $benches; do + f=$(pick "^${b}-[0-9]+-[a-z0-9-]+\.traces\.jsonl$") + if [ -n "$f" ]; then picks="$picks$f"$'\n'; else uncovered="$uncovered benchmark/$b"; fi +done +for a in $agents; do + f=$(pick "^[a-z0-9-]+-[0-9]+-${a}\.traces\.jsonl$") + if [ -n "$f" ]; then picks="$picks$f"$'\n'; else uncovered="$uncovered agent/$a"; fi +done +[ -z "$uncovered" ] || warn "affected components with no replay fixture (untested by this gate):$uncovered" +tests=$(grep . <<< "$picks" | sed 's/\.traces\.jsonl$//' | tr '-' '_' | sed 's/^/replay_/' \ + | LC_ALL=C sort -u | cap "${REPLAY_CAP:-3}" replay | tr '\n' ' ' || true) + +# ── agent smoke: the changed agents' own suite ────────────────────────────── +# --exact idents downstream, so agent_claude_code never matches _rtk. bob has +# no agent_smoke! by design (IBM-backend-tied; see tests/run/agents/test.rs). +smoke="" +for a in $agents; do + id="agent_$(tr '-' '_' <<< "$a")" + if grep -q "agent_smoke!($id," tests/run/agents/test.rs; then smoke="$smoke$id"$'\n' + else warn "agent $a has no agent_smoke! test (untested by this gate)"; fi +done +smoke=$(grep . <<< "$smoke" | LC_ALL=C sort -u | cap "${SMOKE_CAP:-2}" "agent smoke" | tr '\n' ' ' || true) + +# ── oracle: every changed benchmark is a candidate; oracle-filter decides ─── +orc=$(grep . <<< "$benches" | LC_ALL=C sort -u | tr '\n' ' ' || true) + +# Normalize before deciding: an empty list still leaves a separator space +# behind (head emits a newline, tr turns it into ' '), which would make the +# "anything to do?" flag true for a change that selected nothing. +tests=$(tr -s ' ' <<< "$tests" | sed 's/^ *//; s/ *$//') +smoke=$(tr -s ' ' <<< "$smoke" | sed 's/^ *//; s/ *$//') +orc=$(tr -s ' ' <<< "$orc" | sed 's/^ *//; s/ *$//') + +cat "$WARNF" # every warning, on stdout, outside any substitution +out tests "$tests" +out agents "$smoke" +out oracle "$orc" +out any "$([ -z "${tests}${smoke}${orc}" ] && echo false || echo true)" From b365bfe9cf2388d43e6d530870bbc0687f725e71 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Mon, 10 Aug 2026 20:13:38 +0300 Subject: [PATCH 4/5] proof: exercise the affected gate end to end (reverted next commit) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Touches ai2d (replay fixture + oracle check) and openclaw (replay fixture + agent_smoke) so CI runs the smoke and oracle steps for real — they have only ever run the empty path. Also fixes a selection flaw the proof exposed: the diff used GitHub's pull_request.base.sha, which is main's TIP, so anything main merged after the branch point counted as affected by this PR (the preview selected open-interpreter/appworld/mmmu — the chip session's fixes on main, nothing to do with this branch). Now diffs the merge base, so a PR is judged on what it changed. Signed-off-by: Elron Bandel --- containers/agents/openclaw/Dockerfile | 2 ++ containers/benchmarks/ai2d/Dockerfile | 2 ++ tests/select-affected.sh | 4 ++++ 3 files changed, 8 insertions(+) diff --git a/containers/agents/openclaw/Dockerfile b/containers/agents/openclaw/Dockerfile index 3735a815..ab7587c1 100644 --- a/containers/agents/openclaw/Dockerfile +++ b/containers/agents/openclaw/Dockerfile @@ -41,3 +41,5 @@ exec openclaw agent --local --session-id smoke ${EVAL_AGENT_REASONING_EFFORT:+-- E ENTRYPOINT ["/run.sh"] + +# CI proof commit — reverted immediately; exercises the affected gate end to end. diff --git a/containers/benchmarks/ai2d/Dockerfile b/containers/benchmarks/ai2d/Dockerfile index 7a4bd8cd..edde0146 100644 --- a/containers/benchmarks/ai2d/Dockerfile +++ b/containers/benchmarks/ai2d/Dockerfile @@ -106,3 +106,5 @@ ENTRY RUN chmod +x /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] CMD ["/grade.sh"] + +# CI proof commit — reverted immediately; exercises the affected gate end to end. diff --git a/tests/select-affected.sh b/tests/select-affected.sh index fb7ad65c..df511e51 100755 --- a/tests/select-affected.sh +++ b/tests/select-affected.sh @@ -62,6 +62,10 @@ fi # ── affected set ──────────────────────────────────────────────────────────── BASE="${1:-${BASE:-origin/main}}" +# Diff against the MERGE BASE, not the base branch's tip: a PR is responsible +# for what it changed, not for what main changed after it branched (GitHub's +# pull_request.base.sha is the tip, so comparing to it selects unrelated work). +BASE=$(git merge-base "$BASE" HEAD 2>/dev/null || echo "$BASE") changed=$(comm -13 \ <(REF="$BASE" containers/scripts/fleet-hash.sh | LC_ALL=C sort) \ <(containers/scripts/fleet-hash.sh | LC_ALL=C sort) | cut -f1) From 7270686104115597622b9f8c0bb0004c265a78e7 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Mon, 10 Aug 2026 20:41:09 +0300 Subject: [PATCH 5/5] revert the proof commit's Dockerfile edits (keep the merge-base fix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ai2d/openclaw comments existed only to make CI exercise the smoke and oracle steps for real. Proof run: job 'Test affected' ran Select -> Replay (replay_ai2d_1852_openclaw) -> Smoke (agent_openclaw, built from source) -> Oracle (ai2d gold = 1.0), every step success. That was the first end-to-end execution of the smoke and oracle paths. The merge-base selection fix from that commit stays — it is a real fix, not scaffolding. Signed-off-by: Elron Bandel --- containers/agents/openclaw/Dockerfile | 2 -- containers/benchmarks/ai2d/Dockerfile | 2 -- 2 files changed, 4 deletions(-) diff --git a/containers/agents/openclaw/Dockerfile b/containers/agents/openclaw/Dockerfile index ab7587c1..3735a815 100644 --- a/containers/agents/openclaw/Dockerfile +++ b/containers/agents/openclaw/Dockerfile @@ -41,5 +41,3 @@ exec openclaw agent --local --session-id smoke ${EVAL_AGENT_REASONING_EFFORT:+-- E ENTRYPOINT ["/run.sh"] - -# CI proof commit — reverted immediately; exercises the affected gate end to end. diff --git a/containers/benchmarks/ai2d/Dockerfile b/containers/benchmarks/ai2d/Dockerfile index edde0146..7a4bd8cd 100644 --- a/containers/benchmarks/ai2d/Dockerfile +++ b/containers/benchmarks/ai2d/Dockerfile @@ -106,5 +106,3 @@ ENTRY RUN chmod +x /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] CMD ["/grade.sh"] - -# CI proof commit — reverted immediately; exercises the affected gate end to end.