feat(replay): record terminal-bench + skills-bench fixtures

.agents/src/RULES.md

@@ -20,6 +20,7 @@ The CLI shells out only to this fixed set of standard, user-installable tools. "
 | Tool | Used for |
 |------|----------|
 | `docker build` | per-task benchmark variants (outside the static bake graph) |
+| `podman build` | fallback for the per-task **eval combination** when the active buildx builder can't read the local image store (a `docker-container`/remote driver, e.g. a podman-backed Docker) — reads the local store natively, the same reason `benchmarks/<name>/build.sh` uses podman |
 | `docker buildx bake` | the artifact build graph (top-level RULES.md principle 15); in-cluster builds use the *same* command against a `--driver kubernetes` builder |
 | `docker compose` | `run --mode compose`; publishing compose artifacts |
 | `docker run` | `run --mode container` |
@@ -57,7 +58,7 @@ Each tool MUST be a standard release the user can install and invoke themselves
 
 ### Commands
 
-11. **Build.** `eval-containers build agent|bench|model|eval` — each MUST map to a single `docker buildx bake <target>` invocation, which executes the artifact's build graph declared in its `docker-bake.hcl` (top-level RULES.md principle 15). Per-task variants (`--task-id`), which sit outside the static bake graph, fall through to a single `docker build`. Building in a cluster is **not** a separate code path: it is the same `docker buildx bake` pointed at an in-cluster builder (`docker buildx create --driver kubernetes`). The CLI exposes this as `build --builder <name>` — a passthrough of buildx's own `--builder` that implies `--push` (a remote builder can't load into local Docker); a missing builder fails with the one-time `docker buildx create` command to run. The reserved value `build --builder oc` selects the **OpenShift `BuildConfig` backend** instead of buildx: it builds a single artifact in-cluster with `oc start-build` (buildah under the platform's `builder` SCC) — the no-admin path where baseline PodSecurity blocks in-cluster BuildKit. It MUST read the artifact's resolved build spec — `context`, `dockerfile`, and the eval combination's base-image args — from `docker buildx bake --print <target>` (the bake file stays the only build-graph artifact, principle 3); it adds only the OpenShift-specific translation: the single-segment imagestream naming (`core/otel` → `core-otel`, `benchmarks/aime` → `aime`) and the `REGISTRY`/`REGISTRY_SUFFIX` build args that resolve the parameterized `${REGISTRY}/...${REGISTRY_SUFFIX}` FROMs to the internal registry (binary builds ignore `oc start-build --build-arg`, so they live in the BuildConfig spec). Either backend builds **one** artifact: the CLI MUST NOT re-derive or order the build graph for any platform; dependency-ordered cold-graph builds are a thin loop over `build` that lives outside the CLI (e.g. `deploy/examples/openshift/`). Every build also stamps the OCI provenance label `org.opencontainers.image.source` (the pointer GitHub uses to link a package to its repo — on an Actions push or a one-time UI "Connect repository"; a manual push doesn't auto-link) — fleet-wide via `--set "*.labels…"`, and explicit on the per-task `docker build`; never stored in the per-artifact bake files (top-level principle 15.f).
+11. **Build.** `eval-containers build agent|bench|model|eval` — each MUST map to a single `docker buildx bake <target>` invocation, which executes the artifact's build graph declared in its `docker-bake.hcl` (top-level RULES.md principle 15). Per-task variants (`--task-id`), which sit outside the static bake graph, fall through to a single `docker build`. The per-task **eval combination** is built by the same `docker buildx bake eval`, but when the active builder is a `docker-container`/remote driver that can't resolve the per-task base from the local image store (e.g. a podman-backed Docker), the CLI falls back to `podman build` against the spec `docker buildx bake --print eval` resolves — bake stays the single source of truth (principle 3) and `podman build` reads the local store natively. Shared-env eval images and remote `--builder` runs always use bake. Building in a cluster is **not** a separate code path: it is the same `docker buildx bake` pointed at an in-cluster builder (`docker buildx create --driver kubernetes`). The CLI exposes this as `build --builder <name>` — a passthrough of buildx's own `--builder` that implies `--push` (a remote builder can't load into local Docker); a missing builder fails with the one-time `docker buildx create` command to run. The reserved value `build --builder oc` selects the **OpenShift `BuildConfig` backend** instead of buildx: it builds a single artifact in-cluster with `oc start-build` (buildah under the platform's `builder` SCC) — the no-admin path where baseline PodSecurity blocks in-cluster BuildKit. It MUST read the artifact's resolved build spec — `context`, `dockerfile`, and the eval combination's base-image args — from `docker buildx bake --print <target>` (the bake file stays the only build-graph artifact, principle 3); it adds only the OpenShift-specific translation: the single-segment imagestream naming (`core/otel` → `core-otel`, `benchmarks/aime` → `aime`) and the `REGISTRY`/`REGISTRY_SUFFIX` build args that resolve the parameterized `${REGISTRY}/...${REGISTRY_SUFFIX}` FROMs to the internal registry (binary builds ignore `oc start-build --build-arg`, so they live in the BuildConfig spec). Either backend builds **one** artifact: the CLI MUST NOT re-derive or order the build graph for any platform; dependency-ordered cold-graph builds are a thin loop over `build` that lives outside the CLI (e.g. `deploy/examples/openshift/`). Every build also stamps the OCI provenance label `org.opencontainers.image.source` (the pointer GitHub uses to link a package to its repo — on an Actions push or a one-time UI "Connect repository"; a manual push doesn't auto-link) — fleet-wide via `--set "*.labels…"`, and explicit on the per-task `docker build`; never stored in the per-artifact bake files (top-level principle 15.f).
 
 12. **Run.** `eval-containers run {benchmark} --agent {name} --task-id {id}` — maps to the standard command for the chosen `--mode`: `docker compose up` (compose, the default), `docker run` (container), or `helm template <chart> -f <benchmark values> --set … \| kubectl apply -f -` (job). MUST accept both the container-tag axis (`--benchmark-tag`, `--agent-tag`, `--model-tag`) and the internal-version axis (`--benchmark-version`, `--agent-version`, `--litellm-version`), plus `--model`, `--timeout`, `--local`. Cluster- and platform-specific settings for `job` mode (e.g. the service account an OpenShift cluster requires) MUST be supplied as a composable Helm values file via `--overlay <file>` — not encoded per-platform inside the CLI (principle 3); the reference OpenShift overlay is `deploy/values-openshift.yaml`.
 
@@ -89,3 +90,4 @@ Each tool MUST be a standard release the user can install and invoke themselves
 | 2026-06-03 | `run --mode job` drops the per-benchmark `-f values.yaml`; the benchmark is named via `--set benchmark=<x>` and its bespoke topology (if any) lives in the chart at `presets/<x>.yaml`. The chart is now self-contained (renders with no external file), so it can be packaged/published. |
 | 2026-06-03 | Principle 11 (Build) + tools table: added the reserved `build --builder oc` OpenShift `BuildConfig` backend — reads the build spec (context, dockerfile, eval base-image args) from `docker buildx bake --print` (bake stays the source of truth) and translates it to `oc start-build` (buildah), the no-admin in-cluster path where baseline PodSecurity blocks BuildKit. One artifact, no graph ordering in the CLI; the ordered loop lives in `deploy/examples/openshift/`. |
 | 2026-06-11 | Principle 11 (Build): every build stamps the OCI provenance label `org.opencontainers.image.source` (the pointer GitHub uses to link a package to its repo, via an Actions push or UI connect — not a manual push) — fleet-wide via `--set "*.labels…"`, explicit on per-task `docker build`; not in the per-artifact bake files (15.f). |
+| 2026-06-15 | Principle 11 + tools table: the per-task **eval combination** falls back to `podman build` when `docker buildx bake` can't resolve the per-task base from the local image store (a `docker-container`/remote builder, e.g. a podman-backed Docker; CI's `docker` driver is unaffected and keeps bake). The fallback builds the spec `docker buildx bake --print eval` resolves, so bake stays the source of truth (principle 3). Unblocks recording replay fixtures for per-task / built-from-source benchmarks on a container-driver host; `podman build` added to the underlying-tools list. |

.gitleaks.toml

@@ -1,18 +1,21 @@
 # Gitleaks config for Eval Containers.
 #
-# Extends the default ruleset with an allowlist for two patterns that
-# repeatedly trip `generic-api-key` inside recorded LLM trajectories:
+# Recorded LLM trajectory fixtures (tests/run/replay/fixtures/*.trajectory.jsonl)
+# are captured dumps of real agent runs. They legitimately contain high-entropy
+# observability IDs that trip the default `generic-api-key` rule — e.g.
+# `user_api_key_hash` (a one-way hash LiteLLM emits; derived from a key but not
+# the key) and `prompt_cache_key` (a cache-routing UUID). Neither is a credential.
 #
-#   - `user_api_key_hash`: a one-way hash LiteLLM emits for
-#     observability. It is *derived from* a real key but is not the
-#     key itself. Harmless to commit.
-#   - `prompt_cache_key`: an OpenAI / Anthropic cache routing UUID. Not
-#     a credential.
+# The pinned pre-commit hook is gitleaks v8.18.4, which cannot scope an allowlist
+# to "this path AND only these field names": the AND-condition / `[[allowlists]]`
+# forms arrived in 8.19+, and these are JSONL (one large object per line), so a
+# line-target regex matches the whole record anyway. The only mechanism available
+# on 8.18.4 is a `paths` allowlist — which exempts the matched files WHOLESALE.
 #
-# Both live only inside `tests/fixtures/*.trajectory.jsonl` (replay
-# fixtures recorded from release-verification live sweeps — see
-# tests/replay/RULES.md). Allowlist is scoped to that path prefix so a
-# real key leaking into a fixture would still fail the scan.
+# Consequence: this scanner does NOT inspect inside the fixture tree. A trajectory
+# MUST therefore be vetted credential-free *when it is recorded* (see
+# tests/run/replay/RULES.md + provenance.json); gitleaks will not catch a live key
+# committed under this path. Never commit a fixture you have not confirmed clean.
 #
 # Verify after edits:
 #   gitleaks detect --source . --report-format json --report-path /tmp/gitleaks.json
@@ -21,12 +24,13 @@
 [extend]
 useDefault = true
 
-[[allowlists]]
-description = "Trajectory fixture observability IDs are not credentials"
+# Whole-file exemption for the vetted fixture tree (see header). On gitleaks
+# 8.18.4 a `paths` allowlist is a wholesale bypass — intentional here, and the
+# only form available; the safety guarantee is the record-time vetting above,
+# not this scanner. (A bare `regexes` filter cannot narrow it: paths OR regexes
+# are combined, and these single-line JSON records defeat a line-target regex.)
+[allowlist]
+description = "Recorded trajectory fixtures are vetted credential-free at record time"
 paths = [
-  '''tests/fixtures/.*\.trajectory\.jsonl$''',
-]
-regexes = [
-  '''user_api_key_hash''',
-  '''prompt_cache_key''',
+  '''tests/run/replay/fixtures/.*\.trajectory\.jsonl$''',
 ]

.pre-commit-config.yaml

@@ -54,6 +54,9 @@ repos:
       - id: check-merge-conflict
       - id: check-added-large-files
         args: [--maxkb=1024]
+        # Replay fixtures are large recorded trajectories (committed ground
+        # truth, some tens of MB) — not accidental large files.
+        exclude: '\.(jsonl|lock)$'
       - id: check-yaml
         args: [--allow-multiple-documents]   # k8s manifest bundles (e.g. deploy/kueue.yaml)
         # Helm templates are Go-templated, not valid YAML; helm-lint covers them.

cli/src/build.rs

@@ -218,6 +218,38 @@ pub fn execute(registry: &str, args: BuildArgs) -> Result<(), String> {
             if !agent_version.is_empty() {
                 overrides.push(format!("eval.args.AGENT_VERSION={agent_version}"));
             }
+            // Per-task eval images are stitched FROM a per-task BASE that lives
+            // only in the LOCAL image store (build.sh's `podman build`, or the
+            // per-task `docker build` — both outside bake's static graph, loaded to
+            // no registry). buildx bake resolves `FROM` from the local store only
+            // with the `docker` driver (a real Docker daemon, e.g. CI); the
+            // `docker-container` driver (a podman-backed Docker, or a remote
+            // BuildKit) can't. Fall back to `podman build`, which reads the local
+            // store natively (as build.sh does), driving the *same* spec
+            // `bake --print` resolves so bake stays the source of truth (principle
+            // 3). Shared-env evals and remote `--builder` runs keep the bake path.
+            if task_id.is_some() && builder.is_none() {
+                // --dry-run prints BOTH the bake attempt and the podman fallback,
+                // since the fallback is what actually runs on a container-driver
+                // host (principle 2 — print the command it would run).
+                if dry_run {
+                    bake_with_env(registry, "eval", &overrides, &bake_env, builder, true)?;
+                    eprintln!("# …or, when buildx can't resolve the local base, instead:");
+                    let spec = bake_print("eval", &overrides, registry, &bake_env)?;
+                    return podman_build_eval(&spec, true);
+                }
+                if let Err(bake_err) =
+                    bake_with_env(registry, "eval", &overrides, &bake_env, builder, false)
+                {
+                    eprintln!(
+                        "eval: `docker buildx bake` failed ({bake_err}); retrying with \
+                         `podman build` (reads the local image store)"
+                    );
+                    let spec = bake_print("eval", &overrides, registry, &bake_env)?;
+                    return podman_build_eval(&spec, false);
+                }
+                return Ok(());
+            }
             bake_with_env(registry, "eval", &overrides, &bake_env, builder, dry_run)
         }
         BuildTarget::Compose { benchmark } => {
@@ -510,6 +542,55 @@ fn run_build_script(script: &str, image: &str, task_id: &str, dry_run: bool) ->
     }
 }
 
+/// Build a per-task eval combination with `podman build` from the bake-resolved
+/// spec — the fallback when `docker buildx bake` can't resolve the per-task base
+/// because the active builder (a `docker-container`/remote BuildKit) doesn't read
+/// the local image store. `podman build` reads it natively (the same reason
+/// build.sh uses podman). bake stays the single source of truth: `spec` comes
+/// from `docker buildx bake --print` (src/RULES.md principle 3), so this never
+/// re-derives the build graph. amd64 matches the per-task base + the fleet.
+fn podman_build_eval(spec: &BakeTargetSpec, dry_run: bool) -> Result<(), String> {
+    let context = spec.context.clone().unwrap_or_else(|| ".".into());
+    let dockerfile = spec
+        .dockerfile
+        .clone()
+        .unwrap_or_else(|| "Dockerfile".into());
+    // bake emits `dockerfile` relative to `context`; podman's `-f` is relative to cwd.
+    let dockerfile_path = format!("{}/{}", context.trim_end_matches('/'), dockerfile);
+    let tag = spec
+        .tags
+        .as_ref()
+        .and_then(|t| t.first())
+        .ok_or("bake target 'eval' has no tags")?;
+
+    let mut cmd = Command::new("podman");
+    cmd.args(["build", "--platform", "linux/amd64", "-t"])
+        .arg(tag)
+        .arg("-f")
+        .arg(&dockerfile_path);
+    let mut shown = format!("podman build --platform linux/amd64 -t {tag} -f {dockerfile_path}");
+    for (k, v) in spec.args.iter().flatten() {
+        if v.is_empty() {
+            continue;
+        }
+        cmd.arg("--build-arg").arg(format!("{k}={v}"));
+        shown.push_str(&format!(" --build-arg {k}={v}"));
+    }
+    cmd.arg(&context);
+    eprintln!("$ {shown} {context}");
+    if dry_run {
+        return Ok(());
+    }
+
+    let status = cmd
+        .status()
+        .map_err(|e| format!("failed to run podman build: {e}"))?;
+    if !status.success() {
+        return Err(format!("podman build failed with {status}"));
+    }
+    Ok(())
+}
+
 // ─── OpenShift BuildConfig backend (`--builder oc`) ──────────────────────────
 //
 // Builds a SINGLE artifact in-cluster via a binary Docker-strategy

containers/benchmarks/skills-bench/Dockerfile

@@ -38,6 +38,10 @@ LABEL eval.benchmark.paper="https://arxiv.org/abs/2602.12670"
 # Record the pinned upstream revision (rule 3/4) — driven by the same SB_REF that
 # build.sh fetches from, so the label can never disagree with the baked data.
 LABEL eval.benchmark.data_revision="${SB_REF}"
+# Released: proven end-to-end with a recorded replay fixture
+# (tests/run/replay/fixtures/skills-bench-citation-check-claude-code.trajectory.jsonl),
+# benchmarks/RULES.md rule 21a.
+LABEL eval.benchmark.released="true"
 
 ENV EVAL_BENCHMARK=skills-bench \
     EVAL_TASK_ID=${EVAL_TASK_ID} \

containers/benchmarks/terminal-bench/Dockerfile

@@ -20,6 +20,11 @@ LABEL eval.benchmark.tasks="89"
 LABEL eval.benchmark.env="per-task"
 LABEL eval.benchmark.internet="false"
 LABEL eval.benchmark.url="https://github.com/harbor-framework/terminal-bench-2-1"
+LABEL eval.benchmark.description="Terminal-Bench 2.1 (Harbor): real-world terminal tasks, each with its own environment and upstream test suite"
+# Released: proven end-to-end with a recorded replay fixture
+# (tests/run/replay/fixtures/terminal-bench-configure-git-webserver-claude-code.trajectory.jsonl),
+# benchmarks/RULES.md rule 21a.
+LABEL eval.benchmark.released="true"
 # EVAL_TASK_ID is consumed here (the per-task marker, benchmarks/RULES.md 24a/24f).
 # TBENCH_TASK pins the real task name for solution.sh: the oracle/runner override
 # EVAL_TASK_ID at run time (their /tasks index), so the gold fetcher can't rely on
@@ -30,6 +35,9 @@ ENV EVAL_BENCHMARK=tbench \
     EVAL_TASK_ID=${EVAL_TASK_ID} \
     TBENCH_TASK=${EVAL_TASK_ID} \
     TBENCH_REF=${TBENCH_REF}
+# Record the pinned upstream revision (rule 3/4) — driven by the same TBENCH_REF
+# build.sh fetches from, so the label can never disagree with the baked data.
+LABEL eval.benchmark.data_revision="${TBENCH_REF}"
 
 # The instruction the agent sees (TASK) — agent-readable by design.
 COPY instruction.md /task/instruction.md