A runtime-verifiable audit harness that falsifies coding-agent intervention claims: it tests whether a change actually moved an official SWE-bench outcome, or whether the apparent improvement is leakage, noise, a benchmark artifact, or a non-reproducible scaffold difference.
It is not a groundedness / faithfulness / RAG-quality scorer — that lane (Ragas, DeepEval, TruLens, Phoenix, LangSmith, Braintrust) is already crowded and well served. See MEMO.md for the one-page method memo and COVER_NOTE.md for the short version.
preregistration → runtime hook trigger-hit → comparable control/treatment arms → manipulation checks → anti-oracle-leakage → measured per-task noise floor (ε) → isolated official SWE-bench outcome anchoring → post-hoc audit / power boundary → and the discipline to report null.
Runtime guardrails (Guardrails, NeMo), experiment/trace tooling (LangSmith, Braintrust, Phoenix, Langfuse), and the SWE-bench harness each ship pieces of this. The contribution is the assembled closed loop plus null-reporting discipline. Scope is honest: it needs a verifiable ground-truth outcome, so it applies to benchmarkable agent tasks (SWE-bench here), not arbitrary production agents.
Generated from the bundled official-eval reports under models/ by run_intervention_verdict_demo.py,
not hand-authored:
| Intervention | Class | Trigger hit | Leakage check | Official outcome | Verdict |
|---|---|---|---|---|---|
| v1 constraint hook (break-loop / require-repro) | behavioral, deployable | yes | clean | 1 pair, 0 uplift | calibrated null |
| v2 constraint hook (break-recurrence + reproduce) | behavioral, deployable | yes | clean | 4 strict-fresh pairs, 0 uplift | calibrated null (underpowered) |
| B1 deployable-info injection (issue-derived reproduction) | informational, deployable | yes (smoke) | redundant w/ control | no valid pair | design dead-end - no verdict |
| oracle / answer-bearing injection (positive control) | informational, NON-deployable | yes | fail (by design) | 1 resolved (1 patch+outcome improved) | true positive - sensitivity anchor |
| oracle-probe sweep (sphinx-8435 / sphinx-8474) | informational, NON-deployable | yes | fail (by design) | 4/4 unmoved | channel bottleneck / capability ceiling |
| epsilon noise floor (control-arm reruns) | calibration | n/a | n/a | 0 flips / 6 reruns | epsilon point 0.00 (95% upper 0.39) |
Null results prove the harness is not credulous; the oracle-positive control proves it is not blind.
Honest caveat: the oracle/positive control moved 1 task, but a dedicated oracle probe failed on 4/4 runs across 2 other tasks (channel bottleneck / capability ceiling) — so "not blind" is an existence proof, not a sensitivity guarantee. STR and trajectory diagnostics are audit-only, not predictive.
The demo is stdlib-only and reads the recorded official-eval reports bundled under models/:
python3 run_intervention_verdict_demo.py # prints the verdict table above
python3 run_intervention_verdict_demo.py -o out.json # + per-row source path, SHA256, upstream decision
python3 run_intervention_verdict_demo.py --check-docs # README + MEMO drift guard (fails on drift)
pytest tests/ # unit + fail-fast + drift testsEvery row traces to a source report path + SHA256 (provenance.json), and the SHA chain
(MANIFEST.json = provenance.json = bundled files) is closed and self-checked. See
VERIFY.md.
- The OR-Bench over-refusal eval used here as a benchmark: https://github.com/haeliotang/inspect-evals-orbench
(registered to AISI's
inspect_evals).
MIT — see LICENSE.
- active-inference-fairtest-v1 — a preregistered, oracle-checked, power-analyzed fair test of an "Active Inference / free-energy minimization" cognition layer in a production coding agent. Verdict: null, with the mechanism identified at code level (the minimization dynamic was never instantiated as built). Start with PUBLIC_RELEASE_NOTE.md.