Code benchmarks derive the verdict from an exit status the agent's own code controls

[elronbandel]

The hole

aider-polyglot grades with if /tests/run.sh; then echo 1.0, and /tests/run.sh ends in the test command's exit status. The agent's submitted code runs inside that process, so it can just exit 0 itself.

Confirmed on a built image, python/affine-cipher:

import os
os._exit(0)

→ reward=1.0, with no solution. sys.exit(0) does not work (pytest catches SystemExit), but os._exit bypasses it. Equivalents exist per language: func init() { os.Exit(0) } (go), std::process::exit(0) (rust), top-level process.exit(0) (javascript), a __attribute__((constructor)) (cpp). Java is the exception — System.exit from a Gradle test worker fails the build.

This predates the run-tests change and is not specific to this benchmark: any grader whose verdict is "the test command exited 0" while executing agent-authored code has it. Upstream aider is identical (success = result.returncode == 0), so it is not a porting error.

It also has a non-adversarial face: gradle test and (before the change to drop it) ctest exit 0 when no tests run at all, so a packaging slip scores 1.0 rather than failing loudly.

Fix

Stop trusting the exit status; require positive evidence that tests ran and passed. Per language, that means a machine-readable report rather than a summary line the same code could print:

• python: pytest --junit-xml
• go: go test -json
• rust: cargo test -- --format json (or test result: ok. N passed with N > 0)
• javascript: jest --json --outputFile
• java: build/test-results/test/*.xml
• cpp: ctest --output-junit (needs the exercises' CMakeLists to register tests, which they don't — the Catch2 binary runs as a post-build step)

Require tests > 0 && failures == 0. Note the report path is writable by the uid the suite runs as, so this raises the bar rather than closing it completely; closing it fully means running the reporter outside the process that hosts the agent's code.

Worth auditing the other code benchmarks (humaneval, mbpp, bigcodebench, swe-bench) for the same pattern.