What's wrong
Two problems, found by reading upstream's benchmark/benchmark.py and prompts.py against ours.
1. We drop the round of test feedback the published metric is measured after.
Upstream's loop per exercise is: model edits the stub → harness runs the hidden suite → on failure the sanitized test output comes back with "See the testing errors above. The tests are correct, don't try and change them. Fix the code in {file_list} to resolve the errors." → second attempt → grade. The leaderboard sorts on pass_rate_2 — the number after that round. It is worth a lot: Gemini 2.0 Pro is 20.4 (pass_rate_1) → 35.6 (pass_rate_2).
Our runner gives the agent none of it. It has a shell, six toolchains, and a multi-minute EVAL_TIMEOUT, and nothing to execute against — no test, no way to check its own work. We measure one-shot code writing, which is what the chat-model benchmark measured, and we penalize an agent scaffold for its only differentiator.
2. Java tasks hand the agent the test file.
Java keeps its tests in the src/ tree /entrypoint.sh copies wholesale into /app, and #276's chown -R 1002 /app made the copies agent-readable. All 47 java tasks. The same copy also nests a second project tree at /app/src/src (cp -r src /app/src onto a directory the solution-file loop already created).
Smaller deviations from upstream
- The prompt is ours, not upstream's
instructions_addendum.
- Test timeout is 120s; upstream's is 180s.
/grade.sh sends the suite's output to /dev/null, so a toolchain failure and a wrong answer look identical in the logs.
What to do
Give the agent one root-served run of the hidden suite (run-tests) rather than the test files — same information upstream feeds back, same leak surface, one agent invocation. Fix the java leak so the suite is hidden in every language. Align prompt, timeout, and logging with upstream.
What's wrong
Two problems, found by reading upstream's
benchmark/benchmark.pyandprompts.pyagainst ours.1. We drop the round of test feedback the published metric is measured after.
Upstream's loop per exercise is: model edits the stub → harness runs the hidden suite → on failure the sanitized test output comes back with
"See the testing errors above. The tests are correct, don't try and change them. Fix the code in {file_list} to resolve the errors."→ second attempt → grade. The leaderboard sorts onpass_rate_2— the number after that round. It is worth a lot: Gemini 2.0 Pro is 20.4 (pass_rate_1) → 35.6 (pass_rate_2).Our runner gives the agent none of it. It has a shell, six toolchains, and a multi-minute
EVAL_TIMEOUT, and nothing to execute against — no test, no way to check its own work. We measure one-shot code writing, which is what the chat-model benchmark measured, and we penalize an agent scaffold for its only differentiator.2. Java tasks hand the agent the test file.
Java keeps its tests in the
src/tree/entrypoint.shcopies wholesale into/app, and #276'schown -R 1002 /appmade the copies agent-readable. All 47 java tasks. The same copy also nests a second project tree at/app/src/src(cp -r src /app/srconto a directory the solution-file loop already created).Smaller deviations from upstream
instructions_addendum./grade.shsends the suite's output to/dev/null, so a toolchain failure and a wrong answer look identical in the logs.What to do
Give the agent one root-served run of the hidden suite (
run-tests) rather than the test files — same information upstream feeds back, same leak surface, one agent invocation. Fix the java leak so the suite is hidden in every language. Align prompt, timeout, and logging with upstream.