Part of #244 (full-codebase eval testbed).
Why
Judging is emitted once per (eval, condition, run, assertion)
(src/pipeline/grade/judge_tasks.rs). For a mechanical rubric that is fine. For an opinionated one —
"is this code better?" — a single verdict is a coin flip dressed as a measurement.
#244 argues the economics run the other way: executing a real task is expensive, judging it is cheap.
With a question like "is the code better?", we'd likely see more value by running the test once,
and judging it multiple times. If 6/10 judges grade no-skill work as "good", and 10/10 judges grade
with-skill work as "good", we'd count that as marked improvement.
(#244 references pass^k grading from
https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.)
Scope
- Add
samples: N to llm_judge assertions, plus a run-level default.
- Emit N judge tasks per assertion and aggregate them into a vote proportion and a pass^k
figure, rather than collapsing to a single boolean.
- Carry the counts through
grading.json, benchmark.json, and aggregate so a campaign report can
show 6/10 vs 10/10 instead of fail vs pass.
- Update the statistics narrative.
format_minimum_attainable_fisher_p_value
(src/cli/run/statistics.rs) and the run plan's "statistical floor" line currently assume a binary
per-run endpoint. Multi-sample judging changes what the endpoint is, and the printed guidance
must not keep describing the old one.
Files
src/pipeline/grade/{judge_tasks,finalize}.rs
src/pipeline/aggregate.rs
src/cli/run/statistics.rs
schema/{judge-tasks,grading,benchmark}.schema.json
Acceptance criteria
samples: 10 emits ten judge tasks for that assertion in each cell.
grading.json records the vote count, not just a boolean.
benchmark.json and aggregate surface per-condition vote proportions and the pass^k figure.
samples: 1 (the default) produces output equivalent to today's, so existing evals are unaffected.
- Missing judge responses degrade to FAIL for that sample only, as today — not for the assertion.
- The statistical-floor line printed by
run describes the endpoint actually in use.
Cross-cutting requirements
- Mode B parity. Acceptance must hold for both
Mode::NewSkill and Mode::Revision
(src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
src/workspace/promote.rs) must keep working against codebase-backed iterations.
- Provenance reaches the report. Source + resolved SHA must land in
conditions.json, each
run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
workspace.
Verification
cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings
Implementation requires the slow-powers:working-with-tdd skill.
Parent: #244
Part of #244 (full-codebase eval testbed).
Why
Judging is emitted once per
(eval, condition, run, assertion)(
src/pipeline/grade/judge_tasks.rs). For a mechanical rubric that is fine. For an opinionated one —"is this code better?" — a single verdict is a coin flip dressed as a measurement.
#244 argues the economics run the other way: executing a real task is expensive, judging it is cheap.
(#244 references pass^k grading from
https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.)
Scope
samples: Ntollm_judgeassertions, plus a run-level default.figure, rather than collapsing to a single boolean.
grading.json,benchmark.json, andaggregateso a campaign report canshow 6/10 vs 10/10 instead of fail vs pass.
format_minimum_attainable_fisher_p_value(
src/cli/run/statistics.rs) and the run plan's "statistical floor" line currently assume a binaryper-run endpoint. Multi-sample judging changes what the endpoint is, and the printed guidance
must not keep describing the old one.
Files
src/pipeline/grade/{judge_tasks,finalize}.rssrc/pipeline/aggregate.rssrc/cli/run/statistics.rsschema/{judge-tasks,grading,benchmark}.schema.jsonAcceptance criteria
samples: 10emits ten judge tasks for that assertion in each cell.grading.jsonrecords the vote count, not just a boolean.benchmark.jsonandaggregatesurface per-condition vote proportions and the pass^k figure.samples: 1(the default) produces output equivalent to today's, so existing evals are unaffected.rundescribes the endpoint actually in use.Cross-cutting requirements
Mode::NewSkillandMode::Revision(
src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,src/workspace/promote.rs) must keep working against codebase-backed iterations.conditions.json, eachrun.json,benchmark.json, and theBASELINE.mdbuilt bypromote.rs:231— not just in theworkspace.
Verification
Implementation requires the
slow-powers:working-with-tddskill.Parent: #244