Skip to content

Multi-sample judging (pass^k) #262

Description

@slowdini

Part of #244 (full-codebase eval testbed).

Why

Judging is emitted once per (eval, condition, run, assertion)
(src/pipeline/grade/judge_tasks.rs). For a mechanical rubric that is fine. For an opinionated one —
"is this code better?" — a single verdict is a coin flip dressed as a measurement.

#244 argues the economics run the other way: executing a real task is expensive, judging it is cheap.

With a question like "is the code better?", we'd likely see more value by running the test once,
and judging it multiple times. If 6/10 judges grade no-skill work as "good", and 10/10 judges grade
with-skill work as "good", we'd count that as marked improvement.

(#244 references pass^k grading from
https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.)

Scope

  • Add samples: N to llm_judge assertions, plus a run-level default.
  • Emit N judge tasks per assertion and aggregate them into a vote proportion and a pass^k
    figure, rather than collapsing to a single boolean.
  • Carry the counts through grading.json, benchmark.json, and aggregate so a campaign report can
    show 6/10 vs 10/10 instead of fail vs pass.
  • Update the statistics narrative. format_minimum_attainable_fisher_p_value
    (src/cli/run/statistics.rs) and the run plan's "statistical floor" line currently assume a binary
    per-run endpoint. Multi-sample judging changes what the endpoint is, and the printed guidance
    must not keep describing the old one.

Files

  • src/pipeline/grade/{judge_tasks,finalize}.rs
  • src/pipeline/aggregate.rs
  • src/cli/run/statistics.rs
  • schema/{judge-tasks,grading,benchmark}.schema.json

Acceptance criteria

  • samples: 10 emits ten judge tasks for that assertion in each cell.
  • grading.json records the vote count, not just a boolean.
  • benchmark.json and aggregate surface per-condition vote proportions and the pass^k figure.
  • samples: 1 (the default) produces output equivalent to today's, so existing evals are unaffected.
  • Missing judge responses degrade to FAIL for that sample only, as today — not for the assertion.
  • The statistical-floor line printed by run describes the endpoint actually in use.

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions