Skip to content

compare: paired evidence for interactive judging #263

Description

@slowdini

Part of #244 (full-codebase eval testbed). Best after the judge evidence bundle ticket, whose
bundles it pairs.

Why

#244 describes a workflow that has no home in the CLI today:

The user can ask the agent to have eval-magic run both the no-skill and with-skill sides of the
test, then ask vague questions like "did the skill make the code better?" or "did the skill confuse
the agent at any point?". This won't be statistically reliable information, but it can lead to the
discovery of worthwhile test cases.

That is how eval cases get drafted — you look at two comparable transcripts from the same starting
prompt and notice what differs. Right now the evidence for both arms exists but is scattered across
eval-<id>/<condition>/[run-<k>/]{run.json,outputs/,grading.json}, and nothing pairs them.

Every other command in the pipeline requires assertions to be written first. This one deliberately
does not.

Scope

  • eval-magic compare --iteration N --eval <id>.
  • Writes iteration-N/compare/<eval-id>.md pairing both arms' evidence bundles — prompt, final
    message, diff, changed files, conversation transcript — and prints the path.
  • Works with no assertions defined. It answers open questions; it does not grade.
  • Handles multi-run cells by naming which run is being shown.

Files

  • src/cli/commands/compare.rs (new)
  • src/cli/commands/mod.rs
  • src/cli/args.rs, src/cli/help.rs

Acceptance criteria

  • compare works on an iteration whose evals declare no assertions.
  • The output pairs both conditions for the same eval, with the diff for each.
  • A multi-run cell is disambiguated in the output.
  • A missing arm produces a clear message rather than a panic or a silently half-empty document.
  • The path is printed so a driving agent can read it directly.

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions