Part of #244 (full-codebase eval testbed). Best after the judge evidence bundle ticket, whose
bundles it pairs.
Why
#244 describes a workflow that has no home in the CLI today:
The user can ask the agent to have eval-magic run both the no-skill and with-skill sides of the
test, then ask vague questions like "did the skill make the code better?" or "did the skill confuse
the agent at any point?". This won't be statistically reliable information, but it can lead to the
discovery of worthwhile test cases.
That is how eval cases get drafted — you look at two comparable transcripts from the same starting
prompt and notice what differs. Right now the evidence for both arms exists but is scattered across
eval-<id>/<condition>/[run-<k>/]{run.json,outputs/,grading.json}, and nothing pairs them.
Every other command in the pipeline requires assertions to be written first. This one deliberately
does not.
Scope
eval-magic compare --iteration N --eval <id>.
- Writes
iteration-N/compare/<eval-id>.md pairing both arms' evidence bundles — prompt, final
message, diff, changed files, conversation transcript — and prints the path.
- Works with no assertions defined. It answers open questions; it does not grade.
- Handles multi-run cells by naming which run is being shown.
Files
src/cli/commands/compare.rs (new)
src/cli/commands/mod.rs
src/cli/args.rs, src/cli/help.rs
Acceptance criteria
compare works on an iteration whose evals declare no assertions.
- The output pairs both conditions for the same eval, with the diff for each.
- A multi-run cell is disambiguated in the output.
- A missing arm produces a clear message rather than a panic or a silently half-empty document.
- The path is printed so a driving agent can read it directly.
Cross-cutting requirements
- Mode B parity. Acceptance must hold for both
Mode::NewSkill and Mode::Revision
(src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
src/workspace/promote.rs) must keep working against codebase-backed iterations.
- Provenance reaches the report. Source + resolved SHA must land in
conditions.json, each
run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
workspace.
Verification
cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings
Implementation requires the slow-powers:working-with-tdd skill.
Parent: #244
Part of #244 (full-codebase eval testbed). Best after the judge evidence bundle ticket, whose
bundles it pairs.
Why
#244 describes a workflow that has no home in the CLI today:
That is how eval cases get drafted — you look at two comparable transcripts from the same starting
prompt and notice what differs. Right now the evidence for both arms exists but is scattered across
eval-<id>/<condition>/[run-<k>/]{run.json,outputs/,grading.json}, and nothing pairs them.Every other command in the pipeline requires assertions to be written first. This one deliberately
does not.
Scope
eval-magic compare --iteration N --eval <id>.iteration-N/compare/<eval-id>.mdpairing both arms' evidence bundles — prompt, finalmessage, diff, changed files, conversation transcript — and prints the path.
Files
src/cli/commands/compare.rs(new)src/cli/commands/mod.rssrc/cli/args.rs,src/cli/help.rsAcceptance criteria
compareworks on an iteration whose evals declare no assertions.Cross-cutting requirements
Mode::NewSkillandMode::Revision(
src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,src/workspace/promote.rs) must keep working against codebase-backed iterations.conditions.json, eachrun.json,benchmark.json, and theBASELINE.mdbuilt bypromote.rs:231— not just in theworkspace.
Verification
Implementation requires the
slow-powers:working-with-tddskill.Parent: #244