Part of #244 (full-codebase eval testbed). Last ticket — it closes out the migration once
codebase evals are the norm.
Why
#244 is explicit about wanting one architecture, not two:
My feeling is this will present a number of breaking changes, and that there's no reason to support
legacy modes of evals. If we can architecturally handle all our evals in essentially the same way,
we should aim for that goal. The simplicity will be a big benefit in the long run.
Decision taken on #244: files/files_root survive, re-cast as a documented overlay on the
codebase. Seeding a task-specific file into a real repo is a live use, not legacy baggage. Only
isolation is removed — it is already documented in the schema as a no-op:
"Legacy isolation hint retained for config compatibility. Canonical runs already give every
(eval, condition, run) dispatch a private environment […] so 'shared' and 'isolated' currently have
the same effective isolation."
Scope
- Remove the
isolation field from the schema, the Eval type, and validation.
- Re-document
files/files_root as a codebase overlay everywhere they are described — schema
descriptions, --help, and the shipped guides.
- Migrate
eval-magic init scaffolding, integration tests, and golden artifacts onto the single
codebase-backed model.
- Sweep the docs for prose still describing the fixture-only world.
Files
schema/evals.schema.json
src/core/types.rs
src/validation/evals.rs
src/cli/commands/init.rs
tests/, docs/
Acceptance criteria
isolation is gone from the schema and the type; a config still carrying it fails validation with
a message naming the field and saying it is no longer needed.
files/files_root are documented as an overlay in the schema, --help, and the guides.
- No test or golden artifact depends on the fixture-only model as the primary path.
- The docs describe one way to build an eval environment.
Cross-cutting requirements
- Mode B parity. Acceptance must hold for both
Mode::NewSkill and Mode::Revision
(src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
src/workspace/promote.rs) must keep working against codebase-backed iterations.
- Provenance reaches the report. Source + resolved SHA must land in
conditions.json, each
run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
workspace.
Verification
cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings
Implementation requires the slow-powers:working-with-tdd skill.
Parent: #244
Part of #244 (full-codebase eval testbed). Last ticket — it closes out the migration once
codebase evals are the norm.
Why
#244 is explicit about wanting one architecture, not two:
Decision taken on #244:
files/files_rootsurvive, re-cast as a documented overlay on thecodebase. Seeding a task-specific file into a real repo is a live use, not legacy baggage. Only
isolationis removed — it is already documented in the schema as a no-op:Scope
isolationfield from the schema, theEvaltype, and validation.files/files_rootas a codebase overlay everywhere they are described — schemadescriptions,
--help, and the shipped guides.eval-magic initscaffolding, integration tests, and golden artifacts onto the singlecodebase-backed model.
Files
schema/evals.schema.jsonsrc/core/types.rssrc/validation/evals.rssrc/cli/commands/init.rstests/,docs/Acceptance criteria
isolationis gone from the schema and the type; a config still carrying it fails validation witha message naming the field and saying it is no longer needed.
files/files_rootare documented as an overlay in the schema,--help, and the guides.Cross-cutting requirements
Mode::NewSkillandMode::Revision(
src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,src/workspace/promote.rs) must keep working against codebase-backed iterations.conditions.json, eachrun.json,benchmark.json, and theBASELINE.mdbuilt bypromote.rs:231— not just in theworkspace.
Verification
Implementation requires the
slow-powers:working-with-tddskill.Parent: #244