Skip to content

Retire the fixture-only eval model #266

Description

@slowdini

Part of #244 (full-codebase eval testbed). Last ticket — it closes out the migration once
codebase evals are the norm.

Why

#244 is explicit about wanting one architecture, not two:

My feeling is this will present a number of breaking changes, and that there's no reason to support
legacy modes of evals. If we can architecturally handle all our evals in essentially the same way,
we should aim for that goal. The simplicity will be a big benefit in the long run.

Decision taken on #244: files/files_root survive, re-cast as a documented overlay on the
codebase. Seeding a task-specific file into a real repo is a live use, not legacy baggage. Only
isolation is removed — it is already documented in the schema as a no-op:

"Legacy isolation hint retained for config compatibility. Canonical runs already give every
(eval, condition, run) dispatch a private environment […] so 'shared' and 'isolated' currently have
the same effective isolation."

Scope

  • Remove the isolation field from the schema, the Eval type, and validation.
  • Re-document files/files_root as a codebase overlay everywhere they are described — schema
    descriptions, --help, and the shipped guides.
  • Migrate eval-magic init scaffolding, integration tests, and golden artifacts onto the single
    codebase-backed model.
  • Sweep the docs for prose still describing the fixture-only world.

Files

  • schema/evals.schema.json
  • src/core/types.rs
  • src/validation/evals.rs
  • src/cli/commands/init.rs
  • tests/, docs/

Acceptance criteria

  • isolation is gone from the schema and the type; a config still carrying it fails validation with
    a message naming the field and saying it is no longer needed.
  • files/files_root are documented as an overlay in the schema, --help, and the guides.
  • No test or golden artifact depends on the fixture-only model as the primary path.
  • The docs describe one way to build an eval environment.

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions