Skip to content

Default codebase fixtures and shipped docs #265

Description

@slowdini

Part of #244 (full-codebase eval testbed). No hard dependency — the fixture repo can be published
today
; the init scaffolding lands once the codebase block exists.

Why

#244 asks for this directly:

We should provide a default fixture for a simple codebase (maybe just by creating it on github),
and suggest using the eval-magic codebase itself as a complex fixture. Instructions for what
fixtures should be used should be part of the committed eval files in some way.

Without a default, the first thing every new user hits after the codebase feature ships is "…so what
do I point this at?" A published fixture makes the feature usable on day one; documenting eval-magic
itself gives people a genuinely complex target to graduate to.

Scope

  • Publish a small default fixture project on GitHub — enough structure for a realistic task, small
    enough to clone quickly.
  • Scaffold a codebase block from eval-magic init pointing at it, with a pinned ref.
  • New shipped guide docs/guides/codebases.mdeval-magic docs codebases, covering how to source
    a codebase, how to pin it, how the resolved SHA is recorded, and when to reach for a complex
    fixture. build.rs discovers docs/guides/*.md at compile time, so the topic registers
    automatically; the filename stem is the topic name and must be ASCII kebab-case.
  • Document eval-magic itself as the complex fixture.

Per CLAUDE.md, shipped output refers to guides as eval-magic docs <topic>, never repo-relative
paths.

Files

  • docs/guides/codebases.md (new)
  • src/cli/commands/init.rs
  • src/cli/help.rs
  • tests/cli/docs.rs

Acceptance criteria

  • eval-magic docs lists codebases, and eval-magic docs codebases prints the guide.
  • eval-magic init scaffolds a runnable eval with a pinned codebase block.
  • That scaffolded eval runs end to end against the published fixture.
  • The guide covers both the simple fixture and eval-magic-as-fixture.
  • Guide-contract tests in tests/cli/docs.rs cover the new topic.

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions