feat: source a real codebase into each task environment

[slowdini]

Closes #252. Foundation ticket for #244 — the full-codebase eval testbed.

What changes

An eval set can declare a codebase and every (eval, condition, run) environment is built from a
real checkout of it instead of a handful of fixture files.

{
  "skill_name": "working-with-tdd",
  "codebase": { "url": "https://github.com/slowdini/example-project", "ref": "v1.4.0" },
  "evals": [
    { "id": "add-a-feature", "prompt": "...", "expected_output": "...", "files": ["docs/TASK.md"] },
    { "id": "big-refactor",  "prompt": "...", "expected_output": "...",
      "codebase": { "path": "../fixtures/legacy-service" } }
  ]
}

Config-level default, per-eval override, mirroring runs. Sources are a git url + required ref,
or a local path. files/files_root survive, re-cast as an overlay applied on top of the checkout.

Before / after

A prepared environment used to be a fresh git init holding only what the runner placed:

$ git -C env-g1-with_skill log --oneline
9c1f0a2 eval-magic task baseline

It is now the project, with its history, on its own branch:

$ git -C env-g1-with_skill log --oneline | head -3
685317e eval-magic task baseline
9ba3fcd second
8e02560 first
$ git -C env-g1-with_skill remote -v          # empty — nothing can reach the source
$ git -C env-g1-with_skill symbolic-ref --short HEAD
main
$ git -C env-g1-with_skill rev-parse refs/eval-magic/baseline HEAD
685317ea9d9bcce33a31227f623d8635f03b7f3c
685317ea9d9bcce33a31227f623d8635f03b7f3c

Notable decisions

The task-repository lifecycle had two invariants a real codebase breaks by construction. It
git inited every environment from nothing, and verify_task_repository rejected any remote. A
sourced environment now keeps the .git its clone brought, has remotes stripped rather than asserted
absent, and is marked with refs/eval-magic/baseline — which #255 measures against. Fixture-only
evals still get git init on work, unchanged.

The baseline git add drops --force. Forcing made sense when every file present was one the
runner placed; against a real repository it would sweep target/ or node_modules/ into the state
every run starts from. The add now respects the codebase's .gitignore, and runner-placed paths —
harness config directories and the fixture overlay — are forced in on top, so a codebase that ignores
.claude/ cannot hide the staged skill from the baseline and put the condition under test outside
every later diff.

A local path is materialized as a clean checkout of its committed state. History intact,
uncommitted work and ignored build output not carried; resolution warns when the source is dirty.

Host-local provenance is labelled rather than papered over. Nobody else can resolve
../fixtures/legacy-service, so host_local: true is recorded, the run warns, and the BASELINE.md
row says so. Where the directory is a repository, its origin and the resolved commit are recorded
too — those resolve anywhere.

The resolver runs git with the operator's configuration held off. Found while testing: an
inherited url.<base>.insteadOf rewrites the codebase URL, so the tree sourced is not the tree the
report cites — a silent wrong answer rather than a failure. System and global config are off, the
GIT_CONFIG_COUNT mechanism is cleared, and clone/init get an empty template so a configured
init.templateDir cannot seed hooks.

src/source/ is a shared module, not inlined. #253 resolves the skills under test through the
same resolver, so it knows nothing about what a codebase is.

Provenance

The resolved commit — not the declared ref, which moves — reaches conditions.json, each run.json,
benchmark.json, and BASELINE.md. Absent-when-empty throughout, so fixture-only artifacts are
byte-identical.

Also fixed

tests/run/git_isolation.rs pinned %aI as ...Z, which git renders as +00:00 on 2.43 and Z
only on newer versions — green on CI, red on any host with the older git. Normalized.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

865 lib + 178 CLI + 134 run tests pass; fmt and clippy clean. Every acceptance criterion on #252 was
also checked end to end against a real repository: history preserved, no remotes, baseline ref equal
to HEAD, clean tree, on the codebase's own branch, gitignored output excluded from the baseline,
overlay visible at its declared path, and the recorded revision matching the source's HEAD.

Docs

eval-magic docs codebase is the new shipped guide — the feature has no CLI flag, so --help cannot
carry its rules. The isolation guide gains a section separating the two boundaries it would otherwise
be read as covering: what a dispatch can load is not what a dispatch can reach.

Follow-ups (unchanged scope, tracked on #244)

• Provision environments from a per-iteration codebase cache #254 replaces the per-environment copy with git clone --local off a SHA-keyed cache.
• Measure and capture diffs with Git instead of copying a baseline #255 swaps the copy-based diff baseline for Git. Until it lands, capture_iteration_baselines
  still copies each working tree, so a large codebase is correct but slow.

🤖 Generated with Claude Code