feat: source a real codebase into each task environment

docs/developer_overview.md

@@ -43,6 +43,9 @@ preconditions, handoffs, and recovery commands.
   few named capabilities that require harness-specific code.
 - `src/sandbox/`, `src/workspace/`, and `src/validation/` own task isolation, filesystem/workspace
   mechanics, and configuration checks.
+- `src/source/` resolves a declared source — a git URL and ref, or a local directory — to a commit,
+  and materializes it as a tree. It knows nothing about what is being sourced, so both the codebase
+  a task environment is built from and the skills under test resolve through it.
 - `schema/` contains the JSON schemas for user input and generated artifacts.
 - `harnesses/` contains built-in descriptors, descriptor scaffolding, and embedded harness assets.
 - `profiles/` contains shared prompt profiles.
@@ -143,3 +146,5 @@ implementation evidence in an internal note.
   `eval-magic docs byoh`.
 - [Shipped isolation guide](guides/isolation.md) is the repository source for
   `eval-magic docs isolation`.
+- [Shipped codebase guide](guides/codebase.md) is the repository source for
+  `eval-magic docs codebase`.

docs/guides/codebase.md

@@ -0,0 +1,123 @@
+# Sourcing a codebase into a task environment
+
+An eval's environment can be a real project rather than a handful of fixture files. Declare a
+`codebase` in `evals.json` and every `(eval, condition, run)` environment is built from a checkout
+of it — with history, on a branch, ready for the agent under test to work in.
+
+This matters for anything you cannot judge from a toy problem. Whether a skill makes an agent's
+code *better* is not answerable when the task is small enough that any model succeeds.
+
+## Declare one
+
+A git repository, with an explicit ref:
+
+```json
+{
+  "skill_name": "working-with-tdd",
+  "codebase": { "url": "https://github.com/slowdini/example-project", "ref": "v1.4.0" },
+  "evals": [
+    { "id": "add-a-feature", "prompt": "...", "expected_output": "..." }
+  ]
+}
+```
+
+Or a directory on this machine:
+
+```json
+{ "codebase": { "path": "../../fixtures/legacy-service" } }
+```
+
+A relative `path` resolves against the directory holding `evals.json`, so a committed config means
+the same thing in every clone of the skill. Unlike `files_root`, it may be absolute or point
+outside the skill tree — that is the point of it.
+
+The config-level `codebase` is a default. Any eval can override it:
+
+```json
+{
+  "codebase": { "url": "https://github.com/slowdini/example-project", "ref": "main" },
+  "evals": [
+    { "id": "small-fix", "prompt": "...", "expected_output": "..." },
+    { "id": "big-refactor", "prompt": "...", "expected_output": "...",
+      "codebase": { "path": "/srv/projects/monolith" } }
+  ]
+}
+```
+
+## `ref` is required
+
+A git source must name a branch, tag, or full commit SHA. The runner resolves it and records the
+commit, so a report says which tree it measured. An eval tracking whatever `main` happened to be
+could not be re-run against the state it reported on, which is the point of recording provenance at
+all.
+
+Resolution happens before any environment is created. An unreachable repository or a ref that does
+not exist fails the run while it has still built nothing.
+
+## What the environment contains
+
+Each dispatch gets its own private environment holding:
+
+- the codebase, checked out at the resolved commit, with its history intact
+- no remotes — nothing in the environment can reach or push to the source it came from
+- hooks disabled, and a fixed committer identity for the runner's own commit
+- the branch the codebase itself was on: the branch a `ref` names, or the repository's default
+  branch when the ref is a tag or a SHA
+- `refs/eval-magic/baseline`, marking the state the agent started from
+
+An eval that declares no `codebase` still gets a Git repository, initialized on `work`, exactly as
+it always has.
+
+## `files` is an overlay
+
+`files` and `files_root` still work, and are applied *on top* of the codebase at their declared
+paths. Seeding a task-specific file into a real project is the common case:
+
+```json
+{
+  "id": "add-a-feature",
+  "prompt": "Implement what docs/TASK.md describes.",
+  "expected_output": "the feature, with tests",
+  "files": ["docs/TASK.md"]
+}
+```
+
+A fixture overwrites a codebase file of the same path.
+
+The baseline the runner commits respects the codebase's `.gitignore`, so ignored build output stays
+out of it. Fixtures and staged skills are committed regardless of what the codebase ignores.
+
+## A `path` source is not reproducible elsewhere
+
+Someone reading your published results cannot resolve `../../fixtures/legacy-service`. Their machine
+has that directory somewhere else, or not at all. Nothing can fix that, so the artifacts label it:
+the record carries `host_local: true`, the run prints a warning, and the `BASELINE.md` row says so.
+
+Where the directory is itself a Git repository, its `origin` URL and the resolved commit are
+recorded too, and *those* resolve anywhere. Prefer a `url` source for anything you intend to
+publish.
+
+A `path` source is materialized as a clean checkout of its committed state. Uncommitted work in the
+source directory is not carried into the environment; the run warns when the source is dirty.
+
+## Verify the result
+
+From a prepared iteration directory, inspect one environment:
+
+```sh
+cd env-g1-with_skill
+git log --oneline | head
+git remote -v
+git rev-parse refs/eval-magic/baseline HEAD
+git status --porcelain
+```
+
+`git remote -v` and `git status --porcelain` are both empty, and the two revisions match: the
+baseline ref names exactly what the agent started from.
+
+The resolved commit appears in `conditions.json`, each `run.json`, `benchmark.json`, and the
+`BASELINE.md` written by `promote-baseline`:
+
+```sh
+jq '.codebases' conditions.json
+```

docs/guides/isolation.md

@@ -134,6 +134,17 @@ harnesses by checking every rendered eval-agent command in `RUNBOOK.md` and
 dispatch's setting-source selection. A plugin can appear there and remain absent from the dispatch,
 or the reverse. Use the dispatch's init event.
 
+## The task repository is a separate boundary
+
+Skill-source isolation is about what a dispatch can *load*. The task repository is about what it can
+*reach*: every dispatch runs in its own private environment, a Git repository with no remotes and
+hooks disabled, marked with `refs/eval-magic/baseline` at the state the agent started from. That
+holds whether the environment was built from fixture files or from a sourced codebase — see
+`eval-magic docs codebase`.
+
+The two are independent. An environment can be a faithfully isolated repository while the dispatch
+still loads a live skill source, and a shadowed skill is not made safe by the repository boundary.
+
 ## When a source cannot be isolated
 
 Do not declare isolation. Retain the validity warning as the record of a known threat. A symmetric

schema/benchmark.schema.json

@@ -48,6 +48,11 @@
         }
       }
     },
+    "codebases": {
+      "type": "array",
+      "description": "Codebases the compared conditions ran against, echoed from conditions.json. Absent for fixture-only iterations.",
+      "items": { "$ref": "#/definitions/codebaseUse" }
+    },
     "diff_scope": {
       "type": "object",
       "description": "Raw final-environment diff metrics per condition, ordered by eval id and then run index. Omitted for iterations created before diff-scope capture.",
@@ -69,6 +74,42 @@
     }
   },
   "definitions": {
+    "codebaseUse": {
+      "type": "object",
+      "required": ["kind", "source", "branch", "evals"],
+      "additionalProperties": false,
+      "properties": {
+        "kind": {
+          "type": "string",
+          "enum": ["git", "path"],
+          "description": "Whether the codebase came from a repository URL or a directory on the host that ran it."
+        },
+        "source": { "type": "string", "description": "The url or path exactly as declared in evals.json." },
+        "resolved_path": {
+          "type": "string",
+          "description": "Absolute directory a path source resolved to on the host that ran it."
+        },
+        "ref": { "type": "string", "description": "Declared branch, tag, or commit SHA, for a git source." },
+        "revision": {
+          "type": "string",
+          "description": "The commit the run actually ran against. A declared ref does not identify this on its own, because a branch moves. Absent only for a directory carrying no history."
+        },
+        "origin_url": {
+          "type": "string",
+          "description": "The source repository's origin. For a host-local path this is the only handle another reader can resolve: origin_url plus revision names the same tree anywhere."
+        },
+        "branch": { "type": "string", "description": "Branch the task environment was checked out on." },
+        "host_local": {
+          "type": "boolean",
+          "description": "True when the source cannot be resolved off the host that ran it, so a published claim citing it is not reproducible from the eval config alone."
+        },
+        "evals": {
+          "type": "array",
+          "items": { "type": "string" },
+          "description": "Ids of the evals whose environments were built from this codebase."
+        }
+      }
+    },
     "assertionCount": {
       "type": "object",
       "required": ["passed", "n"],

schema/evals.schema.json

@@ -11,13 +11,52 @@
       "type": "string",
       "description": "Name of the skill being evaluated. Should match the skill directory name."
     },
+    "codebase": {
+      "$ref": "#/definitions/codebase",
+      "description": "Default codebase every eval's task environment is built from. A per-eval codebase overrides it."
+    },
     "evals": {
       "type": "array",
       "minItems": 1,
       "items": { "$ref": "#/definitions/eval" }
     }
   },
   "definitions": {
+    "codebase": {
+      "oneOf": [
+        { "$ref": "#/definitions/gitCodebase" },
+        { "$ref": "#/definitions/pathCodebase" }
+      ]
+    },
+    "pathCodebase": {
+      "type": "object",
+      "required": ["path"],
+      "additionalProperties": false,
+      "properties": {
+        "path": {
+          "type": "string",
+          "minLength": 1,
+          "description": "Directory on this host to build the task environment from, resolved relative to this evals.json when relative. Unlike files_root it may be absolute or escape the skill tree, because it deliberately points outside it. A path source is host-local: another machine has the directory elsewhere or not at all, so a run recorded against one is not reproducible from this config alone. When the directory is a Git repository the runner also records its origin URL and resolved SHA, which are."
+        }
+      }
+    },
+    "gitCodebase": {
+      "type": "object",
+      "required": ["url", "ref"],
+      "additionalProperties": false,
+      "properties": {
+        "url": {
+          "type": "string",
+          "minLength": 1,
+          "description": "Git repository to clone the task environment from."
+        },
+        "ref": {
+          "type": "string",
+          "minLength": 1,
+          "description": "Branch, tag, or full commit SHA to check out. Required: the runner records the resolved SHA, so an eval tracking a moving branch could not be re-run against what it measured."
+        }
+      }
+    },
     "eval": {
       "type": "object",
       "required": ["id", "prompt", "expected_output"],
@@ -59,6 +98,10 @@
           "minimum": 1,
           "description": "Runs per condition for this eval, for variance reduction; overrides the --runs flag. Defaults to the flag's value (1 unless raised)."
         },
+        "codebase": {
+          "$ref": "#/definitions/codebase",
+          "description": "Codebase this eval's task environment is built from, overriding the config-level default."
+        },
         "isolation": {
           "type": "string",
           "enum": ["shared", "isolated"],

schema/run-record.schema.json

@@ -84,6 +84,10 @@
     "conversation": {
       "$ref": "#/definitions/conversation",
       "description": "Ordered multi-turn evidence and scripted-delivery outcome. Absent for one-shot runs."
+    },
+    "codebase": {
+      "$ref": "#/definitions/codebase",
+      "description": "The codebase this run's environment was built from. Absent for a fixture-only run."
     }
   },
   "definitions": {
@@ -168,6 +172,37 @@
         "text": { "type": "string" }
       }
     },
+    "codebase": {
+      "type": "object",
+      "required": ["kind", "source", "branch"],
+      "additionalProperties": false,
+      "properties": {
+        "kind": {
+          "type": "string",
+          "enum": ["git", "path"],
+          "description": "Whether the codebase came from a repository URL or a directory on the host that ran it."
+        },
+        "source": { "type": "string", "description": "The url or path exactly as declared in evals.json." },
+        "resolved_path": {
+          "type": "string",
+          "description": "Absolute directory a path source resolved to on the host that ran it."
+        },
+        "ref": { "type": "string", "description": "Declared branch, tag, or commit SHA, for a git source." },
+        "revision": {
+          "type": "string",
+          "description": "The commit the run actually ran against. A declared ref does not identify this on its own, because a branch moves. Absent only for a directory carrying no history."
+        },
+        "origin_url": {
+          "type": "string",
+          "description": "The source repository's origin. For a host-local path this is the only handle another reader can resolve: origin_url plus revision names the same tree anywhere."
+        },
+        "branch": { "type": "string", "description": "Branch the task environment was checked out on." },
+        "host_local": {
+          "type": "boolean",
+          "description": "True when the source cannot be resolved off the host that ran it, so a published claim citing it is not reproducible from the eval config alone."
+        }
+      }
+    },
     "conversationTool": {
       "type": "object",
       "required": ["type", "ordinal", "round", "name"],

src/cli/help.rs

@@ -30,6 +30,11 @@ EXAMPLES:
   # Reduce cost while iterating on the suite
   eval-magic run --only case-a,case-b
 
+  # Run the task against a real project instead of fixture files. The codebase
+  # is declared in evals.json, not on the command line, so it stays a reviewed
+  # property of the eval set
+  eval-magic docs codebase
+
   # Select a built-in harness; `run --help` documents models and environment options
   eval-magic run --harness codex
 

src/cli/run/dispatch.rs

@@ -14,7 +14,9 @@ use serde::{Deserialize, Serialize};
 
 use crate::adapters::{CliManifestContext, adapter_for};
 use crate::core::fs::artifact_path;
-use crate::core::{AvailableSkill, Eval, Harness, POSIX_TOOLING_REQUIREMENT, ScriptedTurn};
+use crate::core::{
+    AvailableSkill, CodebaseRecord, Eval, Harness, POSIX_TOOLING_REQUIREMENT, ScriptedTurn,
+};
 
 use super::RunError;
 
@@ -53,6 +55,10 @@ pub struct DispatchTask {
     /// recipe's `<eval-root>` placeholder resolves to.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub eval_root: Option<String>,
+    /// The codebase this task's environment was built from. Carried here so the
+    /// run record written at ingest names the tree the agent actually worked in.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub codebase: Option<CodebaseRecord>,
     #[serde(default, skip_serializing)]
     pub dispatch_prompt: String,
 }
@@ -90,6 +96,8 @@ pub struct DispatchTaskOpts<'a> {
     /// The task's env dir (the agent-under-test's cwd); `None` only for legacy
     /// callers that do not carry an environment manifest.
     pub eval_root: Option<&'a str>,
+    /// The codebase this task's environment was built from, if any.
+    pub codebase: Option<&'a CodebaseRecord>,
 }
 
 fn render_available_skills_block_for_harness(
@@ -274,6 +282,7 @@ pub fn build_dispatch_task(opts: &DispatchTaskOpts) -> Result<DispatchTask, RunE
         outputs_dir,
         group: opts.group.map(str::to_string),
         eval_root,
+        codebase: opts.codebase.cloned(),
         dispatch_prompt: sections.join(""),
     })
 }
@@ -522,6 +531,7 @@ mod tests {
                 runs: None,
                 isolation: None,
                 turns: None,
+                codebase: None,
             })
             .collect()
     }

src/cli/run/fixtures.rs

@@ -199,6 +199,7 @@ mod tests {
             runs: None,
             isolation: None,
             turns: None,
+            codebase: None,
         }
     }