feat(evals): SkillGym-inspired DX layer (authoring façade, reporter lifecycle, workspace presets, token snapshots)

[ThomasK33]

Summary

Adds a thin, SkillGym-inspired developer-experience layer to the evals/ package without replacing the current eval engine. Case authoring becomes intention-revealing, observability becomes first-class, workspace setup becomes reusable, and token regressions become a dedicated sidecar signal — all while preserving every existing schema, runner, scorer, statistic, and report.

This is the orchestrator-delivered output of a six-phase plan; 36 commits spanning Phases 0+1 through Phase 6. Each phase landed via a dedicated sub-agent task whose patch was dry-run-applied and then integrated here.

What changed, per feature area

1. Authoring façade (evals/authoring/)

Fluent builders that compile to the existing PromptEvalCaseSchema | ExecutionEvalCaseSchema | DogfoodEvalCaseSchema runtime contract.

• promptCase(id) / executionCase(id) / dogfoodCase(id) with a three-tier abstraction: high-level sugar, mid-level typed builders, and raw escape hatches (rawWorkflowCheck, rawVerifier, rawArtifactRequirement, rawReportRequirement).
• Runtime contract is unchanged: build() ends in a strict Zod .parse(), so the lane runners/scorers see identical objects to before.
• Four pilot migrations prove the façade: wait-for-output (prompt), hello-prompt + doctor-gated (execution), exploratory-qa (dogfood). Exported identifiers preserved so runner imports are untouched.
• Golden parity tests compare migrated cases against the pre-migration shape byte-for-byte.

2. Typed reporter lifecycle (evals/reporters/)

Zero-impact by default; additive when opted in.

• Typed Reporter interface with callbacks for run.start / lane.start / case.start / trial.start / trial.finish / case.finish / lane.finish / run.finish.
• ReporterDispatcher validates every event payload through strict Zod schemas, redacts secret-looking env values (*_TOKEN, *_KEY, *_SECRET, *_PASSWORD), and isolates reporter failures.
• Built-in reporters: ConsoleReporter (hierarchical progress on stderr), JsonlReporter (one JSON event per line), FinalReportReporter (adapter wrapping existing generateJsonReport() / generateMarkdownReport()).
• New CLI flags (all additive): --reporter <name> (repeatable), --reporter-output <path>, --progress. Default behavior when no flags are supplied is byte-identical to pre-change.

3. Workspace presets (evals/workspaces/)

Declarative, additive workspace setup for execution + dogfood lanes.

• WorkspacePreset schema: { id, mode, description, templateDir?, bootstrap?, cwd?, env? } with a registry + resolver.
• Resolver redacts secret-looking env in reported payloads, fails fast on missing templateDir, and documents a deterministic merge order (preset env/bootstrap first, then case-level overrides/setup).
• Authoring builders gain .workspace(presetId). CaseStartEvent gains an optional, redacted workspace payload.
• Built-in agent-tty-smoke preset registered via evals/workspaces/builtins.ts; applied to hello-prompt and exploratory-qa as pilots.

4. Token usage + opt-in snapshot guardrails (evals/snapshots/)

Cost regression signal, deliberately orthogonal to quality scoring.

• Optional TokenUsage added to the normalized provider output contract. Populated by claude.ts and codex.ts when the upstream CLI emits usage; fixtures.ts can emit deterministic usage for tests.
• Per-trial token-usage.json sidecar artifacts via evals/lib/artifacts.ts.
• Snapshots are keyed by (provider, model, lane, caseId, condition, caseFingerprint) where caseFingerprint is a stable SHA-256 over the case's semantic fields.
• tokenReport section added to final JSON + Markdown reports when any trial emits usage; the section is cleanly absent otherwise.
• New CLI flags: --snapshot-update, --snapshot-check, --snapshot-threshold <percent> (default 20), --snapshot-dir <path>. Conflicting flags rejected at parse time.
• Invariant verified: snapshot regressions warn in the report but never flip EvalResult.ok or influence scoring / statistics / comparison. No enforce flag ships in this phase by design.

5. Docs

• evals/README.md gains dedicated sections for authoring façade, reporter lifecycle, workspace presets, and token snapshots.
• .mux/skills/eval-guide/SKILL.md updated so agent workflows use the new surfaces.
• dogfood/CATALOG.md references the Phase 5 proof bundle at dogfood/token-usage-phase5-proof/.

Backward compatibility contract

Hard invariants preserved across the entire change:

1. All existing case files still validate + run unchanged.
2. getAllPromptCases() / getAllExecutionCases() / getAllDogfoodCases() registration mechanism unchanged.
3. Default CLI invocation (no new flags) produces byte-identical report.json / report.md paths and shapes.
4. Scoring, statistics, anti-pattern scanning, baseline comparison, and provider comparison are untouched.
5. Token snapshots are opt-in; EvalResult.ok never flips on a regression.
6. Lane runner function signatures stay backward-compatible (new params are optional).

Validation

npm run verify (typecheck + lint + format:check + 976 tests + build + smoke install) passes locally in ~1m55s.

Representative dogfood runs driven by stub + fixture providers exercise:

• Dry-run matrix resolution across 29 cases × 3 lanes × 4 conditions.
• --progress console reporter emitting hierarchical lifecycle lines on stderr.
• --reporter jsonl producing a 20-event stream with consistent runId and full payload schemas.
• Stacked --reporter final --reporter jsonl producing both report files and the event stream.
• Workspace preset (agent-tty-smoke) appearing in case.start events for migrated pilots and absent for unmigrated cases.
• Snapshot workflow: --snapshot-update → baseline, --snapshot-check with same tokens → unchanged, --snapshot-check with +30% tokens at threshold=20 → regressed with ok still true.

Known follow-ups (non-blocking, documented in dogfood/CATALOG.md)

1. Only Phase 5 has a committed dogfood/ proof bundle; Phases 1–4 rely on automated tests and the in-PR walkthrough above. Reviewer-captured bundles would be a small additive follow-up.
2. Per-trial token sidecars key by (lane, caseId, condition) — with --trials > 1, files overwrite and the payload records the last trial. trialIndex is already in the payload; upgrading the path is additive.
3. Snapshot caseFingerprint is in the payload, not the sidecar path. Additive if needed.
4. .tokenBudget(...) authoring sugar was intentionally deferred until snapshot semantics settle in production use.
5. --reporter jsonl alone replaces the implicit final reporter rather than stacking; users who want both must pass --reporter final --reporter jsonl. Documented, but a reviewer-facing gotcha.
6. Only 2 cases actually use .workspace() — intentional Phase 4 pilot scope; broader migration is opportunistic per docs.

Commit map (36 commits)

Phase 0+1  authoring façade + 4 pilot migrations       3 commits
Phase 2    parity + CLI integration tests              2 commits
Phase 3    typed reporter lifecycle                    8 commits
(cleanup)  Prettier drift                              1 commit
Phase 4    workspace presets + pilot usage             5 commits
Phase 5    token usage + snapshot guardrails          14 commits
Phase 6    docs (README, eval-guide, dogfood CATALOG)  3 commits

---

📋 Implementation Plan

SkillGym-inspired eval DX implementation plan

1. Desired outcome

Add a thin, ergonomic developer interface for eval authoring and execution without replacing the current eval engine.

At the end of this work, the repo should have:

1. A new authoring façade for prompt / execution / dogfood cases that compiles to the existing runtime case shapes.
2. A typed reporter lifecycle for live progress, JSONL streaming, and future dashboards.
3. Optional workspace presets for execution and dogfood evals so case setup is less repetitive and easier to reason about.
4. Optional token/cost snapshot guardrails that act as a sidecar regression signal and do not replace current scoring/statistics.
5. A staged migration path that keeps existing cases, runners, scoring, reports, and CLI workflows working throughout.

---

2. Repo facts that shape the plan

The plan is intentionally grounded in the current code layout and verified repo structure:

• evals/run.ts is the main orchestration entrypoint. It parses CLI args, resolves lanes/conditions/cases, runs the lane runners, and writes final JSON/Markdown reports.
• evals/lib/types.ts and evals/lib/schemas.ts are the canonical runtime contracts for cases, results, reports, and provider output.
• evals/prompt/runner.ts, evals/execution/runner.ts, and evals/dogfood/runner.ts each aggregate hard-coded case exports via getAllPromptCases(), getAllExecutionCases(), and getAllDogfoodCases().
• Prompt authoring is currently the most ad hoc (evals/prompt/cases/trigger-agent-tty.ts), execution authoring has the richest helper layer (evals/execution/cases/shared.ts), and dogfood authoring is powerful but regex-heavy (evals/dogfood/cases/exploratory-qa.ts).
• evals/lib/scheduler.ts already centralizes concurrent scheduling and already has a logLine hook, which is a good seam for later reporter events.
• evals/lib/cliHarness.ts already creates isolated eval homes, so workspace presets should extend existing behavior rather than invent a separate setup mechanism.
• The provider layer (evals/providers/base.ts and provider implementations) already normalizes outputs, but token usage is not currently part of the normalized result contract.

These facts strongly favor an additive façade over a rewrite.

---

3. Goals and non-goals

Goals

• Reduce eval authoring boilerplate for common cases.
• Make the intent of a case easier to read than its regex/plumbing details.
• Preserve the current deterministic scoring, anti-pattern scanning, statistics, and comparison engine.
• Improve run observability without forcing a reporting rewrite on day one.
• Make eval setup more DRY and explicit for execution/dogfood lanes.
• Add a lightweight, opt-in cost regression signal that complements rather than replaces core eval quality metrics.
• Keep changes minimal, defensive, and backwards-compatible.

Non-goals

• Do not replace PromptEvalCase | ExecutionEvalCase | DogfoodEvalCase as the runner/scoring contract in the first implementation.
• Do not rewrite all existing eval cases up front.
• Do not introduce a first-class “suite runtime” in v1.
• Do not fold token usage into pass/fail or score by default.
• Do not rewrite generateJsonReport() / generateMarkdownReport() as the first reporter milestone.

---

4. Architectural decisions

Decision	Recommendation	Why
Canonical runtime contract	Keep existing case/result/report schemas as the backend contract	Lowest migration risk; preserves scoring/statistics/reporting semantics
New authoring API	Add evals/authoring/ façade that compiles to existing case objects	Gives better DX without destabilizing runners
Builder design	Use pragmatic fluent builders plus raw escape hatches	Improves readability without type-system overengineering
Case registration	Keep current manual imports/arrays/getters initially	Avoids discovery/runtime refactor while validating the new API
Reporter system	Add typed reporter callbacks, not a stringly typed event emitter	Easier to test, version, and extend safely
Workspace modeling	Make presets optional and execution/dogfood-only at first	Matches current isolated-home behavior and avoids touching prompt lane unnecessarily
Token guardrails	Sidecar snapshots + separate report section, warning-only by default	Keeps “quality” distinct from “cost” and avoids noisy regressions affecting pass/fail
Defensive programming	Validate compiled cases with current schemas; assert impossible states early	Matches repo norms (invariant, strict Zod, fail-fast)

Design rule for the authoring layer

The new authoring surface should offer three levels of abstraction:

1. High-level sugar for the common workflows that keep recurring.
   • Examples: createSession(), input(), waitFor(), snapshot(), destroy(), requiresScreenshot(), findingsWithSeverity().
2. Mid-level typed builders for precise workflow checks, verifiers, artifacts, and report sections.
3. Raw escape hatches for edge cases that do not fit the sugar.
   • Examples: rawWorkflowCheck(...), rawVerifier(...), rawArtifactRequirement(...), rawReportRequirement(...).

That layering prevents DSL overreach from blocking unusual cases.

---

5. Target module layout

New public authoring surface

• evals/authoring/index.ts
• evals/authoring/prompt.ts
• evals/authoring/execution.ts
• evals/authoring/dogfood.ts
• evals/authoring/workflow.ts
• evals/authoring/report.ts
• evals/authoring/compile.ts
• evals/authoring/raw.ts

New reporter surface

• evals/reporters/types.ts
• evals/reporters/dispatch.ts
• evals/reporters/console.ts
• evals/reporters/jsonl.ts
• evals/reporters/final-report.ts

New workspace support

• evals/workspaces/types.ts
• evals/workspaces/registry.ts
• evals/workspaces/resolver.ts

New token snapshot support

• evals/snapshots/schema.ts
• evals/snapshots/store.ts
• evals/snapshots/compare.ts

Existing files likely to change

• evals/lib/types.ts
• evals/lib/schemas.ts
• evals/run.ts
• evals/lib/scheduler.ts
• evals/lib/artifacts.ts
• evals/lib/cliHarness.ts
• evals/providers/base.ts
• evals/providers/claude.ts
• evals/providers/codex.ts
• evals/prompt/runner.ts
• evals/execution/runner.ts
• evals/dogfood/runner.ts
• selected case files for pilot migrations
• evals/README.md
• .mux/skills/eval-guide/SKILL.md
• dogfood/README.md
• dogfood/CATALOG.md

---

6. Phase 0 — guardrails, pilot scope, and parity baseline

Goal

Create the safety rails that keep the implementation from turning into a rewrite.

Work items

1. Document explicit invariants at the top of the implementation work.

   • Existing cases must continue loading and running unchanged.
   • Existing CLI usage must continue working.
   • New authoring builders must emit case objects that validate through the current schemas.
   • Reporter hooks must be additive.
   • Token snapshots must be opt-in and separate from core scoring.

2. Choose a small pilot set of cases to prove the façade.

   • Prompt: one case from evals/prompt/cases/trigger-agent-tty.ts such as wait-for-output.
   • Execution: evals/execution/cases/hello-prompt.ts and evals/execution/cases/doctor-gated.ts.
   • Dogfood: evals/dogfood/cases/exploratory-qa.ts.

3. Capture baseline outputs for pilot cases before migrating them.

   • Dry-run matrix output.
   • JSON report shape.
   • Result artifacts for the fixture/stub providers where possible.
   • This baseline is the comparison target for parity tests in Phase 2.

4. Decide the initial public API shape and freeze it before broad migration.

   • Use pragmatic fluent builders.
   • Avoid advanced type-state/generic machinery unless a simple builder proves insufficient.

Acceptance criteria

• Pilot cases are selected and documented.
• Runtime invariants are written down and agreed to before code changes branch out.
• There is a clear statement that manual case registration remains in place for v1.

---

7. Phase 1 — authoring façade MVP

Goal

Introduce a better developer interface for authoring cases, but compile it down to the current case schemas so the rest of the pipeline can stay intact.

Scope

This phase is only about case authoring ergonomics. It should not yet require reporter, workspace, or token snapshot rewrites.

Implementation approach

7.1 Create lane-specific builder entrypoints

Implement public builders under evals/authoring/:

• promptCase(id)
• executionCase(id)
• dogfoodCase(id)

Each builder should collect author intent and then call build() to produce a schema-validated case object.

7.2 Compile to the current runtime shapes

build() should ultimately call the same schema validation the repo already trusts:

• PromptEvalCaseSchema.parse(...)
• ExecutionEvalCaseSchema.parse(...)
• DogfoodEvalCaseSchema.parse(...)

That preserves the current strict runtime contract.

7.3 Reuse existing helper logic instead of duplicating it

The new builder layer should reuse or wrap existing helper logic where available, especially from:

• evals/execution/cases/shared.ts
• evals/dogfood/cases/shared.ts
• existing pattern constants and anti-pattern defaults

Do not fork pattern logic into two divergent implementations if the repo already has a helper that encodes the intended behavior.

7.4 Start with the recurring “happy path” primitives

Prompt lane primitives

• .category(...)
• .prompt(...)
• .expectSkill(...)
• .context(...)
• .workflow((w) => ...)
• .mustMention(...)
• .mustNotMention(...)
• .budget(...)
• .rawWorkflowCheck(...)

Execution lane primitives

• .category(...)
• .fixture(...)
• .task(...)
• .workflow((w) => ...)
• w.createSession()
• w.run(...) and/or w.input(...)
• w.waitFor(...)
• w.snapshot()
• w.screenshot()
• w.destroy()
• .assertions((a) => ...)
• .artifact(...)
• .budget(...)
• .rawVerifier(...)
• .rawWorkflowCheck(...)

Dogfood lane primitives

• .category(...)
• .fixture(...)
• .task(...)
• .proofBundle((b) => ...)
• b.requiresScreenshot()
• b.requiresRecording()
• b.requiresNotes()
• .report((r) => ...)
• r.title()
• r.reproductionSteps()
• r.findingsWithSeverity()
• r.evidenceReferences()
• .validationProfile(...)
• .budget(...)
• .rawArtifactRequirement(...)
• .rawReportRequirement(...)
• .rawVerifier(...)

7.5 Preserve manual registration in lane runners

Do not add automatic case discovery in v1.

Continue to export regular case constants from case files and continue importing them into:

• getAllPromptCases()
• getAllExecutionCases()
• getAllDogfoodCases()

This keeps the runner/dry-run/reporting behavior stable while validating the new authoring layer.

7.6 Pilot migration strategy

Use the new authoring façade for the pilot cases only:

• re-author wait-for-output
• re-author hello-prompt
• re-author doctor-gated
• re-author exploratory-qa

Preserve the exported identifiers used by the runner imports so the runner diff stays small.

7.7 Target API sketches for the pilot cases

These are target-shape sketches, not final syntax commitments. They exist to keep the implementation aligned with the DX goal.

// prompt pilot
export const waitForOutputCase = promptCase('wait-for-output')
  .category('trigger')
  .prompt("I need to wait until my server prints 'Listening on port 3000' before running tests")
  .expectSkill('agent-tty')
  .workflow((w) =>
    w.step('select-agent-tty').mustMention('agent-tty')
     .step('observe-readiness')
     .mustMention('wait')
     .mustMention('Listening on port 3000')
     .mustNotMention('sleep')
     .mustNotMention('setTimeout'),
  )
  .build();

// execution pilot
export const helloPromptCase = executionCase('hello-prompt')
  .category('session')
  .fixture('hello-prompt')
  .task("Send 'hello world', wait for READY>, capture a snapshot, then destroy the session")
  .workflow((w) =>
    w.createSession()
     .input('hello world')
     .waitFor('READY>')
     .snapshot()
     .destroy(),
  )
  .assertions((a) =>
    a.snapshotContains(/ECHO:\s*hello world/)
     .snapshotContains(/READY>/),
  )
  .build();

// dogfood pilot
export const exploratoryQaCase = dogfoodCase('exploratory-qa')
  .category('qa')
  .fixture('hello-prompt')
  .task('Test three inputs, capture evidence, verify shutdown, and write findings')
  .proofBundle((b) =>
    b.requiresScreenshot()
     .requiresRecording()
     .requiresNotes(),
  )
  .report((r) =>
    r.title()
     .reproductionSteps()
     .findingsWithSeverity()
     .evidenceReferences(),
  )
  .validationProfile('interactive-renderer')
  .build();

The implementation does not need to match the exact method names above if a simpler variant is clearly better, but the resulting authoring experience should be comparably concise and intention-revealing.

Acceptance criteria

• Pilot cases compile to schema-valid runtime case objects.
• The exported runner-facing case constants keep the same IDs and runtime behavior.
• The new authoring layer clearly reduces boilerplate for the pilot cases.
• At least one prompt, execution, and dogfood case are authored through the new façade.
• There are raw escape hatches so edge cases are not blocked.

Notes on minimalism

Avoid turning this phase into a giant type-level DSL. The initial builder implementation should prioritize:

• explicit data structures,
• readable builder code,
• schema validation at the end,
• and straightforward debugging when a builder emits invalid state.

---

8. Phase 2 — golden parity tests and migration safety

Goal

Prove that the authoring façade changes how cases are written, not what the engine sees.

Work items

1. Add unit tests for builder compilation.

   • New tests under test/unit/evals/authoring/.
   • Validate that build() emits schema-valid case objects.
   • Validate that invalid builder states fail fast with good errors.

2. Add golden parity tests for pilot cases.

   • Compare DSL-authored pilot cases against the equivalent legacy runtime case objects or equivalent normalized subsets.
   • Normalize volatile fields if needed, but keep the parity surface meaningful.

3. Add lane-level integration tests.

   • Use fixture/stub providers to run pilot cases through the real runners.
   • Verify that the produced results and reports stay equivalent for the migrated cases.

4. Defensively assert builder invariants.

   • Missing required fields should fail before or during build().
   • Impossible workflow sequences should assert early.
   • Report section and artifact builders should reject malformed states rather than silently guess.

Recommended test files

• test/unit/evals/authoring/prompt.test.ts
• test/unit/evals/authoring/execution.test.ts
• test/unit/evals/authoring/dogfood.test.ts
• test/integration/evals/authoring-parity.test.ts

Acceptance criteria

• The pilot cases pass schema validation and runner execution through the new façade.
• Parity tests prove that migrated cases still behave like their legacy equivalents.
• Builder failure modes are explicit and tested.

---

9. Phase 3 — typed reporter lifecycle and live progress

Goal

Expose the eval lifecycle to built-in and custom reporters without rewriting the core scoring/report generation path.

Implementation approach

9.1 Add a typed reporter contract

Create a small typed reporter surface in evals/reporters/types.ts.

Recommended callback surface:

• onRunStart
• onLaneStart
• onCaseStart
• onTrialStart
• onTrialFinish
• onCaseFinish
• onLaneFinish
• onRunFinish

Keep the event surface coarse-grained. Do not emit every internal implementation detail.

9.2 Add a reporter dispatcher

Implement evals/reporters/dispatch.ts as a small fan-out layer that accepts zero or more reporters and safely invokes them.

Requirements:

• preserve reporter ordering,
• isolate one reporter failure from crashing unrelated reporters unless explicitly configured,
• redact or avoid leaking secret env values in emitted workspace metadata,
• use assertions for malformed event payloads.

9.3 Integrate reporters through evals/run.ts and evals/lib/scheduler.ts

• Thread reporter context from evals/run.ts down into lane execution.
• Extend evals/lib/scheduler.ts with start/finish hooks around scheduled work items.
• Emit lane/case/trial events from the runners.

9.4 Keep current final reports as adapters first

Do not rewrite generateJsonReport() / generateMarkdownReport() at the start of this phase.

Instead:

• keep the existing end-of-run report generation,
• wrap it in a final-report reporter adapter,
• and add new reporters alongside it.

This keeps the diff small and the risk low.

9.5 Ship two built-ins first

1. Console progress reporter

   • concise live terminal progress,
   • case/trial start and finish,
   • lane summaries,
   • optional verbose detail.

2. JSONL reporter

   • one event per line,
   • machine-readable,
   • suitable for CI, dashboards, and later replay.

Suggested CLI additions

Additive flags only, for example:

• --reporter console
• --reporter jsonl
• --reporter-output <path>
• --progress (sugar for console reporter)

9.6 Reporter API sketch

A minimal typed reporter contract should look roughly like this:

export interface EvalReporter {
  onRunStart?(event: RunStartEvent): Promise<void> | void;
  onLaneStart?(event: LaneStartEvent): Promise<void> | void;
  onCaseStart?(event: CaseStartEvent): Promise<void> | void;
  onTrialStart?(event: TrialStartEvent): Promise<void> | void;
  onTrialFinish?(event: TrialFinishEvent): Promise<void> | void;
  onCaseFinish?(event: CaseFinishEvent): Promise<void> | void;
  onLaneFinish?(event: LaneFinishEvent): Promise<void> | void;
  onRunFinish?(event: RunFinishEvent): Promise<void> | void;
}

The first implementation should keep event payloads focused on identities, timing, outcome, and artifact/report paths. Avoid dumping every internal structure into every callback.

Acceptance criteria

• Existing CLI output/report generation still works with no reporter flags.
• Built-in console reporter shows live progress for real runs.
• JSONL reporter emits a stable event stream for the same run.
• Reporter hooks are usable from tests without invoking the full CLI.

---

10. Phase 4 — workspace presets for execution and dogfood

Goal

Make workspace setup explicit, reusable, and easier to dogfood while preserving the current isolated-home model.

Implementation approach

10.1 Introduce a workspace preset model

Create a small preset type in evals/workspaces/types.ts, roughly covering:

• mode: isolated | shared
• template directory (optional)
• bootstrap commands (optional)
• cwd/env overrides (optional)
• human-readable description

Keep it small. Avoid a full-blown environment orchestration framework.

10.2 Resolve presets through existing CLI harness behavior

Use evals/lib/cliHarness.ts as the integration seam.

The preset resolver should extend the existing isolated-home creation path rather than bypass it.

10.3 Integrate presets into execution/dogfood runners first

Prompt lane does not need workspace presets in the first pass.

Execution and dogfood runners should be able to:

• request a preset explicitly from the case authoring layer,
• or derive a default preset from the selected skill condition when appropriate.

10.4 Keep legacy setup steps valid

Workspace presets should augment the current setup approach first, not replace it.

That means:

• old case files with explicit setup steps still run,
• new cases can use presets to avoid repetitive setup wiring,
• if both are present, the merge order must be deterministic and documented.

10.5 Log resolved workspace config for observability

At minimum, expose the resolved workspace plan in:

• dry-run output,
• reporter events,
• and optionally run metadata.

Secrets must be redacted.

10.6 Workspace preset sketch

A deliberately small preset model is enough for the first pass:

export interface WorkspacePreset {
  id: string;
  mode: 'isolated' | 'shared';
  description: string;
  templateDir?: string;
  bootstrap?: ReadonlyArray<{
    command: string;
    argv?: readonly string[];
  }>;
  cwd?: string;
  env?: Readonly<Record<string, string>>;
}

If a case uses a preset plus explicit setup steps, the implementation should document and test the merge order rather than leaving it implicit.

Acceptance criteria

• Legacy execution/dogfood cases still run with their existing setup definitions.
• A pilot execution case and a pilot dogfood case can use a workspace preset successfully.
• Resolved workspace behavior is visible in dry-run/progress output.

---

11. Phase 5 — token usage capture and snapshot guardrails

Goal

Add a lightweight cost regression signal without entangling it with the core quality score.

Implementation approach

11.1 Normalize token usage where providers expose it

Add a small optional token usage type to the provider/result path.

Recommended fields:

• input tokens
• output tokens
• total tokens
• cached tokens if available
• model identifier if helpful

If a provider cannot expose token data, the field should remain absent rather than guessed.

11.2 Store token data as a sidecar artifact first

Do not immediately force token data into the main pass/fail score path.

Instead:

• write token usage sidecar artifacts through evals/lib/artifacts.ts,
• aggregate them in reporting as a separate section,
• and keep the main EvalResult.ok semantics unchanged.

11.3 Add a snapshot store keyed by stable identifiers

Token snapshots should be keyed by at least:

• provider,
• model,
• lane,
• case ID,
• condition,
• and a case fingerprint derived from the compiled case definition.

The fingerprint matters so that prompt or workflow changes do not create bogus regressions against an obsolete baseline.

11.4 Add opt-in CLI flows before per-case budgets

Start with CLI controls such as:

• --snapshot-update
• --snapshot-check
• --snapshot-threshold <percent>
• --snapshot-dir <path>

Only after the storage and reporting semantics are stable should the authoring layer grow convenience helpers such as .tokenBudget(...).

11.5 Keep snapshot outcomes separate from quality scoring

By default:

• snapshot regressions should warn/report,
• they should not flip ok,
• and they should not alter the existing statistics/comparison logic.

If the repo later wants enforcement, add an explicit opt-in mode such as --snapshot-enforce.

11.6 Snapshot CLI sketch

A minimal first-pass CLI could look like this:

npx tsx evals/run.ts --provider codex --model gpt-5.4 --lane execution --snapshot-update
npx tsx evals/run.ts --provider codex --model gpt-5.4 --lane execution --snapshot-check --snapshot-threshold 15

If the repo later wants case-level ergonomics, add that only after the storage/report semantics are stable, for example:

executionCase('hello-prompt')
  .tokenBudget({ maxRegressionPercent: 15 })

That sequence keeps the storage contract ahead of the authoring sugar.

Acceptance criteria

• Providers that expose token usage produce sidecar token artifacts.
• Snapshot check/update flows work for supported providers.
• Snapshot comparisons are visible in reports/reporter output.
• Core pass/fail/score semantics remain unchanged unless an explicit enforce flag is used later.

---

12. Phase 6 — docs, migration, and cleanup

Goal

Turn the pilot architecture into a maintainable, teachable workflow for future contributors.

Work items

1. Document the new authoring path.

   • Update evals/README.md with before/after examples and migration guidance.
   • Update .mux/skills/eval-guide/SKILL.md so agent workflows can use the new interface correctly.

2. Document the reporter model.

   • Show how to enable built-in reporters.
   • Show how to plug in a custom reporter.
   • Show where event payloads are stable and where they are intentionally internal.

3. Document workspace preset usage.

   • Show preset definition,
   • preset resolution,
   • and how it relates to SkillCondition.

4. Document token snapshot semantics.

   • Explain that snapshots are a guardrail, not the benchmark.
   • Explain thresholds, fingerprints, and when snapshots must be refreshed.

5. Migrate additional cases opportunistically, not all at once.

   • Keep the legacy helper path supported during the transition.
   • Encourage new cases to use the façade.
   • Only deprecate old helpers after the new authoring path is proven over multiple real additions.

Acceptance criteria

• The repo has a clear “write new evals this way” path.
• The pilot features are documented and dogfooded.
• There is a low-risk migration path for the remaining cases.

---

13. Detailed file-by-file implementation checklist

Core contracts

evals/lib/types.ts

• Add any additive interfaces needed for:
  • authoring compile helpers (if shared types are useful),
  • reporter events,
  • workspace preset types or references,
  • optional token usage metadata.
• Keep additions optional where possible.
• Use narrow literal unions and ReadonlyArray-style discipline where appropriate.

evals/lib/schemas.ts

• Add matching strict Zod schemas for any new runtime/reporter/token structures.
• Keep .strict() semantics.
• Ensure legacy payloads remain valid when new fields are omitted.

Authoring layer

evals/authoring/index.ts

• Re-export the public builders.
• Keep the public import surface small and obvious.

evals/authoring/prompt.ts

• Build prompt DSL sugar and compilation.
• Reuse prompt lane pattern/reporting expectations where possible.

evals/authoring/execution.ts

• Build execution DSL sugar around the repetitive session workflow primitives.
• Reuse existing execution helper semantics rather than reinventing them.

evals/authoring/dogfood.ts

• Build dogfood DSL sugar for artifacts, report sections, and bundle expectations.

evals/authoring/raw.ts

• Export escape hatches for oddball cases.

Runner/reporter integration

evals/run.ts

• Parse new reporter/snapshot flags.
• Construct reporter dispatcher.
• Emit top-level run/lane lifecycle events.
• Preserve current CLI behavior when the new flags are not used.

evals/lib/scheduler.ts

• Add start/finish callbacks around scheduled items.
• Keep concurrency semantics unchanged.
• Maintain deterministic ordering guarantees where the current tests depend on them.

evals/prompt/runner.ts, evals/execution/runner.ts, evals/dogfood/runner.ts

• Thread reporter context through.
• Emit case/trial lifecycle events.
• Integrate workspace presets in execution/dogfood only.
• Avoid mixing reporter logic with scoring logic more than necessary.

Workspace support

evals/lib/cliHarness.ts

• Extend existing isolated-home creation with preset resolution hooks.
• Keep cleanup behavior intact.

evals/workspaces/*

• Define preset types, registry, and resolution rules.
• Ensure env handling is explicit and redacted in logs/reporter payloads.

Token snapshots

evals/providers/base.ts

• Add optional usage plumbing to normalized provider results if needed.

evals/providers/claude.ts, evals/providers/codex.ts

• Parse usage metadata where available.
• Fail gracefully when usage data is unavailable.

evals/lib/artifacts.ts

• Add validated sidecar write/load support for token usage and snapshots.

evals/snapshots/*

• Define stable snapshot schema,
• storage,
• comparison logic,
• and threshold handling.

Docs and proof

evals/README.md

• New authoring API,
• reporter usage,
• workspace preset usage,
• snapshot semantics.

.mux/skills/eval-guide/SKILL.md

• Update eval workflows and examples.

dogfood/README.md, dogfood/CATALOG.md

• Add reviewer-facing proof bundle references for the new DX features.

---

14. Backward compatibility contract

The implementation should preserve these guarantees throughout the rollout:

1. Existing case files remain valid.
2. Existing runner getter functions remain the registration boundary.
3. Existing CLI invocations keep working.
4. Existing report generation keeps working when reporters are not enabled.
5. Existing scoring/statistics/baseline comparison remain canonical for quality evaluation.
6. Token snapshots remain opt-in and separate from quality scoring.
7. Legacy helper functions stay supported until the new façade is proven in practice.

---

15. Risks and mitigations

Risk	Why it matters	Mitigation
DSL overreach	A fancy builder can become harder to maintain than the current object literals	Keep builders simple, layered, and backed by raw escape hatches
Behavior drift	A “nicer” case authoring API is useless if it changes scoring behavior	Add parity tests and pilot migrations before broader rollout
Reporter complexity	It is easy to accidentally entangle reporters with core execution logic	Use a small typed dispatcher and keep current final report generation as an adapter first
Hidden workspace magic	Presets can make runs harder to debug if they silently mutate setup	Log resolved presets in dry-run/reporter output; redact secrets
Token noise	Token counts can fluctuate and create false alarms	Make snapshots opt-in, threshold-based, and keyed by case fingerprint/provider/model
Schema churn	Strict schemas can make additive changes brittle	Keep new fields optional where possible and validate all new structures explicitly
Migration fatigue	Rewriting too many cases too early increases risk	Pilot only a few cases first; migrate more opportunistically

---

16. Validation strategy

Automated validation gates

Run these before claiming the work complete:

• mise run format
• mise run lint
• mise run typecheck
• mise run test
• mise run ci

If mise is unavailable, fall back to the repo’s documented direct commands.

Targeted test focus

• builder compilation tests,
• parity tests for migrated pilot cases,
• reporter event sequencing tests,
• workspace preset resolution tests,
• token snapshot schema and comparison tests,
• integration tests that run real eval lanes with fixture/stub providers.

Defensive-programming checks

Implementation should deliberately assert:

• required builder fields are set before build(),
• workflow step dependencies are valid,
• duplicate IDs are rejected early,
• workspace preset resolution is valid,
• token snapshot keys contain all required identity components,
• and any impossible runtime states fail loudly.

---

17. Dogfooding and proof plan

This project is CLI-first, so the dogfooding plan needs to show both authoring DX and runtime UX. The proof bundle should be reviewer-friendly and include screenshots plus video/recording evidence.

17.1 Setup environment

1. Prepare the repo with the standard workflow:
   • mise install
   • mise run bootstrap
2. If mise is unavailable:
   • npm ci
   • npx playwright install chromium
3. Use an isolated absolute AGENT_TTY_HOME for dogfood runs.
4. Use fixture/stub providers first for deterministic proof, then run at least one supported real provider path if token usage/reporters need live validation.

17.2 Dogfood milestones

Milestone A — authoring façade proof

• Re-author the pilot cases with the new façade.
• Run dry-run and normal eval executions for the pilot cases.
• Capture:
  • terminal screenshot(s) showing the new case definitions and successful dry-run selection,
  • recording/video of a run that uses migrated cases,
  • JSON report output showing parity with the old behavior.

Milestone B — reporter proof

• Run evals with the built-in console reporter enabled.
• Run evals with the JSONL reporter enabled.
• Capture:
  • screenshot of live progress output,
  • recording/video showing case/trial progress in real time,
  • saved JSONL sample included in the proof bundle.

Milestone C — workspace preset proof

• Run at least one execution case and one dogfood case that use a workspace preset.
• Capture:
  • screenshot or snapshot of dry-run/resolved workspace info,
  • recording/video of the preset-backed run,
  • evidence that the resolved preset does not leak secrets.

Milestone D — token snapshot proof

• Run one provider path with token usage capture enabled.
• Update a snapshot, then run a snapshot check.
• Capture:
  • screenshot of the snapshot check/update output,
  • recording/video of the token snapshot workflow,
  • example sidecar token artifact and comparison result in the proof bundle.

17.3 Suggested proof-bundle contents

Create a dated proof bundle under dogfood/ containing at minimum:

• README.md or index.md summary,
• screenshots (.png),
• terminal recordings (.cast) and exported video (.webm) where relevant,
• sample JSON/Markdown report outputs,
• sample JSONL reporter output,
• token sidecar/snapshot artifacts,
• concise reproduction steps.

Update dogfood/CATALOG.md so reviewers can find it quickly.

17.4 Suggested dogfood commands

Use direct CLI invocations from the repo while developing, for example:

• npx tsx evals/run.ts --provider stub --lane prompt --case wait-for-output --dry-run
• npx tsx evals/run.ts --provider fixture --lane execution --case hello-prompt --json
• npx tsx evals/run.ts --provider fixture --lane dogfood --case exploratory-qa --json
• reporter-enabled variants once implemented
• snapshot update/check variants once implemented

Capture screenshots and recordings with the project’s own terminal workflow where practical so the reviewer can inspect the exact terminal UX being claimed.

Dogfooding acceptance criteria

• Reviewer gets at least one screenshot and one recording/video for each major DX feature area that changes user-visible behavior.
• The proof bundle contains enough detail for a reviewer to replay the workflow.
• Dogfood artifacts are linked from the catalog/docs.

---

18. Recommended implementation order

1. Phase 0 guardrails and pilot scope
2. Phase 1 authoring façade MVP
3. Phase 2 parity tests
4. Phase 3 reporter lifecycle
5. Phase 4 workspace presets
6. Phase 5 token snapshots
7. Phase 6 docs + broader migration

This ordering minimizes risk because it validates the authoring façade before expanding the runtime surface area.

---

19. Exit criteria for the overall initiative

The work is ready to call successful when all of the following are true:

• New evals can be authored through a clearly documented façade rather than by hand-assembling schema-shaped objects.
• The pilot prompt/execution/dogfood cases use the new façade successfully.
• Parity tests prove the façade compiles to the existing engine contract.
• Reporter hooks provide live progress without regressing current report generation.
• Workspace presets reduce setup duplication for at least pilot execution/dogfood cases.
• Token snapshot guardrails work as an optional sidecar signal.
• Docs and dogfood artifacts make the workflow teachable to future contributors.

---

20. Short implementation summary

The safest and most valuable path is:

• keep the current eval engine,
• add a thin evals/authoring façade,
• add a typed reporter bus,
• formalize workspace presets as optional setup metadata,
• add token snapshots as a separate opt-in regression guard,
• and prove everything with pilot migrations, parity tests, and reviewer-friendly dogfood artifacts.

---

Generated with mux • Model: anthropic:claude-opus-4-7 • Thinking: max