feat(evals): SkillGym-inspired DX layer (authoring façade, reporter lifecycle, workspace presets, token snapshots)

.mux/skills/eval-guide/SKILL.md

@@ -179,3 +179,34 @@ npx tsx evals/run.ts --provider stub --lane prompt --trials 3 --concurrency 4
 ```
 
 Use `stub` to validate wiring, not to judge whether a real-provider prompt or skill change helped.
+
+## 7. Writing new eval cases
+
+Checklist:
+
+- Prefer the fluent builders in `evals/authoring/*` (`promptCase()`, `executionCase()`, `dogfoodCase()`) over hand-assembled schema objects.
+- Use the raw escape hatches when the sugar does not fit: `rawWorkflowCheck()`, `rawVerifier()`, `rawArtifactRequirement()`, and `rawReportRequirement()`.
+- After the new case compiles, rerun `npm run test`.
+
+## 8. Reporter lifecycle
+
+Need lifecycle events plus local progress and a machine-readable trace?
+
+```bash
+npx tsx evals/run.ts \
+  --provider stub \
+  --lane execution \
+  --reporter jsonl \
+  --reporter-output evals/reports/execution-events.jsonl \
+  --progress
+```
+
+When you omit `--reporter`, the default `final` reporter still writes `report.json` and `report.md`.
+
+## 9. Workspace presets
+
+Add `.workspace('agent-tty-smoke')` to an `executionCase()` or `dogfoodCase()` when the case needs preset bootstrap/env/template setup. Register custom presets with `registerPreset()` in a module that loads before `runEvalCli()`.
+
+## 10. Token snapshots
+
+Use `--snapshot-update` first, then `--snapshot-check --snapshot-threshold 20` against the same `--snapshot-dir` when you want regression signals over time. Snapshot regressions are warnings only.

dogfood/CATALOG.md

@@ -21,18 +21,21 @@ Paths below are relative to the repository root.
 
 ## Validation and release gates
 
-| Bundle                                         | Why it matters                                                                                                    |
-| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
-| `dogfood/20260326-week9-release-readiness/`    | Current release-signoff bundle for the `0.1.0` bar.                                                               |
-| `dogfood/20260410-release-tarball/`            | Local proof of the shared release tarball packer, checksum, and install flow used by the GitHub release workflow. |
-| `dogfood/20260325-week8-contract-locks/`       | Contract-lock and reporting review evidence.                                                                      |
-| `dogfood/20260325-week8-bundle-validation/`    | Validation of proof-bundle conventions.                                                                           |
-| `dogfood/20260325-week8-capability-inventory/` | Runtime capability inventory/reporting evidence.                                                                  |
-| `dogfood/20260325-week8-inspect-runtime/`      | `inspect --json` runtime reporting review.                                                                        |
-| `dogfood/20260323-week5-platform-closure/`     | Platform/documentation closeout evidence from the earlier hardening phase.                                        |
+| Bundle                                         | Why it matters                                                                                                             |
+| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
+| `dogfood/20260326-week9-release-readiness/`    | Current release-signoff bundle for the `0.1.0` bar.                                                                        |
+| `dogfood/20260410-release-tarball/`            | Local proof of the shared release tarball packer, checksum, and install flow used by the GitHub release workflow.          |
+| `dogfood/20260325-week8-contract-locks/`       | Contract-lock and reporting review evidence.                                                                               |
+| `dogfood/20260325-week8-bundle-validation/`    | Validation of proof-bundle conventions.                                                                                    |
+| `dogfood/20260325-week8-capability-inventory/` | Runtime capability inventory/reporting evidence.                                                                           |
+| `dogfood/20260325-week8-inspect-runtime/`      | `inspect --json` runtime reporting review.                                                                                 |
+| `dogfood/token-usage-phase5-proof/`            | Phase 5 eval DX token-usage proof bundle (commit `91a571de`) with screenshot, WebM recording, snapshot, and replay script. |
+| `dogfood/20260323-week5-platform-closure/`     | Platform/documentation closeout evidence from the earlier hardening phase.                                                 |
 
 | `dogfood/20260330-docs-navigation/` | Repository docs walkthrough with screenshots and a WebM recording of the new navigation path. |
 
+Follow-up: `dogfood/token-usage-phase5-proof/` is the only dedicated eval DX phase bundle currently cataloged. Dedicated proof bundles for the authoring façade, reporter lifecycle, and workspace-preset phases (Phases 1-4) are not currently present under `dogfood/`; reviewers who need fresh artifacts for those phases should capture a local proof bundle.
+
 ## Recovery and hardening
 
 | Bundle                                      | Focus                                                      |

dogfood/README.md

@@ -7,8 +7,9 @@ Some bundles are evergreen workflow scenarios, some are release/contract validat
 
 1. Read [`CATALOG.md`](./CATALOG.md) for the curated bundle map.
 2. For the current release-signoff view, start with `dogfood/20260326-week9-release-readiness/`.
-3. For evergreen workflows, start with bundles such as `dogfood/run-command/`, `dogfood/20260322-dogfood-hello-prompt/`, and `dogfood/20260322-lazyvim-scenario/`.
-4. For recovery and hardening behavior, use the recovery section in the catalog.
+3. For the Phase 5 eval DX token-usage proof from commit `91a571de`, start with `dogfood/token-usage-phase5-proof/`.
+4. For evergreen workflows, start with bundles such as `dogfood/run-command/`, `dogfood/20260322-dogfood-hello-prompt/`, and `dogfood/20260322-lazyvim-scenario/`.
+5. For recovery and hardening behavior, use the recovery section in the catalog.
 
 ## How to treat the directory
 

dogfood/token-usage-phase5-proof/README.md

@@ -0,0 +1,52 @@
+# Phase 5 token-usage proof bundle
+
+This bundle captures a focused `agent-tty` validation run for the Phase 5 token-usage work. It provides reviewer-visible proof for these commits:
+
+- `7beaa84` — `feat(evals): add optional token usage to normalized output`
+- `202244b` — `feat(evals): normalize token usage in claude and codex providers`
+- `e9828d5` — `feat(evals): support deterministic token usage in fixtures provider`
+
+The current repo-local CLI uses `create` / `destroy` rather than `session start` / `session destroy`; recording export is available for created sessions, so there is no separate `--record` flag in this workspace's `--help` output.
+
+## Artifacts
+
+- Snapshot: `dogfood/token-usage-phase5-proof/snapshot.txt`
+- Screenshot: `dogfood/token-usage-phase5-proof/screenshot.png`
+- Recording: `dogfood/token-usage-phase5-proof/recording.webm`
+- Replay script: `dogfood/token-usage-phase5-proof/commands.sh`
+
+## Exact command list used
+
+The executable replay script is `dogfood/token-usage-phase5-proof/commands.sh`. The exact command sequence it ran was:
+
+```bash
+ROOT_DIR="$(git rev-parse --show-toplevel)"
+BUNDLE_DIR="$ROOT_DIR/dogfood/token-usage-phase5-proof"
+export AGENT_TTY_HOME="$(mktemp -d "$BUNDLE_DIR/.home.XXXXXX")"
+
+npx tsx src/cli/main.ts --help
+npx tsx src/cli/main.ts doctor --json
+CREATE_JSON="$(npx tsx src/cli/main.ts create --json --cwd "$ROOT_DIR" --cols 140 --rows 45 --env 'PS1=phase5-proof$ ' -- /bin/bash --noprofile --norc -i)"
+SESSION_ID="$(printf '%s\n' "$CREATE_JSON" | jq -er '.result.sessionId')"
+
+npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 500 --timeout 10000 --json
+npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run typecheck' --timeout 300000 --json
+npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run lint' --timeout 300000 --json
+npx tsx src/cli/main.ts run "$SESSION_ID" 'npx vitest run test/unit/evals/claude.test.ts test/unit/evals/codex.test.ts test/unit/evals/promptRunner.test.ts test/integration/evals/authoring-pilots.test.ts --reporter=verbose' --timeout 300000 --json
+npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 1500 --timeout 10000 --json
+
+SNAPSHOT_JSON="$(npx tsx src/cli/main.ts snapshot "$SESSION_ID" --format text --include-scrollback --json)"
+printf '%s\n' "$SNAPSHOT_JSON" | jq -er '.result.text' > "$BUNDLE_DIR/snapshot.txt"
+
+SCREENSHOT_JSON="$(npx tsx src/cli/main.ts screenshot "$SESSION_ID" --json)"
+SCREENSHOT_PATH="$(printf '%s\n' "$SCREENSHOT_JSON" | jq -er '.result.artifactPath')"
+cp "$SCREENSHOT_PATH" "$BUNDLE_DIR/screenshot.png"
+
+npx tsx src/cli/main.ts record export "$SESSION_ID" --format webm --out "$BUNDLE_DIR/recording.webm" --json
+npx tsx src/cli/main.ts destroy "$SESSION_ID" --json
+rm -rf "$AGENT_TTY_HOME"
+```
+
+## What to verify
+
+Open `screenshot.png` to confirm the terminal finished on a green focused validation run, play `recording.webm` to confirm the full `agent-tty` session covers `typecheck`, `lint`, and the verbose Vitest command end to end, and inspect `snapshot.txt` to confirm the scrollback includes the 27-test success summary plus the token-usage-specific test names from `claude.test.ts`, `codex.test.ts`, and `authoring-pilots.test.ts`.

dogfood/token-usage-phase5-proof/commands.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(git rev-parse --show-toplevel)"
+BUNDLE_DIR="$ROOT_DIR/dogfood/token-usage-phase5-proof"
+mkdir -p "$BUNDLE_DIR"
+rm -f "$BUNDLE_DIR/snapshot.txt" "$BUNDLE_DIR/screenshot.png" "$BUNDLE_DIR/recording.webm"
+
+export AGENT_TTY_HOME="$(mktemp -d "$BUNDLE_DIR/.home.XXXXXX")"
+SESSION_ID=''
+
+cleanup() {
+  if [[ -n "$SESSION_ID" ]]; then
+    npx tsx src/cli/main.ts destroy "$SESSION_ID" --json >/dev/null 2>&1 || true
+  fi
+  if [[ -n "${AGENT_TTY_HOME:-}" && -d "${AGENT_TTY_HOME:-}" ]]; then
+    rm -rf "$AGENT_TTY_HOME"
+  fi
+}
+trap cleanup EXIT
+
+cd "$ROOT_DIR"
+
+npx tsx src/cli/main.ts --help >/dev/null
+npx tsx src/cli/main.ts doctor --json | jq -e '.ok == true and .result.ok == true' >/dev/null
+
+CREATE_JSON="$(npx tsx src/cli/main.ts create --json --cwd "$ROOT_DIR" --cols 140 --rows 45 --env 'PS1=phase5-proof$ ' -- /bin/bash --noprofile --norc -i)"
+SESSION_ID="$(printf '%s\n' "$CREATE_JSON" | jq -er '.result.sessionId')"
+
+npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 500 --timeout 10000 --json | jq -e '.ok == true and .result.timedOut == false' >/dev/null
+
+TYPECHECK_JSON="$(npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run typecheck' --timeout 300000 --json)"
+printf '%s\n' "$TYPECHECK_JSON" | jq -e '.ok == true and .result.accepted == true and .result.completed == true and (.result.timedOut // false) == false' >/dev/null
+
+LINT_JSON="$(npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run lint' --timeout 300000 --json)"
+printf '%s\n' "$LINT_JSON" | jq -e '.ok == true and .result.accepted == true and .result.completed == true and (.result.timedOut // false) == false' >/dev/null
+
+VITEST_COMMAND='npx vitest run test/unit/evals/claude.test.ts test/unit/evals/codex.test.ts test/unit/evals/promptRunner.test.ts test/integration/evals/authoring-pilots.test.ts --reporter=verbose'
+VITEST_JSON="$(npx tsx src/cli/main.ts run "$SESSION_ID" "$VITEST_COMMAND" --timeout 300000 --json)"
+printf '%s\n' "$VITEST_JSON" | jq -e '.ok == true and .result.accepted == true and .result.completed == true and (.result.timedOut // false) == false' >/dev/null
+
+npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 1500 --timeout 10000 --json | jq -e '.ok == true and .result.timedOut == false' >/dev/null
+
+SNAPSHOT_JSON="$(npx tsx src/cli/main.ts snapshot "$SESSION_ID" --format text --include-scrollback --json)"
+printf '%s\n' "$SNAPSHOT_JSON" | jq -er '.result.text' > "$BUNDLE_DIR/snapshot.txt"
+
+SCREENSHOT_JSON="$(npx tsx src/cli/main.ts screenshot "$SESSION_ID" --json)"
+SCREENSHOT_PATH="$(printf '%s\n' "$SCREENSHOT_JSON" | jq -er '.result.artifactPath')"
+cp "$SCREENSHOT_PATH" "$BUNDLE_DIR/screenshot.png"
+
+npx tsx src/cli/main.ts record export "$SESSION_ID" --format webm --out "$BUNDLE_DIR/recording.webm" --json | jq -e '.ok == true and .result.format == "webm"' >/dev/null
+npx tsx src/cli/main.ts destroy "$SESSION_ID" --json | jq -e '.ok == true' >/dev/null
+SESSION_ID=''
+
+trap - EXIT
+rm -rf "$AGENT_TTY_HOME"
+unset AGENT_TTY_HOME

dogfood/token-usage-phase5-proof/snapshot.txt

@@ -0,0 +1,62 @@
+phase5-proof$ npm run typecheck
+
+> agent-tty@0.1.1-beta.3 typecheck
+> tsc -p tsconfig.json --noEmit
+
+phase5-proof$ printf '%s%s\n' '__AT_MARKER_219ee04df48' 'b4d0a9bcaaa71b3c0f578__'
+__AT_MARKER_219ee04df48b4d0a9bcaaa71b3c0f578__
+phase5-proof$ npm run lint
+
+> agent-tty@0.1.1-beta.3 lint
+> eslint src test vitest.config.ts --max-warnings=0
+
+phase5-proof$ printf '%s%s\n' '__AT_MARKER_9eeecfa1bf8' '74baca2e5f837d57f6727__'
+__AT_MARKER_9eeecfa1bf874baca2e5f837d57f6727__
+phase5-proof$ npx vitest run test/unit/evals/claude.test.ts test/unit/evals/codex.test.ts test/unit/evals/promptRunner.test.ts test/integrat
+ion/evals/authoring-pilots.test.ts --reporter=verbose
+
+ RUN  v4.1.2 /home/coder/.mux/src/agent-terminal/agent_exec_f53d8e814f
+
+ ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > normalizes complete JSON usage objects and prefers the last valid record 8ms
+ ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > omits tokenUsage for 'missing total_tokens' 1ms
+ ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > omits tokenUsage for 'fractional totals' 0ms
+ ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > omits tokenUsage for 'negative cached tokens' 0ms
+ ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > does not emit tokenUsage for plain-text output 1ms
+ ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > normalizes command_execution items into shell-style tool calls 11ms
+ ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > normalizes function_call records with parsed arguments and outputs 2ms
+ ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > normalizes complete JSON usage objects and prefers the last valid record 2ms
+ ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > omits tokenUsage for 'missing total_tokens' 1ms
+ ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > omits tokenUsage for 'conflicting cached-token aliases' 0ms
+ ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > does not emit tokenUsage for plain-text output 1ms
+ ✓ test/unit/evals/promptRunner.test.ts > enumeratePromptWorkItems > returns unique prompt work items with stable keys 48ms
+ ✓ test/unit/evals/promptRunner.test.ts > enumeratePromptWorkItems > filters prompt work items by case id 3ms
+ ✓ test/unit/evals/promptRunner.test.ts > enumeratePromptWorkItems > filters prompt work items by condition 2ms
+ ✓ test/unit/evals/promptRunner.test.ts > executePromptWorkItem > builds a prompt request and returns the provider result payload 13ms
+ ✓ test/unit/evals/promptRunner.test.ts > executePromptWorkItem > converts provider errors into failed eval results 3ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves wait-for-output in the dry-run matrix 549ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs wait-for-output end-to-end through the CLI facade
+ with fixture playback 571ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves hello-prompt in the dry-run matrix 579ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs hello-prompt end-to-end through the CLI facade wi
+th fixture playback 441ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves doctor-gated in the dry-run matrix 552ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs doctor-gated end-to-end through the CLI facade wi
+th fixture playback 451ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves exploratory-qa in the dry-run matrix 422ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs exploratory-qa end-to-end through the CLI facade
+with fixture playback 571ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > round-trips tokenUsage from full normalized override f
+ixtures 8ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > preserves inline prompt and agent tokenUsage and keeps
+ omissions undefined 8ms
+ ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > preserves valid legacy normalized tokenUsage and drops
+ invalid tokenUsage payloads 13ms
+
+ Test Files  4 passed (4)
+      Tests  27 passed (27)
+   Start at  11:31:58
+   Duration  4.71s (transform 650ms, setup 0ms, import 1.31s, tests 4.27s, environment 0ms)
+
+phase5-proof$ printf '%s%s\n' '__AT_MARKER_e2b4fb7e307' '7485a89622cec04e0982a__'
+__AT_MARKER_e2b4fb7e3077485a89622cec04e0982a__
+phase5-proof$