feat: add skills evals and optimization framework

[ThomasK33]

feat: Add skills evals and optimization framework

Build a repo-native eval system for agent-tty that measures skill trigger accuracy, execution quality, and dogfood evidence completeness across Claude Code and Codex providers.

What changed

43 files, 21 commits implementing:

• Foundation (evals/lib/): Strict Zod schemas, deterministic scoring with negation-context awareness, anti-pattern detection (7 rules), skill-condition matrix (none/self-load/preloaded/stale), CLI harness, artifact storage, bundle scoring, JSON+Markdown reporting
• Providers (evals/providers/): Base interface, stub/fixture/recording providers, Claude Code adapter (claude --print), Codex adapter (codex exec)
• Lane A — Prompt/routing (evals/prompt/): 24 cases (8 agent-tty trigger, 6 dogfood-tui trigger, 6 should-not-trigger, 4 anti-pattern) + runner
• Lane B — Execution (evals/execution/): 10 fixture-based cases (hello-prompt, resize-demo, alt-screen-demo, color-grid, unicode-grid, scrollback-demo, crash-recovery, export-proof, run-command, doctor-gated) + verifier dispatch + runner
• Lane C — Dogfood (evals/dogfood/): 6 QA cases (exploratory-qa, release-readiness, rendering-bug-repro, navigation-focus-repro, resize-regression, evidence-completeness) + scorers + runner
• Entrypoint (evals/run.ts): CLI with --provider, --lane, --condition, --case, --dry-run, --json, --verbose
• Documentation (evals/README.md): Architecture, quick start, case inventory, scoring, comparison metrics, CI integration

Dogfooding with real Claude Code 2.1.107

7 rounds of fixes discovered by actually running evals against Claude Code:

Round	Issue Found	Fix
1	Execution lane crashes (Zod strict rejects extra scoring fields)	Strip runtime-only fields at source
2	Claude CLI not found (~/.local/bin not in PATH)	Prepend to PATH in providers
3	"instead of tmux" penalized as forbidden pattern	Negation-context heuristic in scoring
4	Score 1.00 still fails (binary AND too strict)	Score threshold ≥ 0.6
5	Anti-pattern detector also context-blind	Extended negation awareness
6	Lanes B/C tested text not actions	Inject skill text, add execution instructions
7	45s timeouts, warnings block pass	120-300s budgets, error-only scoring

Real Claude Code eval results

Lane A (prompt/routing) — 5 cases × 4 conditions:

Case	none	self-load	preloaded	stale
blind-sleep	✗ 0.30	✓ 1.00	✓ 1.00	✗ 0.30
exploratory-testing	✗ 0.00	✗ 0.00	✓ 0.77	✗ 0.00
pure-reasoning	✓ 1.00	✗ 0.30	✓ 1.00	✗ 0.40
session-creation	✗ 0.13	✓ 1.00	✓ 0.63	✗ 0.00
wait-for-output	✗ 0.30	✓ 1.00	✓ 1.00	✗ 0.30

Skill lift metrics: Realized +40pp, Oracle +80pp, Stale harm -100pp

Lane B (execution) — 3 cases, preloaded:

Case	Score	Details
hello-prompt	✅ 4/4	Verifier ✅, Workflow 5/5 ✅, Anti-pattern ✅
crash-recovery	3/4	Workflow 3/3 ✅, verifier exit-code check failed
color-grid	2/5	Workflow 3/3 ✅, renderer dependency issue

Lane C (dogfood) — 1 case, preloaded:

• 39 tool calls running real agent-tty commands before 180s timeout

Validation

Check	Result
npx tsc --noEmit	✅
npx eslint "evals/**/*.ts" --max-warnings=0	✅
npx prettier --check "evals/**/*.ts"	✅
npm run lint	✅
npm run format:check	✅
npm test (634 tests, 62 files)	✅

How to run

# Stub provider (no external deps)
npx tsx evals/run.ts --provider stub --lane prompt --dry-run

# Claude Code
npx tsx evals/run.ts --provider claude --lane prompt --condition preloaded

# Full execution
npx tsx evals/run.ts --provider claude --lane execution --case hello-prompt --condition preloaded

---

📋 Implementation Plan

agent-tty skills evals and optimization — implementation plan

1. Goal

Build a repo-native eval program for agent-tty that answers four questions with hard data:

1. Triggering: when should agent-tty or dogfood-tui load, and does the agent actually load them?
2. Execution: once a skill is available, does the agent complete real terminal/TUI tasks better?
3. Auditability: do skills improve reviewer-facing artifacts such as snapshots, screenshots, casts, WebM exports, notes, and proof bundles?
4. Compatibility: how well do the two highest-priority coding agents — Claude Code (claude CLI) and Codex (codex CLI) — behave with the bootstrap skills, canonical skills, and agent-tty command surface?

The end state is not a single benchmark number. The end state is a repeatable measurement system that can show:

• baseline autonomous capability,
• realized lift from skills in realistic usage,
• upper-bound lift if routing is perfect,
• negative deltas from stale or wrong skills,
• and trend lines that guide skill edits without overfitting.

2. Non-goals

• Do not turn this into a generic coding benchmark for the whole repo.
• Do not build a parallel terminal harness when the repo already has CLI helpers, fixtures, bundles, and artifact validation.
• Do not rely on third-party TUIs first; start with deterministic fixture apps under test/fixtures/apps/.
• Do not use LLM judges for anything that can be checked deterministically via JSON envelopes, snapshots, manifests, event logs, or bundle validators.
• Do not claim Claude Code or Codex compatibility from prompt-only results alone; compatibility claims require execution-lane evidence.

3. Verified repo assets to reuse

Use the existing repo as the foundation instead of building bespoke infrastructure:

• Bootstrap/public skill: skills/agent-tty/SKILL.md
• Canonical core skill: skill-data/agent-tty/SKILL.md
• Specialized QA skill: skill-data/dogfood-tui/SKILL.md
• CLI/test helpers: test/helpers.ts, test/e2e/helpers.ts
• Deterministic fixture apps: test/fixtures/apps/*
• Bundle tooling: src/tools/review-bundle.ts, src/tools/validate-bundle.ts
• Event-log and replay truth: src/host/eventLog.ts, src/host/replay.ts
• Artifact manifests: src/storage/artifactManifest.ts
• Validation guidance: design/20260319_agent-tty-v1/05-dogfooding-and-validation.md

These assets already support:

• isolated AGENT_TTY_HOME setups,
• JSON envelopes,
• terminal/TUI fixtures,
• screenshots and recordings,
• replayable event logs,
• proof-bundle review pages,
• and bundle validation.

Plan principle: build a thin eval layer on top of those primitives.

Research basis that should shape implementation choices

• agent-browser PR #1225 is the template for a lightweight first lane: prompt/routing evals with provider adapters, deterministic regex-style scoring, optional LLM judging, and category-specific reporting.
• SkillsBench argues for explicitly comparing no-skill vs curated-skill vs preloaded/oracle conditions and measuring both positive and negative deltas.
• Terminal-Bench argues for outcome-first evaluation on real terminal tasks with deterministic verifiers.
• Anthropic’s agent-evals guidance argues for balanced should-trigger / should-not-trigger datasets, repeated trials, and careful transcript review.
• The Agentic Benchmark Checklist argues for separate validation of task quality and grading quality so the team does not accidentally reward artifact production without task success.

4. Design principles

1. Outcome over imitation. Prefer final-state success checks over brittle sequence matching.
2. Deterministic first. Use snapshots, manifests, event logs, validators, and custom verifiers before any judge model.
3. Fixture-first. Start with deterministic fixture apps and only later extend to live third-party TUIs.
4. Balanced routing data. Include should-trigger and should-not-trigger cases for both agent-tty and dogfood-tui.
5. Provider parity. Claude Code and Codex must be first-class citizens in the harness, reports, and CI matrix.
6. Skill conditions are part of the experiment. Always compare no-skill vs self-load vs preloaded, and add stale/wrong skill once the basics are stable.
7. Negative deltas matter. Treat skills that hurt success, increase anti-patterns, or reduce auditability as blockers, not noise.
8. Defensive programming. Use strict schemas, assertions, and explicit failure classes everywhere in the eval harness.
9. Reproducibility over one-off demos. Every serious eval run should leave behind artifacts a reviewer can inspect.

5. Target architecture

Implement three eval lanes on top of shared providers, shared case schemas, shared scoring, and shared reporting.

Lane A — Prompt / routing evals

Purpose:

• Measure whether the agent recognizes that a task should use agent-tty.
• Measure whether the agent selects dogfood-tui when the task is explicitly QA / dogfooding / release-readiness oriented.
• Measure whether the agent proposes the right high-level workflow and avoids repo-documented anti-patterns.

What this lane scores:

• skill trigger precision / recall / F1,
• correct skill selection (none, agent-tty, dogfood-tui),
• workflow planning compliance,
• anti-pattern avoidance,
• provider differences between Claude Code and Codex.

This lane should ship first because it is cheap, fast, and directly comparable to agent-browser PR #1225.

Lane B — Closed-loop execution evals

Purpose:

• Run real tasks in real agent-tty sessions against deterministic fixtures.
• Compare autonomous vs skill-assisted performance.
• Score terminal outcomes, workflow correctness, efficiency, error recovery, and artifact quality.

What this lane scores:

• task success,
• workflow compliance,
• efficiency and anti-patterns,
• error recovery,
• artifact completeness,
• realized/oracle lift by provider and skill condition.

This is the core lane for making real compatibility claims about Claude Code and Codex with agent-tty.

Lane C — Dogfood / evidence evals

Purpose:

• Evaluate whether the dogfood-tui skill improves exploratory testing, issue reproduction, evidence capture, and reviewability.
• Score proof bundles, notes, screenshots, recordings, and reviewer usefulness.

What this lane scores:

• bundle validation pass rate,
• evidence completeness,
• report completeness and taxonomy usage,
• issue reproducibility,
• human-review efficiency proxy metrics.

This lane is the differentiator for agent-tty: the system should not only solve terminal tasks, it should leave behind reviewable proof.

6. Comparison framework and experimental conditions

Every eligible case should run under the same prompt, environment, provider, model, and budget across these conditions:

Condition	Meaning	Primary question
A. Autonomous / no skill	No skill text is injected or discoverable	What is the raw baseline?
B. Self-load / realistic	Only the bootstrap/public skill path is available; agent must decide to run agent-tty skills get ...	What lift do users get in realistic deployment?
C. Preloaded / oracle	The relevant canonical skill is injected up front	What is the upper bound if routing is perfect?
D. Wrong or stale skill	An intentionally stale or irrelevant skill is injected	How much harm comes from bad guidance?

Use the repo’s current skill split to implement those conditions:

• Condition B bootstrap: skills/agent-tty/SKILL.md
• Condition C core/oracle: skill-data/agent-tty/SKILL.md
• Condition C QA/oracle: skill-data/dogfood-tui/SKILL.md
• Condition D stale/wrong: archived snapshots of older skill text or intentionally mismatched skill injections

Derived comparison metrics

Compute and publish at least these deltas:

• Realized Skill Lift = success(B) - success(A)
• Oracle Skill Lift = success(C) - success(A)
• Routing Gap = success(C) - success(B)
• Stale-Skill Harm = success(D) - success(C)
• Regression Rate = % of A passes that become B/C failures
• Unlock Rate = % of A failures that become B/C passes
• Routing Efficiency = (success(B) - success(A)) / max(ε, success(C) - success(A))

Report each metric by:

• provider (claude, codex),
• model version,
• lane,
• case category,
• fixture,
• and skill type (agent-tty, dogfood-tui).

7. Provider support plan — Claude Code and Codex are first-class

Provider requirements

Create a provider abstraction that both CLIs implement:

interface EvalProvider {
  id: 'claude' | 'codex';
  detect(): Promise<ProviderRuntimeInfo>;
  invokePlanMode(input: ProviderPromptRequest): Promise<ProviderPromptResult>;
  invokeAgentMode(input: ProviderAgentRequest): Promise<ProviderAgentResult>;
  parse(raw: RawProviderOutput): NormalizedProviderOutput;
}

Each adapter must define:

• command invocation contract,
• environment/config requirements,
• timeout handling,
• raw output capture,
• transcript normalization,
• model/version logging,
• non-zero exit handling,
• and parser behavior.

Claude Code (claude CLI)

Deliverables:

• plan-only prompt adapter,
• execution/agent-run adapter,
• raw stdout/stderr capture,
• model/version logging,
• transcript archival into eval artifacts,
• compatibility smoke tests.

Codex (codex CLI)

Deliverables:

• plan-only prompt adapter,
• execution/agent-run adapter,
• JSONL-aware parser and transcript normalizer,
• model/version logging,
• config/bootstrap detection,
• compatibility smoke tests.

Provider operating modes

Support two provider modes:

1. Plan-only mode

   • Cheap and CI-friendly.
   • Used for Lane A and the first provider integration milestone.
   • Equivalent in spirit to the agent-browser PR approach.

2. Agent-run mode

   • Actually lets Claude Code or Codex operate in the eval sandbox.
   • Required for Lane B and Lane C compatibility claims.
   • Slower and more brittle, so it should enter after the harness is stable.

Provider acceptance policy

• MVP requirement: both Claude Code and Codex work in Lane A.
• V1 requirement: both Claude Code and Codex work in Lane B on the deterministic fixture suite.
• GA requirement for compatibility claims: both providers have visible reports on Lanes A/B/C, with no provider-specific hidden hacks and with raw transcript capture available for debugging.

If one provider temporarily loses stable headless support, keep its schema tests and parser tests green, mark execution-lane support as degraded, and block broad compatibility claims until restored.

8. Proposed repository layout

Create a dedicated evals/ tree, but keep it thin and wired to existing helpers and artifacts.

evals/
├── README.md
├── run.ts
├── lib/
│   ├── types.ts
│   ├── schemas.ts
│   ├── scoring.ts
│   ├── reporting.ts
│   ├── matrix.ts
│   ├── antiPatterns.ts
│   ├── artifacts.ts
│   ├── bundleScoring.ts
│   └── cliHarness.ts
├── providers/
│   ├── base.ts
│   ├── claude.ts
│   ├── codex.ts
│   └── fixtures.ts
├── prompt/
│   ├── cases/
│   │   ├── trigger-agent-tty.ts
│   │   ├── trigger-dogfood-tui.ts
│   │   ├── should-not-trigger.ts
│   │   └── anti-patterns.ts
│   └── runner.ts
├── execution/
│   ├── cases/
│   │   ├── hello-prompt.ts
│   │   ├── resize-demo.ts
│   │   ├── alt-screen-demo.ts
│   │   ├── color-grid.ts
│   │   ├── unicode-grid.ts
│   │   ├── scrollback-demo.ts
│   │   ├── crash-recovery.ts
│   │   └── export-proof.ts
│   ├── verifiers/
│   └── runner.ts
├── dogfood/
│   ├── cases/
│   ├── scorers/
│   └── runner.ts
├── judges/
│   ├── qualityJudge.ts
│   └── calibration.ts
└── reports/

Shared-helper extraction step

Do not make the eval harness depend directly on test/ code forever. In Phase 0, extract the reusable parts of test/helpers.ts and test/e2e/helpers.ts into a shared helper that both tests and evals can consume, such as:

• evals/lib/cliHarness.ts, and/or
• a small internal shared helper module consumed by both tests and evals.

That extraction should preserve current tests while avoiding duplicate logic.

9. Core schemas and data model

Use strict Zod-backed schemas and fail-fast assertions.

Prompt case schema

type ExpectedSkill = 'none' | 'agent-tty' | 'dogfood-tui';

interface PromptEvalCase {
  id: string;
  lane: 'prompt';
  category: 'trigger' | 'selection' | 'workflow' | 'anti-pattern';
  prompt: string;
  expectedSkill: ExpectedSkill;
  context?: string;
  expectedPatterns: string[];
  forbiddenPatterns?: string[];
  rubric?: string[];
  budgets: { timeoutMs: number };
}

Execution case schema

type SkillCondition = 'none' | 'self-load' | 'preloaded' | 'stale';

interface ExecutionEvalCase {
  id: string;
  lane: 'execution';
  category: 'session' | 'tui' | 'artifact' | 'recovery';
  fixture: string;
  prompt: string;
  conditions: SkillCondition[];
  setup: SetupStep[];
  verifier: VerifierSpec;
  workflowChecks: WorkflowCheck[];
  antiPatterns: AntiPatternRule[];
  artifactsRequired: ArtifactRequirement[];
  budgets: {
    timeoutMs: number;
    maxAgentSteps?: number;
    maxWallClockMs?: number;
  };
}

Dogfood case schema

interface DogfoodEvalCase {
  id: string;
  lane: 'dogfood';
  category: 'qa' | 'release-readiness' | 'bug-repro' | 'reporting';
  prompt: string;
  targetFixture: string;
  conditions: SkillCondition[];
  bundleProfile: 'contract-reporting' | 'interactive-renderer';
  reportRequirements: ReportRequirement[];
  artifactsRequired: ArtifactRequirement[];
  verifier: VerifierSpec;
  budgets: { timeoutMs: number; maxWallClockMs?: number };
}

Result schema

Every run should emit a strict result object with at least:

• provider id and detected version,
• model id,
• lane,
• case id,
• condition,
• trial number,
• pass/fail,
• deterministic score breakdown,
• optional judge score,
• workflow findings,
• anti-pattern findings,
• raw transcript paths,
• artifact manifest path,
• bundle path (if any),
• error class,
• and timestamps.

10. Metrics specification

Routing and selection metrics

• Trigger precision / recall / F1
• Overtrigger rate
• Undertrigger rate
• Skill selection accuracy
• Bootstrap-load compliance
• Correct dependency ordering (agent-tty before dogfood-tui when required)

Workflow metrics

Check partial-order invariants rather than exact step-by-step traces.

For agent-tty, track whether the run:

• used isolated home handling,
• used doctor --json before screenshot/recording flows,
• created a session before controlling it,
• used run for setup instead of long simulated typing,
• used wait instead of blind sleeps,
• used snapshot when semantic inspection mattered,
• used screenshot / record export when proof was required,
• destroyed the session when expected,
• and avoided repo-documented anti-patterns.

For dogfood-tui, also score whether the run produced:

• evidence checklist completeness,
• issue taxonomy usage,
• report-template completeness,
• screenshot + cast + WebM + notes bundle quality.

Outcome metrics

• Task success rate
• Task-mean pass rate
• Pass rate by provider and by condition
• Unlock rate / regression rate
• Realized Skill Lift / Oracle Skill Lift / Routing Gap / Stale-Skill Harm
• Flake rate across repeated trials

Efficiency metrics

• Wall-clock time
• Agent step count / tool call count
• agent-tty command count
• Retry count
• Timeout rate
• Anti-pattern incidence rate (sleep, tmux, screen, external screenshots, missing --json, orphaned sessions)

Auditability metrics

• Bundle validation pass rate
• Artifact completeness score
• Notes/report completeness score
• Replayability / manifest integrity checks
• Reviewer-quality score for sampled runs (judge-backed only after calibration)

11. Initial case inventory

Lane A — Prompt / routing MVP (target: 24 cases)

• 8 agent-tty should-trigger prompts
  • session creation
  • interactive CLI automation
  • waiting on observable terminal state
  • snapshot/screenshot/record export requests
• 6 dogfood-tui should-trigger prompts
  • exploratory testing
  • bug hunting
  • release readiness
  • evidence-backed issue reporting
• 6 should-not-trigger prompts
  • pure reasoning
  • non-terminal tasks
  • tasks better handled without terminal automation
• 4 anti-pattern / workflow prompts
  • detect blind sleep
  • detect tmux / screen
  • detect ad hoc screenshots
  • detect missing --json in automation contexts

Lane B — Execution MVP (target: 10 cases)

1. hello-prompt: launch, input, wait, snapshot, exit
2. resize-demo: resize and verify layout recovery
3. alt-screen-demo: enter/exit alt-screen cleanly
4. color-grid: capture screenshot and verify artifact generation
5. unicode-grid: verify semantic snapshot and rendering evidence
6. scrollback-demo: verify scrollback visibility and evidence capture
7. export-proof: cast + WebM export flow
8. crash-recovery: inspect/reconcile/destroy after failure
9. run-command: setup via run, not simulated typing
10. doctor-gated-renderer: ensure screenshot/record flows respect prerequisite checks

Lane C — Dogfood MVP (target: 4–6 cases)

• exploratory QA run against a deterministic fixture,
• release-readiness bundle for a TUI fixture,
• rendering bug repro with notes + evidence,
• navigation/focus issue repro with bundle validation,
• resize/scrollback regression triage,
• optional sampled third-party TUI once fixture suite is stable.

12. Case authoring rules

Every case author must follow these rules:

1. One clear end condition. The verifier must know exactly what success means.
2. Independent grading. The verifier cannot merely restate the skill instructions.
3. Fixture-first. Prefer deterministic local apps before external software.
4. Balanced routing. Add negative controls for every trigger category.
5. No hidden reliance on deferred features. Stay inside shipped agent-tty capabilities.
6. No exact-sequence grading unless unavoidable. Prefer partial-order or end-state checks.
7. Artifact expectations must be explicit. If screenshots or videos matter, state them in the case.
8. Budget expectations must be explicit. Define timeout, retry, and step ceilings.
9. Holdout separation. Maintain a private/hidden holdout set not used for day-to-day optimization.

13. Workstreams for a team of agents

Workstream A — Provider adapters and runtime capture

Owner: Agent 1

Build:

• provider base interfaces and strict schemas,
• Claude Code plan-only adapter,
• Codex plan-only adapter,
• Claude Code agent-run adapter,
• Codex agent-run adapter,
• runtime/version detection,
• transcript/raw-output normalization,
• provider smoke tests.

Dependencies:

• none, except shared schema decisions from Phase 0.

Deliverables:

• provider modules,
• stub/fake providers for tests,
• fixture-based parser tests,
• CLI/config detection docs.

Workstream B — Prompt/routing harness and case authoring

Owner: Agent 2

Build:

• Lane A runner,
• case schemas,
• scoring and regex/forbidden-pattern utilities,
• 24-case routing MVP,
• provider comparison report,
• optional quality-judge plumbing (disabled by default).

Dependencies:

• Workstream A plan-only adapters,
• Workstream E reporting contracts.

Deliverables:

• evals/prompt/*,
• JSON and human-readable reports,
• CI-friendly prompt-only lane.

Workstream C — Execution harness and deterministic verifiers

Owner: Agent 3

Build:

• shared CLI/session harness based on extracted helper logic,
• execution case runner,
• verifier library for snapshots, artifacts, event logs, and manifests,
• 10-case fixture-based execution suite,
• anti-pattern detectors,
• skill-condition matrix runner (A/B/C/D).

Dependencies:

• Workstream A agent-run adapters,
• Phase 0 shared helper extraction.

Deliverables:

• evals/execution/*,
• deterministic verifiers,
• replay/event-log checks,
• execution reports with provider/condition breakdowns.

Workstream D — Dogfood bundle scoring and reviewability

Owner: Agent 4

Build:

• Lane C runner,
• bundle scoring layer on top of review-bundle and validate-bundle,
• report completeness checker,
• evidence-quality metrics,
• 4–6 dogfood cases,
• sampled judge-backed review-quality scorer after calibration.

Dependencies:

• Workstream C execution harness,
• existing bundle tooling.

Deliverables:

• evals/dogfood/*,
• bundle-quality reports,
• proof bundles suitable for reviewer inspection.

Workstream E — Metrics, reporting, CI, and trend storage

Owner: Agent 5

Build:

• shared result schema,
• summary reporters,
• provider × lane × condition dashboards,
• PR/nightly/weekly pipeline definitions,
• artifact retention rules,
• trend storage layout.

Dependencies:

• A/B/C/D output schemas.

Deliverables:

• JSON summary contract,
• markdown/console summary,
• machine-readable trend archives,
• CI matrix docs and scripts.

Workstream F — Skill optimization loop and holdout governance

Owner: Agent 6

Build:

• failure taxonomy,
• negative-delta analysis pipeline,
• A/B skill-edit experiment protocol,
• hidden holdout governance,
• weekly review ritual,
• “do not promote this skill edit unless…” gate rules.

Dependencies:

• stable reports from Workstream E,
• representative results from Lanes A/B/C.

Deliverables:

• optimization SOP,
• holdout registry,
• weekly triage template,
• promotion criteria.

14. Phased rollout

Phase 0 — Foundations and shared plumbing

Scope:

• finalize schemas and result contracts,
• extract shared helper logic from test/helpers.ts / test/e2e/helpers.ts,
• build provider detection and stub providers,
• define report layout and artifact storage,
• establish the condition matrix A/B/C/D.

Tasks:

1. Extract shared CLI/session helper utilities.
2. Define strict schemas for cases and results.
3. Implement provider base interfaces and stub providers.
4. Add raw transcript/result artifact conventions.
5. Add initial reporter skeleton and summary JSON contract.

Acceptance criteria:

• harness compiles with strict schemas,
• stub providers can run end-to-end across all three lanes,
• shared helper extraction does not regress existing tests,
• every eval result has a strict JSON shape and artifact pointers,
• condition matrix can be enumerated for both Claude Code and Codex.

Quality gate / dogfood:

• Run a local smoke harness with an isolated AGENT_TTY_HOME.
• Capture at least one screenshot and one WebM of the harness exercising a simple hello-prompt session.
• Generate a proof bundle for the smoke run, then run review-bundle and validate-bundle.
• Keep the smoke bundle as the first reviewer-facing proof that the eval harness itself is inspectable.

Phase 1 — Prompt / routing MVP

Scope:

• ship Lane A with both Claude Code and Codex,
• author the 24-case prompt suite,
• implement deterministic scoring and optional judge scaffolding,
• establish provider-difference reporting.

Tasks:

1. Implement claude plan-only adapter.
2. Implement codex plan-only adapter.
3. Author routing, selection, should-not-trigger, and anti-pattern cases.
4. Build regex/forbidden-pattern scoring and summary reports.
5. Add local and CI entry points for prompt-only runs.
6. Add sampled transcript review workflow for failed cases.

Acceptance criteria:

• both providers run the full prompt suite,
• reports break down trigger/selection/workflow results by provider,
• failed cases store raw prompt/response artifacts,
• should-trigger and should-not-trigger categories are both present,
• optional judge is disabled by default and never gates deterministic success.

Quality gate / dogfood:

• Run the prompt suite locally on Claude Code and Codex.
• Record terminal sessions of both runs with agent-tty, capture screenshots of the summaries, and export WebM for review.
• Package those artifacts into a bundle proving the prompt lane works for both providers.

Phase 2 — Execution MVP

Scope:

• ship Lane B on deterministic fixtures,
• onboard Claude Code and Codex to agent-run mode,
• measure A/B/C conditions on the 10-case execution suite,
• add deterministic verifiers and anti-pattern detectors.

Tasks:

1. Implement provider agent-run adapters for Claude Code and Codex.
2. Build execution runner around isolated homes and session artifacts.
3. Author 10 execution cases across session, TUI, artifact, and recovery categories.
4. Implement verifiers using snapshots, structured snapshots, manifests, inspect output, event logs, and bundle validation where appropriate.
5. Add anti-pattern detectors and workflow compliance scoring.
6. Publish comparison reports for A/B/C; add D for nightly once stable.

Acceptance criteria:

• both providers complete the full deterministic suite in agent-run mode,
• every case has a deterministic verifier,
• reports include realized lift, oracle lift, routing gap, and regression/unlock rates,
• anti-pattern findings are attached to case results,
• execution artifacts are archived for every failed trial,
• flake rate is measured and visible.

Quality gate / dogfood:

• Produce proof bundles for at least hello-prompt, resize-demo, and alt-screen-demo on both Claude Code and Codex.
• Each bundle must contain notes, snapshots, screenshots, .cast, and WebM.
• Run review-bundle and validate-bundle on each bundle.
• Reviewers must be able to inspect how each provider behaved under A/B/C conditions.

Phase 3 — Dogfood / evidence suite

Scope:

• ship Lane C,
• measure whether dogfood-tui improves QA workflows,
• score bundle completeness and report quality,
• add sampled qualitative judging after calibration.

Tasks:

1. Author 4–6 dogfood cases with explicit evidence requirements.
2. Build bundle-completeness and report-completeness scoring.
3. Integrate with existing review/validation tools.
4. Add sampled judge-backed quality scoring for reports and reviewer usefulness.
5. Compare A/B/C on dogfood tasks, adding D after baseline stabilizes.

Acceptance criteria:

• every dogfood case yields a reviewable proof bundle,
• bundle scores are separated from task-success scores,
• report completeness is deterministic wherever possible,
• judge scoring is calibrated on a human-reviewed sample before being surfaced,
• provider results are broken out separately for Claude Code and Codex.

Quality gate / dogfood:

• Run at least two end-to-end dogfood cases per provider.
• Capture bundle review pages with screenshots and short WebM walkthroughs.
• Confirm that a reviewer can inspect the bundle alone and understand expected vs actual behavior.

Phase 4 — Optimization loop, holdouts, and governance

Scope:

• operationalize recurring skill improvement,
• protect against overfitting,
• add holdout and stale-skill governance,
• turn the eval suite into a continuous optimization system.

Tasks:

1. Define failure taxonomy and tagging.
2. Build a workflow for proposing, testing, and promoting skill edits.
3. Maintain visible suite vs hidden holdout split.
4. Add weekly trend reports and failure review.
5. Require non-regression across providers and conditions before skill promotion.

Acceptance criteria:

• every material skill edit is evaluated on visible suite + holdout,
• negative deltas are surfaced automatically,
• weekly reports highlight biggest routing gaps and biggest stale-skill harms,
• promotion criteria are documented and enforced.

Quality gate / dogfood:

• For every promoted skill change, archive one before/after comparison bundle with screenshots and WebM walkthrough.
• Use those bundles in the weekly review to validate that reported gains are real and reviewer-visible.

15. CI and scheduling strategy

Split the program into tiers so cost and stability are manageable.

PR lane

Run on every PR when possible:

• schema/unit tests for eval harness,
• stub-provider contract tests,
• prompt-lane smoke subset,
• execution smoke subset with deterministic fixtures,
• no judge models,
• no expensive multi-trial sweeps.

Nightly lane

Run nightly:

• full prompt lane on Claude Code and Codex,
• full deterministic execution lane on Claude Code and Codex,
• A/B/C for all core cases,
• D on a sampled subset,
• 3 trials for higher-variance cases,
• summary JSON and markdown report publication.

Weekly / release lane

Run weekly and before releases:

• full prompt + execution + dogfood suite,
• 3–5 trials per selected case,
• judge-enabled sampled report scoring,
• holdout suite,
• stale-skill regression pass,
• reviewer-facing proof bundles with screenshots and WebM walkthroughs.

CI degradation rules

• If provider CLIs or credentials are unavailable, stub-provider tests still run and provider-specific lanes are marked skipped rather than silently passing.
• Parser tests and transcript fixtures must still run for both providers even when live invocations are skipped.
• Never merge a change that breaks result-schema compatibility or silently drops raw transcript capture.

16. Dogfooding and self-verification protocol

Because this is a CLI project, dogfooding must be explicit and artifact-backed.

Setup instructions

Preferred setup:

mise install
mise run bootstrap

Fallback setup:

npm ci
npx playwright install chromium

Provider prerequisites:

• install Claude Code (claude CLI) and ensure auth/config is valid,
• install Codex (codex CLI) and ensure auth/config is valid,
• log detected provider versions at the start of every eval run.

Runtime rules:

• always use an isolated absolute AGENT_TTY_HOME,
• run doctor --json before screenshot/recording-heavy flows,
• prefer run + wait over blind sleeps,
• capture snapshots for semantic truth and screenshots/video for reviewer truth,
• destroy sessions after the run unless the case explicitly tests retention.

Required self-verification artifacts

Every meaningful phase gate must produce:

• exact repro commands,
• summary JSON,
• at least one snapshot,
• at least one screenshot,
• at least one .cast,
• at least one WebM,
• notes explaining the scenario,
• a generated review page via review-bundle,
• and a passing validate-bundle result with the appropriate profile.

Reviewer workflow

For each milestone bundle:

1. open the generated review page,
2. inspect the case summary,
3. inspect the screenshot gallery,
4. play the WebM,
5. inspect snapshot text / JSON,
6. inspect raw transcripts for failures,
7. confirm the bundle validator passed,
8. compare provider behavior side-by-side where relevant.

17. Risks and mitigations

Risk	Impact	Mitigation
Regex/pattern brittleness in Lane A	False positives/negatives	Keep prompt lane narrow, add forbidden patterns, and calibrate against real transcripts
Provider CLI drift	Broken parsers or false failures	Version-detect adapters, parser fixture tests, smoke tests, raw transcript capture
Flaky timing in execution lane	Noisy results	Fixture-first tasks, wait conditions instead of sleeps, visible flake metrics
Stale skill guidance hurts results	False sense of improvement	Explicit D condition, stale-skill harm reporting, promotion gates on negative deltas
Overfitting to visible suite	Inflated benchmark numbers	Hidden holdout set, rotated cases, sampled manual review
Confusing artifact production with success	Misleading scores	Separate task-success, workflow, and bundle-quality scores
Importing test-only helpers permanently into evals	Maintenance drag	Phase 0 shared-helper extraction and reuse contract
Provider-specific hacks distort compatibility claims	Untrustworthy results	Keep provider adapters thin, document differences, ban case-specific hidden overrides

18. Exit criteria

The implementation is complete enough to declare the program operational when all of the following are true:

1. Lane A runs on both Claude Code and Codex and publishes provider-separated trigger/selection/workflow metrics.
2. Lane B runs on both Claude Code and Codex across the deterministic fixture suite with A/B/C comparisons and deterministic verifiers.
3. Lane C produces reviewable proof bundles and provider-separated evidence-quality reports.
4. The harness publishes realized lift, oracle lift, routing gap, stale-skill harm, unlock rate, and regression rate.
5. Every serious eval milestone includes screenshot and WebM evidence, not only JSON.
6. The team has a documented optimization loop with holdout governance and negative-delta blocking.
7. Compatibility claims for Claude Code and Codex are backed by execution-lane evidence, not prompt-only results.

19. Immediate next actions

1. Start Phase 0 with shared-helper extraction, case/result schemas, and provider base interfaces.
2. Assign Workstreams A–F to separate agents with weekly integration points.
3. Land the Lane A MVP first so the team gets early signal on Claude Code and Codex routing behavior.
4. Follow immediately with Lane B execution MVP, because that is the first point where real compatibility claims become credible.
5. Use Lane C to turn agent-tty’s proof-bundle capabilities into a measurable competitive advantage rather than a nice-to-have.
6. Do not promote any material skill rewrite until the A/B/C comparison framework and negative-delta reporting are live.

---

Generated with mux • Model: anthropic:claude-opus-4-6 • Thinking: xhigh