feat: improve eval setup for skill optimization + fix two scoring bugs

[ThomasK33]

Summary

Improves the agent-tty eval setup for skill optimization, and fixes two scoring bugs that were making absolute pass rates misleading (especially for Codex on the execution lane).

Tooling improvements

• Repeatable --condition (evals/run.ts): --condition none --condition preloaded in one invocation, canonically ordered and deduped, with fail-fast validation. Replaces prior set-once CLI behavior.
• Comparison signal in reports (evals/lib/reporting.ts, evals/lib/schemas.ts, evals/run.ts): promoted "Condition comparison" section (per-condition totals + key deltas: realizedSkillLift, oracleSkillLift, routingGap, staleSkillHarm, regressionRate, unlockRate, routingEfficiency) to the top of report.md. Added structured conditionComparisonSummary to report.json. Resolved outputBaseDir is now visible in both human and JSON outputs.
• Lane B readiness docs + environment-blocked classification (evals/README.md, evals/execution/runner.ts, evals/execution/cases/shared.ts): documented the 10 execution cases by readiness tier (battle-tested / non-renderer unproven / renderer-optional / renderer-required), and emit environment-blocked errorClass when transcripts indicate Playwright/Chromium/ghostty-web failures instead of conflating them with skill regressions.

Scoring bug fixes (found while running experiments)

Running the full eval sweep across Claude (opus 4.6, high) and Codex (gpt-5.4, high) with trials surfaced a Codex "self-load collapse" on execution (0% pass rate). Diagnostic revealed this was not a routing/skill problem but two scoring bugs:

1. Anti-pattern detector was flagging tool-call outputs, not just inputs (evals/lib/antiPatterns.ts). In self-load runs, agent-tty skills get agent-tty echoed the skill doc into the transcript. The skill's own "avoid these" warnings (scrot, tmux, xdotool, gnome-screenshot) matched the anti-pattern regexes, producing up to 62 false-positive findings per run and tanking the anti-pattern-avoidance subscore. buildScannableTranscript() now scans only tool-call inputs (what the model intended to execute), not outputs. Prompt-lane behavior unchanged.
2. Skill-selection scorer ignored rejection context (evals/lib/scoring.ts). inferSelectedSkill() returned 'dogfood-tui' on any literal mention, even sentences like "dogfood-tui would be too specialized because the user is not asking for QA." Now uses the existing isInNegationContext() helper to only count non-rejected mentions.

Both fixes land with unit tests.

Paired baseline validation (post-fix vs pre-fix reports)

Using the new --compare-baseline and --trials tooling from main (02cc7dd), with bootstrap-CI paired verdicts:

Run	Pre-fix pass	Post-fix pass	Δ	Verdict
Codex execution (self-load+preloaded, trials=3)	35.0%	68.3%	+33.3pp	IMPROVED (46W / 10L / 4T)
Codex prompt (self-load+preloaded, trials=5)	24.6%	32.9%	+8.3pp	IMPROVED (56W / 51L / 133T)
Claude prompt preloaded (trials=5, control)	27.5%	30.0%	+2.5pp	inconclusive (as expected)

Top per-case improvements (Codex execution, self-load): hello-prompt, scrollback-demo, unicode-grid, resize-demo, crash-recovery all went from 0% → 100% pass rate from the scoring fix alone.

Validation

• npm run typecheck — passes
• npm run lint — passes
• npm run test — 78 test files, 830 tests passing
• npm run format:check — passes (scoped to tracked files; generated evals/reports/ is now gitignored)
• New unit tests added: CLI condition-parser coverage (test/unit/evals/run.test.ts), reporting comparison summary (test/unit/evals/reporting.test.ts), anti-pattern scan regression guard (test/unit/evals/antiPatterns.test.ts), skill-selection negation cases (test/unit/evals/scoring.test.ts).

Dogfooding proof

Cross-provider sweep with the new parallelization tooling (--trials, --concurrency 4):

• Prompt lane: Claude 103/480 (5 trials), Codex 136/480 (5 trials) — ~13 min each, in parallel
• Execution lane: Claude 59/120 (3 trials), Codex 33/120 (3 trials) — ~35/51 min in parallel
• Dogfood lane: Claude 5/48 (2 trials), Codex 3/48 (2 trials)

Post-fix validation runs used --compare-baseline against those baselines and produced the paired verdicts in the table above.

All 8 eval reports (report.json + report.md per run) plus cross-provider COMPARISON.md and the final EXPERIMENTS.md writeup are generated locally and reviewable. Per existing repo convention, they live under evals/reports/{timestamp}/ which is gitignored — they were not committed.

Commits

• c3d71ed feat: allow repeating eval conditions
• 5dd675e test: cover repeated eval conditions
• 545f896 feat: surface renderer blockers in execution evals
• 5027152 docs: document execution readiness tiers
• 93a1a45 feat: surface eval comparison summary
• 53807a7 style: fix formatting in eval files
• 86ff7da fix: ignore echoed tool output in anti-pattern scans
• 00559c8 fix: ignore rejected dogfood-tui mentions in scoring
• 0688484 chore: gitignore generated eval reports

---

📋 Implementation Plan

Plan: Improve the eval setup for skill optimization

Objective

Make the eval loop faster to iterate on skill text, easier to compare none vs preloaded/self-load, and clearer about which lanes and cases produce trustworthy signals.

Repo-backed evidence

• evals/run.ts is the entrypoint. Its help text documents --condition <cond> as singular, and parseCliArgs() currently enforces --condition may only be set once while --case is already repeatable.
• Downstream code already works with SkillCondition[]: resolveRequestedConditions() returns an array and SKILL_CONDITIONS lives in evals/lib/matrix.ts, so the narrowest high-leverage change is CLI ergonomics rather than a matrix rewrite.
• Reports are already persisted by default under evals/reports/{timestamp} or a custom --output via resolveOutputBaseDir() and EvalArtifactStore; the real gap is cross-workspace discoverability/trend tracking, not a literal /tmp write path.
• Lane inventory today is prompt=24 cases, execution=10 cases, dogfood=6 cases. Within execution, doctor-gated requires renderer support, color-grid can use renderer optionally, and the other 8 cases do not depend on Playwright/Chromium.
• Existing eval tests cover matrix/scoring/verifiers/anti-patterns, but there is no dedicated CLI parsing test for evals/run.ts.

Constraints and guiding choices

• Optimize Lane B execution feedback first; treat Lane A pass rates as relative deltas until recalibrated and Lane C as extended/aspirational.
• Keep code minimal and fail-fast: reuse assertString, invariant, strict schemas, and canonical ordering from SKILL_CONDITIONS.
• If any CLI JSON or report shape changes, update schemas/tests/docs in the same change.
• Do not add an external datastore first; expose existing artifacts and only add opt-in history tracking if the lightweight loop still feels insufficient.

Workstream 1 — Make condition comparison a first-class workflow (P0)

Files: evals/run.ts, evals/README.md, new test/unit/evals/run.test.ts (and test/unit/evals/matrix.test.ts only if matrix expectations need adjustment)

1. Change CLI parsing so --condition can be repeated, matching --case.

   if (isConditionFlag(argument)) {
     const parsed = parseOptionValue(...);
     options.conditions.push(parsed.value);
   }

2. Replace CliOptions.condition?: string with conditions: string[] and update resolveRequestedConditions() to:
   • accept zero, one, or many conditions,
   • treat no conditions as all four,
   • reject ambiguous input like --condition all --condition none,
   • normalize to canonical SKILL_CONDITIONS order before execution/reporting.
3. Update help text and README examples so the core optimization loop is one command, e.g. --condition none --condition preloaded.
4. Add fail-fast parser tests for repeated conditions, all exclusivity, invalid values, empty values, and dry-run matrix filtering.

Acceptance:

• A single invocation can compare none vs preloaded (and/or self-load) without shell loops.
• --dry-run, --json, and human output all reflect the selected condition set.
• Parser behavior is covered by targeted unit tests.

Workstream 2 — Surface the comparison signal we actually use (P1)

Files: evals/run.ts, evals/lib/reporting.ts, and evals/lib/schemas.ts only if metadata/report shape changes

1. Make the resolved output location explicit in the run summary and report metadata so users stop losing track of where artifacts landed.
2. Promote the existing comparison math from evals/lib/matrix.ts in report.md/report.json for the selected condition set:
   • condition-by-condition pass/fail counts,
   • efficiency deltas,
   • lift/harm metrics that matter for skill edits (realizedSkillLift, routingGap, staleSkillHarm) when the required conditions are present.
3. If JSON/report shape changes, update schemas and report consumers in the same patch.

Acceptance:

• A reviewer can open one report and immediately answer “did this skill edit improve the execution lane, and by how much?”
• The resolved output directory is visible in both machine-readable and human-readable outputs.
• Report changes preserve schema validation.

Deferred choice: long-term history storage

Current code already persists runs under evals/reports/{timestamp} or a user-supplied --output. The feedback about /tmp does not match the current implementation; the real issue is that workspace-local reports are easy to lose. Start by making --output/resolved paths obvious and by saving important runs to a stable location. Only add an opt-in history file or external store if the team still needs trend aggregation after the faster comparison loop is in place.

Workstream 3 — Make Lane B coverage status explicit and expand it in the right order (P2)

Files: evals/README.md, evals/execution/runner.ts if reporting needs better environment diagnostics, and evals/execution/cases/* or shared case metadata if the readiness labels live with the cases

1. Encode or at least document execution-case readiness:
   • battle-tested today: hello-prompt, crash-recovery, run-command,
   • next non-renderer smoke targets: resize-demo, alt-screen-demo, scrollback-demo, unicode-grid, export-proof,
   • renderer-optional: color-grid,
   • renderer-required: doctor-gated.
2. Make renderer requirements visible in docs and/or dry-run output so environment failures are not mistaken for skill regressions; if runner changes are needed, prefer an explicit environment-blocked outcome over a silent skip.
3. After P0 lands, run the non-renderer cases under real providers first, then add renderer-backed coverage once doctor --json is green.

Acceptance:

• The README/reporting surface clearly distinguishes trustworthy execution cases from cases that still need provider smoke coverage.
• Missing Chromium/Playwright support is easy to diagnose before running renderer-backed cases.
• The expansion order prioritizes the 5 non-renderer unproven cases before the renderer-dependent tail.

Workstream 4 — Recalibrate secondary lanes without blocking skill work (P3)

Files: evals/prompt/runner.ts, evals/prompt/cases/*, evals/dogfood/runner.ts, evals/dogfood/scorers/index.ts, evals/README.md

1. Lane A prompt scoring:
   • collect failing Claude/Codex examples,
   • relax response-style-specific rubrics where the intent is correct but wording varies,
   • keep deltas between conditions/providers as the main metric until calibration improves.
2. Lane C dogfood:
   • keep the lane explicitly non-blocking/extended,
   • consider a later split between “smoke” and “extended” budgets instead of forcing 10-minute runs to pass today.

Acceptance:

• Lane A README text and scoring rules make it clear that absolute pass rates are not ground truth yet.
• Lane C remains available for aspirational coverage without degrading the core skill-optimization loop.

Validation and quality gates

1. Targeted tests while implementing P0/P1:
   • mise run test -- test/unit/evals/run.test.ts
   • mise run test -- test/unit/evals/matrix.test.ts
   • any report/schema unit tests touched by reporting changes
2. Full repo validation before declaring success:
   • mise run test
   • mise run typecheck
   • mise run lint
   • mise run format-check if docs/report templates changed materially

Dogfooding and reviewer proof

1. Use an isolated home for every proof run:
   • mise install && mise run bootstrap (or the documented npm fallback),
   • AGENT_HOME="$(mktemp -d)".
2. Capture the core comparison workflow with real artifacts:
   • run npx tsx evals/run.ts --provider <provider> --lane execution --case hello-prompt --case run-command --case crash-recovery --condition none --condition preloaded --output <stable-dir>,
   • rerun with one or two newly smoked non-renderer cases once P2 begins.
3. Create a reviewer bundle under dogfood/ (or update an existing eval proof bundle) containing:
   • the exact commands used,
   • report.json and report.md,
   • at least one terminal screenshot,
   • at least one full-session recording export (WebM preferred; asciicast acceptable in addition),
   • any doctor --json output used to explain renderer availability.
4. Use the repo’s documented TUI workflow to capture proof artifacts:
   • agent-tty --home "$AGENT_HOME" doctor --json
   • agent-tty --home "$AGENT_HOME" create --json -- /bin/bash
   • agent-tty --home "$AGENT_HOME" run "$SESSION_ID" '<eval command>'
   • agent-tty --home "$AGENT_HOME" wait "$SESSION_ID" --text '<expected text>'
   • agent-tty --home "$AGENT_HOME" screenshot "$SESSION_ID"
   • agent-tty --home "$AGENT_HOME" record export "$SESSION_ID" --format webm
5. If renderer prerequisites are missing, still capture the failure cleanly:
   • save doctor --json,
   • smoke only the non-renderer execution cases,
   • mark renderer-backed cases as environment-blocked instead of skill-regressed.

Non-goals for the first pass

• No external metrics service or database.
• No full rewrite of Lane A or Lane C before the core execution comparison loop is smooth.
• No renderer/backend architecture changes unless later smoke coverage shows a real blocker.

---

Generated with mux • Model: anthropic:claude-opus-4-7 • Thinking: max