fix: harden evals — unit tests, verifier calibration, anti-pattern fixes

[ThomasK33]

fix: harden evals — unit tests, scoring overhaul, provider fixes

Addresses the weak points identified during initial evals dogfooding, then overhauls the scoring system based on real Claude Code and Codex eval runs.

What changed

Unit tests (120 tests across 5 files)

File	Tests	Coverage
scoring.test.ts	34	Pattern matching, negation context, prompt scoring, precision/recall
antiPatterns.test.ts	24	All 7 rules, negation context, missing-json-flag, orphaned-session
matrix.test.ts	23	Condition matrix, comparison metrics (lift, routing gap, stale harm)
verifiers.test.ts	26	Dispatch, snapshot, artifact, exit-code, event-log, bundle verifiers
codex.test.ts	13	Codex JSONL parsing, function_call events, command_execution items

Scoring overhaul (3 connected fixes)

1. Bash-only anti-pattern scanning — The anti-pattern scanner now analyzes only the agent's actual bash tool calls, not the full transcript including injected skill text. This eliminates false positives where the skill's own documentation (mentioning "tmux" as a contrast) was penalizing the agent.

2. Efficiency metric — New score component measures referenceSteps / max(referenceSteps, actualAgentTtyCalls). Each case defines minimum required agent-tty CLI calls. Perfect execution = 1.0, double = 0.5. This makes skill lift directly visible in the score.

3. Outcome-based pass/fail — Workflow checks (command sequence assertions) are now informational only. Pass/fail is determined by: provider invocation succeeded + all required verifiers passed + no error-level anti-patterns. This fixes run-command which failed 100% because it required agent-tty run but agents prefer type/send-keys.

Provider fixes

• Codex sandbox: Changed to --dangerously-bypass-approvals-and-sandbox (Codex 0.120.0 was blocked by bubblewrap)
• Codex tool call parsing: Fixed JSONL parser to extract function_call events and command_execution items, enabling efficiency counting
• Claude permissions: Added --allow-dangerously-skip-permissions before --dangerously-skip-permissions

Verifier calibration

• crash-recovery: Exit-code verifier → optional; status-observation verifier → required
• color-grid: Screenshot → optional; accepts snapshot or screenshot as evidence

📊 Real eval results: Claude opus 4.6 (high) vs Codex gpt-5.4

Pass/Fail Matrix

                  │ Claude opus 4.6 (high)  │ Codex gpt-5.4
                  │ no skill    with skill  │ no skill    with skill
──────────────────┼─────────────────────────┼────────────────────────
hello-prompt      │ ✓ 4.4/5     ✓ 4.7/5    │ ✗ 3.5/5     ✓ 5.0/5
crash-recovery    │ ✓ 4.3/5     ✓ 4.4/5    │ ✓ 4.2/5     ✓ 4.3/5
run-command       │ ✗ 3.0/5     ✓ 4.6/5    │ ✗ 3.3/5     ✓ 4.4/5

Pass rate:          2/3           3/3          1/3           3/3

Efficiency (agent-tty CLI calls)

Provider	Case	No skill	With skill	Reduction
Codex	hello-prompt	10 calls	2 calls	80%
Codex	crash-recovery	19 calls	9 calls	53%
Codex	run-command	17 calls	12 calls	29%
Claude	hello-prompt	12 calls	7 calls	42%
Claude	crash-recovery	10 calls	7 calls	30%
Claude	run-command	4 calls	8 calls	— (more, but succeeds)

Aggregate Skill Lift

Provider	Score without	Score with	Δ	Efficiency without	Efficiency with
Claude	3.91	4.59	+0.68	0.57	0.59
Codex	3.65	4.58	+0.93	0.32	0.58

Key findings

• Both providers can use agent-tty — they create sessions, interact with terminal apps, capture snapshots, and clean up
• The skill gives measurable lift — both providers go from partial to 3/3 pass rate with skill preloaded
• Codex benefits more (+0.93 score lift vs +0.68) — its base model doesn't know agent-tty, so the skill documentation has the most impact
• Codex with skill is remarkably efficient — hello-prompt in 2 CLI calls and 40 seconds
• Without skill, both struggle with run-command — they can complete the task but don't follow the optimal workflow

Validation

Check	Result
npm test	✅ 754 tests (67 files)
npx vitest run test/unit/evals/	✅ 120 tests (5 files)
npx tsc --noEmit	✅
npx eslint "evals/**/*.ts" --max-warnings=0	✅
npx prettier --check "evals/**/*.ts"	✅
Real Claude opus+high eval	✅ 3/3 preloaded
Real Codex gpt-5.4 eval	✅ 3/3 preloaded

---

Generated with mux • Model: anthropic:claude-opus-4-6 • Thinking: xhigh