feat: safely parallelize evals + add statistical rigor for measuring skill changes

[ThomasK33]

Safely parallelize evals + add statistical rigor for measuring skill changes

Summary

This PR delivers two major capabilities for the eval runner:

1. Safe parallelization — --concurrency <n> for within-lane and cross-lane parallel execution
2. Statistical rigor — --trials N, trial aggregation with CIs, and --compare-baseline for paired A/B comparison

Why

A single eval run has a ~16% baseline pass/fail flip rate (measured empirically — two identical serial runs disagree on ~16% of cases). This means you cannot tell whether a skill change helped by comparing two single runs. The eval system needed:

• Multi-trial support to average out noise
• Confidence intervals to quantify uncertainty
• Paired comparison with practical significance thresholds
• And parallelism to keep multi-trial runs affordable time-wise

What changed

Parallelization (Phases 0–5)

Commit	What
Work-item identity + uniqueness assertions	Stable lane:caseId:condition:trial keys
finally cleanup blocks	Execution + dogfood runners clean up temp dirs
Shared bounded scheduler	runScheduled() with all-settled semantics
Lane refactor (all 3 lanes)	Split into enumerate*WorkItems() + execute*WorkItem()
--concurrency <n> CLI flag	Default: 1 (serial), safe up to 20
Cross-lane concurrency	Lanes run in parallel when concurrency > 1

Measured speedups:

• Prompt lane (24 cases, codex): 7m28s → 32s at c=20 (14×)
• Full matrix (160 items): 437min serial equivalent → 16m wall (26.6×)

Parallelism does NOT cause regressions — empirically confirmed:

• Codex serial-vs-serial flip rate: 16.7%
• Codex serial-vs-parallel flip rate: 13.5% (lower!)
• Zero shared regressions between providers

Statistical rigor

Commit	What
--trials N	Trial loops in all 3 lanes (prompt already had infra, execution + dogfood added)
evals/lib/statistics.ts	Pure stats: mean, stddev, CI (t-dist), Wilson interval, paired delta, bootstrap, win rate
Trial aggregation	Reports include per-case pass rate, CI, stddev when trials > 1
--compare-baseline <path>	Paired A/B comparison with bootstrap CIs and verdicts
Eval guide skill	.mux/skills/eval-guide/SKILL.md with learnings and recommended workflows

Verdict system:

• improved = CI excludes 0 AND effect ≥ practical threshold (5pp pass rate / 0.05 score)
• regressed = CI upper bound < 0
• inconclusive = CI crosses 0

Dogfood validation: Two identical 3-trial runs correctly produce inconclusive verdict (23/24 cases inconclusive, 14W/15L/43T — pure noise correctly identified).

New CLI flags

--concurrency <n>        Max work items per lane (default: 1)
--trials <n>             Independent trials per case/condition (default: 1)
--compare-baseline <path> Path to a previous report.json for paired comparison

How to use

# Run baseline with trials + parallelism
BASELINE=$(npx tsx evals/run.ts \
  --provider codex --lane prompt --trials 5 --concurrency 4 \
  --json | jq -r '.jsonReportPath')

# Make skill change, then run candidate with comparison
npx tsx evals/run.ts \
  --provider codex --lane prompt --trials 5 --concurrency 4 \
  --compare-baseline "$BASELINE" --json

Files changed (20 files, +4,080/−527)

New files:

• evals/lib/scheduler.ts — bounded work-item scheduler
• evals/lib/statistics.ts — pure statistics helpers (zero external deps)
• .mux/skills/eval-guide/SKILL.md — learnings + recommended workflows
• test/unit/evals/scheduler.test.ts — 8 tests
• test/unit/evals/statistics.test.ts — 20 tests
• test/unit/evals/reporting.test.ts — 5 tests
• test/unit/evals/workItem.test.ts — 4 tests
• test/unit/evals/promptRunner.test.ts — 5 tests
• test/unit/evals/executionRunner.test.ts — 3 tests
• test/unit/evals/dogfoodRunner.test.ts — 3 tests
• test/integration/evals-parallelization.test.ts — parity test
• test/integration/evals-trials.test.ts — trials wiring test
• test/integration/evals-compare-baseline.test.ts — A/B comparison test

Modified files:

• evals/run.ts — CLI flags, lane dispatch, cross-lane concurrency
• evals/prompt/runner.ts — enumerate + execute + schedule
• evals/execution/runner.ts — enumerate + execute + schedule + cleanup
• evals/dogfood/runner.ts — enumerate + execute + schedule + cleanup
• evals/lib/types.ts — work-item identity, aggregation types, comparison types
• evals/lib/schemas.ts — Zod schemas for new types
• evals/lib/reporting.ts — aggregation, comparison, markdown generation
• evals/lib/matrix.ts — reuse shared computeMean

Validation

Check	Result
npm run typecheck	✅
npm run test	✅ 77 files, 812 tests
npm run lint	✅ 0 errors, 0 warnings
npm run format:check	✅
Stub parity (c=1 vs c=4)	✅ Identical results
Real provider parallelism	✅ Codex + Claude, 160 items each
Baseline non-determinism measured	✅ ~16% flip rate both providers
A/B comparison dogfood	✅ Same-skill → inconclusive verdict

---

📋 Implementation Plan

Plan: safely parallelize evals without breaking determinism or isolation

Recommendation

Do not parallelize evals by sprinkling Promise.all() across evals/run.ts or by running the three lanes independently with ad hoc lane-level concurrency.

Instead, ship this as a scheduler + invariants rollout:

1. Harden isolation and cleanup first while behavior stays serial.
2. Refactor each lane into explicit work items that still run with concurrency = 1.
3. Introduce one shared bounded scheduler with global, provider, and task-kind caps.
4. Roll out by risk level: prompt lane first, execution lane second, dogfood lane last.
5. Only after that, decide whether to allow cross-lane concurrency by feeding multiple lanes into the same scheduler.

That path keeps the current deterministic report model, minimizes artifact collisions, and gives us a single place to enforce provider throttling, logging, cleanup, and error semantics.

Why this plan is the safe one

Evidence from the current repo

• evals/run.ts currently runs lanes sequentially in a for...of loop and aggregates results after each await runLane(...).
• evals/prompt/runner.ts, evals/execution/runner.ts, and evals/dogfood/runner.ts all execute case/condition work serially with nested loops.
• evals/execution/runner.ts already creates isolated temp homes via createIsolatedEvalHome() and isolated output directories; evals/dogfood/runner.ts uses run-id/provider/case/condition-scoped paths.
• evals/lib/artifacts.ts already gives us stable run IDs and validated paths, and reporting sorts results later, so final report generation does not depend on push order.
• evals/lib/cliHarness.ts has cleanup helpers, but current runners do not consistently guarantee cleanup in finally.
• Providers look mostly stateless at the interface level, but there is no built-in rate limiting/backoff, so naive concurrency would increase API-throttling risk.
• Verbose logging currently writes directly to shared stderr, which would become noisy/interleaved under concurrency.
• Existing eval tests cover scoring/matrix/verifiers/providers, but there is little to no direct runner/orchestrator coverage for concurrent execution semantics.

Goals

• Reduce eval wall-clock time safely.
• Preserve deterministic result ordering and report contents.
• Preserve per-case isolation for AGENT_TTY_HOME, temp output, bundles, and artifacts.
• Keep failure handling all-settled, not fail-fast.
• Make concurrency explicit, bounded, and easy to disable.

Non-goals for the first rollout

• Maximizing raw throughput at the expense of provider stability.
• Introducing separate concurrency models for lanes vs. cases.
• Parallelizing inside a single execution/dogfood case.
• Making real-provider concurrency the default on day one.

Proposed architecture

1) Introduce an explicit work-item model

Add a small scheduling abstraction under evals/lib/ and stop treating each lane as an opaque sequential loop.

Likely new concepts:

interface EvalWorkItem {
  key: string; // lane + caseId + condition + trialIndex
  lane: 'prompt' | 'execution' | 'dogfood';
  kind: 'plan' | 'agent';
  caseId: string;
  condition: string;
  trialIndex?: number;
}

interface EvalWorkOutput {
  result: EvalResult;
  logs: string[];
  metadataNotes: string[];
}

Guidance:

• Build work items in lane-specific enumerators:
  • prompt: one work item per case/condition/trial.
  • execution: one work item per case/condition.
  • dogfood: one work item per case/condition.
• Make each runner expose two layers:
  • enumerate*WorkItems(...)
  • execute*WorkItem(...)
• Keep report generation and final artifact writing single-threaded at the end.

2) Use one shared bounded scheduler

Add a scheduler helper, e.g. evals/lib/scheduler.ts, that accepts all work items and enforces:

• a global concurrency cap,
• a provider-specific cap,
• and optionally a task-kind cap (plan vs agent).

Recommended policy:

• default CLI behavior remains concurrency = 1, so current semantics stay unchanged;
• parallel mode is opt-in at first;
• the scheduler returns all-settled results so one failed work item does not cancel siblings.

A rough target shape:

const work = buildEvalWorkPlan(options, metadata);
const scheduler = createEvalScheduler({
  globalConcurrency,
  providerCaps,
  kindCaps,
});
const settled = await scheduler.run(work, executeWorkItem);
const results = sortResults(settled.map((item) => item.result));

3) Preserve isolation boundaries

Parallelize only at the current safe boundaries:

• Prompt lane: case/condition/trial work item.
• Execution lane: whole case+condition work item.
• Dogfood lane: whole case+condition work item.

Do not split a single execution/dogfood case into smaller concurrent subtasks in the first rollout. The per-case isolated home/output directory is the right cleanup and failure boundary.

4) Make concurrency observable and controllable

In evals/run.ts and the CLI option parsing/types:

• add a bounded --concurrency <n> option with default 1;
• consider keeping n > 1 experimental for the first merge if we want an easier rollback story;
• avoid adding a second flag like --lane-concurrency unless a real need appears later.

Phased implementation plan

Phase 0 — harden invariants with no behavior change

Files/symbols to touch

• evals/run.ts
• evals/lib/cliHarness.ts
• evals/lib/artifacts.ts
• lane runners under evals/prompt/runner.ts, evals/execution/runner.ts, evals/dogfood/runner.ts
• tests under test/unit/evals/ and test/integration/

Work

• Define a stable task identity: lane + caseId + condition + trialIndex.
• Assert identity uniqueness during work-plan construction.
• Guarantee temp-home/session cleanup with finally, especially around createIsolatedEvalHome() call sites.
• Stop mutating shared arrays/metadata structures directly from hot loops; collect per-task notes/errors first, then merge.
• Verify that artifact paths are unique at the work-item granularity; if any pathing still keys only on caseId, lane-qualify it or add an invariant/assert.
• Decide how verbose logs remain readable under concurrency:
  • simplest safe option: buffer per-task logs and flush with a task prefix after completion;
  • acceptable fallback: prefix every line with task identity.

Quality gate before Phase 1

• Existing serial behavior still passes with concurrency = 1.
• New tests prove no temp-home leaks and no artifact-path collisions for repeated serial runs.
• No user-facing output changes except possibly better-prefixed verbose logs.

Phase 1 — refactor to work-plan execution, still serial

Files/symbols to touch

• evals/run.ts
• new evals/lib/scheduler.ts or evals/lib/workPlan.ts
• evals/prompt/runner.ts
• evals/execution/runner.ts
• evals/dogfood/runner.ts

Work

• Split each lane into:
  • work-item enumeration,
  • single-work-item execution,
  • lane-level result normalization.
• Keep the executor serial (concurrency = 1) even if the scheduler abstraction exists.
• Sort results by stable identity before JSON/Markdown report generation, rather than relying on push order.
• Keep run metadata/report generation at the end of the run, not inside workers.

Quality gate before Phase 2

• Stub-provider serial runs produce the same sorted results/report contents before and after the refactor.
• New runner/orchestrator tests cover enumeration, error isolation, and stable ordering.
• No new cleanup regressions.

Phase 2 — enable prompt-lane parallelism first

Why first

• Prompt work is the lowest-risk surface: no AGENT_TTY_HOME, no session cleanup, and fewer subprocess/resource concerns.

Files/symbols to touch

• evals/prompt/runner.ts
• evals/run.ts
• provider-related config/types as needed
• test suites for prompt runner/orchestrator parity

Work

• Submit prompt work items through the shared scheduler.
• Add conservative concurrency caps for real providers; allow higher caps for stub/fixture providers.
• Keep failure handling all-settled so one failed trial returns a failed EvalResult without canceling other prompt work.
• Ensure verbose output stays readable with prefixed/buffered logs.

Quality gate before Phase 3

• stub provider: serial vs parallel prompt runs have parity for sorted results and scores.
• Add a stress test that executes multiple prompt work items in parallel and proves deterministic output ordering after sorting.
• Real-provider dogfood run uses a low cap and shows no obvious throttling storm.

Phase 3 — parallelize execution lane at whole-case granularity

Why second

• Execution work already has the right isolation boundary per case, but it is heavier because it creates homes, sessions, transcripts, stdout/stderr, and verifiers.

Files/symbols to touch

• evals/execution/runner.ts
• evals/lib/cliHarness.ts
• evals/lib/artifacts.ts
• possibly provider/task-kind concurrency config
• integration/e2e tests that exercise isolated homes and artifact writes

Work

• Parallelize only one execution case+condition per worker.
• Guarantee cleanupEvalHome() (or equivalent) in finally.
• Keep agent/process-heavy work under a lower cap than prompt work.
• Confirm verifiers and artifact writes are safe when multiple cases finish near-simultaneously.
• Add defensive assertions around paths, manifest existence, and cleanup preconditions.

Quality gate before Phase 4

• Parallel execution runs do not leak temp homes/sessions.
• Parallel execution runs do not corrupt transcripts, stdout/stderr artifacts, or per-case results.
• Repeated stub/fixture runs show stable sorted report parity against serial baseline.

Phase 4 — parallelize dogfood lane last

Why last

• Dogfood is artifact-heavy and reviewer-facing; mistakes here are harder to unwind because they affect proof bundles and bundle organization.

Files/symbols to touch

• evals/dogfood/runner.ts
• bundle/artifact helpers it uses
• dogfood/README.md and/or dogfood/CATALOG.md if proof-bundle workflows change
• eval docs/tests covering bundle layout

Work

• Parallelize at one dogfood case+condition per worker.
• Confirm bundle paths remain unique under the current runId/provider/case/condition directory structure.
• Ensure screenshots/recordings/proof artifacts are still attached to the correct case-condition pair.
• Keep concurrency cap conservative until bundle parity is proven.

Quality gate before Phase 5

• No bundle collisions or missing artifacts under repeated parallel dogfood runs.
• Reviewer-facing proof bundles remain navigable and deterministic.
• Dogfood reports continue to map failures to the correct work-item identity.

Phase 5 — optional cross-lane concurrency via the shared scheduler

Recommendation

• Treat this as optional, not required for the first safe rollout.
• If we want it, do it by feeding work items from multiple lanes into the same scheduler rather than adding a separate Promise.all(lanes) path.

Work

• Allow buildEvalWorkPlan(...) to emit work items across all selected lanes.
• Keep per-kind/provider caps so prompt work cannot starve execution/dogfood work or overload providers.
• Preserve lane-level summaries by deriving them from completed work-item results rather than shared mutable lane arrays.

Quality gate

• Cross-lane parallel runs preserve final report parity and do not regress logs or cleanup.
• A mixed-lane run respects provider and task-kind caps.

Testing strategy

Unit tests

Add focused coverage under test/unit/evals/ for:

• work-item identity generation and uniqueness,
• scheduler all-settled behavior,
• result sorting/deterministic ordering,
• metadata note/error merging,
• prompt parallel parity,
• cleanup behavior and path assertions,
• provider cap selection logic.

Likely new/expanded suites:

• test/unit/evals/runners.test.ts
• test/unit/evals/scheduler.test.ts
• test/unit/evals/isolation.test.ts
• test/unit/evals/provider-concurrency.test.ts

Integration tests

Add integration coverage for:

• serial baseline vs --concurrency > 1 parity on stub,
• repeated execution runs proving isolated AGENT_TTY_HOME usage,
• no artifact collision when multiple work items finish concurrently,
• lane error recovery under all-settled semantics,
• cleanup verification after parallel runs.

Likely suites:

• test/integration/evals-lanes.test.ts
• test/integration/evals-parallelization.test.ts

CI and task-runner updates

If the implementation lands, update:

• .github/workflows/ci.yml
• mise.toml
• evals/README.md

Recommended CI rollout:

• first add a serial stub-provider eval smoke test,
• then add a low-concurrency parallel stub-provider smoke test after parity is proven locally.

Dogfooding plan

Because this is a CLI project, dogfooding should be done against isolated homes and captured as reviewable terminal artifacts.

Setup

• Use an absolute isolated AGENT_TTY_HOME for each dogfood run.
• Prefer the repo-local CLI during development:
  • npx tsx evals/run.ts ... for the eval runner,
  • npx tsx src/cli/main.ts ... if using agent-tty itself to capture proof artifacts.
• Start with the stub provider; use real providers only after the stub parity gates pass.

Dogfood scenarios

1. Serial baseline
   • run selected evals with --concurrency 1;
   • save JSON output, report artifacts, and wall-clock timing.
2. Prompt-only parallel
   • repeat the same selection with prompt lane parallelism enabled;
   • compare sorted reports, timing, and error mapping.
3. Execution parallel
   • run a small execution-only matrix at low concurrency;
   • verify cleanup, per-case outputs, and stable report contents.
4. Dogfood parallel
   • run a small dogfood matrix at low concurrency;
   • verify bundle paths, screenshots, recordings, and final report links.
5. Mixed-lane run (only if Phase 5 is implemented)
   • run at a conservative global cap and verify provider/task-kind caps are respected.

Proof artifacts reviewers should receive

For every dogfood phase above, produce:

• at least one terminal screenshot showing the command and successful completion,
• one video recording/WebM of the run or replay,
• the JSON result/report artifact from the eval run,
• a short timing comparison (serial vs parallel),
• and a note confirming whether temp homes/sessions were cleaned up.

Recommended capture workflow:

1. Launch the eval command inside an agent-tty session.
2. Wait for completion with the machine-readable CLI path.
3. Capture a semantic snapshot and a PNG screenshot.
4. Export a WebM recording for reviewer playback.
5. Store the proof bundle under dogfood/ with a clear run label for the phase and concurrency setting.

Acceptance criteria

The implementation should not be considered complete until all of the following are true:

• --concurrency defaults to 1, preserving current behavior.
• Serial and parallel stub-provider runs have parity for sorted results, scores, and reports.
• Temp homes/sessions are cleaned up reliably after execution/dogfood work.
• No artifact collisions occur under repeated parallel stress runs.
• Provider/task-kind caps prevent unbounded concurrency.
• Verbose logs remain readable and attributable to a specific work item.
• Documentation explains the concurrency model, safety limits, and recommended usage.
• CI includes at least one eval smoke test that exercises the new scheduler path.

Risks and mitigations

• Provider throttling -> start with conservative caps and explicit provider-aware throttles.
• Temp-home leaks/orphaned sessions -> enforce cleanup in finally and add integration tests.
• Artifact collisions -> assert unique work-item identity and unique paths before writing.
• Log interleaving -> buffer or prefix logs per task.
• Hidden shared-state assumptions -> refactor to work items while still serial first, then parallelize only after parity is proven.

Suggested delivery order

1. Invariants + cleanup hardening.
2. Serial work-plan refactor.
3. Shared bounded scheduler behind --concurrency.
4. Prompt-lane parallel rollout.
5. Execution-lane parallel rollout.
6. Dogfood-lane parallel rollout.
7. Optional cross-lane concurrency.

---

Generated with mux • Model: anthropic:claude-opus-4-6 • Thinking: xhigh