feat(runner): ship the fresh-baseline measurement harness (#39)

[myselfsiddharth]

What this is

The harness half of #39: everything needed to measure the §9 kill-line's denominator — "what it costs a model to do the gate task from scratch" — except the measurement itself. Refs #39 (issue stays open; the live run is the remaining half).

What landed

• docs/gate/fresh-baseline.md — the written definition of "fresh reasoning" (page + goal only, no compiled trajectory, no cached locators, no step list), written before any number exists, per the issue's own checklist. States the token-accounting convention explicitly and the cost_fresh is a per-run field used as a one-time cost — #39 will flatten the amortization curve #123 correction (below). Ends with the protocol-record template (model id, effort, run count, date, testbed version, mean and spread) as [PENDING TRACK-1] placeholders.
• src/runner/fresh-baseline.ts — FreshBaselineClient interface + StubFreshBaselineClient (mirrors RepairModelClient/StubRepairModelClient: reports failure and zero tokens, never a no-op success).
• src/runner/fresh-baseline-anthropic.ts — AnthropicFreshBaselineClient, the real SDK wiring. One messages.create call per model turn (structured output, not tool-use, not prose), looping until the model reports done or DEFAULT_MAX_TURNS (30) is hit. Drives the page via the existing executeAction/capturePageState("interactive") — the same privacy-safe capture the repair model already gets (ADR-0012), so a fresh agent never sees more of the page than a repair does.
• src/runner/fresh-baseline-runner.ts — FreshBaselineRunner, the client-agnostic orchestration layer: measures wall-clock around one attempt, sanitizes the reported numbers, emits a RunMetric row through MetricsEmitter.
• experiments/gate-v1/fresh-baseline.ts (npm run gate:baseline) — a separate entry point, not a mode of ReplayRunner/run-matrix.ts. Same --dry-run/live split and bring-up/seed/browser/teardown conventions as run-matrix.ts/live-run.ts, reusing the same testbed/preamble code rather than re-implementing it. Writes out/fresh-baseline/metrics.ndjson and out/fresh-baseline/baseline.json (the protocol record + mean/spread).
• gate:matrix --cost-fresh <path> — reads a measured baseline.json and attaches its mean_cost_fresh to every live run row's cost_fresh via ReplayRunner's existing costFresh option (experiments/gate-v1/run-matrix.ts, live-run.ts). Refuses (exit 2) unless the baseline is usable: true — never silently falls back to a zero that would look identical to never having asked. Ignored under --dry-run.
• package.json: gate:baseline script.

Why fresh-baseline rows never reach gate:matrix's own NDJSON

FreshBaselineRunner emits through the same MetricsEmitter class, but to out/fresh-baseline/metrics.ndjson — never out/metrics.ndjson. A fresh attempt's RunMetric row has no compiled steps and no repair loop, so steps_total/cost_repair/etc. are honest zeros — but repairCostVsFresh() and taskSuccessLe2Repairs() pool every run row in the file they're given, unconditionally. Mixing the two files would silently dilute the real matrix's success rate and mean repair cost. gate:report hard-codes out/metrics.ndjson as its only input, so the two stay apart by construction. Full reasoning in the module doc-comment and docs/gate/fresh-baseline.md.

Why three files under src/runner/ (not one)

Mirrors the existing repair.ts / repair-anthropic.ts split, applied a second time because both clients now share the same constraint (§9 wants them measured identically):

• fresh-baseline.ts — the seam (interface + stub). Lets FreshBaselineRunner depend on an abstraction, never the SDK.
• fresh-baseline-anthropic.ts — everything that touches @anthropic-ai/sdk and nothing else. A reviewer auditing "does this ever make a network call" has exactly one file to read.
• fresh-baseline-runner.ts — orchestration (measure, sanitize, emit), client-agnostic. Tested with a hand-written fake client and never touches the SDK — collapsing this into the Anthropic client would force every cost-mapping/zeros-on-failure test to carry SDK mock scaffolding it doesn't need, the same tax repair-anthropic.test.ts already avoids for the repair side.

The alternative (one file) was considered; fresh-baseline-anthropic.ts alone is ~350 lines of request-shape/turn-loop/error-path logic that runner-level tests have no reason to exercise through a mocked Anthropic client. This isn't a new pattern invented for #39 — it's the existing one, applied because the constraint is now shared.

Token-accounting convention — matched to repair-anthropic.ts exactly

• billedInputTokens() is imported from repair-anthropic.ts, not reimplemented — sums input_tokens + cache_read_input_tokens + cache_creation_input_tokens.
• Prompt caching off on both clients (no cache_control).
• No retries on either — a failed turn/call reports the tokens it actually billed (zero for the part that never returned), never a guess.
• No server-side fallbacks, no temperature/top_p/top_k on either.
• Missing ANTHROPIC_API_KEY throws at construction on both (MissingAnthropicKeyError / MissingFreshBaselineKeyError) rather than degrading to a stub that would look measured.
• Same default model — DEFAULT_FRESH_MODEL = DEFAULT_REPAIR_MODEL — so the ratio isn't secretly comparing two different reasoning systems.

Full table in docs/gate/fresh-baseline.md.

The #123 / ADR-0010 correction

Issue #39's step 4, as filed, predates #123: it says to attach cost_fresh so that repairCostVsFresh() and amortizedTokensOverN() compute. That's no longer correct at HEAD — ADR-0010 split the field. This PR follows the code, not the issue's original wording: --cost-fresh feeds repairCostVsFresh() only. amortizedTokensOverN()'s numerator is cost_program_build (a separate, unmeasured, one-time capital cost) and nothing here wires it or measures it. Stated explicitly in docs/gate/fresh-baseline.md.

What did NOT land (by design)

• No live measurement. No ANTHROPIC_API_KEY was set or used anywhere in this session. cost_fresh stays zeros; gate:report's repair cost vs fresh and amortized tokens/task stay no_data. Per CONTRIBUTING rule 3, there is no number to invent, and none is invented — every protocol-record field in the doc is [PENDING TRACK-1].
• The live measurement (≥3 fresh runs, mean and spread, model id/effort/date/testbed version recorded, real spend) is separate follow-up work against this harness.
• An independent success oracle for the fresh agent's done: true, success: true self-report — flagged as an open question, not built. The model's own claim is what's measured today; nothing external checks it.

Test output

npm run ci — green (secret-scan, validate:contracts, lint, lint:docs, typecheck, 500 unit tests across 35 files, 26 integration tests across 5 files). New coverage: tests/unit/fresh-baseline-anthropic.test.ts (19 tests, real headless browser + mocked SDK — token accounting across turns, no-retry, missing-page zeros, structured-output shape, privacy of what's sent), tests/unit/fresh-baseline-runner.test.ts (11 tests — cost mapping, model_id propagation, and the case #39 calls out by name: a client that throws produces an all-zero cost_fresh row with an explicit note, never partial garbage), tests/unit/fresh-baseline-summary.test.ts (11 tests — mean/spread over measured runs only, usable flag), and a loadCostFreshBaseline block added to tests/unit/gate-matrix.test.ts (6 tests — refuses a missing/invalid/not-usable baseline by name rather than degrading silently).

npm run test:canary — green (52 tests, 8 files). No network calls anywhere in either suite.

Test Files  35 passed (35)  |  Tests  500 passed (500)   [ci: test]
Test Files   5 passed (5)   |  Tests   26 passed (26)    [ci: test:integration]
Test Files   8 passed (8)   |  Tests   52 passed (52)    [test:canary]

Branch: track1/b4-fresh-baseline, rebased onto origin/main after #160/#161 landed (both docs/README.md rows kept).

🤖 Generated with Claude Code

[myselfsiddharth]

Opus review — approved

Reviewed the seam, the runner, the Anthropic client's accounting, the matrix wiring, and the definition doc. This is the strongest of today's four PRs and I want to be specific about the two places it did better than the brief asked.

1. The NDJSON separation is a hazard the issue never mentioned, and it is real. FreshBaselineRunner emits a schema-valid RunMetric whose steps_total and repair_count are zero — and repairCostVsFresh() / taskSuccessLe2Repairs() pool every run row in whatever file they are handed, unconditionally. Mixed into the matrix's NDJSON, each fresh attempt would land as a zero-repair, zero-step "run" and quietly dilute both the success rate and the mean repair cost. I verified the separation actually holds rather than trusting the comment: generate-amortized.ts:238 hard-codes out/metrics.ndjson, while the entry point writes out/fresh-baseline/metrics.ndjson. Different file, no glob, no collision. Finding that before it corrupted a number is worth more than the harness itself.

2. The accounting convention is shared, not merely matched. billedInputTokens is imported from repair-anthropic.ts (line 48), not reimplemented, and DEFAULT_FRESH_MODEL = DEFAULT_REPAIR_MODEL. The issue asked for the conventions to be identical; making them the same function means they cannot drift apart later, which is the difference between a convention and a guarantee. Cache-read and cache-creation tokens are therefore summed on both sides by construction.

Refuses rather than degrades. loadCostFreshBaseline() throws on a missing file, unparsable JSON, or usable !== true, each naming why. A dry-run baseline cannot be wired into a gate run silently — correct, and consistent with how the cache-MISS and unbound-param checks already behave.

Honest about what it is not. cost_fresh stays zeros, the aggregates stay no_data, every protocol field in docs/gate/fresh-baseline.md reads [PENDING TRACK-1], and the doc carries the #123/ADR-0010 correction explaining why amortizedTokensOverN is untouched (its numerator is cost_program_build, not cost_fresh). No live call was made and no number was invented. Refs #39, so the issue stays open for the measurement — which is the actual remaining work.

File layout — three modules mirroring repair.ts / repair-anthropic.ts / orchestration. The justification holds: keeping the runner client-agnostic is what lets fresh-baseline-runner.test.ts exercise the zeros-on-failure path with a hand-written fake and never touch the SDK. I'd have questioned a two-module split more than this three-module one.

Out of scope, now filed as #165

The flagged assignValue gap is real and I confirmed it: --repair-model, --from-cache, --site-key, and --task-key sit in parseArgs's valued set, have their values consumed, and are then dropped by assignValue, which has no branch for any of them. No error — they are in valued, so they miss the unknown argument throw.

--repair-model is the dangerous one: a run invoked with it silently uses StubRepairModelClient, reporting self-heal 0 and cost_repair zero, which against a 70%-of-fresh kill line reads as a spectacular pass. That is the exact failure AnthropicRepairModelClient's constructor-time throw exists to prevent — the CLI degrades silently in front of a client that refuses to. Filed as #165 with the structural fix (the valued set and assignValue's branches are two lists that must agree and don't) rather than a patch of four branches.

Correct call to leave it untouched here. Fixing it in this PR would have mixed an unrelated severity-1 correctness fix into a harness PR.

Verdict: approve, merge on green CI.

[myselfsiddharth]

Opus re-review — approved, merging

The integration break is fixed correctly and documented.

experiments/gate-v1/fresh-baseline.ts:424 now calls SessionAuthorization.authorize(baseUrl) with no consent argument, matching the reasoning and comment style of the two existing call sites — the fresh baseline runs against the local test-bed with fixture credentials the project owns, so it authorizes unconditionally.

The doc note is the part I'd have accepted the PR without and am glad is there: docs/gate/fresh-baseline.md now has "Pointing this harness at a non-local target requires recorded consent," explaining that a non-local fresh run would refuse with ConsentRequiredError and that this is correct behaviour rather than an obstacle to route around. A future reader aiming this harness at a real site learns it from the doc instead of from a thrown error.

Worth recording why this break happened, because it is the one class of failure a green local run cannot catch: #162 changed establishSession's signature while this PR was open, and this PR introduced a third call site that did not exist when #162 was written. Neither author could have seen the other's change, and git auto-merged the two cleanly because they touch different files. Only a typecheck against the combined state catches it — which is why this branch was updated and re-run rather than merged from a BEHIND status.

That is also the argument for #162's approach. A convention would have let this through; a required SessionAuthorization parameter made it a build error.

All checks green: lint-typecheck-test-secrets, privacy-canary, conventional-commit title, CodeQL, testbed-smoke. Merging via admin bypass — #39 stays open for the measurement itself.