feat(runner): real Anthropic repair client, opt-in, with token accounting

[OM152002]

Closes #27 except its stated exit criterion — see "What I could not do" at the bottom, which is the first thing to read.

Rebased onto main now that #150 has landed. #150 was squash-merged, so its original commits are not ancestors of main and the auto-retarget alone left this PR replaying them (22 files). git rebase --onto main drops it back to the 8 files that are actually this change. serializeRepairContext(), which this imports, now comes from main.

StubRepairModelClient proposes null and reports zero tokens, so self-heal rate is structurally 0 and cost_repair structurally zero — blocking two §9 metrics outright, one of which is a ratio with a kill line at 70%.

What it sees, and what it cannot

AnthropicRepairModelClient consumes only serializeRepairContext()'s output. It never touches RepairContext, which carries params — the runtime bindings with secrets in them. A client trusted to pick the safe fields itself is a convention, not a boundary, and tests/canary/repair-egress.test.ts is merge-blocking on exactly that.

Opt-in, everywhere

The stub stays the default: npm run ci, dry runs, and every existing path make no network call and spend nothing. Enabling it is gate:matrix --repair-model <m>, and the client throws at construction when ANTHROPIC_API_KEY is unset rather than degrading — a run that silently used the stub would report a self-heal rate of 0 that looks measured, which is worse than a run that failed.

The decisions that move a verdict

Decision	Why
Prompt caching off	cache_read_input_tokens / cache_creation_input_tokens bill differently from plain input. A repair cost quietly excluding cache writes would understate against the 70% line. All four fields are summed anyway, so enabling caching later cannot silently change what the number means
A refusal reports the tokens it burned	A 200-with-refusal returns null with its consumed tokens, never zero — free-looking repair understates against the same line. The catch path is the one exception and reports zero deliberately: a thrown error carries no usage to read, so zero is the honest floor rather than a guess. Neither path is ever retried silently — a hidden retry hides cost
stop_reason checked before reading content	A decline is HTTP 200 with possibly empty content; indexing content[0] would throw
A proposal carrying an assertion is dropped whole	Partially honouring it would look like a repair while corrupting the measurement. The output schema offers no assertion field at all, so the ask is never made; assertAssertionUnchanged stays the runtime guard
model_id and effort on every proposal	A cost figure without the model and effort that produced it is not reproducible
No server-side fallbacks	A fallback would let a different model serve the repair while model_id is doing reproducibility work for cost_repair — the recorded model would stop being the one that was billed. A refusal is reported as a refusal instead
effort typed as the SDK's own union	Typing it string forced a cast on the whole request object, taking output_config, messages, system, and max_tokens out of the checker too. No live call has been made, so the compiler is the only thing between a malformed request and the first run that spends money. A typo ("maximum") now fails at build — verified by sabotage (TS2322)
No minimum on wait_ms	Numerical constraints aren't supported by structured outputs, and schemas compile server-side on first use — a rejection would land on exactly the paid call this is saving up for. wait_ms is optional and unvalidated by sanitizeProposedAction regardless

max_tokens caps adaptive thinking and response text together on claude-opus-5; the docstring says so, because a truncation there is a paid call that yields nothing and surfaces as "no text block in response".

Tests: 21, SDK injected, no network

Guard-proven, each sabotage run and observed:

Sabotage	Result
Drop cache tokens from billed input	1 fails
Merge a tampering proposal instead of dropping it	4 fail
Report zero tokens on a refusal	1 fails
DEFAULT_EFFORT = "maximum"	typecheck fails (TS2322)

new Anthropic( appears in exactly one place in the tree — the production path. Every test injects a fake.

What I could not do

#27's exit criterion is "a single observed real self-heal", with the metric row quoted. No ANTHROPIC_API_KEY was available and running it spends money, so no live repair has been observed and none is claimed. Self-heal rate remains structurally 0 until someone runs:

export ANTHROPIC_API_KEY=...
npm run gate:matrix -- --versions 9.5.21 --repair-model claude-opus-5 \
  --program artifacts/compiled/traj-gate-live-create-stat-dashboard-from-testdata-9.5.21.bundle.json

then confirms in out/metrics.ndjson: at least one REPAIRED_PASS, non-zero tokens_in/tokens_out in cost_repair, and model_id populated. docs/gate/runner.md records this as built but unmeasured rather than resolved, and the issue should stay open on that item.

I could have written a plausible-looking metric row. This repo's whole culture is that an unearned number is worse than a missing one.

Dependency

Adds @anthropic-ai/sdk — mandated by the issue as the only permitted way to call the API. npm audit --omit=dev reports 0 vulnerabilities. The lockfile also picks up main's promotion of ajv/ajv-formats to runtime deps, which happened after this branch was cut.

One thing worth its own issue

scripts/secret-scan.mjs's env-assignment pattern flags const apiKey = process.env["ANTHROPIC_API_KEY"] — the correct way to read a key. Fourth trip on that pattern this session (see #100 and the note in tests/unit/page-context.test.ts). Worked around by naming the local credential, with a comment. The pattern is over-broad for source code and teaches people to rename variables rather than fix leaks.

npm run ci            # green — 453 unit, 26 integration, secret-scan clean, lint-docs clean
npm run test:canary   # 52 pass

🤖 Generated with Claude Code

[myselfsiddharth]

Review — not merging yet; needs a rebase, and one fix I'd want before a paid call

#150 is merged, so GitHub auto-retargeted this to main. That is not enough on its own — see (1). The code itself is good and I verified it runs; the blockers are mechanical plus one type-safety issue that matters specifically because the exit criterion is unmet.

The "What I could not do" framing is the right call, and npm audit --omit=dev before adding the first non-Playwright runtime dep is the right instinct.

---

1. Blocker — this needs a rebase onto main, not just the retarget

#150 was squash-merged (repo has allow_merge_commit: false, delete_branch_on_merge: true), so its original commits are not ancestors of main. This branch was cut from them, so after the auto-retarget the diff replays them:

$ gh pr diff 151 --name-only | wc -l
22

That is your 8 files plus all of #150's — ADR-0012-repair-context-budget.md, src/shared/page-context.ts, src/runner/repair-egress.ts, contracts/metrics.schema.json, and the rest. Merging as-is would re-apply changes that are already on main.

git rebase --onto main <old-base> HEAD + force-push drops it back to the 8 files that are actually yours.

2. Blocker — no CI has ever run on this PR

.github/workflows/ci.yml triggers on pull_request: branches: [main]. This PR opened against track1/b4-repair-context-budget, so lint-typecheck-test-secrets, privacy-canary, testbed-smoke, and CodeQL never fired — the five green checks are assign, auto-merge, path labels, size label, and the title lint. Two of main's three required checks have no result at all.

Retargeting alone does not re-run them (the runs still linked here are from the original 12:26 push). The rebase in (1) fixes this too, since the force-push is a synchronize.

I ran the suite locally at d853d8c so this isn't a guess about the code:

npm run ci          # green — 377 unit, 16 integration, secret-scan/contracts/lint/lint:docs/typecheck all pass
npm run test:canary # 42 pass

So the substance holds. The point is that the checkmarks on this PR were never evidence of it, and nothing would have blocked a red one.

3. as never switches off type-checking for the one call that has never been executed

response = await this.client.messages.create({ ... } as never);

The root cause is narrow: the SDK types effort as a union, and this widens it.

// node_modules/@anthropic-ai/sdk/resources/messages/messages.d.ts
export interface OutputConfig {
  effort?: 'low' | 'medium' | 'high' | 'xhigh' | 'max' | null;
  format?: JSONOutputFormat | null;
}

AnthropicRepairClientOptions.effort and DEFAULT_EFFORT are string, which won't assign — so the cast goes on the whole request object, taking output_config, messages, system, and max_tokens out of the checker with it.

That trade is worse here than it normally would be. output_config is typed on MessageCreateParams (line 2260), and by the PR's own account no live call has been made — so the compiler is currently the only thing standing between a malformed request and the first run that spends money. The tests can't cover it either: they assert the shape the client writes against a hand-rolled fake, not against what the SDK accepts.

Narrowing the type lets the cast go entirely. I applied this locally — npm run typecheck clean, 377 tests still pass:

+import type { OutputConfig } from "@anthropic-ai/sdk/resources/messages/messages";
+type Effort = NonNullable<OutputConfig["effort"]>;

-export const DEFAULT_EFFORT = "medium";
+export const DEFAULT_EFFORT: Effort = "medium";

-  effort?: string;
+  effort?: Effort;

-  readonly effort: string;
+  readonly effort: Effort;

-      } as never);
+      });

Bonus: an invalid effort ("maximum", a typo) now fails at build instead of at the API.

4. minimum: 0 is not a supported structured-output constraint

REPAIR_OUTPUT_SCHEMA has wait_ms: { type: "integer", minimum: 0 }. Numerical constraints (minimum/maximum/multipleOf) aren't supported by structured outputs. The SDK strips unsupported keywords when you go through its schema helpers; this hands a raw schema straight to messages.create, and I couldn't find stripping logic on that path in 0.116.0.

Schemas are compiled server-side on first use, so if this is rejected it's rejected on the first paid call — the exact call you're saving up for. Dropping minimum: 0 costs nothing (wait_ms is optional and unvalidated by sanitizeProposedAction regardless).

5. The PR body overstates the error path

Failure paths report the tokens they burned — A refusal or network error returns null with consumed tokens, never zero.

The refusal path does. The catch returns tokens_in: 0, tokens_out: 0, and its own comment says so ("Tokens consumed are unknowable here, so they are reported as zero rather than guessed"). That is the right behaviour — there's no usage to read off a thrown error — but "never zero" isn't what the code does. Worth a word in the table, only because this PR is otherwise careful about exactly this distinction and the table is what a reader will trust.

6. Two smaller things

• max_tokens covers thinking too. claude-opus-5 runs adaptive thinking when thinking is omitted, and max_tokens caps thinking + response text together. 16k at effort: "medium" is very likely fine for one CompiledAction, but a truncation here is a paid call that yields nothing, and the failure looks like "no text block in response". Worth a sentence in the docstring so the next reader knows the ceiling isn't just about output size.
• No fallbacks. Omitting server-side refusal fallbacks is defensible here — a fallback would change which model served the repair while model_id is doing reproducibility work — but that's a real decision and it isn't written down. One row in the decisions table would cover it.

---

Summary: rebase onto main and force-push (fixes 1 and 2 together), take the type fix in (3) and the schema drop in (4), and I'll re-review. 5 and 6 are documentation. Nothing here is about the design — the egress discipline and the refuse-to-fabricate-a-metric posture are both right.

🤖 Reviewed with Claude Code

[myselfsiddharth]

All six items addressed — merging

Took the fixes myself rather than round-tripping. Merging with admin bypass on the approval rule; the three required checks are green on their own merits, which is the part that actually mattered.

1 & 2 — rebase onto main, and CI that actually ran

git rebase --onto main 51bc136 + force-push. The diff is 22 files → 8; #150's files (ADR-0012, src/shared/page-context.ts, src/runner/repair-egress.ts, contracts/metrics.schema.json, …) are gone from it.

Three conflicts, all additive — --repair-model alongside --from-cache/--pool-only in run-matrix.ts, repairClient alongside programSource/cacheProgramInvalidated in live-run.ts, and the dependency block. Both sides kept in each.

The force-push fired synchronize, so the checks ran for the first time:

lint-typecheck-test-secrets   pass  1m34s
privacy-canary                pass  19s
testbed-smoke (grafana 11)    pass  32s
analyze (javascript-ts)       pass  1m5s

One thing the rebase turned up that the textual merge missed: main promoted ajv/ajv-formats from dev to runtime deps after this branch was cut, so package-lock.json needed reconciling. npm audit --omit=dev still reports 0 vulnerabilities with the SDK added.

3 — as never is gone

Narrowed to the SDK's own union and the cast came off the whole request, exactly as sketched:

import type { OutputConfig } from "@anthropic-ai/sdk/resources/messages/messages";
type Effort = NonNullable<OutputConfig["effort"]>;

npm run typecheck is clean without the cast, which is the confirmation that effort really was the only mismatch — output_config was never the problem. Sabotage-checked per repo convention: DEFAULT_EFFORT = "maximum" now fails at build with TS2322, where before it would have failed on the first paid call.

4 — minimum: 0 dropped

Gone from wait_ms, with a comment recording why it can't come back. Confirmed nothing depended on it: sanitizeProposedAction doesn't validate wait_ms and no test asserted the constraint.

5 — error-path row corrected

The table now separates the two paths instead of claiming "never zero" for both: the refusal path reports its consumed tokens, and the catch path reports zero deliberately, because a thrown error carries no usage to read. That was always the right behaviour — only the description was wrong.

6 — both decisions written down

max_tokens capping adaptive thinking and response text is now in the DEFAULT_MAX_TOKENS docstring, including that a truncation surfaces as "no text block in response". The fallbacks omission got a docstring section and a decisions-table row: a fallback would let a different model serve the repair while model_id is doing reproducibility work for cost_repair.

---

#27 stays open. The exit criterion is unchanged and unmet — no live repair has been observed, self-heal rate is still structurally 0, and docs/gate/runner.md continues to record this as built but unmeasured. This merges the mechanism, not the measurement.

Local verification before merge:

npm run ci          # green — 453 unit, 26 integration
npm run test:canary # 52 pass
npm run build       # clean

The secret-scan env-assignment false positive is worth its own issue, as you noted — the workaround is still a rename, which is the wrong lesson.

🤖 Generated with Claude Code