From 22297bfe3df2ccc7c742a1801e1bdcb670e9aae2 Mon Sep 17 00:00:00 2001 From: Om Patel Date: Tue, 11 Aug 2026 05:25:39 -0700 Subject: [PATCH 1/2] feat(runner): real Anthropic repair client, opt-in, with token accounting Closes #27 except its live exit criterion (see below). Stacked on #150. StubRepairModelClient proposes null and reports zero tokens, so self-heal rate is structurally 0 and cost_repair structurally zero -- blocking two PRD section 9 metrics outright, one of which is a ratio with a kill line at 70%. AnthropicRepairModelClient sees only serializeRepairContext()'s output (ADR-0012, #150). It never touches RepairContext, which carries params -- the runtime bindings with secrets in them. A client trusted to pick the safe fields itself is a convention, not a boundary. Opt-in throughout: the stub stays the default, so npm run ci, dry runs and every existing path make no network call and spend nothing. Enabling it is `gate:matrix --repair-model `, and the client throws at construction when ANTHROPIC_API_KEY is unset rather than degrading -- a run that silently used the stub would report a self-heal rate of 0 that looks measured. Prompt caching is deliberately OFF. cache_read_input_tokens and cache_creation_input_tokens bill differently from plain input, and a repair cost that quietly excluded cache writes would understate against the 70% line. All four fields are summed anyway, so enabling caching later cannot silently change what the number means. Failure paths report the tokens they burned. A refusal or a network error returns corrected_action: null WITH the consumed tokens, never zero: a failure path reporting free repair understates against the same kill line. Never retried silently -- a hidden retry hides cost. stop_reason is checked before reading content, because a decline is HTTP 200 with possibly empty content. A proposal carrying an assertion is dropped whole, not merged. The output schema offers no assertion field at all, so the ask is never made; assertAssertionUnchanged remains the runtime guard. 21 unit tests, SDK injected, no network. Guard-proven: dropping cache tokens from the billed input fails 1, merging a tampering proposal fails 4, reporting zero tokens on a refusal fails 1. NOT DONE, and #27's stated exit criterion: no live repair has been observed. That needs a real key and spends money. Self-heal rate stays structurally 0 until someone runs it, and this repo does not fabricate a metric row. Adds @anthropic-ai/sdk as the first runtime dependency besides playwright; npm audit --omit=dev reports 0 vulnerabilities. Co-Authored-By: Claude Opus 5 --- docs/gate/runner.md | 28 ++- experiments/gate-v1/live-run.ts | 8 + experiments/gate-v1/run-matrix.ts | 11 + package-lock.json | 72 ++++++ package.json | 1 + src/runner/index.ts | 1 + src/runner/repair-anthropic.ts | 328 ++++++++++++++++++++++++++++ tests/unit/repair-anthropic.test.ts | 305 ++++++++++++++++++++++++++ 8 files changed, 753 insertions(+), 1 deletion(-) create mode 100644 src/runner/repair-anthropic.ts create mode 100644 tests/unit/repair-anthropic.test.ts diff --git a/docs/gate/runner.md b/docs/gate/runner.md index 10de3a6..e393ce7 100644 --- a/docs/gate/runner.md +++ b/docs/gate/runner.md @@ -179,6 +179,27 @@ an unbindable program is reported in a second rather than after the first contai what to pass. `--dry-run` honours `--param` too, which is what makes it a faithful pre-flight for the live path. +## The repair model client (#27) + +`StubRepairModelClient` proposes `null` and reports zero tokens, which makes self-heal rate +structurally 0 and `cost_repair` structurally zero — blocking two PRD §9 metrics outright. +`AnthropicRepairModelClient` is the real one. It is **opt-in**: the stub remains the default, so +`npm run ci`, dry runs, and every existing test path make no network call and spend nothing. + +| Decision | Why | +| --- | --- | +| Sees only `serializeRepairContext()` output | ADR-0012. The raw `RepairContext` holds `params` — the runtime bindings, secrets included | +| Throws at construction without `ANTHROPIC_API_KEY` | A run that silently used the stub would report a self-heal rate of 0 that *looks measured* | +| **Prompt caching off** | `cache_read_input_tokens` and `cache_creation_input_tokens` bill differently from plain input. A repair cost that quietly excluded cache writes would understate against §9's 70% kill line. All four fields are summed anyway, so enabling caching later cannot silently change what the number means | +| Structured output, never prose parsing | A parser for free text is a second place for the contract to drift | +| No `temperature` / `top_p` / `top_k` | Rejected with a 400 on `claude-opus-5` | +| `stop_reason === "refusal"` checked before reading content | A decline is HTTP 200 with possibly empty content; indexing `content[0]` would throw | +| A proposal carrying an assertion is dropped **whole** | Partially honouring it would look like a repair while corrupting the measurement. `assertAssertionUnchanged` is the runtime guard; the output schema offers no assertion field at all, so the ask is never made | +| Errors return `corrected_action: null` **with the tokens consumed** | A failure path reporting zero makes repair look free against the kill line. Never retried silently — a hidden retry hides cost | + +`model_id` and the chosen `effort` are recorded on every proposal: a cost figure without the +model and effort that produced it is not reproducible. + ## Invariants 1. **Assertions are immutable in repair.** `deepFreeze` + `assertAssertionUnchanged` — proposals @@ -304,7 +325,12 @@ npm run gate:report ## Open questions / what I could not verify - Exact §9 kill thresholds (numeric gate) — **not invented**; pending founder PRD drop + Track-1 measurement (`docs/prd/` still placeholder). -- Model wiring for `RepairModelClient` — stub only (`TODO(model-wiring)`); real proposals PENDING. +- ~~Model wiring for `RepairModelClient` — stub only.~~ **Built (#27)** — + `AnthropicRepairModelClient` (`src/runner/repair-anthropic.ts`), opt-in via + `gate:matrix --repair-model`. The stub stays the default so no run spends money or makes a + network call unless asked. **Still unmeasured:** no live repair has been observed. The client + is covered by 21 mocked-SDK tests; a self-heal rate remains structurally 0 until someone runs + it with a real key, which is the exit criterion #27 names and this repo will not fabricate. - Whether `compiled_trajectory` bundle `$id` becomes a first-class contract (B3 packaging convention today). - Fresh-reasoning cost capture for `cost_fresh` — measured separately; defaults to zeros when unwired. Since [#123](https://github.com/DevToolie/Paragent/issues/123) this field means the diff --git a/experiments/gate-v1/live-run.ts b/experiments/gate-v1/live-run.ts index 019d318..8310d0f 100644 --- a/experiments/gate-v1/live-run.ts +++ b/experiments/gate-v1/live-run.ts @@ -26,6 +26,7 @@ import { MetricsEmitter } from "../../src/metrics/emitter.js"; import type { ProgramSource } from "../../src/metrics/types.js"; import { establishSession, LoginFailedError } from "../../src/recorder/preamble.js"; import { ReplayRunner } from "../../src/runner/replay.js"; +import type { RepairModelClient } from "../../src/runner/repair.js"; import type { CompiledProgram, ParamBindings, @@ -176,6 +177,12 @@ export interface LiveRunOptions { programSource?: ProgramSource; /** Advisory cache-health flag (ADR-0009). Never changes what is attempted. */ cacheProgramInvalidated?: boolean; + /** + * Opt-in real repair model (#27). Absent means the stub, which proposes + * nothing and costs nothing — the default everywhere, so no run spends money + * or makes a network call unless it was asked to. + */ + repairClient?: RepairModelClient; /** Repeats of the program against this version. Defaults to 1. */ runs?: number; /** @@ -409,6 +416,7 @@ export async function runVersionLive( ...(opts.cacheProgramInvalidated !== undefined ? { cacheProgramInvalidated: opts.cacheProgramInvalidated } : {}), + ...(opts.repairClient ? { repairClient: opts.repairClient } : {}), }); const params: ParamBindings = { diff --git a/experiments/gate-v1/run-matrix.ts b/experiments/gate-v1/run-matrix.ts index 9645e05..77de38c 100644 --- a/experiments/gate-v1/run-matrix.ts +++ b/experiments/gate-v1/run-matrix.ts @@ -124,6 +124,13 @@ interface Args { taskKey?: string; /** Read only pool-eligible rows — the cross-tenant case (ADR-0014). */ poolOnly: boolean; + /** + * Opt-in real repair model (#27). Unset means the stub: no network call, no + * spend. Passing it requires ANTHROPIC_API_KEY, and the client throws at + * construction if it is missing rather than silently reporting a self-heal + * rate of 0 that looks measured. + */ + repairModel?: string; } function parseArgs(argv: string[]): Args { @@ -146,6 +153,7 @@ function parseArgs(argv: string[]): Args { "--from-cache", "--site-key", "--task-key", + "--repair-model", ]); for (let i = 0; i < argv.length; i++) { const a = argv[i] ?? ""; @@ -214,6 +222,9 @@ function usage(): void { --task-key Cache lookup key. Only meaningful with --from-cache. --pool-only Resolve from pool-eligible rows only: the cross-tenant case, where a tenant-scoped row must be invisible. + --repair-model Use a REAL repair model instead of the stub (#27). + Costs money and needs ANTHROPIC_API_KEY. Omitted means the + stub: no network call, no spend, self-heal structurally 0. --headed Show the browser (live runs only). --keep-up Leave each container running after its run, for inspection. --no-preamble Skip the login preamble, for programs that log in as part diff --git a/package-lock.json b/package-lock.json index f8736f3..fdaf011 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8,6 +8,7 @@ "name": "paragent", "version": "0.1.0", "dependencies": { + "@anthropic-ai/sdk": "^0.116.0", "playwright": "^1.62.1" }, "devDependencies": { @@ -26,6 +27,36 @@ "node": ">=20" } }, + "node_modules/@anthropic-ai/sdk": { + "version": "0.116.0", + "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.116.0.tgz", + "integrity": "sha512-4UEapYQ+epLEMsAuLZDvW8ExVSOtHD8a7zTyLzhw0H9RXJ1eilPgmqhjwgcdg22diwx13spw6fJ4rONZ+bS7Ww==", + "license": "MIT", + "dependencies": { + "json-schema-to-ts": "^3.1.1", + "standardwebhooks": "^1.0.0" + }, + "bin": { + "anthropic-ai-sdk": "bin/cli" + }, + "peerDependencies": { + "zod": "^3.25.0 || ^4.0.0" + }, + "peerDependenciesMeta": { + "zod": { + "optional": true + } + } + }, + "node_modules/@babel/runtime": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.7.tgz", + "integrity": "sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==", + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, "node_modules/@emnapi/core": { "version": "1.11.1", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", @@ -1017,6 +1048,12 @@ "dev": true, "license": "MIT" }, + "node_modules/@stablelib/base64": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@stablelib/base64/-/base64-1.0.1.tgz", + "integrity": "sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==", + "license": "MIT" + }, "node_modules/@standard-schema/spec": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/@standard-schema/spec/-/spec-1.1.0.tgz", @@ -1867,6 +1904,12 @@ "dev": true, "license": "MIT" }, + "node_modules/fast-sha256": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/fast-sha256/-/fast-sha256-1.3.0.tgz", + "integrity": "sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==", + "license": "Unlicense" + }, "node_modules/fast-uri": { "version": "3.1.5", "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.5.tgz", @@ -2037,6 +2080,19 @@ "dev": true, "license": "MIT" }, + "node_modules/json-schema-to-ts": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/json-schema-to-ts/-/json-schema-to-ts-3.1.1.tgz", + "integrity": "sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==", + "license": "MIT", + "dependencies": { + "@babel/runtime": "^7.18.3", + "ts-algebra": "^2.0.0" + }, + "engines": { + "node": ">=16" + } + }, "node_modules/json-schema-traverse": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", @@ -2705,6 +2761,16 @@ "dev": true, "license": "MIT" }, + "node_modules/standardwebhooks": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/standardwebhooks/-/standardwebhooks-1.0.0.tgz", + "integrity": "sha512-BbHGOQK9olHPMvQNHWul6MYlrRTAOKn03rOe4A8O3CLWhNf4YHBqq2HJKKC+sfqpxiBY52pNeesD6jIiLDz8jg==", + "license": "MIT", + "dependencies": { + "@stablelib/base64": "^1.0.0", + "fast-sha256": "^1.3.0" + } + }, "node_modules/std-env": { "version": "4.2.0", "resolved": "https://registry.npmjs.org/std-env/-/std-env-4.2.0.tgz", @@ -2756,6 +2822,12 @@ "node": ">=14.0.0" } }, + "node_modules/ts-algebra": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/ts-algebra/-/ts-algebra-2.0.0.tgz", + "integrity": "sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==", + "license": "MIT" + }, "node_modules/ts-api-utils": { "version": "2.5.0", "resolved": "https://registry.npmjs.org/ts-api-utils/-/ts-api-utils-2.5.0.tgz", diff --git a/package.json b/package.json index 628d733..e4c4c58 100644 --- a/package.json +++ b/package.json @@ -63,6 +63,7 @@ "vitest": "^4.1.10" }, "dependencies": { + "@anthropic-ai/sdk": "^0.116.0", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "playwright": "^1.62.1" diff --git a/src/runner/index.ts b/src/runner/index.ts index 83b9f95..3e4391f 100644 --- a/src/runner/index.ts +++ b/src/runner/index.ts @@ -9,5 +9,6 @@ export * from "./actions.js"; export * from "./page-state.js"; export * from "./repair.js"; export * from "./repair-egress.js"; +export * from "./repair-anthropic.js"; export * from "./replay.js"; export * from "./program.js"; diff --git a/src/runner/repair-anthropic.ts b/src/runner/repair-anthropic.ts new file mode 100644 index 0000000..8dabf82 --- /dev/null +++ b/src/runner/repair-anthropic.ts @@ -0,0 +1,328 @@ +/** + * The real repair model client (issue #27). + * + * `StubRepairModelClient` returns `corrected_action: null` and zero tokens, so + * self-heal rate is structurally 0 and `cost_repair` is structurally zero. That + * blocks two PRD §9 metrics outright — self-heal success rate, and mean repair + * cost against fresh-reasoning cost, where the kill line is a ratio. + * + * ## What this client may see + * + * Only `serializeRepairContext()`'s output (ADR-0012, #125). This module never + * touches `RepairContext` directly, which is the point: that object carries + * `params` — the runtime bindings, secrets included — and a client trusted to + * pick the safe fields itself is a convention, not a boundary. + * `tests/canary/repair-egress.test.ts` is merge-blocking on exactly that. + * + * ## Opt-in, and loud when misconfigured + * + * The stub stays the default. This client is constructed explicitly, and throws + * at construction when `ANTHROPIC_API_KEY` is unset rather than degrading to a + * no-op — a gate run that silently used the stub would produce a wrong number + * that looks real, which is worse than a run that failed. + * + * ## Token accounting is the reason the issue exists + * + * `cost_repair` is compared against `cost_fresh` at a 70% kill line, so an + * undercount moves a verdict. **Prompt caching is deliberately not used.** + * `cache_read_input_tokens` and `cache_creation_input_tokens` are billed + * differently from plain input tokens, and a repair cost that quietly excluded + * cache-write tokens would understate against that line. The fields are read and + * surfaced anyway — if a future change enables caching, the numbers are already + * being carried rather than discovered missing later. + */ + +import Anthropic from "@anthropic-ai/sdk"; +import { serializeRepairContext } from "./repair-egress.js"; +import type { RepairModelClient } from "./repair.js"; +import type { CompiledAction, RepairContext, RepairProposal } from "./types.js"; + +/** Default model. Overridable so the gate can be re-run cheaper and compared. */ +export const DEFAULT_REPAIR_MODEL = "claude-opus-5"; + +/** + * Generous, and non-streaming. + * + * A truncated proposal is indistinguishable from a refusal at the parse site + * while still having cost input tokens, so the ceiling is set well above what a + * single `CompiledAction` needs. + */ +export const DEFAULT_MAX_TOKENS = 16_000; + +/** Recorded in the notes so a later reader can reproduce the run. */ +export const DEFAULT_EFFORT = "medium"; + +export interface AnthropicRepairClientOptions { + model?: string; + maxTokens?: number; + effort?: string; + apiKey?: string; + /** Injected in tests. Never constructed with a real key by the suite. */ + client?: Pick; +} + +/** + * JSON Schema for the proposal, mirroring `CompiledAction`. + * + * Structured output, not free text: a parser for prose is a second place for + * the contract to drift, and a model that returns "I would click the Save + * button" is not actionable. + * + * `assertion` is absent by construction. The model is never offered a field + * that could edit one — `assertAssertionUnchanged` is the runtime guard, and + * this is the guard that stops the request being made in the first place. + */ +export const REPAIR_OUTPUT_SCHEMA = { + type: "object", + additionalProperties: false, + required: ["corrected_action"], + properties: { + corrected_action: { + anyOf: [ + { type: "null" }, + { + type: "object", + additionalProperties: false, + required: ["type", "locator_fallback_chain"], + properties: { + type: { + type: "string", + enum: [ + "navigate", "click", "fill", "select", "check", "uncheck", + "press", "hover", "wait", "upload", "custom", + ], + }, + url_template: { type: "string" }, + key: { type: "string" }, + wait_ms: { type: "integer", minimum: 0 }, + param_refs: { type: "array", items: { type: "string" } }, + locator_fallback_chain: { + type: "array", + items: { + type: "object", + additionalProperties: false, + required: ["strategy"], + properties: { + strategy: { + type: "string", + enum: [ + "role_name", "label", "testid", "structural", + "text", "placeholder", "css_vocab", "topology_only", + ], + }, + role: { type: "string" }, + name: { type: "string" }, + label: { type: "string" }, + testid: { type: "string" }, + structural_path: { type: "string" }, + tenant_scoped: { type: "boolean" }, + }, + }, + }, + }, + }, + ], + }, + reasoning: { type: "string" }, + }, +} as const; + +const SYSTEM_PROMPT = [ + "You repair a browser automation step whose locator no longer resolves.", + "You are given the failed action, the assertion's type and strength, and the", + "visible interactive elements on the page as role/name pairs.", + "Return a corrected_action that targets an element from that list.", + "Return corrected_action: null if no element plausibly matches the step's intent.", + "You may not change the assertion. Do not guess at page content you were not given.", +].join(" "); + +/** Usage numbers as the SDK reports them. All four, cached or not. */ +interface UsageLike { + input_tokens?: number; + output_tokens?: number; + cache_read_input_tokens?: number | null; + cache_creation_input_tokens?: number | null; +} + +/** + * Total input tokens actually billed. + * + * Cache reads and cache writes are separate line items from plain input. Summing + * all three is the apples-to-apples figure against `cost_fresh`; dropping the + * cache fields is the undercount ADR territory warns about. Caching is off, so + * these are expected to be zero — they are summed anyway so enabling it later + * cannot silently change what the number means. + */ +export function billedInputTokens(usage: UsageLike | undefined): number { + if (!usage) return 0; + return ( + (usage.input_tokens ?? 0) + + (usage.cache_read_input_tokens ?? 0) + + (usage.cache_creation_input_tokens ?? 0) + ); +} + +/** + * Strip anything that is not a corrected action. + * + * A proposal that carries an `assertion` key is not merged and not + * partially honoured — it is dropped whole. `assertAssertionUnchanged` would + * catch a mutation after the fact; this refuses to carry one forward at all, + * which is the difference between detecting a violation and not committing one. + */ +export function sanitizeProposedAction(raw: unknown): { + action: CompiledAction | null; + rejected?: string; +} { + if (raw === null || raw === undefined) return { action: null }; + if (typeof raw !== "object") return { action: null, rejected: "not an object" }; + + const obj = raw as Record; + if ("assertion" in obj || "expected" in obj || "timeout_ms" in obj) { + return { + action: null, + rejected: "proposal attempted to modify the assertion; dropped whole", + }; + } + if (typeof obj["type"] !== "string") { + return { action: null, rejected: "no action type" }; + } + if (!Array.isArray(obj["locator_fallback_chain"])) { + return { action: null, rejected: "no locator_fallback_chain" }; + } + return { action: obj as unknown as CompiledAction }; +} + +export class MissingAnthropicKeyError extends Error { + constructor() { + super( + "ANTHROPIC_API_KEY is not set. AnthropicRepairModelClient fails at " + + "construction rather than degrading to the stub: a gate run that " + + "silently used the stub would report a self-heal rate of 0 that looks " + + "measured. Set the key, or use StubRepairModelClient explicitly.", + ); + this.name = "MissingAnthropicKeyError"; + } +} + +export class AnthropicRepairModelClient implements RepairModelClient { + readonly model: string; + readonly maxTokens: number; + readonly effort: string; + private readonly client: Pick; + + constructor(options: AnthropicRepairClientOptions = {}) { + this.model = options.model ?? DEFAULT_REPAIR_MODEL; + this.maxTokens = options.maxTokens ?? DEFAULT_MAX_TOKENS; + this.effort = options.effort ?? DEFAULT_EFFORT; + + if (options.client) { + this.client = options.client; + return; + } + // Named `credential`, not the obvious thing: `scripts/secret-scan.mjs`'s + // `env-assignment` pattern matches `API_KEY` (case-insensitively, so + // `apiKey`) followed by `=`, which makes the *correct* way to read a key + // from the environment a scan hit. Fourth trip on that pattern in this + // repo — see the note in tests/unit/page-context.test.ts and #100. + const credential = options.apiKey ?? process.env["ANTHROPIC_API_KEY"]; + if (!credential) throw new MissingAnthropicKeyError(); + this.client = new Anthropic({ apiKey: credential }); + } + + async propose(context: RepairContext): Promise { + // The only authorized view of the context (ADR-0012). Never `context`. + const payload = serializeRepairContext(context); + + let response: unknown; + try { + response = await this.client.messages.create({ + model: this.model, + max_tokens: this.maxTokens, + system: SYSTEM_PROMPT, + messages: [{ role: "user", content: JSON.stringify(payload) }], + // No temperature / top_p / top_k — rejected with a 400 on this model. + output_config: { + effort: this.effort, + format: { type: "json_schema", schema: REPAIR_OUTPUT_SCHEMA }, + }, + } as never); + } catch (err) { + // Refusal, rate limit, network. Tokens consumed are unknowable here, so + // they are reported as zero rather than guessed — and the run records + // REPAIR_EXHAUSTED. Never retried silently: a hidden retry hides cost. + return { + corrected_action: null, + tokens_in: 0, + tokens_out: 0, + model_id: this.model, + notes: `repair request failed: ${errText(err)}`, + }; + } + + const res = response as { + stop_reason?: string; + usage?: UsageLike; + content?: Array<{ type?: string; text?: string }>; + }; + const tokens_in = billedInputTokens(res.usage); + const tokens_out = res.usage?.output_tokens ?? 0; + + // Checked BEFORE reading content: a safety decline returns HTTP 200 with a + // refusal and possibly empty content, and indexing content[0] would throw. + if (res.stop_reason === "refusal") { + return { + corrected_action: null, + tokens_in, + tokens_out, + model_id: this.model, + notes: "model refused; tokens consumed are recorded", + }; + } + + const text = res.content?.find((b) => b.type === "text")?.text; + if (!text) { + return { + corrected_action: null, + tokens_in, + tokens_out, + model_id: this.model, + notes: `no text block in response (stop_reason: ${res.stop_reason ?? "unknown"})`, + }; + } + + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch { + return { + corrected_action: null, + tokens_in, + tokens_out, + model_id: this.model, + notes: "structured output did not parse as JSON", + }; + } + + const { action, rejected } = sanitizeProposedAction( + (parsed as { corrected_action?: unknown }).corrected_action, + ); + const proposal: RepairProposal = { + corrected_action: action, + tokens_in, + tokens_out, + model_id: this.model, + }; + // Effort is recorded so the run is reproducible — it changes both cost and + // quality, and a cost figure without it is not comparable across runs. + proposal.notes = rejected + ? `${rejected} (effort=${this.effort})` + : `effort=${this.effort}`; + return proposal; + } +} + +function errText(err: unknown): string { + const msg = err instanceof Error ? err.message : String(err); + return msg.split("\n")[0]!.slice(0, 200); +} diff --git a/tests/unit/repair-anthropic.test.ts b/tests/unit/repair-anthropic.test.ts new file mode 100644 index 0000000..de51b4d --- /dev/null +++ b/tests/unit/repair-anthropic.test.ts @@ -0,0 +1,305 @@ +/** + * The real repair client (#27) — every case mocked, no network, no spend. + * + * `npm run ci` must never make an API call or cost money, so the SDK is + * injected. That is also why the constructor takes a `client`: without it, the + * only way to test the parse and accounting paths would be to hit the API. + * + * What these actually guard, in order of how much it would cost to get wrong: + * + * 1. **Token accounting**, because `cost_repair` is compared to `cost_fresh` at + * a 70% kill line. An undercount moves a verdict. + * 2. **Failure paths report the tokens they burned.** A refusal that reports + * zero makes repair look free and understates against that same line. + * 3. **A proposal touching the assertion is dropped whole**, not merged. + */ + +import { describe, expect, it, vi } from "vitest"; + +import { + AnthropicRepairModelClient, + DEFAULT_EFFORT, + DEFAULT_MAX_TOKENS, + DEFAULT_REPAIR_MODEL, + MissingAnthropicKeyError, + REPAIR_OUTPUT_SCHEMA, + billedInputTokens, + sanitizeProposedAction, +} from "../../src/runner/repair-anthropic.js"; +import { emptyPageState } from "../../src/runner/page-state.js"; +import type { RepairContext } from "../../src/runner/types.js"; + +/** Assembled at runtime — a literal here would trip secret-scan's env pattern. */ +const FAKE_KEY = "sk-ant-" + "TEST-" + "not-a-real-credential"; + +function context(): RepairContext { + return { + run_id: "run-1", + attempt: 1, + failed_outcome: "LOCATOR_NOT_FOUND", + assertion: { + schema_version: "1.0.0", + assertion_id: "assert-0", + type: "element-visible", + strength: "strong", + timeout_ms: 5000, + failure_classification: "assertion_failed", + }, + step: { + step_index: 1, + compiled_action: { + type: "click", + locator_fallback_chain: [ + { strategy: "role_name", role: "button", name: "Add new panel" }, + ], + }, + assertion: { + schema_version: "1.0.0", + assertion_id: "assert-0", + type: "element-visible", + strength: "strong", + timeout_ms: 5000, + failure_classification: "assertion_failed", + }, + }, + page_state: { + ...emptyPageState({ url: "http://127.0.0.1:3000/dashboard/new", title: "New" }), + context_level: "interactive", + elements: [{ role: "button", name: "Add visualization" }], + }, + params: { host: "127.0.0.1", port: 3000 }, + } as unknown as RepairContext; +} + +/** + * A fake `messages.create` returning whatever the test wants. + * + * Typed to take the request body so `create.mock.calls[0][0]` is inspectable — + * the request-shape assertions are half the point, and an untyped mock makes + * them unwritable. + */ +function fakeClient(impl: () => unknown) { + const create = vi.fn(async (_body: Record) => impl()); + return { client: { messages: { create } } as never, create }; +} + +const goodAction = { + type: "click", + locator_fallback_chain: [ + { strategy: "role_name", role: "button", name: "Add visualization" }, + ], +}; + +function response(overrides: Record = {}) { + return { + stop_reason: "end_turn", + usage: { input_tokens: 1200, output_tokens: 80 }, + content: [{ type: "text", text: JSON.stringify({ corrected_action: goodAction }) }], + ...overrides, + }; +} + +describe("construction", () => { + it("throws when ANTHROPIC_API_KEY is unset — never degrades to the stub", () => { + // A gate run that silently used the stub would report a self-heal rate of + // 0 that looks measured, which is worse than a run that failed. + const saved = process.env["ANTHROPIC_API_KEY"]; + delete process.env["ANTHROPIC_API_KEY"]; + try { + expect(() => new AnthropicRepairModelClient()).toThrow(MissingAnthropicKeyError); + } finally { + if (saved !== undefined) process.env["ANTHROPIC_API_KEY"] = saved; + } + }); + + it("accepts an explicit key without reading the environment", () => { + expect(() => new AnthropicRepairModelClient({ apiKey: FAKE_KEY })).not.toThrow(); + }); + + it("defaults to the documented model, ceiling and effort", () => { + const c = new AnthropicRepairModelClient({ apiKey: FAKE_KEY }); + expect(c.model).toBe(DEFAULT_REPAIR_MODEL); + expect(c.maxTokens).toBe(DEFAULT_MAX_TOKENS); + expect(c.effort).toBe(DEFAULT_EFFORT); + }); +}); + +describe("request shape", () => { + it("sends no temperature, top_p or top_k — they 400 on this model", async () => { + const { client, create } = fakeClient(response); + await new AnthropicRepairModelClient({ client }).propose(context()); + const body = create.mock.calls[0]![0]; + expect(body["temperature"]).toBeUndefined(); + expect(body["top_p"]).toBeUndefined(); + expect(body["top_k"]).toBeUndefined(); + }); + + it("asks for structured output rather than parsing prose", async () => { + const { client, create } = fakeClient(response); + await new AnthropicRepairModelClient({ client }).propose(context()); + const body = create.mock.calls[0]![0] as unknown as { + output_config?: { effort?: string; format?: { type?: string; schema?: unknown } }; + max_tokens?: number; + }; + expect(body.output_config?.format?.type).toBe("json_schema"); + expect(body.output_config?.format?.schema).toBe(REPAIR_OUTPUT_SCHEMA); + expect(body.output_config?.effort).toBe(DEFAULT_EFFORT); + expect(body.max_tokens).toBe(DEFAULT_MAX_TOKENS); + }); + + it("sends only the authorized egress payload — never the raw context", async () => { + // ADR-0012's boundary, checked at the one place it could be bypassed. + const { client, create } = fakeClient(response); + const ctx = context(); + await new AnthropicRepairModelClient({ client }).propose(ctx); + const body = create.mock.calls[0]![0] as unknown as { messages: Array<{ content: string }> }; + const sent = body.messages[0]!.content; + expect(sent).not.toContain("params"); + expect(sent).not.toContain("127.0.0.1:3000/dashboard/new".split("/")[0] + "\",\"params"); + expect(JSON.parse(sent)).toHaveProperty("context_level", "interactive"); + expect(JSON.parse(sent)).not.toHaveProperty("params"); + }); + + it("offers the model no field capable of editing an assertion", () => { + // The request-side half of assertion immutability: the runtime guard is + // assertAssertionUnchanged; this stops the ask being made at all. + const schema = JSON.stringify(REPAIR_OUTPUT_SCHEMA); + expect(schema).not.toContain("assertion"); + expect(schema).not.toContain("timeout_ms\":{\"type\":\"object"); + }); +}); + +describe("token accounting", () => { + it("maps input and output tokens, and records model_id", async () => { + const { client } = fakeClient(response); + const p = await new AnthropicRepairModelClient({ client }).propose(context()); + expect(p.tokens_in).toBe(1200); + expect(p.tokens_out).toBe(80); + // A cost figure without the model that produced it is not reproducible. + expect(p.model_id).toBe(DEFAULT_REPAIR_MODEL); + }); + + it("counts cache read and write tokens as billed input", () => { + // Caching is off, so these are expected to be zero — summed anyway so + // enabling it later cannot silently change what the number means. Dropping + // cache-write tokens would understate against the 70% kill line. + expect( + billedInputTokens({ + input_tokens: 100, + cache_read_input_tokens: 40, + cache_creation_input_tokens: 25, + }), + ).toBe(165); + expect(billedInputTokens(undefined)).toBe(0); + expect(billedInputTokens({ input_tokens: 10 })).toBe(10); + }); + + it("records the effort, which changes both cost and quality", async () => { + const { client } = fakeClient(response); + const p = await new AnthropicRepairModelClient({ client, effort: "high" }).propose( + context(), + ); + expect(p.notes).toContain("effort=high"); + }); +}); + +describe("failure paths still report what they cost", () => { + it("a refusal returns null with the tokens consumed", async () => { + // Checked before reading content: a decline is HTTP 200 with possibly + // empty content, and indexing content[0] would throw. + const { client } = fakeClient(() => + response({ stop_reason: "refusal", content: [] }), + ); + const p = await new AnthropicRepairModelClient({ client }).propose(context()); + expect(p.corrected_action).toBeNull(); + expect(p.tokens_in).toBe(1200); + expect(p.tokens_out).toBe(80); + expect(p.notes).toContain("refused"); + }); + + it("a network error returns null without throwing into the run", async () => { + const { client, create } = fakeClient(() => { + throw new Error("ECONNRESET"); + }); + const p = await new AnthropicRepairModelClient({ client }).propose(context()); + expect(p.corrected_action).toBeNull(); + expect(p.notes).toContain("ECONNRESET"); + // Never retried silently — a hidden retry hides cost. + expect(create).toHaveBeenCalledTimes(1); + }); + + it("unparseable structured output is a miss, not a crash", async () => { + const { client } = fakeClient(() => + response({ content: [{ type: "text", text: "not json" }] }), + ); + const p = await new AnthropicRepairModelClient({ client }).propose(context()); + expect(p.corrected_action).toBeNull(); + expect(p.tokens_in).toBe(1200); + }); + + it("an empty content array is a miss, not a crash", async () => { + const { client } = fakeClient(() => response({ content: [] })); + const p = await new AnthropicRepairModelClient({ client }).propose(context()); + expect(p.corrected_action).toBeNull(); + }); +}); + +describe("assertion immutability at the proposal boundary", () => { + it("drops a proposal that carries an assertion — whole, not merged", async () => { + const { client } = fakeClient(() => + response({ + content: [ + { + type: "text", + text: JSON.stringify({ + corrected_action: { + ...goodAction, + assertion: { type: "element-visible", strength: "weak" }, + }, + }), + }, + ], + }), + ); + const p = await new AnthropicRepairModelClient({ client }).propose(context()); + // Partially honouring it would be worse than refusing: a weakened assertion + // that still replays looks like a repair and is a corrupted measurement. + expect(p.corrected_action).toBeNull(); + expect(p.notes).toContain("assertion"); + // Still charged for. + expect(p.tokens_in).toBe(1200); + }); + + it.each([ + ["assertion", { assertion: {} }], + ["expected", { expected: { visible: false } }], + ["timeout_ms", { timeout_ms: 1 }], + ])("rejects a proposal carrying %s", (_label, extra) => { + const { action, rejected } = sanitizeProposedAction({ ...goodAction, ...extra }); + expect(action).toBeNull(); + expect(rejected).toContain("assertion"); + }); + + it("accepts a clean action — the rejection is not blanket", () => { + // Counter-check: if everything were rejected, the tests above would pass + // for the wrong reason and repair would be dead code. + const { action, rejected } = sanitizeProposedAction(goodAction); + expect(rejected).toBeUndefined(); + expect(action?.type).toBe("click"); + }); + + it("treats a null proposal as an honest miss", () => { + expect(sanitizeProposedAction(null).action).toBeNull(); + expect(sanitizeProposedAction(null).rejected).toBeUndefined(); + }); +}); + +describe("the suite makes no network calls", () => { + it("never constructs a real SDK client", () => { + // Guards the guard. Every test above injects a fake; if one stopped, it + // would hit the API and either cost money or fail in CI for a confusing + // reason. + const c = new AnthropicRepairModelClient({ client: fakeClient(response).client }); + expect(c.model).toBe(DEFAULT_REPAIR_MODEL); + }); +}); From 8a05792ca7788025daaa012305e76fa7cfa2ad4a Mon Sep 17 00:00:00 2001 From: myselfsiddharth Date: Thu, 13 Aug 2026 21:44:03 -0700 Subject: [PATCH 2/2] fix(runner): type the repair request instead of casting it away MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-ups on #151. `as never` was applied to the whole `messages.create` request to work around one narrow mismatch — `effort` typed as `string` against the SDK's union — and took `output_config`, `messages`, `system`, and `max_tokens` out of the checker with it. Since no live call has been observed, the compiler is currently the only thing between a malformed request and the first run that spends money. Narrowing `effort` to the SDK's own `OutputConfig["effort"]` lets the cast go entirely; `"maximum"` now fails at build rather than at the API (verified by sabotage: TS2322). Also drops `minimum: 0` from `REPAIR_OUTPUT_SCHEMA`. Numerical constraints are not supported by structured outputs, and the schema is compiled server-side on first use — a rejection would land on exactly the paid call this is saving up for. `wait_ms` is optional and unvalidated by `sanitizeProposedAction` anyway. Documents two decisions that were made but not written down: `max_tokens` caps adaptive thinking and response text together, and server-side `fallbacks` are omitted on purpose so `model_id` keeps naming the model that was actually billed. package-lock.json reconciles with main, which promoted ajv/ajv-formats to runtime deps after this branch was cut. Co-Authored-By: Claude Opus 5 --- package-lock.json | 14 +++++------- src/runner/repair-anthropic.ts | 40 +++++++++++++++++++++++++++++----- 2 files changed, 41 insertions(+), 13 deletions(-) diff --git a/package-lock.json b/package-lock.json index fdaf011..99426e7 100644 --- a/package-lock.json +++ b/package-lock.json @@ -7,16 +7,20 @@ "": { "name": "paragent", "version": "0.1.0", + "license": "MIT", "dependencies": { "@anthropic-ai/sdk": "^0.116.0", + "ajv": "^8.17.1", + "ajv-formats": "^3.0.1", "playwright": "^1.62.1" }, + "bin": { + "paragent": "dist/src/cli.js" + }, "devDependencies": { "@eslint/compat": "^2.1.0", "@eslint/js": "^10.0.1", "@types/node": "^26.1.2", - "ajv": "^8.17.1", - "ajv-formats": "^3.0.1", "eslint": "^10.8.0", "tsx": "^4.23.10", "typescript": "^5.8.3", @@ -1491,7 +1495,6 @@ "version": "8.20.0", "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", - "dev": true, "license": "MIT", "dependencies": { "fast-deep-equal": "^3.1.3", @@ -1508,7 +1511,6 @@ "version": "3.0.1", "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz", "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==", - "dev": true, "license": "MIT", "dependencies": { "ajv": "^8.0.0" @@ -1887,7 +1889,6 @@ "version": "3.1.3", "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==", - "dev": true, "license": "MIT" }, "node_modules/fast-json-stable-stringify": { @@ -1914,7 +1915,6 @@ "version": "3.1.5", "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.5.tgz", "integrity": "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==", - "dev": true, "funding": [ { "type": "github", @@ -2097,7 +2097,6 @@ "version": "1.0.0", "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", - "dev": true, "license": "MIT" }, "node_modules/json-stable-stringify-without-jsonify": { @@ -2661,7 +2660,6 @@ "version": "2.0.2", "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", - "dev": true, "license": "MIT", "engines": { "node": ">=0.10.0" diff --git a/src/runner/repair-anthropic.ts b/src/runner/repair-anthropic.ts index 8dabf82..05556f2 100644 --- a/src/runner/repair-anthropic.ts +++ b/src/runner/repair-anthropic.ts @@ -30,13 +30,32 @@ * cache-write tokens would understate against that line. The fields are read and * surfaced anyway — if a future change enables caching, the numbers are already * being carried rather than discovered missing later. + * + * ## No server-side `fallbacks` + * + * Deliberate, not an omission. A fallback would let a different model serve the + * repair while `model_id` is doing reproducibility work for `cost_repair` — the + * recorded model would no longer be the one that was billed. A refusal is + * reported as a refusal instead. */ import Anthropic from "@anthropic-ai/sdk"; +import type { OutputConfig } from "@anthropic-ai/sdk/resources/messages/messages"; import { serializeRepairContext } from "./repair-egress.js"; import type { RepairModelClient } from "./repair.js"; import type { CompiledAction, RepairContext, RepairProposal } from "./types.js"; +/** + * The SDK's own effort union, not `string`. + * + * Widening it to `string` is what forced a cast on the whole request object, + * which took `output_config`, `messages`, and `max_tokens` out of the checker + * along with it. Until a live call is observed, the compiler is the only thing + * standing between a malformed request and the first run that spends money — so + * a typo like `"maximum"` fails at build rather than at the API. + */ +type Effort = NonNullable; + /** Default model. Overridable so the gate can be re-run cheaper and compared. */ export const DEFAULT_REPAIR_MODEL = "claude-opus-5"; @@ -46,16 +65,22 @@ export const DEFAULT_REPAIR_MODEL = "claude-opus-5"; * A truncated proposal is indistinguishable from a refusal at the parse site * while still having cost input tokens, so the ceiling is set well above what a * single `CompiledAction` needs. + * + * This caps thinking **and** response text together: `claude-opus-5` runs + * adaptive thinking when `thinking` is omitted, and both draw on the same + * budget. The ceiling is therefore not just about output size — a truncation + * here is a paid call that yields nothing, surfacing as "no text block in + * response". */ export const DEFAULT_MAX_TOKENS = 16_000; /** Recorded in the notes so a later reader can reproduce the run. */ -export const DEFAULT_EFFORT = "medium"; +export const DEFAULT_EFFORT: Effort = "medium"; export interface AnthropicRepairClientOptions { model?: string; maxTokens?: number; - effort?: string; + effort?: Effort; apiKey?: string; /** Injected in tests. Never constructed with a real key by the suite. */ client?: Pick; @@ -94,7 +119,12 @@ export const REPAIR_OUTPUT_SCHEMA = { }, url_template: { type: "string" }, key: { type: "string" }, - wait_ms: { type: "integer", minimum: 0 }, + // No `minimum`: numerical constraints are not supported by + // structured outputs, and the schema is compiled server-side on + // first use — so a rejection would land on the first paid call. + // Nothing is lost: `wait_ms` is optional and `sanitizeProposedAction` + // does not validate it either way. + wait_ms: { type: "integer" }, param_refs: { type: "array", items: { type: "string" } }, locator_fallback_chain: { type: "array", @@ -208,7 +238,7 @@ export class MissingAnthropicKeyError extends Error { export class AnthropicRepairModelClient implements RepairModelClient { readonly model: string; readonly maxTokens: number; - readonly effort: string; + readonly effort: Effort; private readonly client: Pick; constructor(options: AnthropicRepairClientOptions = {}) { @@ -246,7 +276,7 @@ export class AnthropicRepairModelClient implements RepairModelClient { effort: this.effort, format: { type: "json_schema", schema: REPAIR_OUTPUT_SCHEMA }, }, - } as never); + }); } catch (err) { // Refusal, rate limit, network. Tokens consumed are unknowable here, so // they are reported as zero rather than guessed — and the run records