Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion docs/gate/runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,27 @@ an unbindable program is reported in a second rather than after the first contai
what to pass. `--dry-run` honours `--param` too, which is what makes it a faithful pre-flight for
the live path.

## The repair model client (#27)

`StubRepairModelClient` proposes `null` and reports zero tokens, which makes self-heal rate
structurally 0 and `cost_repair` structurally zero — blocking two PRD §9 metrics outright.
`AnthropicRepairModelClient` is the real one. It is **opt-in**: the stub remains the default, so
`npm run ci`, dry runs, and every existing test path make no network call and spend nothing.

| Decision | Why |
| --- | --- |
| Sees only `serializeRepairContext()` output | ADR-0012. The raw `RepairContext` holds `params` — the runtime bindings, secrets included |
| Throws at construction without `ANTHROPIC_API_KEY` | A run that silently used the stub would report a self-heal rate of 0 that *looks measured* |
| **Prompt caching off** | `cache_read_input_tokens` and `cache_creation_input_tokens` bill differently from plain input. A repair cost that quietly excluded cache writes would understate against §9's 70% kill line. All four fields are summed anyway, so enabling caching later cannot silently change what the number means |
| Structured output, never prose parsing | A parser for free text is a second place for the contract to drift |
| No `temperature` / `top_p` / `top_k` | Rejected with a 400 on `claude-opus-5` |
| `stop_reason === "refusal"` checked before reading content | A decline is HTTP 200 with possibly empty content; indexing `content[0]` would throw |
| A proposal carrying an assertion is dropped **whole** | Partially honouring it would look like a repair while corrupting the measurement. `assertAssertionUnchanged` is the runtime guard; the output schema offers no assertion field at all, so the ask is never made |
| Errors return `corrected_action: null` **with the tokens consumed** | A failure path reporting zero makes repair look free against the kill line. Never retried silently — a hidden retry hides cost |

`model_id` and the chosen `effort` are recorded on every proposal: a cost figure without the
model and effort that produced it is not reproducible.

## Invariants

1. **Assertions are immutable in repair.** `deepFreeze` + `assertAssertionUnchanged` — proposals
Expand Down Expand Up @@ -304,7 +325,12 @@ npm run gate:report
## Open questions / what I could not verify

- Exact §9 kill thresholds (numeric gate) — **not invented**; pending founder PRD drop + Track-1 measurement (`docs/prd/` still placeholder).
- Model wiring for `RepairModelClient` — stub only (`TODO(model-wiring)`); real proposals PENDING.
- ~~Model wiring for `RepairModelClient` — stub only.~~ **Built (#27)** —
`AnthropicRepairModelClient` (`src/runner/repair-anthropic.ts`), opt-in via
`gate:matrix --repair-model`. The stub stays the default so no run spends money or makes a
network call unless asked. **Still unmeasured:** no live repair has been observed. The client
is covered by 21 mocked-SDK tests; a self-heal rate remains structurally 0 until someone runs
it with a real key, which is the exit criterion #27 names and this repo will not fabricate.
- Whether `compiled_trajectory` bundle `$id` becomes a first-class contract (B3 packaging convention today).
- Fresh-reasoning cost capture for `cost_fresh` — measured separately; defaults to zeros when
unwired. Since [#123](https://github.com/DevToolie/Paragent/issues/123) this field means the
Expand Down
8 changes: 8 additions & 0 deletions experiments/gate-v1/live-run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import { MetricsEmitter } from "../../src/metrics/emitter.js";
import type { ProgramSource } from "../../src/metrics/types.js";
import { establishSession, LoginFailedError } from "../../src/recorder/preamble.js";
import { ReplayRunner } from "../../src/runner/replay.js";
import type { RepairModelClient } from "../../src/runner/repair.js";
import type {
CompiledProgram,
ParamBindings,
Expand Down Expand Up @@ -176,6 +177,12 @@ export interface LiveRunOptions {
programSource?: ProgramSource;
/** Advisory cache-health flag (ADR-0009). Never changes what is attempted. */
cacheProgramInvalidated?: boolean;
/**
* Opt-in real repair model (#27). Absent means the stub, which proposes
* nothing and costs nothing — the default everywhere, so no run spends money
* or makes a network call unless it was asked to.
*/
repairClient?: RepairModelClient;
/** Repeats of the program against this version. Defaults to 1. */
runs?: number;
/**
Expand Down Expand Up @@ -409,6 +416,7 @@ export async function runVersionLive(
...(opts.cacheProgramInvalidated !== undefined
? { cacheProgramInvalidated: opts.cacheProgramInvalidated }
: {}),
...(opts.repairClient ? { repairClient: opts.repairClient } : {}),
});

const params: ParamBindings = {
Expand Down
11 changes: 11 additions & 0 deletions experiments/gate-v1/run-matrix.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,13 @@ interface Args {
taskKey?: string;
/** Read only pool-eligible rows — the cross-tenant case (ADR-0014). */
poolOnly: boolean;
/**
* Opt-in real repair model (#27). Unset means the stub: no network call, no
* spend. Passing it requires ANTHROPIC_API_KEY, and the client throws at
* construction if it is missing rather than silently reporting a self-heal
* rate of 0 that looks measured.
*/
repairModel?: string;
}

function parseArgs(argv: string[]): Args {
Expand All @@ -146,6 +153,7 @@ function parseArgs(argv: string[]): Args {
"--from-cache",
"--site-key",
"--task-key",
"--repair-model",
]);
for (let i = 0; i < argv.length; i++) {
const a = argv[i] ?? "";
Expand Down Expand Up @@ -214,6 +222,9 @@ function usage(): void {
--task-key <k> Cache lookup key. Only meaningful with --from-cache.
--pool-only Resolve from pool-eligible rows only: the cross-tenant
case, where a tenant-scoped row must be invisible.
--repair-model <m> Use a REAL repair model instead of the stub (#27).
Costs money and needs ANTHROPIC_API_KEY. Omitted means the
stub: no network call, no spend, self-heal structurally 0.
--headed Show the browser (live runs only).
--keep-up Leave each container running after its run, for inspection.
--no-preamble Skip the login preamble, for programs that log in as part
Expand Down
86 changes: 78 additions & 8 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@
"vitest": "^4.1.10"
},
"dependencies": {
"@anthropic-ai/sdk": "^0.116.0",
"ajv": "^8.17.1",
"ajv-formats": "^3.0.1",
"playwright": "^1.62.1"
Expand Down
1 change: 1 addition & 0 deletions src/runner/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,6 @@ export * from "./actions.js";
export * from "./page-state.js";
export * from "./repair.js";
export * from "./repair-egress.js";
export * from "./repair-anthropic.js";
export * from "./replay.js";
export * from "./program.js";
Loading
Loading