diff --git a/apps/loopover-ui/content/docs/verify-this-review.mdx b/apps/loopover-ui/content/docs/verify-this-review.mdx index 1cace0e72..0bd3d1479 100644 --- a/apps/loopover-ui/content/docs/verify-this-review.mdx +++ b/apps/loopover-ui/content/docs/verify-this-review.mdx @@ -14,30 +14,46 @@ its checksum, replay the same scorer over it, and compare what you get against w Everything below runs read-only against a corpus export and pure functions from `@loopover/engine`. Nothing posts anywhere, and nothing needs a LoopOver API key. - - **Step 1 is not yet runnable by a stranger.** Exporting the corpus reads the deployment's own - database — with `--remote` that means the operator's Cloudflare D1, which needs *their* Cloudflare - credentials, not a LoopOver key. So today steps 1–3 are reproducible by an operator or by anyone - running a self-host deployment against their own data, and step 4 is the part a third party can - check unauthenticated. No anonymous corpus download exists yet; until one does, this page marks - each step with who can actually run it rather than implying everyone can run all of them. + + **Every step below is runnable by a stranger.** The corpus download needs no credentials of any + kind. It is *redacted*, and deliberately so: `targetKey` — the `owner/repo#number` a case came from + — is dropped outright rather than hashed, since `scoreBacktest` never reads it and a hash would + still be a stable per-PR identifier. Metadata is narrowed to the single `confidence` field the + shipped classifier reads, and timestamps are truncated to the day. What you lose is the ability to + point at *which* pull request a case was; what you keep is every input the scorer actually + consumes. The operator CLI further down is still there for anyone running their own deployment + against their own database. For the wider contract — every claim, its artifact, and its trust assumption, including the ones you cannot check — see [what you can verify](/docs/what-you-can-verify). -## 1. Export the corpus snapshot *(operator / self-host)* +## 1. Download the corpus *(anyone)* + +One rule's labeled fired/override history, over the same trailing window the fairness report covers, +as a checksummed JSON document ([backtest & calibration](/docs/backtest-calibration) explains how +that history is recorded): + +```bash +curl -s "https://api.loopover.ai/v1/public/eval-corpus?ruleId=ai_consensus_defect" > corpus.json +``` + +`ruleId` is required — a corpus only means anything for one rule, and the rule ids are the ones the +[fairness report](/fairness) lists. Each case carries `ruleId`, `outcome`, `label`, day-truncated +`firedAt`/`decidedAt`, and `metadata.confidence` when the firing recorded one. `caseCount` and +`truncated` describe the array; `truncated` is `true` only if a rule ever exceeds the published cap, +and it is reported rather than hidden. + +### Operator / self-host: the same corpus, unredacted -Every rule's fired/override history exports as a versioned, checksummed JSON snapshot -([backtest & calibration](/docs/backtest-calibration) explains how that history is recorded). -`--remote` shells out to `wrangler d1 execute … --remote`, so it reads the deployed database and -requires that deployment's own Cloudflare credentials: +Running your own deployment, the CLI reads your own database and keeps the target keys: ```bash npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --remote ``` -On a self-host deployment, point the same CLI at your own Postgres instead: +`--remote` shells out to `wrangler d1 execute … --remote`, so it needs that deployment's own +Cloudflare credentials. On a self-host deployment, point the same CLI at your own Postgres instead: ```bash npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --pg "$DATABASE_URL" @@ -50,23 +66,38 @@ covered — re-exporting that rule over the same window reproduces the same chec It is one run's freeze point, not a per-rule commitment for every rule on the report, so compare it against an export of the rule the run actually covered. -## 2. Verify the checksum *(anyone, given a snapshot)* +## 2. Verify the checksum *(anyone)* -The manifest is self-verifying: recompute the hash over its own `cases` array and compare it to -the recorded `checksum`. The canonicalization lives in `scripts/backtest-corpus-export-core.ts` -(`buildBacktestCorpusManifest`), so the check is one short script: +The document is self-verifying: recompute the hash over its own `cases` array and compare it to the +recorded `checksum`. **This step deliberately imports nothing from LoopOver** — canonical JSON is +eight lines, and a verification that runs our code to check our numbers is not much of a check: ```bash -node --experimental-strip-types -e ' -import { readFileSync } from "node:fs"; -import { buildBacktestCorpusManifest } from "./scripts/backtest-corpus-export-core.ts"; -const saved = JSON.parse(readFileSync("corpus.json", "utf8")); -const recomputed = buildBacktestCorpusManifest(saved.ruleId, saved.cases); -console.log(recomputed.checksum === saved.checksum ? "checksum OK" : "CHECKSUM MISMATCH"); +node -e ' +const { createHash } = require("node:crypto"); +const canonical = (v) => + Array.isArray(v) ? `[${v.map(canonical).join(",")}]` + : v && typeof v === "object" ? `{${Object.keys(v).sort().map((k) => `${JSON.stringify(k)}:${canonical(v[k])}`).join(",")}}` + : JSON.stringify(v); +const saved = JSON.parse(require("node:fs").readFileSync("corpus.json", "utf8")); +const recomputed = createHash("sha256").update(canonical(saved.cases)).digest("hex"); +console.log(recomputed === saved.checksum ? "checksum OK" : "CHECKSUM MISMATCH"); ' ``` -## 3. Replay the scorer *(anyone, given a snapshot)* +The rule is: recursively sort object keys, drop all insignificant whitespace, leave array order +alone (order is meaning there), then SHA-256 the UTF-8 bytes. That is the same canonicalization +every other digest in this system uses, including each EvalScoreRecord's `recordDigest`. + + + The **operator** export from step 1 uses a different, older canonicalization + (`buildBacktestCorpusManifest` in `scripts/backtest-corpus-export-core.ts`: shallow key-sort, then + `JSON.stringify` the array). The two checksums are therefore not comparable, by design — each one + commits to the artifact it ships with. Verify the downloaded document with the snippet above, and + the CLI export with `buildBacktestCorpusManifest`. + + +## 3. Replay the scorer *(anyone)* The published precision comes from the same pure functions any Node script can import: `scoreBacktest` replays a classifier over the labeled cases, and `compareBacktestScores` applies diff --git a/apps/loopover-ui/content/docs/what-you-can-verify.mdx b/apps/loopover-ui/content/docs/what-you-can-verify.mdx index beefaa3d2..55558d235 100644 --- a/apps/loopover-ui/content/docs/what-you-can-verify.mdx +++ b/apps/loopover-ui/content/docs/what-you-can-verify.mdx @@ -202,15 +202,22 @@ replayable history — not hand-entered numbers. This is the walkthrough in [Verify this review](/docs/verify-this-review): export the checksummed corpus, verify its checksum, re-run the same public scoring functions from `@loopover/engine`, and -compare. The checksum verification and the scorer replay are pure functions **anyone** can run over -a snapshot they hold, and the published numbers themselves are fetchable unauthenticated — but the -export step that produces the snapshot reads the deployment's own database and needs that -deployment's credentials, so it is an operator / self-host step today rather than an anonymous one. +compare. **Anyone** can now do all of it: the corpus downloads from +`https://api.loopover.ai/v1/public/eval-corpus?ruleId=...` with no credentials, the checksum check +imports nothing from LoopOver, and the scorer is a pure function from a published package. The +downloaded corpus is redacted — `targetKey` is dropped outright rather than hashed, and metadata is +narrowed to the one `confidence` field the classifier reads — so you can reproduce every number +without being able to point at which pull request any individual case was. - **Private repositories are the exception.** A hosted tenant's review history cannot be published - for anonymous third-party re-runs. Per-tenant checksummed export — so *you* can verify your own - numbers even when the public cannot — is tracked and not yet shipped. + **What private repositories do and do not contribute.** A private repo's cases DO appear in the + public corpus, because the published precision counts them and a corpus that omitted them would + not explain those numbers. They appear stripped of identity: no `targetKey`, no repo, no PR + number, no metadata beyond `confidence`, and timestamps truncated to the day — a row says "this + rule fired, a human called it right or wrong, at roughly this time", and nothing that ties it to a + repository. What is still *not* published is any identifiable per-tenant history; a per-tenant + checksummed export — so *you* can verify your own numbers against your own target keys — is + tracked and not yet shipped. ### 4. Attested execution diff --git a/apps/loopover-ui/public/openapi.json b/apps/loopover-ui/public/openapi.json index 8ec90f739..50b8e2576 100644 --- a/apps/loopover-ui/public/openapi.json +++ b/apps/loopover-ui/public/openapi.json @@ -27176,6 +27176,36 @@ } } } + }, + "/v1/public/eval-corpus": { + "get": { + "operationId": "getPublicEvalCorpus", + "tags": [ + "Public" + ], + "summary": "The redacted, checksummed corpus behind one rule's published precision — downloadable without credentials", + "parameters": [ + { + "schema": { + "type": "string" + }, + "required": true, + "name": "ruleId", + "in": "query" + } + ], + "responses": { + "200": { + "description": "{ ruleId, windowDays, caseCount, truncated, checksum, cases: [{ ruleId, outcome, label, firedAt, decidedAt, metadata?: { confidence } }] } — `targetKey` and every other metadata key are DROPPED, not hashed, so no repo or PR identity is published; `metadata.confidence` stays nested where buildConfidenceThresholdClassifier reads it. Timestamps are truncated to the UTC day. `checksum` is sha256 over canonicalJson(cases), i.e. it commits to the artifact you can actually download" + }, + "400": { + "description": "`ruleId` is required — a corpus is only meaningful for one rule" + }, + "404": { + "description": "Public stats are disabled for this deployment" + } + } + } } }, "servers": [ diff --git a/src/api/routes.ts b/src/api/routes.ts index 250e03142..b23cecbac 100644 --- a/src/api/routes.ts +++ b/src/api/routes.ts @@ -313,6 +313,7 @@ import { getPublicStats, isPublicStatsEnabled, resolvePublicStatsManifestOverrid import { loadPublicAccuracyTrend } from "../services/public-accuracy-trend"; import { loadPublicFleetAccuracyTrend } from "../services/public-fleet-accuracy-trend"; import { loadPublicRulePrecision } from "../review/public-rule-precision"; +import { loadPublicEvalCorpus } from "../review/public-eval-corpus"; import { loadCalibrationTrend } from "../services/rule-calibration-trend"; import { isSatisfactionFloorAutotuneEnabled, loadSatisfactionFloorStatus, runSatisfactionFloorLoosening } from "../services/satisfaction-floor-loosening-run"; import { loadAllKnobStatuses } from "../services/knob-loosening-run"; @@ -874,6 +875,23 @@ export function createApp() { // per-record, digest-committed form; it is not a new scoring surface). Wired for the // outcome_confirmed_precision source only today -- a future benchmark_run source (#9265) feeds the same // endpoint, never a second response format. + // #9636: the corpus behind the published per-rule precision, redacted and downloadable WITHOUT + // credentials -- the read path that makes the verifiability walkthrough's step 1 true for a stranger + // instead of only for an operator holding this deployment's Cloudflare keys. See + // public-eval-corpus.ts's header for why `targetKey` is dropped rather than hashed, and why + // `metadata.confidence` stays nested exactly where the shipped classifier reads it. + app.get("/v1/public/eval-corpus", async (c) => { + const publicStatsManifestOverride = await resolvePublicStatsManifestOverride(c.env); + if (!isPublicStatsEnabled(c.env, publicStatsManifestOverride)) return c.json({ error: "not_found" }, 404); + const ruleId = c.req.query("ruleId"); + // Required, not defaulted: a corpus is only meaningful for one rule, and silently picking one would + // publish a checksum for a rule the caller never asked about. + if (!ruleId) return c.json({ error: "rule_id_required" }, 400); + const corpus = await loadPublicEvalCorpus(c.env, ruleId); + c.header("Cache-Control", "public, max-age=60, stale-while-revalidate=300"); + return c.json(corpus); + }); + app.get("/v1/public/eval-scores", async (c) => { const publicStatsManifestOverride = await resolvePublicStatsManifestOverride(c.env); if (!isPublicStatsEnabled(c.env, publicStatsManifestOverride)) return c.json({ error: "not_found" }, 404); diff --git a/src/auth/route-auth.ts b/src/auth/route-auth.ts index 8df42732f..f54b2cb58 100644 --- a/src/auth/route-auth.ts +++ b/src/auth/route-auth.ts @@ -40,6 +40,10 @@ export function requiresApiToken(path: string): boolean { // #9266: the eval-scores transport is unauthenticated by the same design as every /v1/public/* sibling // above -- committed to a corpus checksum, independently re-derivable, nothing gated behind a token. if (path === "/v1/public/eval-scores") return false; + // #9636: same posture as its sibling above -- the corpus is redacted at the source (no target keys, no + // repos, no metadata beyond the one confidence field replay needs), so there is nothing here an + // Authorization header would be protecting. + if (path === "/v1/public/eval-corpus") return false; // #9269: the single-row read, added in the SAME PR as its route so the two can never drift the way #9120's // sibling did. Regex (not a literal) because of the :seq path parameter. if (/^\/v1\/public\/decision-ledger\/row\/[^/]+$/.test(path)) return false; diff --git a/src/openapi/spec.ts b/src/openapi/spec.ts index 8e36ce144..02a21e55f 100644 --- a/src/openapi/spec.ts +++ b/src/openapi/spec.ts @@ -1969,6 +1969,22 @@ export function buildOpenApiSpec() { 404: { description: "No decision record persisted yet for this PR" }, }, }); + registry.registerPath({ + method: "get", + path: "/v1/public/eval-corpus", + operationId: "getPublicEvalCorpus", + tags: ["Public"], + summary: "The redacted, checksummed corpus behind one rule's published precision — downloadable without credentials", + request: { query: z.object({ ruleId: z.string() }) }, + responses: { + 200: { + description: + "{ ruleId, windowDays, caseCount, truncated, checksum, cases: [{ ruleId, outcome, label, firedAt, decidedAt, metadata?: { confidence } }] } — `targetKey` and every other metadata key are DROPPED, not hashed, so no repo or PR identity is published; `metadata.confidence` stays nested where buildConfidenceThresholdClassifier reads it. Timestamps are truncated to the UTC day. `checksum` is sha256 over canonicalJson(cases), i.e. it commits to the artifact you can actually download", + }, + 400: { description: "`ruleId` is required — a corpus is only meaningful for one rule" }, + 404: { description: "Public stats are disabled for this deployment" }, + }, + }); registry.registerPath({ method: "get", path: "/v1/public/eval-scores", diff --git a/src/review/public-eval-corpus.ts b/src/review/public-eval-corpus.ts new file mode 100644 index 000000000..9681e1494 --- /dev/null +++ b/src/review/public-eval-corpus.ts @@ -0,0 +1,151 @@ +// Public, anonymously-downloadable eval corpus (#9636). The verifiability walkthrough +// (apps/loopover-ui/content/docs/verify-this-review.mdx) tells a skeptic to "export the same corpus the +// numbers come from" -- but the only exporter is scripts/backtest-corpus-export.ts, which shells out to +// `wrangler d1 execute --remote` against the deployment's own database and needs THAT deployment's +// Cloudflare credentials. Step 1 was never runnable by a third party, which made the page's "nothing +// needs an API key" claim false. This is the read path that makes it true. +// +// REDACTION IS THE WHOLE DESIGN. A raw BacktestCase carries `targetKey` (literally `owner/repo#number`) +// and the firing's entire `metadata` bag. public-rule-precision.ts's own header states this surface +// publishes "no target keys, no repos, no confidence distributions, no corpus content" -- so publishing +// the raw corpus would leak exactly what the rest of the surface is careful to withhold, for PRIVATE +// repositories included. Two facts make redaction cheap rather than lossy: +// 1. `scoreBacktest` never reads `targetKey`. It only needs `ruleId` + `label` plus whatever the +// caller's classifier reads. So the identity field can be dropped outright, not hashed -- a hash +// would still be a stable per-PR identifier and still correlatable, for no replay benefit. +// 2. `buildConfidenceThresholdClassifier` reads `metadata.confidence` (backtest-threshold.ts:21), so +// that one key is kept NESTED exactly where the shipped classifier looks. Flattening it to a +// top-level field would make every replay silently degrade to the `?? 1` fallback and classify +// every case "confirmed" -- a wrong answer that looks like a working one. +// +// Timestamps are published truncated to the DAY. Nothing in the replay path reads them (they exist for +// a human sanity-checking the window), while full-precision pairs are the one remaining vector for +// correlating a case back to a specific private-repo PR by lining it up against that repo's timeline. +// +// The checksum commits to THIS artifact -- the redacted, published bytes -- not to the internal manifest +// scripts/backtest-corpus-export.ts produces. That is the correct commitment: a reader can only re-derive +// what they can actually download, and a checksum over a preimage nobody can obtain verifies nothing +// (the exact failure #9636 fixed on the eval-score records). +import { buildBacktestCorpus } from "@loopover/engine/calibration/backtest-corpus"; +import { canonicalJson, sha256Hex } from "./decision-record"; +import { NON_ATTRIBUTABLE_OVERRIDE_PROVENANCES, PUBLIC_PRECISION_WINDOW_DAYS } from "./public-rule-precision"; +import { createSignalStore } from "./signal-tracking-wire"; + +/** Hard cap on published cases. The window is already bounded, but an unbounded array on an + * unauthenticated route is a footgun the moment a rule gets noisy; a truncated corpus is reported + * honestly via `truncated` rather than silently trimmed. */ +export const PUBLIC_EVAL_CORPUS_MAX_CASES = 5_000; + +/** One published case: a {@link BacktestCase} minus `targetKey`, with `metadata` narrowed to the single + * key the shipped classifier reads. `metadata` is omitted entirely (never `undefined`) when the firing + * recorded no confidence, matching BacktestCase's own optional-property discipline. */ +export type PublicEvalCorpusCase = { + ruleId: string; + outcome: string; + label: "reversed" | "confirmed"; + firedAt: string; + decidedAt: string; + metadata?: { confidence: number }; +}; + +export type PublicEvalCorpus = { + ruleId: string; + windowDays: number; + caseCount: number; + truncated: boolean; + checksum: string; + cases: PublicEvalCorpusCase[]; +}; + +/** ISO instant truncated to its UTC day. See the module header for why. */ +function toUtcDay(iso: string): string { + const parsed = Date.parse(iso); + // A row whose timestamp doesn't parse keeps its original string rather than becoming "Invalid Date": + // the value is already only advisory here, and silently emitting a broken date would be worse. + if (!Number.isFinite(parsed)) return iso; + return `${new Date(parsed).toISOString().slice(0, 10)}T00:00:00.000Z`; +} + +/** PURE. Strip a raw case down to its publishable shape. */ +export function redactBacktestCase(input: { + ruleId: string; + outcome: string; + label: "reversed" | "confirmed"; + firedAt: string; + decidedAt: string; + metadata?: Record | undefined; +}): PublicEvalCorpusCase { + const confidence = input.metadata?.["confidence"]; + return { + ruleId: input.ruleId, + outcome: input.outcome, + label: input.label, + firedAt: toUtcDay(input.firedAt), + decidedAt: toUtcDay(input.decidedAt), + ...(typeof confidence === "number" ? { metadata: { confidence } } : {}), + }; +} + +/** PURE. Deterministic order for the checksum. `targetKey` -- the natural tiebreak -- is deliberately + * gone, so ordering falls back to the published fields themselves; two genuinely identical cases are + * interchangeable, so a stable total order over those fields is sufficient and reproducible. */ +export function sortPublicEvalCorpusCases(cases: readonly PublicEvalCorpusCase[]): PublicEvalCorpusCase[] { + return [...cases].sort( + (a, b) => + a.firedAt.localeCompare(b.firedAt) || + a.decidedAt.localeCompare(b.decidedAt) || + a.outcome.localeCompare(b.outcome) || + a.label.localeCompare(b.label) || + (a.metadata?.confidence ?? -1) - (b.metadata?.confidence ?? -1), + ); +} + +/** PURE. Apply {@link PUBLIC_EVAL_CORPUS_MAX_CASES}, reporting truncation rather than silently trimming. + * Split out from the loader so both arms are exercised directly -- driving the truthy side through the + * real store would mean seeding thousands of fired/override pairs for one boolean. */ +export function applyPublicEvalCorpusCap(cases: readonly PublicEvalCorpusCase[]): { cases: PublicEvalCorpusCase[]; truncated: boolean } { + const truncated = cases.length > PUBLIC_EVAL_CORPUS_MAX_CASES; + return { cases: truncated ? cases.slice(0, PUBLIC_EVAL_CORPUS_MAX_CASES) : [...cases], truncated }; +} + +/** The published checksum: SHA-256 over the canonical JSON of the published `cases` array. Uses the same + * canonicalJson + Web Crypto pair every other digest in this system uses, so a consumer needs one + * serialization rule, not two. */ +export async function checksumPublicEvalCorpus(cases: readonly PublicEvalCorpusCase[]): Promise { + return sha256Hex(canonicalJson(cases)); +} + +/** + * Load one rule's publishable corpus over the same trailing window `/v1/public/stats`'s rulePrecision + * block reports. Fail-safe: a read error yields an empty corpus (checksummed as such), never a thrown + * public endpoint -- the same degradation contract as loadPublicRulePrecision. + */ +export async function loadPublicEvalCorpus(env: Env, ruleId: string, nowMs: number = Date.now()): Promise { + const sinceMs = nowMs - PUBLIC_PRECISION_WINDOW_DAYS * 24 * 60 * 60 * 1000; + let fired: Awaited["queryRuleHistory"]>>["fired"] = []; + let overrides: Awaited["queryRuleHistory"]>>["overrides"] = []; + try { + ({ fired, overrides } = await createSignalStore(env).queryRuleHistory(ruleId, sinceMs)); + } catch { + // Fall through to an empty corpus rather than 500ing an unauthenticated route. + } + + // Same exclusion the published per-rule precision applies: an override whose verdict was a human + // decision about a DIFFERENT rule cannot support a claim about this one, so it must not appear in the + // corpus backing that claim either -- otherwise the corpus and the precision it explains disagree. + const attributable = overrides.filter((override) => { + const provenance = override.metadata?.["provenance"]; + return typeof provenance !== "string" || !(NON_ATTRIBUTABLE_OVERRIDE_PROVENANCES as readonly string[]).includes(provenance); + }); + + const { cases, truncated } = applyPublicEvalCorpusCap(sortPublicEvalCorpusCases(buildBacktestCorpus(ruleId, fired, attributable).map(redactBacktestCase))); + + return { + ruleId, + windowDays: PUBLIC_PRECISION_WINDOW_DAYS, + caseCount: cases.length, + truncated, + checksum: await checksumPublicEvalCorpus(cases), + cases, + }; +} diff --git a/src/review/public-rule-precision.ts b/src/review/public-rule-precision.ts index 4e2cb1c3f..8747ee0e8 100644 --- a/src/review/public-rule-precision.ts +++ b/src/review/public-rule-precision.ts @@ -70,7 +70,7 @@ export type PublicRulePrecision = { * NOT excluded: `review_targets_decision_level` (#8083's own backfill), whose labels come from what actually * happened to the PRs THAT rule fired on -- synthesized rows, but genuinely about that rule. */ -const NON_ATTRIBUTABLE_OVERRIDE_PROVENANCES = ["slop_replay_backfill_v1"] as const; +export const NON_ATTRIBUTABLE_OVERRIDE_PROVENANCES = ["slop_replay_backfill_v1"] as const; /** * Load the public per-rule precision block. Fail-safe per section (the same degradation contract as diff --git a/test/integration/public-decision-ledger-routes.test.ts b/test/integration/public-decision-ledger-routes.test.ts index 82e110cb7..9b86df7d6 100644 --- a/test/integration/public-decision-ledger-routes.test.ts +++ b/test/integration/public-decision-ledger-routes.test.ts @@ -84,6 +84,7 @@ describe("public decision-ledger/decision-records routes answer WITHOUT credenti ["repos/:owner/:repo/badge.json", "/v1/public/repos/acme/widgets/badge.json"], ["repos/:owner/:repo/quality", "/v1/public/repos/acme/widgets/quality"], ["eval-scores", "/v1/public/eval-scores"], + ["eval-corpus", "/v1/public/eval-corpus?ruleId=ai_consensus_defect"], ]; it.each(exemptPublicRoutes)("%s answers without credentials (never 401)", async (_name, path) => { diff --git a/test/integration/public-eval-corpus-route.test.ts b/test/integration/public-eval-corpus-route.test.ts new file mode 100644 index 000000000..0927e250f --- /dev/null +++ b/test/integration/public-eval-corpus-route.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from "vitest"; +import { createApp } from "../../src/api/routes"; +import { createTestEnv } from "../helpers/d1"; +import { createSignalStore } from "../../src/review/signal-tracking-wire"; + +// #9636: the route that makes the verifiability walkthrough's step 1 runnable by a stranger. The +// load-bearing properties are that it needs no Authorization header, that it publishes nothing +// identifying, and that its checksum commits to the bytes the caller just received. + +function enabledEnv(): Env { + return createTestEnv({ LOOPOVER_PUBLIC_STATS: "true" }); +} + +async function seed(env: Env, targetKey: string, verdict: "confirmed" | "reversed"): Promise { + const store = createSignalStore(env); + await store.recordRuleFired({ ruleId: "ai_consensus_defect", targetKey, outcome: "close", occurredAt: new Date().toISOString(), metadata: { confidence: 0.6 } }); + await store.recordHumanOverride({ ruleId: "ai_consensus_defect", targetKey, verdict, occurredAt: new Date().toISOString() }); +} + +describe("GET /v1/public/eval-corpus (#9636)", () => { + it("answers 200 with NO Authorization header -- the whole point of the endpoint", async () => { + const env = enabledEnv(); + await seed(env, "acme/widgets#1", "confirmed"); + const response = await createApp().request("/v1/public/eval-corpus?ruleId=ai_consensus_defect", {}, env); + expect(response.status).toBe(200); + const body = (await response.json()) as { ruleId: string; caseCount: number; checksum: string; truncated: boolean }; + expect(body.ruleId).toBe("ai_consensus_defect"); + expect(body.caseCount).toBe(1); + expect(body.truncated).toBe(false); + expect(body.checksum).toMatch(/^[0-9a-f]{64}$/); + }); + + it("INVARIANT: the response body never carries a target key, repo, or PR number", async () => { + const env = enabledEnv(); + await seed(env, "acme/private-widgets#4242", "reversed"); + const response = await createApp().request("/v1/public/eval-corpus?ruleId=ai_consensus_defect", {}, env); + expect(await response.text()).not.toMatch(/acme|private-widgets|4242|targetKey/i); + }); + + it("the published checksum is recomputable from the published cases alone", async () => { + const env = enabledEnv(); + await seed(env, "acme/widgets#2", "confirmed"); + const body = (await (await createApp().request("/v1/public/eval-corpus?ruleId=ai_consensus_defect", {}, env)).json()) as { + checksum: string; + cases: unknown[]; + }; + // Exactly what a third party runs: hash the canonicalized cases array they were handed. + const { canonicalJson, sha256Hex } = await import("../../src/review/decision-record"); + expect(await sha256Hex(canonicalJson(body.cases))).toBe(body.checksum); + }); + + it("400s without a ruleId rather than silently picking one", async () => { + const response = await createApp().request("/v1/public/eval-corpus", {}, enabledEnv()); + expect(response.status).toBe(400); + expect(await response.json()).toEqual({ error: "rule_id_required" }); + }); + + it("404s when LOOPOVER_PUBLIC_STATS is off (default) -- same flag as its siblings", async () => { + const response = await createApp().request("/v1/public/eval-corpus?ruleId=ai_consensus_defect", {}, createTestEnv()); + expect(response.status).toBe(404); + }); + + it("returns an empty but still-checksummed corpus for a rule with no history", async () => { + const response = await createApp().request("/v1/public/eval-corpus?ruleId=never_fired", {}, enabledEnv()); + expect(response.status).toBe(200); + const body = (await response.json()) as { caseCount: number; cases: unknown[]; checksum: string }; + expect(body).toMatchObject({ caseCount: 0, cases: [] }); + // sha256("[]") -- an empty corpus is still committed to, distinguishably. + expect(body.checksum).toBe("4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945"); + }); + + it("sets the same Cache-Control posture as its public siblings", async () => { + const response = await createApp().request("/v1/public/eval-corpus?ruleId=ai_consensus_defect", {}, enabledEnv()); + expect(response.headers.get("Cache-Control")).toBe("public, max-age=60, stale-while-revalidate=300"); + }); +}); diff --git a/test/unit/public-eval-corpus.test.ts b/test/unit/public-eval-corpus.test.ts new file mode 100644 index 000000000..6cc223df0 --- /dev/null +++ b/test/unit/public-eval-corpus.test.ts @@ -0,0 +1,209 @@ +import { describe, expect, it } from "vitest"; +import { + applyPublicEvalCorpusCap, + checksumPublicEvalCorpus, + loadPublicEvalCorpus, + PUBLIC_EVAL_CORPUS_MAX_CASES, + redactBacktestCase, + sortPublicEvalCorpusCases, + type PublicEvalCorpusCase, +} from "../../src/review/public-eval-corpus"; +import { canonicalJson, sha256Hex } from "../../src/review/decision-record"; +import { PUBLIC_PRECISION_WINDOW_DAYS } from "../../src/review/public-rule-precision"; +import { createSignalStore } from "../../src/review/signal-tracking-wire"; +import { createTestEnv } from "../helpers/d1"; + +// #9636: the anonymously-downloadable corpus. The load-bearing properties are that NOTHING identifying +// is published, that the one field the shipped classifier reads survives in the shape it reads it from, +// and that the checksum commits to the bytes a reader can actually obtain. + +const NOW = Date.parse("2026-07-23T12:00:00.000Z"); + +async function seedPair( + env: Env, + input: { ruleId: string; targetKey: string; verdict: "confirmed" | "reversed"; confidence?: number; provenance?: string; firedOffsetMs?: number }, +): Promise { + const store = createSignalStore(env); + await store.recordRuleFired({ + ruleId: input.ruleId, + targetKey: input.targetKey, + outcome: "close", + occurredAt: new Date(NOW - (input.firedOffsetMs ?? 20_000)).toISOString(), + ...(input.confidence === undefined ? {} : { metadata: { confidence: input.confidence } }), + }); + await store.recordHumanOverride({ + ruleId: input.ruleId, + targetKey: input.targetKey, + verdict: input.verdict, + occurredAt: new Date(NOW - 10_000).toISOString(), + ...(input.provenance === undefined ? {} : { metadata: { provenance: input.provenance } }), + }); +} + +describe("redactBacktestCase (#9636)", () => { + const raw = { + ruleId: "ai_consensus_defect", + targetKey: "acme/private-widgets#42", + outcome: "close", + label: "reversed" as const, + firedAt: "2026-07-20T14:37:11.482Z", + decidedAt: "2026-07-21T09:02:55.001Z", + metadata: { confidence: 0.4, prompt: "secret prompt text", repo: "acme/private-widgets" }, + }; + + it("INVARIANT: drops targetKey and every metadata key except confidence", () => { + const redacted = redactBacktestCase(raw); + const serialized = JSON.stringify(redacted); + expect(serialized).not.toMatch(/acme|private-widgets|#42|secret prompt|targetKey|repo/i); + expect(redacted.metadata).toEqual({ confidence: 0.4 }); + expect(Object.keys(redacted).sort()).toEqual(["decidedAt", "firedAt", "label", "metadata", "outcome", "ruleId"]); + }); + + it("keeps confidence NESTED under metadata, where buildConfidenceThresholdClassifier reads it", () => { + // Flattening this to a top-level `confidence` would make the shipped classifier hit its `?? 1` + // fallback and label every case "confirmed" -- a wrong replay that looks like a working one. + expect(redactBacktestCase(raw).metadata?.confidence).toBe(0.4); + }); + + it("omits metadata entirely (never undefined) when the firing recorded no confidence", () => { + const redacted = redactBacktestCase({ ...raw, metadata: { prompt: "x" } }); + expect("metadata" in redacted).toBe(false); + }); + + it("omits metadata when confidence is present but not a number", () => { + expect("metadata" in redactBacktestCase({ ...raw, metadata: { confidence: "0.4" } })).toBe(false); + }); + + it("truncates both timestamps to the UTC day", () => { + const redacted = redactBacktestCase(raw); + expect(redacted.firedAt).toBe("2026-07-20T00:00:00.000Z"); + expect(redacted.decidedAt).toBe("2026-07-21T00:00:00.000Z"); + }); + + it("passes an unparseable timestamp through rather than emitting an Invalid Date", () => { + const redacted = redactBacktestCase({ ...raw, firedAt: "not-a-date" }); + expect(redacted.firedAt).toBe("not-a-date"); + }); +}); + +describe("sortPublicEvalCorpusCases", () => { + const base: PublicEvalCorpusCase = { ruleId: "r", outcome: "close", label: "confirmed", firedAt: "2026-07-02T00:00:00.000Z", decidedAt: "2026-07-03T00:00:00.000Z" }; + + it("is a deterministic total order over the published fields (targetKey is gone as a tiebreak)", () => { + const cases: PublicEvalCorpusCase[] = [ + { ...base, firedAt: "2026-07-05T00:00:00.000Z" }, + { ...base, metadata: { confidence: 0.9 } }, + { ...base, metadata: { confidence: 0.1 } }, + { ...base, label: "reversed" }, + { ...base, outcome: "advise" }, + { ...base, decidedAt: "2026-07-01T00:00:00.000Z" }, + ]; + const once = sortPublicEvalCorpusCases(cases); + // Stable regardless of input order -- the property the checksum depends on. + expect(sortPublicEvalCorpusCases([...cases].reverse())).toEqual(once); + expect(once[0]?.decidedAt).toBe("2026-07-01T00:00:00.000Z"); + expect(once[once.length - 1]?.firedAt).toBe("2026-07-05T00:00:00.000Z"); + }); + + it("orders a case with no confidence before one that has it (both sides of the ?? tiebreak)", () => { + // Identical in every earlier field, so the comparator actually reaches the confidence tiebreak with + // one side absent -- the only way the `?? -1` fallbacks are exercised at all. + const withConfidence: PublicEvalCorpusCase = { ...base, metadata: { confidence: 0.3 } }; + const sorted = sortPublicEvalCorpusCases([withConfidence, base]); + expect(sorted[0]).toEqual(base); + expect(sorted[1]).toEqual(withConfidence); + // And the reverse input yields the same order, so the fallback is symmetric. + expect(sortPublicEvalCorpusCases([base, withConfidence])).toEqual(sorted); + }); + + it("does not mutate its input", () => { + const cases = [{ ...base, firedAt: "2026-07-09T00:00:00.000Z" }, base]; + sortPublicEvalCorpusCases(cases); + expect(cases[0]?.firedAt).toBe("2026-07-09T00:00:00.000Z"); + }); +}); + +describe("applyPublicEvalCorpusCap", () => { + const one = (i: number): PublicEvalCorpusCase => ({ ruleId: "r", outcome: "close", label: "confirmed", firedAt: `2026-07-20T00:00:00.00${i % 10}Z`, decidedAt: "2026-07-21T00:00:00.000Z" }); + + it("passes an under-cap corpus through untruncated", () => { + const result = applyPublicEvalCorpusCap([one(1), one(2)]); + expect(result).toEqual({ cases: [one(1), one(2)], truncated: false }); + }); + + it("caps an over-cap corpus and says so", () => { + const result = applyPublicEvalCorpusCap(Array.from({ length: PUBLIC_EVAL_CORPUS_MAX_CASES + 1 }, (_, i) => one(i))); + expect(result.truncated).toBe(true); + expect(result.cases).toHaveLength(PUBLIC_EVAL_CORPUS_MAX_CASES); + }); + + it("returns exactly the cap without truncating at the boundary", () => { + const result = applyPublicEvalCorpusCap(Array.from({ length: PUBLIC_EVAL_CORPUS_MAX_CASES }, (_, i) => one(i))); + expect(result.truncated).toBe(false); + expect(result.cases).toHaveLength(PUBLIC_EVAL_CORPUS_MAX_CASES); + }); +}); + +describe("checksumPublicEvalCorpus", () => { + it("is sha256 over canonicalJson of the published cases -- independently recomputable", async () => { + const cases: PublicEvalCorpusCase[] = [{ ruleId: "r", outcome: "close", label: "confirmed", firedAt: "2026-07-02T00:00:00.000Z", decidedAt: "2026-07-03T00:00:00.000Z" }]; + expect(await checksumPublicEvalCorpus(cases)).toBe(await sha256Hex(canonicalJson(cases))); + }); + + it("commits to an empty corpus distinguishably (the value #9636 made unpublishable elsewhere)", async () => { + expect(await checksumPublicEvalCorpus([])).toBe(await sha256Hex("[]")); + }); +}); + +describe("loadPublicEvalCorpus — end to end over the real signal tables", () => { + it("builds a redacted, checksummed corpus from real fired/override pairs", async () => { + const env = createTestEnv(); + await seedPair(env, { ruleId: "ai_consensus_defect", targetKey: "acme/widgets#1", verdict: "confirmed", confidence: 0.8 }); + await seedPair(env, { ruleId: "ai_consensus_defect", targetKey: "acme/widgets#2", verdict: "reversed", confidence: 0.2, firedOffsetMs: 30_000 }); + + const corpus = await loadPublicEvalCorpus(env, "ai_consensus_defect", NOW); + expect(corpus.ruleId).toBe("ai_consensus_defect"); + expect(corpus.windowDays).toBe(PUBLIC_PRECISION_WINDOW_DAYS); + expect(corpus.caseCount).toBe(2); + expect(corpus.truncated).toBe(false); + expect(corpus.checksum).toBe(await checksumPublicEvalCorpus(corpus.cases)); + expect(corpus.cases.map((c) => c.label).sort()).toEqual(["confirmed", "reversed"]); + }); + + it("INVARIANT: the serialized corpus never carries a target key, repo, or PR number", async () => { + const env = createTestEnv(); + await seedPair(env, { ruleId: "ai_consensus_defect", targetKey: "acme/private-widgets#4242", verdict: "confirmed", confidence: 0.5 }); + const serialized = JSON.stringify(await loadPublicEvalCorpus(env, "ai_consensus_defect", NOW)); + expect(serialized).not.toMatch(/acme|private-widgets|4242|targetKey/i); + }); + + it("excludes counterfactual-replay overrides, so the corpus agrees with the precision it explains", async () => { + const env = createTestEnv(); + await seedPair(env, { ruleId: "slop_gate_score", targetKey: "acme/widgets#7", verdict: "confirmed", provenance: "slop_replay_backfill_v1" }); + expect((await loadPublicEvalCorpus(env, "slop_gate_score", NOW)).caseCount).toBe(0); + }); + + it("keeps a synthesized override whose labels ARE about the rule it is filed under", async () => { + const env = createTestEnv(); + await seedPair(env, { ruleId: "ai_consensus_defect", targetKey: "acme/widgets#8", verdict: "confirmed", provenance: "review_targets_decision_level" }); + expect((await loadPublicEvalCorpus(env, "ai_consensus_defect", NOW)).caseCount).toBe(1); + }); + + it("returns an empty, still-checksummed corpus for a rule with no history", async () => { + const corpus = await loadPublicEvalCorpus(createTestEnv(), "never_fired", NOW); + expect(corpus).toMatchObject({ caseCount: 0, truncated: false, cases: [] }); + expect(corpus.checksum).toBe(await sha256Hex("[]")); + }); + + it("degrades to an empty corpus (never throws) when the store read fails", async () => { + const broken = createTestEnv(); + broken.DB = { prepare: () => { throw new Error("boom"); } } as never; + const corpus = await loadPublicEvalCorpus(broken, "ai_consensus_defect", NOW); + expect(corpus.caseCount).toBe(0); + expect(corpus.checksum).toBe(await sha256Hex("[]")); + }); + + it("reports truncation honestly rather than silently trimming", async () => { + expect((await loadPublicEvalCorpus(createTestEnv(), "r", NOW)).truncated).toBe(false); + }); +});