feat(api): publish the eval corpus so a stranger can actually verify a review

apps/loopover-ui/content/docs/verify-this-review.mdx

@@ -14,30 +14,46 @@ its checksum, replay the same scorer over it, and compare what you get against w
 Everything below runs read-only against a corpus export and pure functions from `@loopover/engine`.
 Nothing posts anywhere, and nothing needs a LoopOver API key.
 
-<Callout variant="warn">
-  **Step 1 is not yet runnable by a stranger.** Exporting the corpus reads the deployment's own
-  database — with `--remote` that means the operator's Cloudflare D1, which needs *their* Cloudflare
-  credentials, not a LoopOver key. So today steps 1–3 are reproducible by an operator or by anyone
-  running a self-host deployment against their own data, and step 4 is the part a third party can
-  check unauthenticated. No anonymous corpus download exists yet; until one does, this page marks
-  each step with who can actually run it rather than implying everyone can run all of them.
+<Callout variant="note">
+  **Every step below is runnable by a stranger.** The corpus download needs no credentials of any
+  kind. It is *redacted*, and deliberately so: `targetKey` — the `owner/repo#number` a case came from
+  — is dropped outright rather than hashed, since `scoreBacktest` never reads it and a hash would
+  still be a stable per-PR identifier. Metadata is narrowed to the single `confidence` field the
+  shipped classifier reads, and timestamps are truncated to the day. What you lose is the ability to
+  point at *which* pull request a case was; what you keep is every input the scorer actually
+  consumes. The operator CLI further down is still there for anyone running their own deployment
+  against their own database.
 </Callout>
 
 For the wider contract — every claim, its artifact, and its trust assumption, including the ones you
 cannot check — see [what you can verify](/docs/what-you-can-verify).
 
-## 1. Export the corpus snapshot *(operator / self-host)*
+## 1. Download the corpus *(anyone)*
+
+One rule's labeled fired/override history, over the same trailing window the fairness report covers,
+as a checksummed JSON document ([backtest &amp; calibration](/docs/backtest-calibration) explains how
+that history is recorded):
+
+```bash
+curl -s "https://api.loopover.ai/v1/public/eval-corpus?ruleId=ai_consensus_defect" > corpus.json
+```
+
+`ruleId` is required — a corpus only means anything for one rule, and the rule ids are the ones the
+[fairness report](/fairness) lists. Each case carries `ruleId`, `outcome`, `label`, day-truncated
+`firedAt`/`decidedAt`, and `metadata.confidence` when the firing recorded one. `caseCount` and
+`truncated` describe the array; `truncated` is `true` only if a rule ever exceeds the published cap,
+and it is reported rather than hidden.
+
+### Operator / self-host: the same corpus, unredacted
 
-Every rule's fired/override history exports as a versioned, checksummed JSON snapshot
-([backtest &amp; calibration](/docs/backtest-calibration) explains how that history is recorded).
-`--remote` shells out to `wrangler d1 execute … --remote`, so it reads the deployed database and
-requires that deployment's own Cloudflare credentials:
+Running your own deployment, the CLI reads your own database and keeps the target keys:
 
 ```bash
 npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --remote
 ```
 
-On a self-host deployment, point the same CLI at your own Postgres instead:
+`--remote` shells out to `wrangler d1 execute … --remote`, so it needs that deployment's own
+Cloudflare credentials. On a self-host deployment, point the same CLI at your own Postgres instead:
 
 ```bash
 npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --pg "$DATABASE_URL"
@@ -50,23 +66,38 @@ covered — re-exporting that rule over the same window reproduces the same chec
 It is one run's freeze point, not a per-rule commitment for every rule on the report, so compare it
 against an export of the rule the run actually covered.
 
-## 2. Verify the checksum *(anyone, given a snapshot)*
+## 2. Verify the checksum *(anyone)*
 
-The manifest is self-verifying: recompute the hash over its own `cases` array and compare it to
-the recorded `checksum`. The canonicalization lives in `scripts/backtest-corpus-export-core.ts`
-(`buildBacktestCorpusManifest`), so the check is one short script:
+The document is self-verifying: recompute the hash over its own `cases` array and compare it to the
+recorded `checksum`. **This step deliberately imports nothing from LoopOver** — canonical JSON is
+eight lines, and a verification that runs our code to check our numbers is not much of a check:
 
 ```bash
-node --experimental-strip-types -e '
-import { readFileSync } from "node:fs";
-import { buildBacktestCorpusManifest } from "./scripts/backtest-corpus-export-core.ts";
-const saved = JSON.parse(readFileSync("corpus.json", "utf8"));
-const recomputed = buildBacktestCorpusManifest(saved.ruleId, saved.cases);
-console.log(recomputed.checksum === saved.checksum ? "checksum OK" : "CHECKSUM MISMATCH");
+node -e '
+const { createHash } = require("node:crypto");
+const canonical = (v) =>
+  Array.isArray(v) ? `[${v.map(canonical).join(",")}]`
+  : v && typeof v === "object" ? `{${Object.keys(v).sort().map((k) => `${JSON.stringify(k)}:${canonical(v[k])}`).join(",")}}`
+  : JSON.stringify(v);
+const saved = JSON.parse(require("node:fs").readFileSync("corpus.json", "utf8"));
+const recomputed = createHash("sha256").update(canonical(saved.cases)).digest("hex");
+console.log(recomputed === saved.checksum ? "checksum OK" : "CHECKSUM MISMATCH");
 '
 ```
 
-## 3. Replay the scorer *(anyone, given a snapshot)*
+The rule is: recursively sort object keys, drop all insignificant whitespace, leave array order
+alone (order is meaning there), then SHA-256 the UTF-8 bytes. That is the same canonicalization
+every other digest in this system uses, including each EvalScoreRecord's `recordDigest`.
+
+<Callout variant="note">
+  The **operator** export from step 1 uses a different, older canonicalization
+  (`buildBacktestCorpusManifest` in `scripts/backtest-corpus-export-core.ts`: shallow key-sort, then
+  `JSON.stringify` the array). The two checksums are therefore not comparable, by design — each one
+  commits to the artifact it ships with. Verify the downloaded document with the snippet above, and
+  the CLI export with `buildBacktestCorpusManifest`.
+</Callout>
+
+## 3. Replay the scorer *(anyone)*
 
 The published precision comes from the same pure functions any Node script can import:
 `scoreBacktest` replays a classifier over the labeled cases, and `compareBacktestScores` applies

apps/loopover-ui/content/docs/what-you-can-verify.mdx

@@ -202,15 +202,22 @@ replayable history — not hand-entered numbers.
 
 This is the walkthrough in [Verify this review](/docs/verify-this-review): export the checksummed
 corpus, verify its checksum, re-run the same public scoring functions from `@loopover/engine`, and
-compare. The checksum verification and the scorer replay are pure functions **anyone** can run over
-a snapshot they hold, and the published numbers themselves are fetchable unauthenticated — but the
-export step that produces the snapshot reads the deployment's own database and needs that
-deployment's credentials, so it is an operator / self-host step today rather than an anonymous one.
+compare. **Anyone** can now do all of it: the corpus downloads from
+`https://api.loopover.ai/v1/public/eval-corpus?ruleId=...` with no credentials, the checksum check
+imports nothing from LoopOver, and the scorer is a pure function from a published package. The
+downloaded corpus is redacted — `targetKey` is dropped outright rather than hashed, and metadata is
+narrowed to the one `confidence` field the classifier reads — so you can reproduce every number
+without being able to point at which pull request any individual case was.
 
 <Callout variant="warn">
-  **Private repositories are the exception.** A hosted tenant's review history cannot be published
-  for anonymous third-party re-runs. Per-tenant checksummed export — so *you* can verify your own
-  numbers even when the public cannot — is tracked and not yet shipped.
+  **What private repositories do and do not contribute.** A private repo's cases DO appear in the
+  public corpus, because the published precision counts them and a corpus that omitted them would
+  not explain those numbers. They appear stripped of identity: no `targetKey`, no repo, no PR
+  number, no metadata beyond `confidence`, and timestamps truncated to the day — a row says "this
+  rule fired, a human called it right or wrong, at roughly this time", and nothing that ties it to a
+  repository. What is still *not* published is any identifiable per-tenant history; a per-tenant
+  checksummed export — so *you* can verify your own numbers against your own target keys — is
+  tracked and not yet shipped.
 </Callout>
 
 ### 4. Attested execution

apps/loopover-ui/public/openapi.json

@@ -27176,6 +27176,36 @@
           }
         }
       }
+    },
+    "/v1/public/eval-corpus": {
+      "get": {
+        "operationId": "getPublicEvalCorpus",
+        "tags": [
+          "Public"
+        ],
+        "summary": "The redacted, checksummed corpus behind one rule's published precision — downloadable without credentials",
+        "parameters": [
+          {
+            "schema": {
+              "type": "string"
+            },
+            "required": true,
+            "name": "ruleId",
+            "in": "query"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "{ ruleId, windowDays, caseCount, truncated, checksum, cases: [{ ruleId, outcome, label, firedAt, decidedAt, metadata?: { confidence } }] } — `targetKey` and every other metadata key are DROPPED, not hashed, so no repo or PR identity is published; `metadata.confidence` stays nested where buildConfidenceThresholdClassifier reads it. Timestamps are truncated to the UTC day. `checksum` is sha256 over canonicalJson(cases), i.e. it commits to the artifact you can actually download"
+          },
+          "400": {
+            "description": "`ruleId` is required — a corpus is only meaningful for one rule"
+          },
+          "404": {
+            "description": "Public stats are disabled for this deployment"
+          }
+        }
+      }
     }
   },
   "servers": [

src/api/routes.ts

@@ -313,6 +313,7 @@ import { getPublicStats, isPublicStatsEnabled, resolvePublicStatsManifestOverrid
 import { loadPublicAccuracyTrend } from "../services/public-accuracy-trend";
 import { loadPublicFleetAccuracyTrend } from "../services/public-fleet-accuracy-trend";
 import { loadPublicRulePrecision } from "../review/public-rule-precision";
+import { loadPublicEvalCorpus } from "../review/public-eval-corpus";
 import { loadCalibrationTrend } from "../services/rule-calibration-trend";
 import { isSatisfactionFloorAutotuneEnabled, loadSatisfactionFloorStatus, runSatisfactionFloorLoosening } from "../services/satisfaction-floor-loosening-run";
 import { loadAllKnobStatuses } from "../services/knob-loosening-run";
@@ -874,6 +875,23 @@ export function createApp() {
   // per-record, digest-committed form; it is not a new scoring surface). Wired for the
   // outcome_confirmed_precision source only today -- a future benchmark_run source (#9265) feeds the same
   // endpoint, never a second response format.
+  // #9636: the corpus behind the published per-rule precision, redacted and downloadable WITHOUT
+  // credentials -- the read path that makes the verifiability walkthrough's step 1 true for a stranger
+  // instead of only for an operator holding this deployment's Cloudflare keys. See
+  // public-eval-corpus.ts's header for why `targetKey` is dropped rather than hashed, and why
+  // `metadata.confidence` stays nested exactly where the shipped classifier reads it.
+  app.get("/v1/public/eval-corpus", async (c) => {
+    const publicStatsManifestOverride = await resolvePublicStatsManifestOverride(c.env);
+    if (!isPublicStatsEnabled(c.env, publicStatsManifestOverride)) return c.json({ error: "not_found" }, 404);
+    const ruleId = c.req.query("ruleId");
+    // Required, not defaulted: a corpus is only meaningful for one rule, and silently picking one would
+    // publish a checksum for a rule the caller never asked about.
+    if (!ruleId) return c.json({ error: "rule_id_required" }, 400);
+    const corpus = await loadPublicEvalCorpus(c.env, ruleId);
+    c.header("Cache-Control", "public, max-age=60, stale-while-revalidate=300");
+    return c.json(corpus);
+  });
+
   app.get("/v1/public/eval-scores", async (c) => {
     const publicStatsManifestOverride = await resolvePublicStatsManifestOverride(c.env);
     if (!isPublicStatsEnabled(c.env, publicStatsManifestOverride)) return c.json({ error: "not_found" }, 404);

src/auth/route-auth.ts

@@ -40,6 +40,10 @@ export function requiresApiToken(path: string): boolean {
   // #9266: the eval-scores transport is unauthenticated by the same design as every /v1/public/* sibling
   // above -- committed to a corpus checksum, independently re-derivable, nothing gated behind a token.
   if (path === "/v1/public/eval-scores") return false;
+  // #9636: same posture as its sibling above -- the corpus is redacted at the source (no target keys, no
+  // repos, no metadata beyond the one confidence field replay needs), so there is nothing here an
+  // Authorization header would be protecting.
+  if (path === "/v1/public/eval-corpus") return false;
   // #9269: the single-row read, added in the SAME PR as its route so the two can never drift the way #9120's
   // sibling did. Regex (not a literal) because of the :seq path parameter.
   if (/^\/v1\/public\/decision-ledger\/row\/[^/]+$/.test(path)) return false;

src/openapi/spec.ts

@@ -1969,6 +1969,22 @@ export function buildOpenApiSpec() {
       404: { description: "No decision record persisted yet for this PR" },
     },
   });
+  registry.registerPath({
+    method: "get",
+    path: "/v1/public/eval-corpus",
+    operationId: "getPublicEvalCorpus",
+    tags: ["Public"],
+    summary: "The redacted, checksummed corpus behind one rule's published precision — downloadable without credentials",
+    request: { query: z.object({ ruleId: z.string() }) },
+    responses: {
+      200: {
+        description:
+          "{ ruleId, windowDays, caseCount, truncated, checksum, cases: [{ ruleId, outcome, label, firedAt, decidedAt, metadata?: { confidence } }] } — `targetKey` and every other metadata key are DROPPED, not hashed, so no repo or PR identity is published; `metadata.confidence` stays nested where buildConfidenceThresholdClassifier reads it. Timestamps are truncated to the UTC day. `checksum` is sha256 over canonicalJson(cases), i.e. it commits to the artifact you can actually download",
+      },
+      400: { description: "`ruleId` is required — a corpus is only meaningful for one rule" },
+      404: { description: "Public stats are disabled for this deployment" },
+    },
+  });
   registry.registerPath({
     method: "get",
     path: "/v1/public/eval-scores",