From aefcb2cb3582cffbf015b15f1c2e37ed119b7034 Mon Sep 17 00:00:00 2001 From: JSONbored <49853598+JSONbored@users.noreply.github.com> Date: Thu, 30 Jul 2026 16:03:52 -0700 Subject: [PATCH 1/5] ci(verifiability): verify our own public claims nightly, as an outsider LoopOver tells a stranger they can check its fairness claims without asking permission. Nothing checked that this stayed true, so a regression in the public verification path was found by outside readers whenever they happened to look. The job runs the verifier PUBLISHED ON NPM against production, from a runner holding no credentials. Both parts are load-bearing. Building from the checkout would assert something weaker -- an outsider has the published CLI, so a fix merged but unpublished must not turn this green. #9962 is exactly that gap: the shipped verifier asked for `?rule_id=` while the route reads `ruleId`, every corpus fetch 400'd, and the tool called production unverifiable while production was fine. Both sides were individually tested and individually correct; only running the real pair catches it. The verify step is given no token at all -- `issues: write` lives on the step that files the tracking issue, after verification is done -- because a verifier running with repo credentials is not reproducing an outsider's view. Both surfaces are checked (#9940): `api.loopover.ai` for aggregates and eval-score records, `shots.loopover.ai` for the anchored ledger. One tracking issue, found by exact title: filed on failure, commented on while it persists, closed automatically on recovery. Comments rather than body rewrites, since which claim failed on which night is the useful history. Self-verified by pointing the real verifier at a deliberately wrong base URL, which found a false-green in the first draft of the decision logic. Every claim came back `skip` ("unavailable (HTTP 404)") and the CLI exited 0 -- judged claim-by-claim, four legitimate skips and a clean exit, so the monitor would have reported GREEN against a completely dead endpoint. The verifier is right to call them skips; from inside one claim, "disabled surface" and "wrong host" are indistinguishable. The monitor has the context the claim lacks, so a run that positively verified NOTHING is now red. The floor is one passing claim rather than a proportion because the two surfaces are legitimately lopsided: measured against production, the API passes three of four and the Orb passes one of four, and "most claims pass" would fail the Orb every healthy night. Three further false-greens are rejected: a run that produced no readable report (nothing failed because nothing ran), a non-zero exit with no failing claim, and an unreadable capture file. Skips alone never file an issue -- the API's ledger is empty by design and skips nightly -- but are always listed, since a new skip beside a failure is often its cause. The decision logic is a tested script rather than inline bash. That is also how an output-interleaving bug was caught: the workflow wrapping the script's stdout in a heredoc while the script appended its own scalars buried VERIFY_OK inside the body block, leaving the workflow reading an empty verdict and filing an outage on a healthy night. The script now writes all outputs itself, in order. Closes #9724 --- .github/workflows/verify-public-claims.yml | 157 ++++++++++ .../content/docs/verify-this-review.mdx | 9 + scripts/verify-public-claims-report.ts | 278 ++++++++++++++++ test/unit/verify-public-claims-report.test.ts | 296 ++++++++++++++++++ 4 files changed, 740 insertions(+) create mode 100644 .github/workflows/verify-public-claims.yml create mode 100644 scripts/verify-public-claims-report.ts create mode 100644 test/unit/verify-public-claims-report.test.ts diff --git a/.github/workflows/verify-public-claims.yml b/.github/workflows/verify-public-claims.yml new file mode 100644 index 000000000..23b41d8a9 --- /dev/null +++ b/.github/workflows/verify-public-claims.yml @@ -0,0 +1,157 @@ +# Nightly anonymous verification (#9724, epic #9722). +# +# LoopOver publishes fairness claims and tells a stranger they can check them without asking permission. This +# job IS that stranger, on a schedule. It runs the verifier PUBLISHED ON NPM against PRODUCTION with no +# credentials of any kind, so a regression in the public verification path is caught by us within a cycle +# instead of by an outside reader whenever they happen to look. +# +# THE PUBLISHED TOOL, NOT THE ONE IN THIS TREE. `npx -p @loopover/mcp` deliberately resolves the release on npm +# rather than building from the checkout. What is being asserted is "an outsider succeeds today", and an +# outsider has the published CLI -- so a fix that is merged but unpublished must NOT turn this green. #9962 is +# exactly that failure: the shipped verifier asked `/v1/public/eval-corpus?rule_id=`, the route reads `ruleId`, +# every corpus fetch 400'd, and the tool reported production as unverifiable while production was fine. Both +# sides were individually tested and individually correct. Only running the real pair catches it. +# +# NO CREDENTIALS ON THE VERIFY STEP. The job-level permission is `contents: read`, and the verify step is given +# no token in its environment at all -- `issues: write` and the token live only on the step that files the +# tracking issue, after verification is finished. A verifier that runs with repo credentials is not reproducing +# an outsider's view, and could pass on access a stranger does not have. +# +# TWO SURFACES, because they hold different things (#9940): `api.loopover.ai` serves aggregate stats and +# eval-score records, while the anchored decision ledger lives on the Orb at `shots.loopover.ai`. Checking only +# one leaves the other free to regress silently, so both are verified and reported independently. +name: Verify Public Claims + +on: + schedule: + # 04:12 UTC. Off the hour on purpose: the top of the hour is the busiest slot on GitHub's shared scheduler + # and the most likely to be delayed or dropped, which for a nightly is a silently skipped check. + - cron: "12 4 * * *" + workflow_dispatch: + inputs: + api-base-url: + description: "Base URL for the aggregate/eval surface" + required: false + default: "https://api.loopover.ai" + orb-base-url: + description: "Base URL for the anchored decision ledger" + required: false + default: "https://shots.loopover.ai" + dry-run: + description: "Report the verdict without filing, updating or closing the tracking issue" + required: false + default: "false" + +permissions: + contents: read + +concurrency: + group: verify-public-claims + cancel-in-progress: false + +jobs: + verify: + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: read + issues: write + env: + API_BASE_URL: ${{ inputs.api-base-url || 'https://api.loopover.ai' }} + ORB_BASE_URL: ${{ inputs.orb-base-url || 'https://shots.loopover.ai' }} + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + + - name: Setup Node + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 + with: + node-version-file: .nvmrc + + # No `npm ci`: this step must not depend on the repository's own dependency tree, because a stranger has + # none. `npx -p @loopover/mcp` fetches the published package exactly as the docs tell a reader to. + - name: Verify published claims anonymously + id: verify + run: | + # No `set -e`: a failing claim exits the verifier non-zero, and that is a RESULT to be captured and + # reported, not a reason to abort before the tracking issue is filed. + set -uo pipefail + run_surface() { + base="$1" + out="$2" + echo "::group::loopover-verify --base-url $base" + # `code` is assigned on its OWN line. `local code=$?` / `code=$(...)` would capture the assignment's + # own status instead of the command's, quietly making every run look like a clean exit 0. + npx --yes -p @loopover/mcp loopover-verify --base-url "$base" --json > "$out" 2>"$out.err" + code=$? + echo "$code" > "$out.code" + cat "$out" + # stderr matters most when stdout is unparseable -- that is the "the verifier itself broke" case, + # and it is the one where the JSON report cannot explain anything. + if [ -s "$out.err" ]; then echo "--- stderr ---"; cat "$out.err"; fi + echo "exit=$code" + echo "::endgroup::" + } + run_surface "$API_BASE_URL" api.json + run_surface "$ORB_BASE_URL" orb.json + # The script appends VERIFY_OK, VERIFY_SUMMARY and the heredoc-delimited body to $GITHUB_OUTPUT + # itself, as one write. Wrapping its stdout in a heredoc here instead would interleave the two + # writers and bury VERIFY_OK inside the body block. + npx --yes tsx scripts/verify-public-claims-report.ts \ + "public API" "$API_BASE_URL" "$(cat api.json.code)" api.json \ + "Orb ledger" "$ORB_BASE_URL" "$(cat orb.json.code)" orb.json + + - name: Report verdict + run: | + { + echo "### Anonymous verification — ${{ steps.verify.outputs.VERIFY_SUMMARY }}" + echo "" + echo "- \`${{ env.API_BASE_URL }}\`" + echo "- \`${{ env.ORB_BASE_URL }}\`" + } >> "$GITHUB_STEP_SUMMARY" + + # Open-or-update on failure, close on recovery. ONE issue, found by exact title, so a persistent outage + # is a single tracked thing rather than a nightly pile -- and so recovery has something unambiguous to + # close. This step is the only one holding a token. + - name: File or close the tracking issue + if: ${{ inputs.dry-run != 'true' }} + env: + GITHUB_TOKEN: ${{ github.token }} + VERIFY_OK: ${{ steps.verify.outputs.VERIFY_OK }} + VERIFY_BODY: ${{ steps.verify.outputs.body }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + TITLE: "verifiability: the nightly anonymous verification run is failing" + run: | + set -euo pipefail + existing=$(gh issue list --state open --search "\"$TITLE\" in:title" --json number,title \ + --jq "map(select(.title == env.TITLE)) | .[0].number // empty") + if [ "$VERIFY_OK" = "true" ]; then + if [ -n "$existing" ]; then + gh issue comment "$existing" --body "Recovered. The nightly anonymous verification run is green again as of [this run]($RUN_URL); closing automatically." + gh issue close "$existing" --reason completed + echo "Closed #$existing on recovery." + else + echo "Green, and nothing was open. Nothing to do." + fi + exit 0 + fi + body="$VERIFY_BODY + + [Failing run]($RUN_URL)" + if [ -n "$existing" ]; then + # Comment rather than overwrite the body: the history of WHICH claims failed on WHICH night is the + # most useful thing on a flapping issue, and rewriting the body destroys it. + gh issue comment "$existing" --body "$body" + echo "::error::Anonymous verification is failing -- updated #$existing." + else + gh issue create --title "$TITLE" --label maintainer-only --assignee JSONbored --body "$body" + echo "::error::Anonymous verification is failing -- tracking issue filed." + fi + + # Last, so the issue is always filed or closed even when the job is about to go red. A red run with no + # issue would be a monitor that notices a problem and then drops it. + - name: Fail the run when a claim failed + if: ${{ steps.verify.outputs.VERIFY_OK != 'true' }} + run: | + echo "::error::One or more published claims could not be verified anonymously." + exit 1 diff --git a/apps/loopover-ui/content/docs/verify-this-review.mdx b/apps/loopover-ui/content/docs/verify-this-review.mdx index 86be9335f..b024ac165 100644 --- a/apps/loopover-ui/content/docs/verify-this-review.mdx +++ b/apps/loopover-ui/content/docs/verify-this-review.mdx @@ -16,6 +16,15 @@ figure stops supporting a conclusion — see the [fairness methodology](/docs/fa For a single command that recomputes the published commitments and exits non-zero if any fail, run `npx -p @loopover/mcp loopover-verify`. +[![Nightly anonymous verification](https://github.com/JSONbored/loopover/actions/workflows/verify-public-claims.yml/badge.svg)](https://github.com/JSONbored/loopover/actions/workflows/verify-public-claims.yml) + +**We run this against ourselves every night** (#9724). That badge is a scheduled job that executes +the verifier *published on npm* against production, from a runner holding no credentials of any +kind — the same position you are in. If it is green, an outsider following this page succeeded +within the last 24 hours. If it is red, something on this page does not currently work, we already +know, and there is an open issue saying which claim broke. The badge links to every run, including +the failures; you do not have to take our word for the history. + **Two surfaces, because they hold different things** (#9940). The commands on this page use `api.loopover.ai`, which serves aggregate stats and eval-score records. The **anchored decision ledger** — the hash chain, its signed checkpoints, and the Rekor anchors — lives on the Orb that diff --git a/scripts/verify-public-claims-report.ts b/scripts/verify-public-claims-report.ts new file mode 100644 index 000000000..12ddc35db --- /dev/null +++ b/scripts/verify-public-claims-report.ts @@ -0,0 +1,278 @@ +#!/usr/bin/env node +// #9724: turn one or more `loopover-verify --json` reports into the nightly job's verdict and, when something +// is wrong, the body of the tracking issue a maintainer will actually have to act on. +// +// WHY THIS IS A SCRIPT AND NOT INLINE BASH. The interesting logic here is a judgement -- which claims count as +// a regression, and what a reader needs in order to fix it -- and that judgement is exactly the part that must +// not be discovered to be wrong on the night it first fires. As a pure function it is unit-tested against real +// report shapes, including the shapes that must NOT open an issue. +// +// SKIP IS NOT FAILURE, AND IS NOT SUCCESS EITHER. `loopover-verify` reports a surface it could not obtain as +// `skip`, and the verifier's own header is emphatic that collapsing "could not check" into "checked, fine" is +// how you build a monitor that always says green. So skips never open an issue -- a deployment legitimately +// serves no corpus for a rule, and the public API's ledger is empty BY DESIGN (#9940), which is a permanent, +// expected skip -- but they are always reported in the body, because "we verified nothing at all tonight" is +// something a maintainer reading a green run deserves to see. + +import { pathToFileURL } from "node:url"; +import { readFileSync } from "node:fs"; + +/** One claim as `loopover-verify --json` emits it. Structural, matching the verifier's own reasoning about + * older deployments: the nightly must be able to parse a report from a CLI version it does not control. */ +export type VerifyClaim = { + id?: unknown; + claim?: unknown; + status?: unknown; + detail?: unknown; +}; + +export type VerifyRun = { + /** Human label for the surface, e.g. "public API". Appears in the issue body and the job log. */ + label: string; + baseUrl: string; + /** The CLI's exit code. Non-zero with no failing claim means the tool itself broke -- see below. */ + exitCode: number; + /** Parsed `--json` payload, or null when the CLI produced nothing parseable. */ + report: { results?: unknown } | null; +}; + +export type ClaimLine = { surface: string; status: string; claim: string; detail: string }; + +export type VerificationReport = { + ok: boolean; + failures: ClaimLine[]; + skips: ClaimLine[]; + passes: ClaimLine[]; + title: string; + body: string; +}; + +const str = (value: unknown, fallback = ""): string => (typeof value === "string" ? value : fallback); + +/** PURE. Flatten one run's claims into labelled lines. A report whose `results` is absent or not an array + * yields nothing here; {@link buildVerificationReport} turns that into its own failure rather than letting an + * unreadable report read as "no failures". */ +export function claimLinesFor(run: VerifyRun): ClaimLine[] { + const results = run.report?.results; + if (!Array.isArray(results)) return []; + return results.map((entry) => { + const claim = entry as VerifyClaim; + return { + surface: run.label, + status: str(claim.status, "unknown").toLowerCase(), + claim: str(claim.claim, str(claim.id, "(unnamed claim)")), + detail: str(claim.detail, "(no detail published)"), + }; + }); +} + +/** + * PURE. Decide whether the public verification path is healthy, and write the issue body if it is not. + * + * Four independent things count as broken, and every one after the first is a case that would otherwise + * present as a silent green: + * + * 1. any claim the verifier reports as `fail`; + * 2. a run that produced NO readable report at all (the CLI crashed, the JSON was malformed, npx could not + * resolve the package). Nothing failed, because nothing ran -- which is a regression in the published + * verification path itself, precisely what this job exists to notice; + * 3. a non-zero exit code with no failing claim to explain it. The tool is telling us something went wrong + * in a way its own claim list does not describe, and trusting the empty claim list over the exit code + * would be choosing the more comfortable of two contradictory signals; + * 4. a run in which NOTHING PASSED. This is the one that matters most, and it was found by pointing the real + * verifier at a deliberately wrong base URL: every claim came back `skip` ("/v1/public/eval-scores + * unavailable (HTTP 404)") and the CLI exited 0. Judged claim-by-claim that is four legitimate skips and + * a clean exit -- so the monitor would have reported GREEN against a completely dead endpoint, which is + * the exact failure this job exists to prevent. + * + * The verifier is right to call them skips: from inside a single claim, "this surface is disabled" and + * "this host is wrong" are indistinguishable. The MONITOR has the context the claim lacks -- production is + * supposed to be serving these -- so requiring at least one positively verified claim per surface is a + * judgement that belongs here rather than in the CLI. + * + * The floor is one, not a proportion, because the two surfaces are legitimately lopsided and both must + * clear it with room to spare (measured against production): `api.loopover.ai` passes record-digests, + * corpus-commitments and stats-parity while skipping anchor-checkpoint (its ledger is empty BY DESIGN, + * #9940), and `shots.loopover.ai` passes anchor-checkpoint while skipping the other three (it does not + * publish stats). A threshold like "most claims pass" would fail the Orb on a healthy night. + */ +export function buildVerificationReport(runs: readonly VerifyRun[]): VerificationReport { + const all = runs.flatMap(claimLinesFor); + const failures = all.filter((line) => line.status === "fail"); + const skips = all.filter((line) => line.status === "skip"); + const passes = all.filter((line) => line.status === "pass"); + + const brokenRuns = runs.filter((run) => run.report === null || !Array.isArray(run.report.results)); + const unexplained = runs.filter( + (run) => run.exitCode !== 0 && !brokenRuns.includes(run) && !claimLinesFor(run).some((line) => line.status === "fail"), + ); + // Case 4. Only for runs that produced a readable report -- a broken run is already reported as such, and + // saying "and also nothing passed" about it would be two findings for one cause. + const verifiedNothing = runs.filter((run) => !brokenRuns.includes(run) && !claimLinesFor(run).some((line) => line.status === "pass")); + + const ok = failures.length === 0 && brokenRuns.length === 0 && unexplained.length === 0 && verifiedNothing.length === 0; + const title = TRACKING_ISSUE_TITLE; + if (ok) { + return { ok, failures, skips, passes, title, body: "" }; + } + + const lines: string[] = [ + "The nightly anonymous verification run failed. These are LoopOver's own published claims, checked the way", + "an outsider checks them: the verifier published on npm, run against production, with no credentials of any", + "kind. A failure here means a stranger following the walkthrough right now gets the same result.", + "", + ]; + + for (const run of brokenRuns) { + lines.push(`### ${run.label} — the verifier produced no readable report`, ""); + lines.push( + `\`${run.baseUrl}\` exited ${run.exitCode} without emitting parseable \`--json\` output. Nothing was checked, which`, + "is itself the regression: the published verification path did not run. Check the job log for the raw output.", + "", + ); + } + + for (const run of unexplained) { + lines.push(`### ${run.label} — non-zero exit with no failing claim`, ""); + lines.push( + `\`${run.baseUrl}\` exited ${run.exitCode} but reported no \`fail\` claim. The tool is signalling a problem its own`, + "claim list does not describe; the exit code is the signal to trust here, not the empty list.", + "", + ); + } + + for (const run of verifiedNothing) { + lines.push(`### ${run.label} — nothing was verified`, ""); + lines.push( + `Every claim against \`${run.baseUrl}\` came back skipped, so not one published commitment was actually`, + "recomputed. The most common cause is that the surface is not reachable at that host at all -- a wrong or", + "renamed base URL returns 404 for every endpoint, which the verifier reports claim-by-claim as \"unavailable\"", + "and exits 0. A healthy surface always positively verifies at least one claim.", + "", + ); + } + + if (failures.length > 0) { + lines.push("### Failing claims", ""); + lines.push("| Surface | Claim | Detail |", "| --- | --- | --- |"); + for (const line of failures) lines.push(`| ${line.surface} | ${escapeCell(line.claim)} | ${escapeCell(line.detail)} |`); + lines.push(""); + } + + if (skips.length > 0) { + // Reported, never counted. A permanent skip is expected (the public API's ledger is empty by design), but a + // NEW one next to a failure is often the actual cause, so it has to be visible in the same body. + lines.push("### Skipped (not counted as failures)", ""); + lines.push("| Surface | Claim | Detail |", "| --- | --- | --- |"); + for (const line of skips) lines.push(`| ${line.surface} | ${escapeCell(line.claim)} | ${escapeCell(line.detail)} |`); + lines.push(""); + } + + lines.push("### Reproduce", ""); + lines.push("```bash"); + for (const run of runs) lines.push(`npx -p @loopover/mcp loopover-verify --base-url ${run.baseUrl}`); + lines.push("```", ""); + lines.push( + "This issue is updated in place while the failure persists and closed automatically by the next green run,", + "so it never needs manual triage to stay accurate. Filed by `.github/workflows/verify-public-claims.yml` (#9724).", + ); + + return { ok, failures, skips, passes, title, body: lines.join("\n") }; +} + +/** The single tracking issue's title. A CONSTANT, not templated with a date or a claim name: the workflow finds + * the existing issue by exact title, and anything varying per run would file a new issue every night instead of + * updating one. */ +export const TRACKING_ISSUE_TITLE = "verifiability: the nightly anonymous verification run is failing"; + +/** Markdown table cells cannot contain a raw pipe or newline; verifier details legitimately contain both. */ +export function escapeCell(value: string): string { + return value.replace(/\|/g, "\\|").replace(/\r?\n/g, " "); +} + +/** PURE. The one-line job summary, printed whether or not anything failed. */ +export function summarize(report: VerificationReport): string { + return `${report.passes.length} passed, ${report.failures.length} failed, ${report.skips.length} skipped`; +} + +/** Parse one run from the CLI's stdout. Never throws: unparseable output IS a result (see case 2 above). */ +export function parseRun(label: string, baseUrl: string, exitCode: number, stdout: string): VerifyRun { + try { + const parsed: unknown = JSON.parse(stdout); + if (parsed === null || typeof parsed !== "object") return { label, baseUrl, exitCode, report: null }; + return { label, baseUrl, exitCode, report: parsed as { results?: unknown } }; + } catch { + return { label, baseUrl, exitCode, report: null }; + } +} + +/** The heredoc delimiter for the multi-line `body` output. GitHub requires a delimiter that does not occur in + * the value; this one is long and structured enough that verifier prose cannot collide with it, and + * {@link renderGithubOutput} asserts that rather than assuming it. */ +export const OUTPUT_DELIMITER = "VERIFY_BODY_EOF_9d3f1c"; + +/** + * PURE. The exact bytes to append to `$GITHUB_OUTPUT`. + * + * Built here, as one string, rather than by the workflow shell echoing around the script's stdout. That shape + * was subtly broken: the script appending its own scalars while the shell wrapped its stdout in a heredoc + * interleaves the two writers, and `VERIFY_OK=` lands INSIDE the body block -- leaving the workflow reading an + * empty verdict and, because an empty string is not `"true"`, filing an outage issue on a healthy night. + */ +export function renderGithubOutput(report: VerificationReport): string { + if (report.body.includes(OUTPUT_DELIMITER)) { + // Unreachable with the delimiter above, but an unguarded heredoc is an output-injection primitive: a body + // containing the delimiter would terminate the block early and let the rest be parsed as more outputs. + throw new Error("verification body contains the output delimiter"); + } + return [`VERIFY_OK=${report.ok ? "true" : "false"}`, `VERIFY_SUMMARY=${summarize(report)}`, `body<<${OUTPUT_DELIMITER}`, report.body, OUTPUT_DELIMITER, ""].join("\n"); +} + +/** PURE. Parse the CLI's positional arguments into runs. Four per surface. */ +export function parseArgs(argv: readonly string[], readFile: (path: string) => string): VerifyRun[] { + const runs: VerifyRun[] = []; + for (let index = 0; index < argv.length; index += 4) { + const label = argv[index] ?? ""; + const baseUrl = argv[index + 1] ?? ""; + const exitCode = Number.parseInt(argv[index + 2] ?? "", 10); + const file = argv[index + 3] ?? ""; + let stdout = ""; + try { + stdout = readFile(file); + } catch { + // An unreadable capture file is the same finding as unparseable output: nothing was verified. + } + // A non-numeric exit code is treated as a failure rather than a zero -- "we could not tell how it exited" + // must never be the optimistic branch. + runs.push(parseRun(label, baseUrl, Number.isFinite(exitCode) ? exitCode : 1, stdout)); + } + return runs; +} + +/** CLI: `verify-public-claims-report.ts