Skip to content

ci(verifiability): verify our own public claims nightly, as an outsider #9973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JSONbored merged 5 commits into from
Jul 31, 2026
Merged

ci(verifiability): verify our own public claims nightly, as an outsider #9973

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
aefcb2c
ci(verifiability): verify our own public claims nightly, as an outsider
JSONbored Jul 30, 2026
e55fb01
ci(verifiability): run the report script with node's own type stripping
JSONbored Jul 30, 2026
54f8ced
ci(verifiability): stop actions/checkout leaving a token on disk
JSONbored Jul 31, 2026
a69c16c
ci(verifiability): pin the anonymous-run guarantees with a test, not …
JSONbored Jul 31, 2026
7c3607b
ci(verifiability): drop actions/setup-node, the last action that can …
JSONbored Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
192 changes: 192 additions & 0 deletions .github/workflows/verify-public-claims.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
# Nightly anonymous verification (#9724, epic #9722).
#
# LoopOver publishes fairness claims and tells a stranger they can check them without asking permission. This
# job IS that stranger, on a schedule. It runs the verifier PUBLISHED ON NPM against PRODUCTION with no
# credentials of any kind, so a regression in the public verification path is caught by us within a cycle
# instead of by an outside reader whenever they happen to look.
#
# THE PUBLISHED TOOL, NOT THE ONE IN THIS TREE. `npx -p @loopover/mcp` deliberately resolves the release on npm
# rather than building from the checkout. What is being asserted is "an outsider succeeds today", and an
# outsider has the published CLI -- so a fix that is merged but unpublished must NOT turn this green. #9962 is
# exactly that failure: the shipped verifier asked `/v1/public/eval-corpus?rule_id=`, the route reads `ruleId`,
# every corpus fetch 400'd, and the tool reported production as unverifiable while production was fine. Both
# sides were individually tested and individually correct. Only running the real pair catches it.
#
# NO CREDENTIALS ON THE VERIFY STEP. The job-level permission is `contents: read`, and the verify step is given
# no token in its environment at all -- `issues: write` and the token live only on the step that files the
# tracking issue, after verification is finished. A verifier that runs with repo credentials is not reproducing
# an outsider's view, and could pass on access a stranger does not have.
#
# TWO SURFACES, because they hold different things (#9940): `api.loopover.ai` serves aggregate stats and
# eval-score records, while the anchored decision ledger lives on the Orb at `shots.loopover.ai`. Checking only
# one leaves the other free to regress silently, so both are verified and reported independently.
name: Verify Public Claims

on:
schedule:
# 04:12 UTC. Off the hour on purpose: the top of the hour is the busiest slot on GitHub's shared scheduler
# and the most likely to be delayed or dropped, which for a nightly is a silently skipped check.
- cron: "12 4 * * *"
workflow_dispatch:
inputs:
api-base-url:
description: "Base URL for the aggregate/eval surface"
required: false
default: "https://api.loopover.ai"
orb-base-url:
description: "Base URL for the anchored decision ledger"
required: false
default: "https://shots.loopover.ai"
dry-run:
description: "Report the verdict without filing, updating or closing the tracking issue"
required: false
default: "false"

permissions:
contents: read

concurrency:
group: verify-public-claims
cancel-in-progress: false

jobs:
verify:
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
issues: write
env:
API_BASE_URL: ${{ inputs.api-base-url || 'https://api.loopover.ai' }}
ORB_BASE_URL: ${{ inputs.orb-base-url || 'https://shots.loopover.ai' }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
Comment thread
JSONbored marked this conversation as resolved.
with:
# `persist-credentials: false` is load-bearing here, not hygiene. actions/checkout defaults to TRUE,
# which writes an authenticated `http.extraheader` into `.git/config` -- so the verify step would run
# with a usable GitHub token sitting on disk, and this job's whole claim is that it reproduces what a
# stranger sees. Withholding the token from the step's `env:` while leaving it in `.git/config` is
# theatre: the credential is still there for anything the step runs to pick up.
#
# Nothing here needs it. The only step that talks to GitHub is the tracking-issue step, which uses
# `gh` with an explicit `GITHUB_TOKEN` in its own environment -- `gh` reads that variable directly and
# never consults git's credential config.
persist-credentials: false

# DELIBERATELY NO actions/setup-node. This job needs a Node RUNTIME, not this repository's toolchain:
# it runs a published CLI through npx and one checked-in script, and builds nothing. The runner image
# already provides both node and npm, and three workflows here (ci.yml, mcp-release-watch.yml,
# package-release-watch.yml) already rely on that.
#
# Dropping it removes the last action that can write credentials. `actions/setup-node` writes an
# authenticated .npmrc via `auth.configAuthentication(registryUrl)` -- gated behind `if (registryUrl)`
# in its `src/main.ts`, so it was unreachable while that input stayed unset, but "unreachable because
# nobody has added one line yet" is a weaker property than "not present". For a job whose entire claim
# is that it holds no credentials, not present is the one worth having.
#
# The tradeoff is that the Node version is now the runner's rather than .nvmrc's, so it is ASSERTED
# rather than assumed. `--experimental-strip-types` needs 22.6+; if an image ever regresses below that,
# this fails immediately with the fix in the message instead of the nightly quietly breaking.
- name: Require a Node that can strip types
run: |
node -e '
const [major, minor] = process.versions.node.split(".").map(Number);
if (major > 22 || (major === 22 && minor >= 6)) process.exit(0);
console.error("::error::Node " + process.versions.node + " cannot run --experimental-strip-types (needs 22.6+). Re-add actions/setup-node with node-version-file: .nvmrc.");
process.exit(1);
'
echo "node $(node --version), npm $(npm --version)"

# No `npm ci`: this step must not depend on the repository's own dependency tree, because a stranger has
# none. `npx -p @loopover/mcp` fetches the published package exactly as the docs tell a reader to.
- name: Verify published claims anonymously
id: verify
run: |
# No `set -e`: a failing claim exits the verifier non-zero, and that is a RESULT to be captured and
# reported, not a reason to abort before the tracking issue is filed.
set -uo pipefail
run_surface() {
base="$1"
out="$2"
echo "::group::loopover-verify --base-url $base"
# `code` is assigned on its OWN line. `local code=$?` / `code=$(...)` would capture the assignment's
# own status instead of the command's, quietly making every run look like a clean exit 0.
npx --yes -p @loopover/mcp loopover-verify --base-url "$base" --json > "$out" 2>"$out.err"
code=$?
echo "$code" > "$out.code"
cat "$out"
# stderr matters most when stdout is unparseable -- that is the "the verifier itself broke" case,
# and it is the one where the JSON report cannot explain anything.
if [ -s "$out.err" ]; then echo "--- stderr ---"; cat "$out.err"; fi
echo "exit=$code"
echo "::endgroup::"
}
run_surface "$API_BASE_URL" api.json
run_surface "$ORB_BASE_URL" orb.json
# The script appends VERIFY_OK, VERIFY_SUMMARY and the heredoc-delimited body to $GITHUB_OUTPUT
# itself, as one write. Wrapping its stdout in a heredoc here instead would interleave the two
# writers and bury VERIFY_OK inside the body block.
#
# `node --experimental-strip-types`, not `npx tsx`: this file is OURS and already checked out, so
# fetching an unpinned TypeScript runner from the registry to read it would add a second unvetted
# runtime dependency to a job whose one intentional unpinned fetch is the published verifier itself.
# Same invocation `npm run actionlint` already uses for its own TS script.
node --experimental-strip-types scripts/verify-public-claims-report.ts \
"public API" "$API_BASE_URL" "$(cat api.json.code)" api.json \
"Orb ledger" "$ORB_BASE_URL" "$(cat orb.json.code)" orb.json

- name: Report verdict
run: |
{
echo "### Anonymous verification — ${{ steps.verify.outputs.VERIFY_SUMMARY }}"
echo ""
echo "- \`${{ env.API_BASE_URL }}\`"

@superagent-security superagent-security Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Template injection in Report verdict step via workflow_dispatch inputs

workflow_dispatch inputs are interpolated into a shell run block with backtick command substitution.

Use shell variables ($API_BASE_URL, $ORB_BASE_URL) instead of template substitution.

AI prompt 
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/verify-public-claims.yml">
<violation number="1" location=".github/workflows/verify-public-claims.yml:136">
<priority>P1</priority>
<title>Template injection in Report verdict step via workflow_dispatch inputs</title>
<evidence>The Report verdict step interpolates ${{ env.API_BASE_URL }} and ${{ env.ORB_BASE_URL }} directly into a shell script. These values derive from ${{ inputs.api-base-url }} and ${{ inputs.orb-base-url }}, which are workflow_dispatch inputs. The surrounding backticks create a bash command-substitution context, allowing arbitrary command execution when a malicious input is provided.</evidence>
<recommendation>Replace the template substitutions with shell variable references ($API_BASE_URL, $ORB_BASE_URL). These environment variables are already set at the job level, so this simply removes the template injection vector.</recommendation>
</violation>
</file>

echo "- \`${{ env.ORB_BASE_URL }}\`"
} >> "$GITHUB_STEP_SUMMARY"

# Open-or-update on failure, close on recovery. ONE issue, found by exact title, so a persistent outage
# is a single tracked thing rather than a nightly pile -- and so recovery has something unambiguous to
# close. This step is the only one holding a token.
- name: File or close the tracking issue
if: ${{ inputs.dry-run != 'true' }}
env:
GITHUB_TOKEN: ${{ github.token }}
VERIFY_OK: ${{ steps.verify.outputs.VERIFY_OK }}
VERIFY_BODY: ${{ steps.verify.outputs.body }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
TITLE: "verifiability: the nightly anonymous verification run is failing"
run: |
set -euo pipefail
existing=$(gh issue list --state open --search "\"$TITLE\" in:title" --json number,title \
--jq "map(select(.title == env.TITLE)) | .[0].number // empty")
if [ "$VERIFY_OK" = "true" ]; then
if [ -n "$existing" ]; then
gh issue comment "$existing" --body "Recovered. The nightly anonymous verification run is green again as of [this run]($RUN_URL); closing automatically."
gh issue close "$existing" --reason completed
echo "Closed #$existing on recovery."
else
echo "Green, and nothing was open. Nothing to do."
fi
exit 0
fi
body="$VERIFY_BODY

[Failing run]($RUN_URL)"
if [ -n "$existing" ]; then
# Comment rather than overwrite the body: the history of WHICH claims failed on WHICH night is the
# most useful thing on a flapping issue, and rewriting the body destroys it.
gh issue comment "$existing" --body "$body"
echo "::error::Anonymous verification is failing -- updated #$existing."
else
gh issue create --title "$TITLE" --label maintainer-only --assignee JSONbored --body "$body"
echo "::error::Anonymous verification is failing -- tracking issue filed."
fi

# Last, so the issue is always filed or closed even when the job is about to go red. A red run with no
# issue would be a monitor that notices a problem and then drops it.
- name: Fail the run when a claim failed
if: ${{ steps.verify.outputs.VERIFY_OK != 'true' }}
run: |
echo "::error::One or more published claims could not be verified anonymously."
exit 1
9 changes: 9 additions & 0 deletions apps/loopover-ui/content/docs/verify-this-review.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,15 @@ figure stops supporting a conclusion — see the [fairness methodology](/docs/fa
For a single command that recomputes the published commitments and exits non-zero if any fail,
run `npx -p @loopover/mcp loopover-verify`.

[![Nightly anonymous verification](https://github.com/JSONbored/loopover/actions/workflows/verify-public-claims.yml/badge.svg)](https://github.com/JSONbored/loopover/actions/workflows/verify-public-claims.yml)

**We run this against ourselves every night** (#9724). That badge is a scheduled job that executes
the verifier *published on npm* against production, from a runner holding no credentials of any
kind — the same position you are in. If it is green, an outsider following this page succeeded
within the last 24 hours. If it is red, something on this page does not currently work, we already
know, and there is an open issue saying which claim broke. The badge links to every run, including
the failures; you do not have to take our word for the history.

**Two surfaces, because they hold different things** (#9940). The commands on this page use
`api.loopover.ai`, which serves aggregate stats and eval-score records. The **anchored decision
ledger** — the hash chain, its signed checkpoints, and the Rekor anchors — lives on the Orb that
Expand Down
Loading
Loading