Provision the ledger-anchor keypair so self-host anchoring can actually run

[JSONbored]

What's wrong

GET https://api.loopover.ai/v1/public/decision-ledger/anchors returns {"anchors":[],"nextBefore":null} and always has. The docs present anchoring as a live trust surface.

This is not a scheduler or Rekor bug. Both are wired correctly and the job runs.

Root cause: topology

The anchor-decision-ledger job is enqueued unconditionally on every 2-minute cron tick (src/index.ts:141-145, wrangler.jsonc:398-402) with no feature flag, reaches processJob, and calls runScheduledLedgerAnchor. It then early-returns on the first guard, every time:

if (input.currentTip.seq === 0) return { shouldAnchor: false, reason: "empty_ledger" };

— src/review/ledger-anchor-scheduler.ts:37

Confirmed live: GET /v1/public/decision-ledger/verify reports tipSeq: 0, totalCount: 0, ok: true.

The ledger is empty because appendDecisionLedger is only reached via persistDecisionRecord, called from src/queue/processors.ts:3810 and src/services/agent-action-executor.ts:358 — both on the review-execution path, which the hosted Worker acks-and-drops (src/index.ts:61-75). Self-host runtimes that do execute reviews write anchors into their own database, invisible to api.loopover.ai.

The API that publishes anchors is not the runtime that produces the ledger being anchored. No code fix in ledger-anchor-scheduler.ts or ledger-anchor-rekor.ts changes that.

(The anchors route returning 200 [] rather than 500 also confirms migrations/0195_decision_ledger_anchors.sql is applied — the table exists and is genuinely empty. Producer and reader are the same table on the same binding, so this is not a store mismatch.)

Second, independent blocker: no signing key

Even with a non-empty ledger, src/review/ledger-anchor-scheduler.ts:111-121 returns early with ledger_anchor_skipped_unconfigured / no_current_signing_key_published, because LOOPOVER_LEDGER_ANCHOR_KEYS and LOOPOVER_LEDGER_ANCHOR_PRIVATE_KEY are unset. Confirmed live: GET /v1/public/decision-ledger/anchor-key returns {"keys":[],"currentKeyId":null}.

No file outside src/ and test/ mentions LEDGER_ANCHOR at all — not wrangler.jsonc, not docker-compose.yml, not any runbook. There is no documented procedure for generating the P-256 keypair or publishing the public half. Rekor itself needs no credentials (ledger-anchor-rekor.ts:13, public shard default), so once a key exists and a ledger is non-empty, submission should work.

Observability gap that hid this

Both early returns are invisible from outside: :37 returns without writing anything, :113-121 writes only a console.log. From the public endpoint, "never configured", "never had a ledger to anchor", and "never deployed" are indistinguishable — all three render as {"anchors":[]}. ledger-anchor-persistence.ts:1-6 states the design goal ("an operator whose anchoring silently fails could quietly regress the ledger back to tamper-evident-only with no visible signal"), but that guarantee only holds after both guards pass.

Relatedly, no test covers the production shape: every scheduler test calls seedOneDecision(env) and keyedEnv() first. empty_ledger is asserted only as correct behavior (test/unit/ledger-anchor-scheduler.test.ts:120-126), never as an alarm.

What needs deciding

1. Where anchoring lives. Either the hosted Worker gets a ledger to anchor (ingest decision records from self-host instances, the way orb_pr_outcomes already ingests outcomes), or anchoring is a self-host-only capability and the public endpoint plus docs should say so instead of implying a live central anchor log.
2. Provision the keypair. Generate the P-256 key, publish the public half via LOOPOVER_LEDGER_ANCHOR_KEYS, set LOOPOVER_LEDGER_ANCHOR_PRIVATE_KEY as a Worker secret, and write the runbook that currently does not exist.
3. Make the silence visible — distinguish "not configured" from "nothing to anchor" on the public surface, so this cannot look identical to a healthy empty state again.

Same class as #9636 and #9676: a reader still runs on the hosted Worker and renders the absence of a retired writer's output as a healthy state.

[linear-code]

JSO-6780

[JSONbored]

Decomposition, and a blocker that changes the plan

I traced the full path. The "reuse the existing POST /v1/decision-ledger/anchor-attempts endpoint" plan does not work as-is, for a reason worth recording before anyone tries it.

The blocker

ingestBittensorAnchorReport verifies an ok report against the receiving deployment's own ledger:

const row = await loadPublicLedgerRow(env, report.signed.payload.seq);
if (!row) return { recorded: false, reason: "row_not_found" };
if (row.rowHash !== report.signed.payload.rowHash) return { recorded: false, reason: "row_hash_mismatch" };

— src/review/ledger-anchor-bittensor.ts:113-115

api.loopover.ai's decision_ledger is empty. So every ok anchor reported by a self-host runtime would 422 with row_not_found, forever. The check is correct for its designed purpose — it stops anyone claiming an anchor for a row that doesn't exist — but it structurally cannot verify a foreign ledger's row.

What already exists (so nobody re-investigates)

• The route, its validation, auth (LOOPOVER_LEDGER_ANCHOR_REPORT_TOKEN, fails closed), persistence into the same decision_ledger_anchors table the public read uses, and tests — all complete.
• A self-host container already runs the anchor scheduler every ~2 minutes (src/index.ts:145 → src/server.ts:1322-1339 → processJob directly, bypassing the hosted ack-and-drop) and already has a populated ledger. It is one keypair away from genuinely anchoring its own ledger.
• All nine LOOPOVER_LEDGER_ANCHOR_* vars are documented and read; the self-host Env is {...process.env} so no plumbing is needed.
• No client anywhere POSTs to the route — not in src/, packages/, scripts/, or selfhost. It is a server-side contract awaiting an off-repo process.
• runScheduledLedgerAnchor has no reporting seam: deps is {submitRekor, submitGit} only, and it returns just the scheduling decision, not per-backend outcomes. Success rows are written inside each backend, where the scheduler never sees them.

Filed as decision-independent sub-issues

• anchors: self-host preflight and config-lint say nothing about the nine LOOPOVER_LEDGER_ANCHOR_* vars #9769 — self-host preflight/config-lint says nothing about the nine anchor vars. The failure is silent (ledger_anchor_skipped_unconfigured logs and returns), so a partially-configured operator cannot discover it. Worth doing under any option below.
• anchors: POST /v1/decision-ledger/anchor-attempts publishes no request-body schema #9770 — the anchor-attempts route publishes no request-body schema, despite existing solely to be called by code this repo does not ship.

The decision

Everything else depends on one question: when the verifier does not hold the ledger, how is a reported anchor verified?

A — Anchoring is per-deployment; stop implying otherwise. Self-host anchors its own ledger and serves its own /v1/public/decision-ledger/anchors. The hosted endpoint keeps reporting empty_ledger, which is now correct rather than misleading (#9721), and the docs/proof surface point a verifier at the deployment that actually made the decision. Preserves the chain's meaning exactly: an anchor attests the ledger of the runtime that produced it. Smallest change, strongest guarantee, and the hosted anchor list stays empty on purpose.

B — Mirror self-host ledger rows into the hosted deployment. Add an ingest for decision-ledger rows so the hosted side can verify what it is asked to anchor. Makes the central endpoint genuinely non-empty. The cost is real and should not be glossed: the hosted "ledger" becomes an aggregation of self-reported rows, so its tamper-evidence proves instances reported consistently, not that decisions were honest. Also needs the row-binding check reworked around a ledger-identity dimension, since seq numbers collide across instances.

C — Record foreign anchors signature-only, labelled unverified. Cheapest path to a non-empty list: carry a ledger/instance id, verify the signature, skip the row binding. I would argue against it — publishing anchors the receiver cannot check is weaker than publishing none, and this whole area is being fixed because an empty state was being read as a healthy one.

My recommendation is A, with #9769 and #9770 done regardless. B is defensible if central aggregation is a product requirement, but it should be chosen with the trust cost stated out loud rather than discovered later.

Whichever is picked, the keypair still has to be provisioned — LOOPOVER_LEDGER_ANCHOR_KEYS plus the LOOPOVER_LEDGER_ANCHOR_PRIVATE_KEY secret — and there is no runbook for generating or rotating it. That is a third sub-issue I will file once the direction is chosen, since its content differs per option.