Skip to content

fix(selfhost): count the Orb's own decision ledger in published totals - #9970

Closed
JSONbored wants to merge 1 commit into
mainfrom
fix/self-host-totals-read-own-ledger-9963
Closed

fix(selfhost): count the Orb's own decision ledger in published totals#9970
JSONbored wants to merge 1 commit into
mainfrom
fix/self-host-totals-read-own-ledger-9963

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

The contradiction

An Orb published, in one payload:

  • reviewParity.verdicts = 2123 (from decision_records)
  • totals.handled = 0

Root cause — not the one the issue guessed

The issue supposed totals.* reads D1 while the Orb's primary store is Postgres. It does not: both reads hit the same store on that box. The real split is which tables, and the answer is that neither of totals' two sources exists on a self-hosted Orb.

Source What it reads On the Orb
own-ledger audit_events published-surface, scoped to LOOPOVER_PUBLIC_STATS_REPOS skipped — the allowlist is unset, and getPublicStats skips those queries outright when it is empty
fleet fold orb_pr_outcomes for other registered installations empty — an Orb is not the aggregator
the actual ledger decision_records 2,123 rows, not a source at all

LOOPOVER_PUBLIC_STATS_REPOS is a frozen snapshot of the repos the old central App used to process. It is a hosted-Worker concept; no self-hoster has a reason to fill it in. Confirmed directly against the deployment — the Orb sets LOOPOVER_REVIEW_REPOS (three repos) and does not set LOOPOVER_PUBLIC_STATS_REPOS at all. So every totals.* figure was structurally zero on the deployment that does the work, while reviewParity — which has no allowlist — reported reality.

The fix

decision_records becomes a third disposition source, emitted in the same shape the published-surface query produces so totals, byProject and the ordering all fold it identically. It is deliberately evaluated outside the allowlist branch: an empty allowlist is precisely the case it exists to cover, so gating it there would reproduce the bug.

The three sources stay disjoint by construction rather than reconciled afterwards — the discipline the file header already states for the first two. Two anti-joins do it, each excluding exactly what another source counts:

  • the published-surface set, but only where that repo is allowlisted, because that is the exact condition under which the first query counts a pair. Excluding an un-allowlisted published pair would drop it from every source at once.
  • the registered-install outcomes the fleet fold counts, matched on the same (repo, pr_number) key its own anti-join uses, and on the same registered = 1 population.

Verdicts are counted per distinct PR, so a re-evaluated PR (which appends rows) does not inflate the headline.

On the hosted Worker decision_records is empty by design (review execution retired), so its published figures do not move at all — pinned by a test.

Deliberately not changed

accuracyPct still pairs windowedMerged/windowedClosed with totals.reversed. Those share the numerator's bounds by design (own-ledger, inside the retention window); widening the denominator with ledger rows whose reversals are not in that population is exactly the #7449 error the file warns against. On the Orb it therefore reports null — honest — and the UI already prefers fleetAccuracy.

Tests

Against a real migrated D1, not a SQL-shape stub: the bug was a real query returning nothing for a real reason, and a stub asks the wrong question and answers it happily.

The parity invariant is asserted by calling the verifier's own checkStatsParity — the function that printed the original FAIL — rather than restating it, so it cannot drift from the tool that decides whether production is publishing a contradiction.

Every guard is mutation-tested; each fails, baseline restored green:

Mutation Result
Ledger source gated behind the allowlist (the original bug) 4 failed
Published-surface anti-join dropped 1 failed
Fleet anti-join dropped 1 failed
Anti-join matches unregistered installs too 1 failed
Counts ledger rows instead of distinct PRs 1 failed

The fleet anti-join had no covering test on the first pass; the two orb_pr_outcomes cases were added specifically because that mutation survived.

Acceptance criteria

  • On a self-hosted Orb with a populated ledger, totals.handled reflects reality rather than 0.
  • loopover-verify reports the stats-parity claim as PASS (pinned via the verifier's own checker).
  • A regression test pins that totals.handled and reviewParity.verdicts cannot disagree — both directions, including the failing one.

Verification

  • npx tsc --noEmit clean; static checker sweep green
  • npx vitest run test/unit1299 files, 25323 tests passed

LOOPOVER_PUBLIC_STATS remains off on the Orb; enabling it is a separate operator decision once this ships.

Closes #9963

An Orb published `totals.handled: 0` in the same payload as
`reviewParity.verdicts: 2123`, both derived from its own ledger.

`totals` had exactly two sources and both are hosted-Worker concepts. The
own-ledger side reads `audit_events` scoped to `LOOPOVER_PUBLIC_STATS_REPOS` --
a frozen snapshot of the repos the old central App used to process, which no
self-hoster has a reason to set, and when it is unset those queries are skipped
outright. The other side folds in what OTHER registered installations reported
via `orb_pr_outcomes`, which on an Orb is nobody. Both are structurally empty
there, while `decision_records` -- the anchored ledger the Orb writes a row to
per verdict -- was not a source at all. Every `totals.*` figure inherited it.

The issue guessed the split was Postgres versus D1. It is not: both reads hit
the same store on that box. `LOOPOVER_PUBLIC_STATS_REPOS` is simply unset there,
which is confirmed by the deployment's own environment.

The decision ledger is now a third disposition source, in the same shape the
published-surface query produces, so totals, byProject and the ordering all fold
it identically. It is deliberately evaluated OUTSIDE the allowlist branch --
an empty allowlist is precisely the case it exists to cover.

The three sources stay disjoint by construction rather than being reconciled
afterwards, which is the discipline the file header already states for the first
two. Two anti-joins do it, each excluding exactly what another source counts:
the published-surface set but only where the repo is allowlisted (excluding an
un-allowlisted published pair would drop it from every source at once), and the
registered-install outcomes the fleet fold counts. Verdicts are counted per
distinct PR, so a re-evaluated PR does not inflate the headline.

On the hosted Worker `decision_records` is empty by design, so its published
figures do not move at all.

Tests run against a real migrated D1 rather than a SQL-shape stub, because the
bug was a real query returning nothing for a real reason. The parity invariant
is asserted through the verifier's own `checkStatsParity` rather than a
restatement of it, so it cannot drift from the tool that flagged production.
Each guard was mutation-tested: gating the source behind the allowlist, dropping
either anti-join, matching unregistered installs, and counting rows instead of
distinct PRs all fail.

Closes #9963
@loopover-orb

loopover-orb Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 LoopOver is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 30, 2026
@JSONbored

Copy link
Copy Markdown
Owner Author

Superseded by #9967, which landed the same root-cause analysis (the LOOPOVER_PUBLIC_STATS_REPOS gate, not a D1/Postgres split) and a better fix.

Two things #9967 gets right that this PR did not:

  • byProject stays gated on the allowlist. This PR populated it from the ledger, which would have published repo names that were never opted in. fix(stats): derive the published totals from the ledger instead of publishing a false zero #9967's framing is the correct one — the privacy control is about naming repos, and totals carries no repo identity while byProject does.
  • Ledger-as-fallback rather than a third summed source. Using the ledger only when nothing is allowlisted removes the double-counting question by construction, instead of answering it with two anti-joins that then have to be kept correct.

Closing rather than rebasing: the remaining delta is not the fix, it is the cross-surface regression test #9963's acceptance asked for, which is tracked separately and lands on its own.

@JSONbored JSONbored closed this Jul 30, 2026
@JSONbored
JSONbored deleted the fix/self-host-totals-read-own-ledger-9963 branch July 30, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

selfhost(stats): an Orb reports totals.handled = 0 while its ledger holds 2,123 verdicts

1 participant