fix(selfhost): count the Orb's own decision ledger in published totals - #9970
Closed
JSONbored wants to merge 1 commit into
Closed
fix(selfhost): count the Orb's own decision ledger in published totals#9970JSONbored wants to merge 1 commit into
JSONbored wants to merge 1 commit into
Conversation
An Orb published `totals.handled: 0` in the same payload as `reviewParity.verdicts: 2123`, both derived from its own ledger. `totals` had exactly two sources and both are hosted-Worker concepts. The own-ledger side reads `audit_events` scoped to `LOOPOVER_PUBLIC_STATS_REPOS` -- a frozen snapshot of the repos the old central App used to process, which no self-hoster has a reason to set, and when it is unset those queries are skipped outright. The other side folds in what OTHER registered installations reported via `orb_pr_outcomes`, which on an Orb is nobody. Both are structurally empty there, while `decision_records` -- the anchored ledger the Orb writes a row to per verdict -- was not a source at all. Every `totals.*` figure inherited it. The issue guessed the split was Postgres versus D1. It is not: both reads hit the same store on that box. `LOOPOVER_PUBLIC_STATS_REPOS` is simply unset there, which is confirmed by the deployment's own environment. The decision ledger is now a third disposition source, in the same shape the published-surface query produces, so totals, byProject and the ordering all fold it identically. It is deliberately evaluated OUTSIDE the allowlist branch -- an empty allowlist is precisely the case it exists to cover. The three sources stay disjoint by construction rather than being reconciled afterwards, which is the discipline the file header already states for the first two. Two anti-joins do it, each excluding exactly what another source counts: the published-surface set but only where the repo is allowlisted (excluding an un-allowlisted published pair would drop it from every source at once), and the registered-install outcomes the fleet fold counts. Verdicts are counted per distinct PR, so a re-evaluated PR does not inflate the headline. On the hosted Worker `decision_records` is empty by design, so its published figures do not move at all. Tests run against a real migrated D1 rather than a SQL-shape stub, because the bug was a real query returning nothing for a real reason. The parity invariant is asserted through the verifier's own `checkStatsParity` rather than a restatement of it, so it cannot drift from the tool that flagged production. Each guard was mutation-tested: gating the source behind the allowlist, dropping either anti-join, matching unregistered installs, and counting rows instead of distinct PRs all fail. Closes #9963
Contributor
|
Important 🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪 🔍 LoopOver is reviewing…AI analysis is in progress. This comment will update when the review is complete. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Owner
Author
|
Superseded by #9967, which landed the same root-cause analysis (the Two things #9967 gets right that this PR did not:
Closing rather than rebasing: the remaining delta is not the fix, it is the cross-surface regression test #9963's acceptance asked for, which is tracked separately and lands on its own. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The contradiction
An Orb published, in one payload:
reviewParity.verdicts= 2123 (fromdecision_records)totals.handled= 0Root cause — not the one the issue guessed
The issue supposed
totals.*reads D1 while the Orb's primary store is Postgres. It does not: both reads hit the same store on that box. The real split is which tables, and the answer is that neither oftotals' two sources exists on a self-hosted Orb.audit_eventspublished-surface, scoped toLOOPOVER_PUBLIC_STATS_REPOSgetPublicStatsskips those queries outright when it is emptyorb_pr_outcomesfor other registered installationsdecision_recordsLOOPOVER_PUBLIC_STATS_REPOSis a frozen snapshot of the repos the old central App used to process. It is a hosted-Worker concept; no self-hoster has a reason to fill it in. Confirmed directly against the deployment — the Orb setsLOOPOVER_REVIEW_REPOS(three repos) and does not setLOOPOVER_PUBLIC_STATS_REPOSat all. So everytotals.*figure was structurally zero on the deployment that does the work, whilereviewParity— which has no allowlist — reported reality.The fix
decision_recordsbecomes a third disposition source, emitted in the same shape the published-surface query produces sototals,byProjectand the ordering all fold it identically. It is deliberately evaluated outside the allowlist branch: an empty allowlist is precisely the case it exists to cover, so gating it there would reproduce the bug.The three sources stay disjoint by construction rather than reconciled afterwards — the discipline the file header already states for the first two. Two anti-joins do it, each excluding exactly what another source counts:
(repo, pr_number)key its own anti-join uses, and on the sameregistered = 1population.Verdicts are counted per distinct PR, so a re-evaluated PR (which appends rows) does not inflate the headline.
On the hosted Worker
decision_recordsis empty by design (review execution retired), so its published figures do not move at all — pinned by a test.Deliberately not changed
accuracyPctstill pairswindowedMerged/windowedClosedwithtotals.reversed. Those share the numerator's bounds by design (own-ledger, inside the retention window); widening the denominator with ledger rows whose reversals are not in that population is exactly the #7449 error the file warns against. On the Orb it therefore reportsnull— honest — and the UI already prefersfleetAccuracy.Tests
Against a real migrated D1, not a SQL-shape stub: the bug was a real query returning nothing for a real reason, and a stub asks the wrong question and answers it happily.
The parity invariant is asserted by calling the verifier's own
checkStatsParity— the function that printed the original FAIL — rather than restating it, so it cannot drift from the tool that decides whether production is publishing a contradiction.Every guard is mutation-tested; each fails, baseline restored green:
The fleet anti-join had no covering test on the first pass; the two
orb_pr_outcomescases were added specifically because that mutation survived.Acceptance criteria
totals.handledreflects reality rather than 0.loopover-verifyreports the stats-parity claim as PASS (pinned via the verifier's own checker).totals.handledandreviewParity.verdictscannot disagree — both directions, including the failing one.Verification
npx tsc --noEmitclean; static checker sweep greennpx vitest run test/unit— 1299 files, 25323 tests passedLOOPOVER_PUBLIC_STATSremains off on the Orb; enabling it is a separate operator decision once this ships.Closes #9963