Reversal signal is unavailable on the hosted runtime, so own-ledger accuracy is unmeasurable there

[JSONbored]

What's wrong

loopover.ai/fairness currently shows a headline 94.6% decision accuracy while every by-repository row and every one of the 8 weekly-trend rows shows 100% with Reversed: 0, and the per-rule table shows 62.4% for two different rules. Four numbers, four different tables, four different definitions of "wrong" — and only one of them is measuring anything live.

Live payload (GET https://api.loopover.ai/v1/public/stats) confirms totals.reversed: 0, every byProject[].accuracyPct: 100, and every accuracyTrend[].reversed: 0.

Root cause

Every writer of ground-truth "the bot was wrong" evidence lives in the review-execution pipeline, which the hosted Worker acks-and-drops — while every reader of that evidence still runs there and renders its absence as perfection.

recordReversalSignals is called from exactly one place (src/queue/processors.ts:7039, inside handlePullRequestWebhookEvent), reached only via the github-webhook job. That job type is in REVIEW_EXECUTION_JOB_TYPES (src/selfhost/review-runtime.ts:4), so on the hosted Worker src/index.ts:61-75 logs retired_review_job_ignored and acks it. The Orb App webhook path that does still run writes only orb_pr_outcomes (src/orb/outcomes.ts:9-32) and has no reversal concept.

The repo already records the empirical result — src/review/reversal-superseded.ts:3-4: "verified in production: zero reversal events ever."

Three separate defects fall out of this

1. By-repository: lifetime denominator ÷ 90-day-pruned numerator. reviewed counts github_app.pr_public_surface_published, the one event type exempt from retention (src/db/retention.ts:14,213-216), so it is lifetime and immortal. The reversed numerator reads reversal_* events from audit_events, pruned at 90 days (src/db/retention.ts:25). Even with a working signal, that ratio drifts toward 100% by construction.

2. Weekly trend: the denominator grows via Orb, the numerator is own-ledger-only. merged/closed fold in Orb fleet rows (src/services/public-accuracy-trend.ts:181-182) which keep growing, while reversed comes only from loadReversalDayRows (:111-139), which additionally requires an agent.action.close|merge row created in the last 8 weeks — also written only by the retired pipeline. This is the same asymmetry that src/review/public-stats.ts:463-470 already fixed for the lifetime number and did not fix for the trend. Result: rising volume, permanently 100%.

3. Per-rule precision is one number printed twice. ai_consensus_defect and slop_gate_score both report decided: 460, confirmed: 287 because slop_gate_score's override rows are a relabeled copy of ai_consensus_defect's: scripts/backfill-slop-corpus.ts:2-4 reads the exported ai_consensus_defect corpus, and manifestToSourceCases (:51-58) passes backtestCase.label through verbatim into the new rule's verdict (scripts/backfill-slop-corpus-core.ts:133-134,165). Only the score side is recomputed; the human verdict is copied.

Separately, slop_gate_score has no live path to a confirmed override at all — recordImplicitTerminalConfirmations excludes GATE_SCORE_SIGNAL_CODES (src/review/outcomes-wire.ts:654, src/rules/advisory.ts:207), and recordConfiguredGateBlockerOverrides only ever writes reversed. So a non-zero precision for it is proof the rows are backfilled, not live.

4. The UI drops the honesty label the backend computes. With instanceCount === 1 the API correctly sets basis: "single_instance_self_report" (src/review/public-stats.ts:599, src/orb/analytics.ts:49,402) — the field #9168 added precisely to stop this being read as fleet corroboration. fairness-report-page.tsx never renders basis, decidedCount, accuracyCiPct, or coveragePct; the caption says only "across 1 self-hosted instance". The headline is also decisionAccuracy (confirmed verdicts / verdicts) while the explainer directly beneath it (fairness-report-page.tsx:152-157) describes the reversal formula, i.e. a different quantity.

Also worth noting: loadPublicRulePrecision returns a reversals: {reopened, reverted, superseded} block that would read 0,0,0 for these reasons, and the UI is the only reason it isn't visible — it renders the rules table and freeze point only (fairness-report-page.tsx:261-324).

What needs deciding

The page should apply the same structural-zero discipline it already applies elsewhere — "insufficient data" is rendered for a sparse rule rather than 0%, and gamingFlagsCaught is correctly null below 3 instances (src/orb/analytics.ts:343). A 100% derived from 1 − 0/N where the numerator is unwritable should not render as a measurement. Whether that means hiding the tables, labeling them "not measurable on this deployment", or restoring the reversal signal on the hosted path is the call to make.

Unambiguous regardless of that decision: the trend's Orb/own-ledger denominator asymmetry (defect 2), and surfacing basis / CI bounds in the UI (defect 4).

Related: #9636 (the same class of problem on the corpus freeze point).

[linear-code]

JSO-6739

[JSONbored]

Addressed across two PRs:

• fix(stats): withhold reversal-grounded accuracy where a reversal cannot be recorded #9718 — defects 1, 2 and 4: accuracy is withheld (null, with an on-page explanation) where a reversal cannot be recorded, the weekly trend's Orb-vs-own-ledger denominator asymmetry is fixed, and the UI now surfaces basis/decidedCount/accuracyCiPct plus a methodology note that describes both measures rather than one.
• fix(stats): drop counterfactual-replay rows from the public per-rule precision #9720 — defect 3: the counterfactual-replay rows are dropped from the public per-rule precision by provenance, so slop_gate_score no longer publishes ai_consensus_defect's label set under its own name.

What is deliberately not closed by either: restoring a real reversal signal on the hosted surface. Both PRs stop the page claiming a measurement it cannot make; neither makes the measurement possible. That needs either running the reversal writer on that runtime or ingesting reversals from self-host instances the way orb_pr_outcomes already ingests outcomes — the same architectural question as #9719, and the reason this issue should stay open after both merge.