Epic: provable decision accuracy — measurement, labels, calibrated abstention, and a legible decision record

[JSONbored]

Context

We just found and fixed three independent distortions in ORB's published decision accuracy (#8820, #8823/#8824, #8825/#8826/#8827). That work makes the number honest. It does not make it provable, and it does not get us to 99.5%.

Two research sweeps (measurement/calibration; verifiable-agent architecture) produced findings that change how this goal should be framed. The three that matter most:

1. "99.5% accuracy" is not achievable as a scalar — it is achievable as a guarantee on the decisions we act on. ORB's merge/close-vs-hold split already is a selective-prediction system. The literature has an exact tool for "hit a target accuracy on the subset you decide": distribution-free risk control (Trust or Escalate, ICLR 2025; Learn then Test). The honest target is "≥99.5% guaranteed accuracy at N% coverage", never "99.5% accurate".

2. Our current metric is gameable and statistically under-powered. Because holds are excluded from the denominator (correctly — they are deferrals), raising the hold rate mechanically raises accuracy. Separately, by the statistical rule of three, claiming ≤0.5% error needs ~600 consecutive clean decisions — with even one error, substantially more. At current volumes the confidence interval is wider than the gap between 96% and 99.5%, i.e. we cannot currently detect whether we hit the target. Standard errors must also be clustered by repo and contributor; naive SEs can understate by 3x on clustered data.

3. Our ground truth is a lower bound on error, not a label. "No human reversed it" scores a silently-wrong merge as correct. Humans reverse only what they notice. Reversal-derived labels measure reversal-propensity, not correctness.

There is also a hard negative result: off-policy evaluation / IPS does not work on ORB as deployed. ORB is deterministic, so logged propensities are 0/1, there is no overlap, and IPS is undefined. It becomes usable only if we inject randomization. Do not build OPE infrastructure first.

And the largest single accuracy gain available is not an ML technique: most of our gate's criteria are objectively checkable (CI green, mergeable-clean, linked-issue eligibility, path allowlist, patch coverage). Every criterion moved from LLM judgment to a deterministic check is 100% accurate by construction and raises the ceiling.

Requirements

Sequenced deliberately — measurement before labels before mechanism. Tuning against an instrument we cannot trust is how we got here.

Phase 1 — measurement (blocks everything else)

• Publish per-class precision (merge arm and close arm separately), coverage, and Wilson score intervals — not Wald, which is badly wrong near p→1. Quote every accuracy claim as "X% at Y% coverage".
• Cluster standard errors by repo and contributor.
• Compute the achievable claim at current volume and state it publicly; if under ~600 clean decisions, 99.5% is a goal, not a measurable state.

Phase 2 — trustworthy labels

• Weekly stratified human audit sample (oversample low-confidence, merges, new contributors) against a frozen rubric. Treat no_reversal as unlabeled.
• Randomized ε-holdout (~3-5%) on would-close PRs routed to human adjudication — the only unbiased close-precision estimate available, and the only thing that makes any counterfactual estimator well-defined (selective labels, KDD 2017).
• Golden replay corpus seeded from real failures (the reversal class, CI-retrigger loophole cases, near-threshold holds), gating every gate/prompt/model/config change.

Phase 3 — mechanism

• Rubric decomposition: move every objectively-checkable criterion out of the LLM into deterministic checks; grade the subjective residual with per-dimension isolated judges and an explicit abstain option.
• Real confidence signal via self-consistency / simulated annotators (rotated few-shot exemplars, inter-run agreement). Verbalized confidence alone is poorly calibrated.
• Distribution-free risk control to set the act/hold threshold at α=0.005, with separate Neyman-Pearson thresholds per arm so the merge arm's tighter bound reflects that a wrong merge costs more than a wrong close.

Phase 4 — trust surface (architecture)

• Content-addressed decision records published on the PR: ruleset digest, prompt-template digest, model ID + dated snapshot, config-snapshot hash, and the clause that fired. Shape follows SLSA VSA. Publish the digest even where config contents stay private.
• Hash-chained append-only decision ledger + a verify endpoint.
• Generator/verifier separation: irreversible auto-close must be a deterministic function of structured findings, never a free-text model judgment. The "verifying is easier than generating" assumption is reversing for 2026 coding agents.
• Deterministic replay, treating the model call as the one irreducible nondeterministic input. Report an action-match rate; hosted inference is not bit-deterministic even at temperature 0.

Explicitly NOT doing

• TEE / confidential computing. Wrong boundary — it would attest our orchestration binary while the contested judgment executes at the model provider. The attestation protocol is also broken in the field (relay attack, CVE-2026-33697; seven binding methods tested, none prevented it). Revisit only if we self-host inference and post-handshake attestation gets fixed.
• Blockchain anchoring as the general trust mechanism for this ledger — a self-operated chain plus third-party transparency-log/git anchoring already closes the real gap (orb(trust): external decision-ledger anchoring — Rekor + git-commit, from tamper-evident to tamper-proof #9267) without asking every verifier to trust or understand a chain. Exception, added 2026-07-27: an optional, Gittensor/SN74-audience-only on-chain commitment backend (ledger: Bittensor on-chain commitment backend — optional, Gittensor/SN74-audience corroboration #9277) is in scope specifically because that audience already trusts and verifies via subtensor, using infrastructure the operator already runs — never the default or required verification path for the general contributor/maintainer audience this repo otherwise serves.
• Reproducible builds of ORB (proves the binary matches source — undisputed; our contested surface is ruleset + prompt, which the decision record pins for near-zero cost).
• Bitwise-deterministic inference, multi-agent debate (bounded by the strongest agent; majority voting explains most of the claimed gain), ECE as a headline KPI, and position-bias mitigation (that literature is pairwise; our gate is pointwise).

Test Coverage Requirements

99%+ patch coverage, branch-counted, on all changed src/** lines, both arms of every conditional. Note the meta-point this epic exists to fix: ORB's decision policy currently ships without an evaluation gate in a repo that enforces 99% patch coverage on everything else.

Expected Outcome

A published figure of the form "≥99.5% guaranteed accuracy at N% coverage", defensible in both directions, backed by human-audited labels, gated against regression, and accompanied by a per-decision record a contributor can inspect and challenge.

[linear-code]

JSO-6108