fix(review): correct risk-control calibration and the published guarantee's fleet integrity#9228
Merged
Merged
Conversation
…calibration delta (#9048, #9066) calibrateActThreshold returned "insufficient_labels" from two branches with different "have" semantics: a genuine label shortfall (have = total pairs) and a residual high-confidence stratum too small to certify despite ample total labels (have = that stratum's size). The latter now returns a distinct "no_certifiable_threshold" status carrying totalPairs/bestN/bestLambda/bestUpperBound, rendered through its own message and a distinct risk_control_no_certifiable_threshold audit event so the label burn-down no longer conflates "needs labels" with "needs a better error rate". Separately, the ascending-lambda scan reported whichever of K observed-confidence candidates passed first without correcting for testing K of them — the advertised 1-delta confidence overstated what the scan actually delivered. Each candidate is now tested at a Bonferroni-split delta/K, so the certified lambda is valid at the full, originally-advertised delta regardless of which candidate passes. Chosen over a true fixed-sequence rewrite as the smaller, safer diff against the existing ascending scan. Also adds AND dr2.action = dal.verdict to the calibration join (latent-risk hardening: today's data is not mis-joined, but a later HOLD/MERGE record on the same PR could otherwise shadow the acted CLOSE record a label adjudicates) and tags each calibration pair's provenance (backfilled vs live, via the backfill's configDigest sentinel) so a published guarantee can later say how much of its evidence is reconstructed history.
…ction, pool decisionAccuracy (#9068) orb_signals ingest stored any object-shaped risk_control payload verbatim, so a registered instance with a stale or misconfigured alpha (or an outright refused calibration) could reach the public guarantee unchecked. handleOrbIngest now runs the payload through validateCalibrationPayload (status === "calibrated", alpha/ lambda/coverage in range, nAtLambda clearing the zero-error floor for its own alpha/delta) before it reaches orb_risk_control_arms at all. gamingPatternFlags compared each eligible instance against the fleet median, which is structurally unfireable below 3 eligible instances (an instance IS the median at n=1) and can never flag "low reversal" once the fleet's own reversal-rate median is exactly 0 (a common, healthy-fleet case) since a fraction of zero can never be undercut. computeFleetAnalytics now gates detection on eligible.length >= 3 (surfaced via the new gamingDetectionEligible field) and falls back to an absolute reversal-rate floor when the fleet median is zero. fleet.decisionAccuracy published the per-instance MEDIAN while accuracyCiPct (public-stats.ts) is a Wilson interval over the POOLED counts — different estimands that only coincide at equal per-instance volumes. fleet.decisionAccuracy now publishes the pooled proportion directly (the same population the interval describes); the per-instance median survives as the new decisionAccuracyMedian diagnostic field. The underlying per-instance last-writer-wins fleet-key bug this issue also describes was already fixed by #9177 (orb_risk_control_arms is keyed per instance_id/arm and public-stats already aggregates across registered instances at read time) — this change covers the remaining validation-before-publish and detection-floor gaps.
…date it before serving (#9050) readGuarantee published "coveragePct" as if it were a share of all closes, adjacent to a different fleetAccuracy.coveragePct that IS a share of all decided signals -- one word, two denominators. The guarantee's own field is actually the share of the arm's AI-JUDGED sub-population the threshold covers (loadCalibrationPairs can only join a confidence to decisions an AI-judgment blocker ran on, a minority of real closes). Renamed to aiJudgedCoveragePct and the homepage string now names the sub-population explicitly instead of leaving a bare percentage next to its sibling. readGuarantee also now re-validates every stored orb_risk_control_arms row through risk-control.ts's validateCalibrationPayload (defense in depth alongside the ingest-side check from the companion #9068 fix) and walks all registered rows for an arm in nAtLambda-descending order instead of trusting only the top one, so a single malformed or stale peer can no longer hide a good row behind it. Each calibration pair now carries whether it's backfilled (the 2026-07 calibration-corpus backfill's configDigest sentinel) or live; calibrateActThreshold surfaces the split as backfilledPairs, and the public guarantee renders it as backfilledPct so a guarantee resting mostly on reconstructed history says so. Regenerated apps/loopover-ui/public/openapi.json for the schema rename/nullability.
…ions npm run ui:lint's format:check caught unformatted lines from the #9050 fleetAccuracy.guaranteed rename in proof-of-power-stats-model.ts and its new test.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | b89464d | Commit Preview URL Branch Preview URL |
Jul 27 2026, 08:06 AM |
Bundle ReportChanges will increase total bundle size by 952 bytes (0.01%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
❌ 4 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four related fixes to the published statistical guarantee subsystem:
insufficient_labelsoutcomes (genuinely too few labels vs. ample labels but no certifiable threshold) into distinct outcomes with distinct messages and event types, so the label burn-down dashboard stops conflating "needs labels" with "needs better precision."gamingFlagsCaughtcomputation to require enough eligible instances, and publishes the pooled proportion asdecisionAccuracy's point estimate instead of conflating it with the cross-instance median.Test plan
npx tsc --noEmit -p tsconfig.json --incremental false— cleanmain(picked up fix(orb): installation/permission/config integrity — suspension backfill, broker identity, numeric env validation, migration content hash #9222 — no conflicts, re-verified)Closes #9048
Closes #9050
Closes #9066
Closes #9068