Two public-stats defects that become wrong the moment a second instance registers.
1. riskcontrol:fleet:<arm> is last-writer-wins, stored verbatim, never validated. src/orb/ingest.ts ~218-230 does INSERT OR REPLACE INTO system_flags (key,...) VALUES ('riskcontrol:fleet:' || arm, ...) — no per-instance keying, no shape validation beyond typeof value === "object", gated only on registeredRow?.registered === 1.
readGuarantee (src/review/public-stats.ts ~462-464) type-checks four numbers but never checks status === "calibrated", never range-checks alpha/lambda/coverageAtLambda, and never records which instance it came from.
So with two registered instances, whichever POSTs last overwrites the key — the homepage's "fleet guarantee" becomes one arbitrary instance's number, rendered next to decidedCount/instanceCount drawn from the pooled fleet. A registered instance with a stale or misconfigured LOOPOVER_RISK_CONTROL_CLOSE_ALPHA publishes its α straight to the public site.
Fix: key by riskcontrol:fleet:<instance_id>:<arm> and aggregate conservatively at read (max α, min coverage); validate status === "calibrated", 0 < alpha ≤ 0.05, 0 ≤ lambda ≤ 1, nAtLambda ≥ minimumCalibrationLabels(alpha, delta) before publishing.
2. gamingFlagsCaught cannot fire on the current fleet. src/orb/analytics.ts ~279-284 computes thresholds as the median over eligible, then compares each eligible instance to that median. With one eligible instance, fleetMedianDecided === i.decided, so highVolume = i.decided > i.decided * 2 is always false. Separately lowReversal = i.reversalRate < fleetReversalRate * 0.5 can never fire when the median reversal rate is 0 — the common case.
It is published as "proof the fleet actively polices for gaming, not just a claim of it" (public-stats.ts ~265-268) and rendered when > 0. A structurally guaranteed zero presented as a positive safety signal.
Fix: require eligible.length >= 3 before computing gaming flags and say "not enough instances to compare" otherwise; use an absolute reversal-rate floor rather than a fraction of a frequently-zero median.
3. Related, lower: decisionAccuracy is published as a median across instances (analytics.ts ~319) while accuracyCiPct is computed from pooled counts (public-stats.ts ~505-506). Different estimands — with >1 instance the interval need not contain the point estimate. The code documents that they coincide at one instance today; it is a live footgun the moment a second registers. Publish the pooled proportion as the point estimate and keep the median as a separate diagnostic.
Two public-stats defects that become wrong the moment a second instance registers.
1.
riskcontrol:fleet:<arm>is last-writer-wins, stored verbatim, never validated.src/orb/ingest.ts~218-230 doesINSERT OR REPLACE INTO system_flags (key,...) VALUES ('riskcontrol:fleet:' || arm, ...)— no per-instance keying, no shape validation beyondtypeof value === "object", gated only onregisteredRow?.registered === 1.readGuarantee(src/review/public-stats.ts~462-464) type-checks four numbers but never checksstatus === "calibrated", never range-checksalpha/lambda/coverageAtLambda, and never records which instance it came from.So with two registered instances, whichever POSTs last overwrites the key — the homepage's "fleet guarantee" becomes one arbitrary instance's number, rendered next to
decidedCount/instanceCountdrawn from the pooled fleet. A registered instance with a stale or misconfiguredLOOPOVER_RISK_CONTROL_CLOSE_ALPHApublishes its α straight to the public site.Fix: key by
riskcontrol:fleet:<instance_id>:<arm>and aggregate conservatively at read (max α, min coverage); validatestatus === "calibrated",0 < alpha ≤ 0.05,0 ≤ lambda ≤ 1,nAtLambda ≥ minimumCalibrationLabels(alpha, delta)before publishing.2.
gamingFlagsCaughtcannot fire on the current fleet.src/orb/analytics.ts~279-284 computes thresholds as the median overeligible, then compares each eligible instance to that median. With one eligible instance,fleetMedianDecided === i.decided, sohighVolume = i.decided > i.decided * 2is always false. SeparatelylowReversal = i.reversalRate < fleetReversalRate * 0.5can never fire when the median reversal rate is 0 — the common case.It is published as "proof the fleet actively polices for gaming, not just a claim of it" (
public-stats.ts~265-268) and rendered when > 0. A structurally guaranteed zero presented as a positive safety signal.Fix: require
eligible.length >= 3before computing gaming flags and say "not enough instances to compare" otherwise; use an absolute reversal-rate floor rather than a fraction of a frequently-zero median.3. Related, lower:
decisionAccuracyis published as a median across instances (analytics.ts~319) whileaccuracyCiPctis computed from pooled counts (public-stats.ts~505-506). Different estimands — with >1 instance the interval need not contain the point estimate. The code documents that they coincide at one instance today; it is a live footgun the moment a second registers. Publish the pooled proportion as the point estimate and keep the median as a separate diagnostic.