Skip to content

orb(data): fleet guarantee is a single last-writer-wins key with no validation, and gamingFlagsCaught is structurally always 0 #9068

Description

@JSONbored

Two public-stats defects that become wrong the moment a second instance registers.

1. riskcontrol:fleet:<arm> is last-writer-wins, stored verbatim, never validated. src/orb/ingest.ts ~218-230 does INSERT OR REPLACE INTO system_flags (key,...) VALUES ('riskcontrol:fleet:' || arm, ...)no per-instance keying, no shape validation beyond typeof value === "object", gated only on registeredRow?.registered === 1.

readGuarantee (src/review/public-stats.ts ~462-464) type-checks four numbers but never checks status === "calibrated", never range-checks alpha/lambda/coverageAtLambda, and never records which instance it came from.

So with two registered instances, whichever POSTs last overwrites the key — the homepage's "fleet guarantee" becomes one arbitrary instance's number, rendered next to decidedCount/instanceCount drawn from the pooled fleet. A registered instance with a stale or misconfigured LOOPOVER_RISK_CONTROL_CLOSE_ALPHA publishes its α straight to the public site.

Fix: key by riskcontrol:fleet:<instance_id>:<arm> and aggregate conservatively at read (max α, min coverage); validate status === "calibrated", 0 < alpha ≤ 0.05, 0 ≤ lambda ≤ 1, nAtLambda ≥ minimumCalibrationLabels(alpha, delta) before publishing.

2. gamingFlagsCaught cannot fire on the current fleet. src/orb/analytics.ts ~279-284 computes thresholds as the median over eligible, then compares each eligible instance to that median. With one eligible instance, fleetMedianDecided === i.decided, so highVolume = i.decided > i.decided * 2 is always false. Separately lowReversal = i.reversalRate < fleetReversalRate * 0.5 can never fire when the median reversal rate is 0 — the common case.

It is published as "proof the fleet actively polices for gaming, not just a claim of it" (public-stats.ts ~265-268) and rendered when > 0. A structurally guaranteed zero presented as a positive safety signal.

Fix: require eligible.length >= 3 before computing gaming flags and say "not enough instances to compare" otherwise; use an absolute reversal-rate floor rather than a fraction of a frequently-zero median.

3. Related, lower: decisionAccuracy is published as a median across instances (analytics.ts ~319) while accuracyCiPct is computed from pooled counts (public-stats.ts ~505-506). Different estimands — with >1 instance the interval need not contain the point estimate. The code documents that they coincide at one instance today; it is a live footgun the moment a second registers. Publish the pooled proportion as the point estimate and keep the median as a separate diagnostic.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions