feat(stats): publish a fleet accuracy trend, so the weekly table means something again - #9775
Merged
Conversation
…s something again After #9718 and #9768 the own-ledger weekly trend is correct and, for recent weeks, correctly null: the hosted Worker does not execute reviews, so that ledger is frozen while Orb volume keeps growing. A column of nulls beside a rising volume column is honest and tells a reader nothing about how the gate behaves today. orb_signals already carries everything needed, with no new ingest, no new column and no new secret: self-host runtimes export gate_verdict/outcome/reversal_flag per PR, the hosted side validates and stores them, and computeFleetAnalytics already turns them into the 90-day headline. This buckets the SAME rows weekly. Two deliberate non-choices, both of which would have reintroduced the bug class this surface is being corrected for: - It is a SEPARATE series, never blended with the own-ledger one. orb_signals is keyed by per-instance HMACs of repo/PR; audit_events and orb_pr_outcomes use raw owner/repo#number. The populations cannot be joined, so a reversal from one can never be attributed to a decided row in the other, and one number over a mixed denominator would be exactly the defect #9676 reports. - It does NOT use 1 - reversalRate. #8820 established decisionAccuracy as the fleet estimand because the reversal formula divides by holds -- deferrals that cannot be right or wrong -- and misses mispredictions carrying no reversal marker. A weekly series on a different estimand than the headline directly above it would make the page disagree with itself. The weekly fold calls foldInstance rather than reimplementing the confusion matrix. That function is already exported for the federated bundle with a doc comment warning that a second copy makes comparisons apples-to-oranges, and the accounting is subtle: policy_action rows are excluded from scoring but still count as activity, and a superseded close is disconfirmed exactly like a reopen. Only registered instances count, matching computeFleetAnalytics' own trust gate -- the ingest is open, so a stranger's signals must not move a published number until a human opts them in. There is a test pinning that. Refs #9676
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | f7777b1 | Commit Preview URL Branch Preview URL |
Jul 29 2026, 07:55 AM |
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-29 08:20:19 UTC
Review summary Nits — 5 non-blocking
Concerns raised — review before merging
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Bundle ReportChanges will increase total bundle size by 2.43kB (0.03%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9775 +/- ##
==========================================
- Coverage 90.28% 89.47% -0.82%
==========================================
Files 907 908 +1
Lines 113354 113385 +31
Branches 26893 26899 +6
==========================================
- Hits 102339 101446 -893
- Misses 9683 10849 +1166
+ Partials 1332 1090 -242
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This was referenced Jul 29, 2026
orb_signals grows unbounded, and adding retention would silently corrupt two existing surfaces
#9783
Closed
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After #9718 and #9768 the own-ledger weekly trend is correct and, for recent weeks, correctly null — the hosted Worker doesn't execute reviews, so that ledger is frozen while Orb volume keeps growing. A column of nulls beside a rising volume column is honest and tells a reader nothing about how the gate behaves today.
orb_signalsalready carries everything needed, with no new ingest, no new column and no new secret: self-host runtimes already exportgate_verdict/outcome/reversal_flagper PR, the hosted side already validates and stores them, andcomputeFleetAnalyticsalready turns them into the 90-day headline. This buckets the same rows weekly.Refs #9676
Two deliberate non-choices
Both would have reintroduced the bug class this surface is being corrected for:
It is a SEPARATE series, never blended.
orb_signalsis keyed by per-instance HMACs of repo/PR;audit_eventsandorb_pr_outcomesuse rawowner/repo#number. The populations cannot be joined, so a reversal from one can never be attributed to a decided row in the other. One number over a mixed denominator is exactly the defect #9676 reports.It does NOT use
1 - reversalRate. #8820 establisheddecisionAccuracyas the fleet estimand, because the reversal formula divides by holds — deferrals that can be neither right nor wrong — and misses mispredictions carrying no reversal marker. A weekly series on a different estimand than the headline directly above it would make the page disagree with itself. There's an invariant test pinning this: 5 confirmed merges plus 95 holds reports 5 decisions at 100%, not ~100% over 100.The weekly fold calls
foldInstancerather than reimplementing the confusion matrix. That function is already exported for the federated bundle with a doc comment warning that a second copy makes comparisons apples-to-oranges — and the accounting is subtle:policy_actionrows are excluded from scoring but still count as activity (#8825), and asupersededclose is disconfirmed exactly like a literal reopen (#8166). Both have tests.Only registered instances count, matching
computeFleetAnalytics' own trust gate — the ingest is open, so a stranger's signals must not move a published number until a human opts them in. Pinned by an invariant test seeding 40 rows from an unregistered instance and asserting every week stays null.Validation
public-fleet-accuracy-trend.tstypecheck·ui:typecheck·ui:openapi:check·contract:api-schemas:check·ui-derived-types:check·docs:drift-check·ui:lint(0 errors) ·ui:buildBucketing is by when the gate decided, not when the row arrived — a late export must not pile weeks of decisions into the week it landed. There's a test for that, and one for a malformed
decision_timestamp(the column isTEXTwith no format constraint, so it's reachable) being dropped rather than bucketed to the epoch.Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.Aggregates only — week, a count, and a percentage.
orb_signals' repo/PR columns are per-instance HMACs and are never read here; nothing identifying can reach the response. The registered-instance join is the security-relevant boundary and has its own invariant test. The contract field is added to@loopover/contract, so the UI got a compile error until its fixtures were updated — the intended behaviour of that package.UI Evidence
Rendered against the real component with a captured production payload plus a representative
fleetAccuracyTrend, at 1280×800,deviceScaleFactor: 2. The section is hidden entirely when no week has a scored verdict (tested), so a deployment with no fleet data sees no empty table.There is no "before": this section does not exist on
main. The own-ledger table it sits beside is unchanged.