metrics(ledger): weekly automation-rate series, published on /fairness - #9938
Merged
Conversation
#9727, #9728) "How much of this is actually automated?" underwrites every other claim the gate makes, and it was only answerable by ad-hoc query. It is a standing series now, computed from decision_records alone so an outsider holding the export recomputes the same numbers. The definition is the load-bearing part. A PR is automated only if EVERY verdict for it was a merge or close with no human in the path -- a hold, a named reevaluation actor, or a maintainer_request all make it manual. So a PR that was held and later merged is manual, not automated: counting the final disposition would let the rate be inflated by holding everything and merging it by hand, which is precisely the failure this number exists to expose. Weeks are bucketed by a PR's FIRST verdict, so a PR does not migrate between weeks as it accrues re-evaluations, and it is counted once however many verdicts it has. BACKFILL HORIZON, stated rather than hidden: the re-evaluation provenance fields arrived with migration 0204, so earlier weeks can only detect manual work that took the form of a hold and are published as basis "holds_only". A week straddling that date is labelled holds_only too -- understating confidence rather than overstating it, which is the direction of error a fairness figure has to take. /fairness renders the rate, the weekly table and the definition, with the same zero-state conventions as the sibling series: an empty window is a MEASURED zero with its dates, never a blank.
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-30 19:01:01 UTC
Review summary Blockers
Nits — 6 non-blocking
Concerns raised — review before merging
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | e429fd9 | Commit Preview URL Branch Preview URL |
Jul 30 2026, 06:09 PM |
Bundle ReportChanges will increase total bundle size by 10.1kB (0.13%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9938 +/- ##
==========================================
- Coverage 91.87% 91.00% -0.87%
==========================================
Files 923 924 +1
Lines 113436 113496 +60
Branches 27331 27348 +17
==========================================
- Hits 104219 103287 -932
- Misses 7928 9117 +1189
+ Partials 1289 1092 -197
Flags with carried forward coverage won't be shown. Click here to find out more.
|
The series folded EVERY decision_records row in the window, with no filter on action. `action` also carries non-deciding classes -- `label`, `update_branch`, `approve`, and the error/no-op classes -- none of which is `hold`, so a pull request that only ever drew those carried no human signal and was counted as an AUTOMATED decision. That inflated the published rate with pull requests the gate never decided at all, contradicting the definition published alongside it. A PR now enters the series only when it reached an enacting disposition (`merge`/`close`) or a hold, and is automated only when it enacted one with no human signal. A non-deciding verdict alongside a real one does not flip the PR to manual -- that is the bot acting, not a person, and treating it as manual would under-count automation just as wrongly. The read filters on the same exported constant the fold classifies on, so the WHERE clause and the predicate cannot drift apart.
This was referenced Jul 30, 2026
JSONbored
added a commit
that referenced
this pull request
Jul 30, 2026
…n tree (#9944) * fix(ci): drop two unreferenced exports that turned main red #9938 exported AUTOMATION_ENACTING_ACTIONS and AUTOMATION_HOLD_ACTION, but AUTOMATION_COUNTED_ACTIONS is the surface both the fold and the read actually consume, so the other two are referenced only inside their own module. dead-exports:check runs in test:ci and fails on them. Module-private now; the exported vocabulary and every behaviour are unchanged. * fix(ci): re-derive the replay-runner image manifest after a lockfile change A merge changed package-lock.json without re-running replay-runner-manifest:write, so the committed packageLockSha256 no longer matched the committed lockfile and replay-runner-manifest:check -- which runs in test:ci -- failed on clean main. Regenerated; only the lockfile hash and the rolled-up digest move.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9727. Closes #9728. Completes epic #9726.
Problem
"How much of this is actually automated?" underwrites every other claim the gate makes, and it was only answerable by ad-hoc query. The ledger records every disposition and whether a human touched the decision path, but nothing rolled it up.
The definition is the load-bearing part
Computed from
decision_recordsalone, so an outsider holding the export recomputes the same numbers. A PR is automated only if every verdict for it was a merge or close with no human in the path. Three things mark a human:action = 'hold'— the gate declined to decide and handed it to a personreevaluation_actor— a named person caused a re-evaluation (ledger(policy): verdict immutability per head SHA — re-evaluations require a reason code and leave a trail #9742)reevaluation_reason = 'maintainer_request'— a human asked for the re-runSo a PR that was held and later merged is manual, not automated. Counting the final disposition instead would let the rate be inflated by holding everything and then merging it by hand — precisely the failure mode this number exists to expose. There is a test named for that.
Machine-paced re-evaluation causes (
scheduled_recheck,pipeline_error,upstream_state_change,config_change) explicitly do not count as human action: that is the automation working, not someone intervening.PRs are counted once, bucketed by their first verdict, so a PR does not migrate between weeks as it accrues re-evaluations.
Backfill horizon, stated rather than hidden
#9727 asks for the horizon to be stated. The re-evaluation provenance fields arrived with migration 0204, so earlier weeks can only detect manual work that took the form of a
hold— they under-count manual work. Those weeks are published asbasis: "holds_only"rather than silently mixed in with complete ones.A week straddling the horizon is labelled
holds_onlytoo: understating confidence rather than overstating it, which is the direction of error a fairness figure has to take. That is a test, not a comment.The horizon is a single dated constant rather than something inferred — "the column did not exist yet" and "this was a first evaluation" both read as
NULL, so the data genuinely cannot tell them apart. Stating the date is honest; deriving it from the rows would be a guess dressed as a computation./fairness (#9728)
The rate, the weekly table, and the definition, readable without opening source — that is #9728's acceptance. Reduced-basis weeks are labelled inline and explained underneath. Same zero-state conventions as the sibling series: an empty window is a measured zero with its dates, never a blank. Added to the verifier walkthrough's cross-checks.
The payload shape went into the contract's zod schema, so the UI type is derived — and adding it made both existing UI fixtures fail to compile, which is the #9282/#9521 guard doing its job again.
Validation
automation-rate.ts. Includes the Sunday off-by-one in week bucketing, the truncation of a fractional window, and the outer.catchthat protects an injected reader (safeAllswallows the real query's own failures, so that guard was the one genuinely unprotected path).tsc --noEmit,ui:build,ui:openapi:check,contract:api-schemas:check,ui-derived-types:check,dead-exports:check,maintainer-associations:check,docs:drift-check,git diff --checkall clean.One note: mid-work I hit
contract:api-schemas:checkfailing and a contract build error onMAX_PRIORITY_ELIGIBILITY_WINDOW_MINUTES. I confirmed it reproduced on clean main and left it alone rather than carrying a regenerated-but-broken artifact; #9936 has since landed and fixed it, and this branch is rebased on top.