docs(fairness): write down what the automation rate actually counts - #10066
Merged
Conversation
#9726's second sub-issue asked to publish the series WITH a methodology note. The series shipped and the note did not, so the metric the 97% target is stated against was the one figure on /fairness whose definition existed only in a source comment. Its definition contains three choices a reader cannot guess, and would plausibly assume the other way round: Non-deciding verdicts are excluded from BOTH halves rather than counted as automated. `action` also carries label, update_branch, approve and the error classes; counting those would let work the gate never ruled on read as an automated decision. A PR that was held and later merged is MANUAL. The question is whether a human had to act, not how it ended -- scoring the final disposition would let the rate be inflated by holding everything and then merging it by hand. Three human signals count, not just `hold`: a named reevaluation_actor, and a reevaluation_reason of maintainer_request, each make a PR manual. The backfill horizon gets its own section, because the page already surfaces a per-week `basis` field that nothing explained. The provenance fields arrived in migration 0204 on 2026-07-29; earlier weeks can only observe `hold`, so they can only under-count manual work and therefore only OVER-state automation. Those weeks are published as `holds_only` rather than mixed in, and the horizon is a stated constant rather than inferred -- "the column did not exist yet" and "this was a first evaluation" both read as NULL, so guessing it from the rows would be a fabrication dressed as a derivation. Every number and date here was read out of automation-rate.ts and the migrations rather than recalled, including which migration added the fields. /fairness's automation section now links to the definition, as its sibling sections already do, and the code pointer joins the list at the foot of the methodology page. Closes #10065
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
loopover-ui | f4a3d4d | Jul 31 2026, 06:04 AM |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
|
Caution 🛑 LoopOver review result - fixes requiredReview updated: 2026-07-31 06:12:34 UTC
Review summary Nits — 6 non-blocking
CI checks failing
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Bundle ReportChanges will increase total bundle size by 9.69kB (0.12%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#9726's second sub-issue asked to publish the automation-rate series with a methodology note. The series shipped; the note did not — leaving the metric the 97% target is stated against as the one figure on
/fairnesswhose definition lived only in a source comment.Three choices a reader would guess wrong
The definition is not obvious, and each of these closes a way the number could be made to look better than it is:
actionalso carrieslabel,update_branch,approveand the error classes. Counting them would let work the gate never ruled on read as an automated decision.hold— a namedreevaluation_actorand areevaluation_reasonofmaintainer_requesteach make a PR manual. Counting only holds would miss both.The backfill horizon gets its own section
The page already renders a per-week
basisfield that nothing explained. The re-evaluation provenance fields arrived in migration0204on 2026-07-29; earlier weeks can only observehold, so they can only under-count manual work — which means they can only over-state automation. Those weeks publish asholds_onlyrather than being quietly mixed in.The horizon is a stated constant rather than inferred, and the note says why: "the column did not exist yet" and "this was a first evaluation" both read as
NULL, so guessing it from the rows would be a fabrication dressed as a derivation.Verified, not recalled
Every number and date came out of the source, not memory:
src/review/automation-rate.tsAUTOMATION_RATE_PROVENANCE_HORIZON_ISO = "2026-07-29T00:00:00.000Z"— read from the constant0204— confirmed withgrep -l reevaluation_actor migrations/*.sqlAlso
/fairness's automation section now links to the definition, matching the pattern its sibling sections already use, andautomation-rate.tsjoins the code-pointer list at the foot of the methodology page — which exists so a definition change and a doc change land in the same review.Scope
Documentation and one link. No behaviour, no computation, no published figure changes.
This does not close the #9726 epic: its third sub-issue is #9729, which is blocked on #9991 (the ledger records
reason_code: "success"for 518 holds, so the manual-review surface cannot yet be split by cause).Verification
ui:typecheck,ui:lint,docs:drift-checkand the fullui:testsuite green.Closes #10065