From f4a3d4d608b61e83ea23d19e9459a788dc9c922a Mon Sep 17 00:00:00 2001 From: JSONbored <49853598+JSONbored@users.noreply.github.com> Date: Thu, 30 Jul 2026 23:03:54 -0700 Subject: [PATCH] docs(fairness): write down what the automation rate actually counts #9726's second sub-issue asked to publish the series WITH a methodology note. The series shipped and the note did not, so the metric the 97% target is stated against was the one figure on /fairness whose definition existed only in a source comment. Its definition contains three choices a reader cannot guess, and would plausibly assume the other way round: Non-deciding verdicts are excluded from BOTH halves rather than counted as automated. `action` also carries label, update_branch, approve and the error classes; counting those would let work the gate never ruled on read as an automated decision. A PR that was held and later merged is MANUAL. The question is whether a human had to act, not how it ended -- scoring the final disposition would let the rate be inflated by holding everything and then merging it by hand. Three human signals count, not just `hold`: a named reevaluation_actor, and a reevaluation_reason of maintainer_request, each make a PR manual. The backfill horizon gets its own section, because the page already surfaces a per-week `basis` field that nothing explained. The provenance fields arrived in migration 0204 on 2026-07-29; earlier weeks can only observe `hold`, so they can only under-count manual work and therefore only OVER-state automation. Those weeks are published as `holds_only` rather than mixed in, and the horizon is a stated constant rather than inferred -- "the column did not exist yet" and "this was a first evaluation" both read as NULL, so guessing it from the rows would be a fabrication dressed as a derivation. Every number and date here was read out of automation-rate.ts and the migrations rather than recalled, including which migration added the fields. /fairness's automation section now links to the definition, as its sibling sections already do, and the code pointer joins the list at the foot of the methodology page. Closes #10065 --- .../content/docs/fairness-methodology.mdx | 51 +++++++++++++++++++ .../components/site/fairness-report-page.tsx | 9 ++++ 2 files changed, 60 insertions(+) diff --git a/apps/loopover-ui/content/docs/fairness-methodology.mdx b/apps/loopover-ui/content/docs/fairness-methodology.mdx index ecc6ec2db8..9cf55ec9f9 100644 --- a/apps/loopover-ui/content/docs/fairness-methodology.mdx +++ b/apps/loopover-ui/content/docs/fairness-methodology.mdx @@ -188,6 +188,56 @@ because a methodology page that omitted them would be marketing. recorded gets its own row rather than being bucketed into maintainer or contributor — bucketing it would bias exactly the comparison the block exists to make. +## Automation rate: what counts as automated + +The share of pull requests decided with no human in the path. It is the metric every claim about the +gate ultimately rests on, so its definition is stated here rather than left to the label on the chart. + +Computed from `decision_records` — the anchored ledger — **and nothing else**, so anyone holding the +export recomputes the same numbers. A pull request is counted once per week, by the week of its +**first** verdict. + +| Bucket | Definition | +| --- | --- | +| **decided** | The PR reached a disposition at all: at least one verdict whose action *enacts* one (`merge` / `close`), or one that `hold`s. | +| **manual** | A decided PR where **any** verdict shows a human in the decision path. | +| **automated** | A decided PR that reached `merge` or `close` with none of those signals. | + +Three things about that definition are deliberate, and each one closes a way the number could be +made to look better than it is. + +**Non-deciding verdicts are excluded from both halves, not counted as automated.** `action` also +carries `label`, `update_branch`, `approve` and the error/no-op classes. A PR that only ever drew +those was never *decided*, so counting it would let work the gate never ruled on read as an +automated decision. + +**A PR that was held and later merged is manual.** The question is whether a human had to act, not +what the end state was. Scoring the final disposition instead would let the rate be inflated by +holding everything and then merging it by hand. + +**Three distinct human signals count, not just one.** A `hold` is the gate declining to decide and +handing the PR to a person. A named `reevaluation_actor` is a person who caused a re-evaluation. A +`reevaluation_reason` of `maintainer_request` is a human asking for the re-run. Counting only holds +would miss the last two entirely. + +A bot action alongside a decision does **not** make it manual — a `label` or an `update_branch` is +the bot acting, not a person. + +### The backfill horizon + +The two re-evaluation fields arrived with migration 0204, on **2026-07-29**. Weeks starting before +that date can only observe `hold`, so they can only **under-count** manual work — which means they +can only over-state automation. + +Those weeks are published with `basis: "holds_only"` rather than being quietly mixed in with +complete ones. A series whose definition changes partway along, without saying where, is worse than +one that is explicit about it: the earlier points are not wrong, they are measuring something +narrower, and a reader comparing this quarter to last needs to know which. + +The horizon is a stated constant rather than something inferred from the rows, because it cannot be +inferred: "the column did not exist yet" and "this was a first evaluation" both read as `NULL`. +Guessing the date from the data would be a fabrication dressed as a derivation. + ## Where this lives in the code Kept adjacent to the scoring code it describes, so a change to a definition and a change to this page @@ -199,3 +249,4 @@ show up in the same review: - Wilson intervals and the fleet fold — `src/orb/analytics.ts` - Review-parity rollups — `src/review/review-parity-rollups.ts` - The published corpus and its checksum — `src/review/public-eval-corpus.ts` +- The automation rate, its buckets and the provenance horizon — `src/review/automation-rate.ts` diff --git a/apps/loopover-ui/src/components/site/fairness-report-page.tsx b/apps/loopover-ui/src/components/site/fairness-report-page.tsx index 193232936a..b2170ff65e 100644 --- a/apps/loopover-ui/src/components/site/fairness-report-page.tsx +++ b/apps/loopover-ui/src/components/site/fairness-report-page.tsx @@ -462,6 +462,15 @@ export function FairnessReportPage() { > verify this review + . For the full definition — which verdicts count, why a held-then-merged PR is + manual, and what the pre-2026-07-29 weeks can and cannot measure — see the{" "} + + fairness methodology + .