feat(orb): weekly stratified decision-audit sampling with a frozen adjudication rubric

[JSONbored]

What

The human-label pipeline: a weekly stratified sample of the gate's decisions, adjudicated by a human against a frozen rubric, stored as calibration data. Closes #8830 (epic #8828, Phase 2 — labels).

Why

Every accuracy figure we publish is currently confirmed by "no human reversed it" — a lower bound on error, not a label. Humans reverse only what they notice; a silently-wrong merge scores as correct forever. The audit sample is the independent measurement: its divergence from the reversal-based number is the label bias, quantified. It also becomes the calibration set the risk-control thresholds (#8835) train on — without it, "≥99.5% guaranteed" has nothing to stand on.

How

• decision_audit_labels (migration 0178): one label per PR ever (UNIQUE(target_id) — a re-draw can never double-label), adjudications carry the rubric version they were made under.
• Pure stratified sampler (planAuditSample): merges oversampled at 50% (the costly arm), 20% first-time-author stratum (where the gate's priors are weakest), 30% closes; deterministic under an injected RNG; thin strata spill into the rest rather than shrinking the week's sample. Policy closes (fix(orb): score enforcement closes as their own class, not quality mispredictions #8827) are excluded at the candidate query — enforcement is not a quality decision to audit.
• Weekly cron (Tuesday 08:00 UTC, its own slot): flag-gated by LOOPOVER_DECISION_AUDIT, default OFF — the job is never enqueued and the tick is byte-identical to today. Self-host only. The dispatch re-checks the flag so a stale queued job after a flip does zero work.
• Adjudication routes (/v1/internal/audit-labels, bearer-gated): list (filterable by status) + adjudicate. A second adjudication 409s — labels are calibration data; corrections require a rubric-version bump, never a silent rewrite.
• docs/decision-audit-rubric.md v1: judge the decision at decision-time knowledge, not the outcome; ignore reversal state (that bias is what the audit measures); uncertain is a first-class label excluded from both numerator and denominator and tracked as its own rate.
• First-time detection reads pull_requests.author_login (the audit ledger carries no logins by design); a PR with no cached row degrades to not-first-time rather than guessing.

Verification

• TSC clean; 14 sampler/IO tests + 3 route tests + the enqueue-window test (Tuesday-08:00 × flag × runtime matrix) + 196 queue/migration tests, all green.
• Sampler pinned: determinism under a seed, exact strata quotas, spill behavior (including the close-tagged-spill arm), no-duplicate guarantee, pool-smaller-than-k, concurrent double-run inserting exactly once, failing insert logged and skipped.
• Changed-line coverage: 0 uncovered statements/branches.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.84%. Comparing base (cd2258d) to head (881b53a).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8840   +/-   ##
=======================================
  Coverage   93.83%   93.84%           
=======================================
  Files         800      801    +1     
  Lines       79879    79953   +74     
  Branches    24209    24227   +18     
=======================================
+ Hits        74956    75030   +74     
  Misses       3558     3558           
  Partials     1365     1365           

Flag	Coverage Δ
backend	95.12% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/api/routes.ts	95.35% <100.00%> (+0.03%)	⬆️
src/index.ts	96.26% <100.00%> (+0.07%)	⬆️
src/queue/job-dispatch.ts	99.38% <100.00%> (+0.01%)	⬆️
src/review/decision-audit.ts	100.00% <100.00%> (ø)
src/types.ts	100.00% <ø> (ø)

[loopover-orb]

Warning

⏸️ LoopOver review result - manual review recommended

_{Review updated: 2026-07-26 11:45:04 UTC}

12 files · 1 AI reviewer · 1 blocker · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This adds the full human-label pipeline for #8830: a new decision_audit_labels table with a matching D1-safe migration, a pure/deterministic stratified sampler, a flag-gated weekly cron job, and bearer-gated list/adjudicate routes with proper atomic UPDATE-based 409 handling for double-adjudication. The migration, schema-drift allowlist entry, flag wiring (dispatch re-check + cron enqueue), and test coverage are all solid and internally consistent. The one real defect is in first-time-author detection: it counts ALL cached pull_requests rows for an author rather than only their previously-decided ones, which can silently starve the first_time_author stratum of its intended targets, and the added tests happen to never expose this because they always insert one pull_requests row per decision in lockstep.

Blockers

• src/review/decision-audit.ts (runDecisionAuditSample, the authorRows/decidedCountByAuthor block): decidedCountByAuthor is built from an unfiltered `SELECT ... FROM pull_requests WHERE author_login IS NOT NULL` query, so it counts every cached PR (including open/undecided ones) for an author, not just previously-decided PRs as the code comment claims — a contributor with several open-but-undecided PRs and exactly one decided PR is wrongly marked NOT first-time, undermining the stratum whose whole purpose is targeting weak-prior newcomers; test/unit/decision-audit.test.ts's seedDecision helper always inserts exactly one pull_requests row per decision, so this drift between stated intent and actual count is never exercised.

Nits — 6 non-blocking

• src/api/routes.ts POST /v1/internal/audit-labels/adjudicate accepts any `reasonCategory` string up to 100 chars without validating it against the rubric's defined categories (missed_defect/false_block/stale_signal/scope_misread/policy_misapplied/other) in docs/decision-audit-rubric.md, letting an operator record a category the rubric doesn't recognize.
• The `audit:${row.targetId}`.slice(0,190) / project.slice(0,200) / errorMessage(error).slice(0,160) truncation lengths in src/review/decision-audit.ts are unexplained magic numbers — consider named constants alongside DECISION_AUDIT_SAMPLE_SIZE.
• planAuditSample's nested take() closures (src/review/decision-audit.ts) push control-flow depth to 5; a small helper extraction would make the spill logic easier to follow.
• Fix decidedCountByAuthor to count only PRs that actually have a prior gate_decision (e.g. join against review_audit / an existing decided-PR set) rather than every row in the pull_requests cache, so first_time_author reflects 'no earlier decided PR' as documented.
• Validate reasonCategory against the rubric's fixed category list in the adjudicate route, returning 400 on an unrecognized value.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Concerns raised — review before merging

• src/review/decision-audit.ts (runDecisionAuditSample, the authorRows/decidedCountByAuthor block): decidedCountByAuthor is built from an unfiltered `SELECT ... FROM pull_requests WHERE author_login IS NOT NULL` query, so it counts every cached PR (including open/undecided ones) for an author, not just previously-decided PRs as the code comment claims — a contributor with several open-but-undecided PRs and exactly one decided PR is wrongly marked NOT first-time, undermining the stratum whose whole purpose is targeting weak-prior newcomers; test/unit/decision-audit.test.ts's seedDecision helper always inserts exactly one pull_requests row per decision, so this drift between stated intent and actual count is never exercised.

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. src/review/decision-audit.ts \(runDecisionAuditSample, the authorRows/decidedCountByAuthor block\): decidedCountByAuthor is built from an unfiltered \`SELECT ... FROM pull\_requests WHERE author\_login IS NOT NULL\` query, so it counts every cached PR \(including open/undecided ones\) for an author, not just previously-decided PRs as the code comment claims — a contributor with several open-but-undecided PRs and exactly one decided PR is wrongly marked NOT first-time, undermining the stratum whose whole purpose is targeting weak-prior newcomers; test/unit/decision-audit.test.ts's seedDecision helper always inserts exactly one pull\_requests row per decision, so this drift between stated intent and actual count is never exercised.

Decision drivers

• ❌ Code review — 1 blocker (1 reviewer)
• ❌ Gate result — Blocking (Repo-configured hard blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#8830
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 13 registered-repo PR(s), 13 merged, 221 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 13 PR(s), 221 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Addressed
The PR implements a deterministic, property-tested stratified sampler with merge/close/first-time-author quotas and spill logic, a decision_audit_labels migration (0178) with UNIQUE(target_id), a flag-gated weekly Tuesday cron, bearer-gated list/adjudicate CLI-equivalent routes with 409-on-rewrite semantics, and a frozen versioned rubric doc requiring a version bump on edits — all matching the iss

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
• Official Gittensor activity: 13 PR(s), 221 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.