feat(orb): randomized close-audit holdout — the selective-labels fix for close precision

[JSONbored]

What

A randomized ε-holdout on would-auto-close PRs — the only mechanism that can produce an unbiased close-precision estimate. Closes #8831 (epic #8828, Phase 2). Stacked on #8840 (uses its decision_audit_labels table); mergeable once that lands.

Why

The selective-labels problem: when ORB closes a PR, we never observe whether merging would have been fine. Close-precision today is estimated only on the closes humans happened to contest — a biased sample. And because ORB is deterministic, every off-policy estimator (IPS, doubly-robust) is undefined: logged propensities are 0/1, there is no overlap. Randomization is not one option among several; it is the only fix (Lakkaraju et al., KDD 2017). This was called out as the single most important negative result in the epic's research basis — do not build OPE infrastructure without this.

How

gate.closeAuditHoldoutPct (manifest-only, 0–20%, fractional allowed, default 0 = byte-identical): that fraction of would-auto-close PRs is HELD for human adjudication instead of closing, with ε and the draw persisted — the propensity record.

Invariants, each pinned by a test:

• The draw consumes the FINAL post-breaker plan. It runs after every gate/breaker decision is settled, so it can divert the decided close to a human but can never influence the decision. (maybeApplyCloseAuditHoldout sits immediately after applyPrecisionBreakers in the processor; all downstream consumers — disposition labels, the recorded verdict, execution — read the post-holdout plan, so a diverted PR honestly records hold.)
• Heuristic closes only. Policy closes are enforcement, not quality predictions (fix(orb): score enforcement closes as their own class, not quality mispredictions #8827) — holding one would suspend enforcement for zero measurement gain. Staged (auto_with_approval) closes already get a human.
• Hard ordering rule: no unlogged holds. The plan is diverted only after the propensity record writes. An unlogged hold is invisible to every estimator and silently biases coverage — so on a write failure the close proceeds and the instrument degrades, never the gate.
• Held PRs land in decision_audit_labels as stratum holdout_close with a NULL outcome (the PR is still open; the human adjudication is its ground truth) and surface the manual-review label via the exact conversion downgradeCloseToHold uses.

Verification

• TSC clean; 1,352 tests green across holdout/manifest/queue/queue-2/lifecycle/merge-facts, plus 798 manifest tests including the new knob's parse/fold/round-trip and out-of-range rejection.
• Changed-line coverage: 0 uncovered statements/branches (including the real-RNG production arm and the ε-boundary draw ≥ ε case).
• Branding-drift and schema-drift checks green.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[github-actions]

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (881b53a) and head (9c92c59) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	loopover-ui	9c92c59	Commit Preview URL Branch Preview URL	Jul 26 2026, 11:51 AM

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
21798	2	21796	21

View the top 2 failed test(s) by shortest run time

test/unit/config-templates.test.ts > config/examples review templates (#1682) > loopover.full.yml body matches .loopover.yml.example from WHERE IT LIVES onward

Stack Traces | 0.0649s run time

AssertionError: expected '# WHERE IT LIVES (first match wins):\…' to be '# WHERE IT LIVES (first match wins):\…' // Object.is equality

- Expected
+ Received

@@ -91,13 +91,10 @@
  #   off      — the dimension is not evaluated.
  #   advisory — the finding is surfaced (comment/context) but never blocks.
  #   block    — the finding can become a hard `LoopOver Gate` blocker
  #              (always confirmed-contributor-gated).
  gate:
-   # Percent (0-20) of would-auto-close PRs randomly HELD for human adjudication instead of closing --
-   # the randomized holdout behind the unbiased close-precision estimate (#8831). 0/absent disables.
-   # closeAuditHoldoutPct: 5
    # Legacy check-run publish switch (#5355) — despite the name, this does NOT turn the deterministic
    # gate itself on or off. Gate evaluation, comments, labels, audit records, spend, and autonomous
    # merge/close all run identically whether this is true, false, or unset; the per-dimension modes
    # below are what actually configure gate evaluation. `enabled` is only a boolean shorthand for
    # `checkMode` below: true maps to `required`, false maps to `disabled`, and it cannot express

 ❯ test/unit/config-templates.test.ts:39:57

test/unit/selfhost-metrics.test.ts > DEFAULT_METRIC_META completeness (drift guard, 2026-07 fix) > every literal metric name emitted anywhere in src/ has a registered DEFAULT_METRIC_META entry

Stack Traces | 0.178s run time

AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "loopover_close_audit_holdouts_total",
+ ]

 ❯ test/unit/selfhost-metrics.test.ts:385:21

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[loopover-orb]

Caution

🛑 LoopOver review result - fixes required

_{Review updated: 2026-07-26 11:42:25 UTC}

9 files · 1 AI reviewer · no blockers · CI failing · unstable

🛑 Suggested Action - Manual Review

Review summary
This implements a randomized ε-holdout on would-auto-close heuristic PRs, correctly placed after `applyPrecisionBreakers` so the draw only diverts an already-decided close, never influences it — the ordering contract is well-traced through the diff (breakerOnPlan → holdoutOnPlan flows into disposition, native gate decision, hold-audit detail, and the merge-eligibility checks consistently). The hard ordering rule (propensity record must write before the plan is diverted, else the close proceeds unheld) is a sound design against silently biased coverage, and it's directly pinned by a test that mocks a DB write failure. The manifest wiring (schema field, parser with 0-20 bound, JSON round-trip, effective-settings fold) is complete and field-exhaustiveness tests were updated accordingly.

Nits — 6 non-blocking

• src/review/close-audit-holdout.ts:98 hardcodes the id truncation length 190 and repoFullName truncation 200 as magic numbers with no named constant or comment on why those specific bounds were chosen.
• src/review/close-audit-holdout.ts:15 and test/unit/close-audit-holdout.test.ts:35 use `blacklist` as a closeKind value; consider `denylist` for consistency with more neutral terminology used elsewhere.
• src/review/close-audit-holdout.ts:101 logs via console.warn on the record-write failure path — worth confirming this matches the logging convention used by other best-effort write failures elsewhere in the codebase (e.g. structured logger vs raw console).
• Consider extracting the 0-20 bound in focus-manifest.ts's normalizeOptionalHoldoutPct into a named constant (e.g. MAX_CLOSE_AUDIT_HOLDOUT_PCT) so the epsilon ceiling reads as an intentional policy choice rather than an arbitrary literal.
• The test suite for close-audit-holdout.ts is thorough (eligibility, pure transform, full step, ordering-rule failure, real-RNG arm) — no changes needed there.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

CI checks failing

• validate
• validate-code
• validate-tests

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#8831
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 13 registered-repo PR(s), 13 merged, 221 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 13 PR(s), 221 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: significant

Linked issue satisfaction

Addressed
The PR adds the gate.closeAuditHoldoutPct manifest knob with per-repo override and 0-disables semantics, implements a post-breaker RNG draw that logs the propensity (draw, epsilon, counterfactual action) into audit_events and decision_audit_labels tagged 'holdout_close', and surfaces holds via the existing manual-review label queue, matching nearly all stated requirements including deterministic i

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
• Official Gittensor activity: 13 PR(s), 221 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review