feat(orb): golden gate corpus — production failure archetypes pinned as replayable verdicts

[JSONbored]

What

A versioned golden corpus of decision-input snapshots, each drawn from a real production failure archetype and replayed through the pure evaluateGateCheck with its conclusion and blocker set pinned. Closes #8832 (epic #8828, Phase 2 — the regression gate).

Why

The meta-point from the epic: ORB's decision policy shipped without an evaluation gate, in a repo that enforces 99% patch coverage on everything else. Every accuracy incident this week (inconclusive-AI closes, mode-flip hardening, verdict clobbering) would have been caught by exactly this suite had it existed. Now: a production failure becomes a labeled corpus entry in the same PR that fixes it — the online-failure → golden-set loop the research identified as converged practice.

What's pinned (15 entries)

• The inconclusive-AI incident class (fix(selfhost): force JSON mode on OpenAI-compatible review calls and stop retrying deterministic-identical outputs #8794–fix(ai-review): scope the incoherent-diff bail to mechanical breakage and reclassify evidence-bearing bails #8796 / PR 8735 et al.): no usable verdict → HOLD, never auto-pass — and inconclusive can never bury a real deterministic blocker.
• knownBad invariants: a secret_leak (alone or alongside an inconclusive AI) may never evaluate to success — under its own policy or the permissive default. This double-check means a future "maintainer relaxes every optional mode" config can't resurrect the failure either.
• Mode-flip symmetry (same finding: block fails / advisory passes), the merge-readiness composite escalation, dry-run preview-without-enforcement (displayConclusion), slop floor both sides, guardrail holds, and the not-synced-is-neutral contract.

Two of my own draft expectations were corrected against reality during construction (guardrail holds are policy-driven, not finding-driven; the slop blocker code is slop_risk_above_threshold) — which is precisely the drift-detection this corpus exists to perform, working on its author first.

Deliberate scope notes

• No dedicated CI job: the suite runs in validate-tests on every PR — the corpus is pure and millisecond-cheap, so it is the fast tier the issue requires; a path-filtered job would add machinery without protection.
• The full AI-inclusive tier (replay on model-version/prompt changes) requires captured model inputs, which land with gate: per-decision confidence signal via rotated-exemplar self-consistency #8834/trust: deterministic replay harness with the model call as the sole recorded nondeterministic input #8838; noted on the issue as the tracked follow-up.

Verification

TSC clean; 17 tests green (schema well-formedness, per-entry replay, the knownBad invariant sweep); branding drift clean. Pure test-side change — no src/** modified, no migration.

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
21383	2	21381	21

View the top 2 failed test(s) by shortest run time

test/unit/config-templates.test.ts > config/examples review templates (#1682) > loopover.full.yml body matches .loopover.yml.example from WHERE IT LIVES onward

Stack Traces | 0.0281s run time

AssertionError: expected '# WHERE IT LIVES (first match wins):\…' to be '# WHERE IT LIVES (first match wins):\…' // Object.is equality

- Expected
+ Received

@@ -91,13 +91,10 @@
  #   off      — the dimension is not evaluated.
  #   advisory — the finding is surfaced (comment/context) but never blocks.
  #   block    — the finding can become a hard `LoopOver Gate` blocker
  #              (always confirmed-contributor-gated).
  gate:
-   # Percent (0-20) of would-auto-close PRs randomly HELD for human adjudication instead of closing --
-   # the randomized holdout behind the unbiased close-precision estimate (#8831). 0/absent disables.
-   # closeAuditHoldoutPct: 5
    # Legacy check-run publish switch (#5355) — despite the name, this does NOT turn the deterministic
    # gate itself on or off. Gate evaluation, comments, labels, audit records, spend, and autonomous
    # merge/close all run identically whether this is true, false, or unset; the per-dimension modes
    # below are what actually configure gate evaluation. `enabled` is only a boolean shorthand for
    # `checkMode` below: true maps to `required`, false maps to `disabled`, and it cannot express

 ❯ test/unit/config-templates.test.ts:39:57

test/unit/selfhost-metrics.test.ts > DEFAULT_METRIC_META completeness (drift guard, 2026-07 fix) > every literal metric name emitted anywhere in src/ has a registered DEFAULT_METRIC_META entry

Stack Traces | 0.181s run time

AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "loopover_close_audit_holdouts_total",
+ ]

 ❯ test/unit/selfhost-metrics.test.ts:385:21

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}