feat(orb): gate determinism — deterministic test-evidence facts, decision confidence, and replayable time (#8833, #8834, #9028) by JSONbored · Pull Request #9256 · JSONbored/loopover

JSONbored · 2026-07-27T10:22:31Z

Summary

Three related pieces of the Phase-3/4 determinism work: shrink what the LLM is allowed to decide (#8833), measure how reproducibly it decided the rest (#8834), and make the replay harness cover time (#9028).

Two of the three are deliberately scoped down — see Scoping decisions below. Only #9028's requirement 2 and #8833's next extraction step are claimed; both parent issues stay open with the remainder named explicitly.

#8833 — one more objectively-checkable criterion leaves LLM judgment

Whether a PR carries test-path evidence is a fact: signals/test-evidence.ts's isTestPath owns it, and slop.ts's missing_test_evidence finding already decides its severity. The reviewer prompt nonetheless fed the same fact as free text and let the model re-decide blocker-vs-nit — the exact double-judging the issue's own input inventory flagged ("still fed as free text, model re-decides blocker-vs-nit").

Extends the established parse-time fact-authority pattern (demoteCiClaimBlockers #8845, demoteEvidenceAbsenceBlockers #8961) to this criterion. A whole-PR "no tests were added" blocker is demoted to a nit when the classifier confirms the PR does change test paths.

Two properties worth calling out, both mirroring the existing demotions:

It fires only where the claim is provably a fact error. A genuine test-absence blocker on a genuinely test-free PR keeps its severity — the rule never weakens a true finding.
Coverage-depth claims are deliberately excluded. "no tests for the nullish branch" / "no test covers the error path" narrow to a specific target; the path classifier cannot check those, so they remain the model's judgment to make and keep blocking. Only existence claims are demoted.

#8834 — a per-decision confidence signal

The persistence half of this issue already shipped (DECISION_RECORD_SCHEMA_VERSION v2 credits #8834 for aiConfidence). What was missing is the agreement signal.

scoreJudgmentAgreement scores inter-run agreement across the reviewer stances the engine already produces (#8229's reviewerVotes) and folds it into the verbalized confidence. The combined score multiplies the two, so it is monotonically below either input — a judgment is only as trustworthy as both how sure the judge said it was and how reproducibly the judges reached it, which is the property an abstention threshold depends on.

A lone run is recorded as uncorroborated at a 0.5 agreement floor rather than fabricated unanimity, so a single-reviewer or budget-degraded review records a strictly lower confidence than a corroborated one. Zero samples refuses to invent a score at all. (Same "silence is not certainty" discipline fix(orb): enforce the model/fact boundary — CI claims demoted at parse, silence is not certainty #8845 established.)
Rides to DecisionRecord.aiAgreement (schema v5) so every decision joins the risk-control calibration set with its reproducibility attached.
Deliberately additive — it does not re-route the gate. "Disagreement → hold" already happens: differing stances are the ai_review_split finding, which blocks or holds via the existing confidence floor. A second parallel route would double-count the same evidence rather than measure it.

#9028 — wall-clock capture (requirement 2)

gate.requireFreshRebaseWindow compared the base branch's tip against an inline Date.now() read inside maybeForceFreshRebase. Time is a decision input, and nothing recorded which instant the comparison used — so re-deriving such a decision later could reach the opposite answer purely because the wall clock had moved, and report it as a match.

The decision pass now takes one Date.now() reading, records it into decision_replay_inputs.replay_json as clock, and passes it to every clock-dependent rule instead of each calling the clock itself.
Both staleness rules move to a pure, clock-injected module. isWithinFreshRebaseWindow takes the instant explicitly; isBaseStaleByAheadBy is stated as the commit-count comparison it is.
replayDecision gains a stage-0 clock check: replaying at the recorded instant (the CLI default) is bit-exact, while naming a different instant reports a clock divergence rather than silently certifying a re-derivation that never reproduced the original evaluation. Pre-Replay v2: re-query action-match mode + wall-clock capture for staleness rules #9028 records carry no instant, so the stage is skipped rather than guessed. Exposed as replay-decision.ts --at <epoch ms>.

Honest finding on the second named rule: staleBaseAheadByThreshold reads no clock at all — it is a commit-count comparison (aheadBy >= threshold), so wall-clock capture does not apply to it. Rather than pretend otherwise, it is stated as a pure function and the suite pins its instant-independence (same inputs, wildly different instants, identical answer). That is the correct analogue of "bit-exact at the recorded instant" for a rule that has no instant. Note this also means neither staleness rule is fully replayable yet for a second reason outside this issue's scope: both make live GitHub reads (fetchLiveBaseBranchAdvancedAt, fetchBaseAheadBy) that are not captured either.

Scoping decisions

Issue	Claimed	Deliberately deferred, and why
#8833	Advances	The issue's own comment splits the remainder into five items, "each its own PR". This lands the test-evidence half of item 2. Deferred: item 1 (structured verifiable blocker schema — needs a `REVIEW_PROMPT_VERSION` bump and is the largest single piece), the size-vs-norm and base-staleness halves of item 2, item 3 (scanner-confirmed CVE/secret/lockfile facts by rule — REES returns only free-text `promptSection` today, so this needs a structured findings channel across a separately-deployed service), item 4 (explicit `"unknown"` in the finding schema), item 5 (screenshot-vision laundering). The full rubric-shrink + per-dimension isolated judge calls the issue describes is a multi-PR refactor; a speculative sweep here would have been unbounded.
#8834	Advances	The paid rotated-exemplar sampling is NOT implemented. The issue asks for N=2–3 runs of the same judge with exemplars rotated out of the golden corpus. That is a strictly better signal than two different models voting once each — it isolates the judge's own reproducibility instead of confounding it with two models' differing priors — but every extra run is a real per-review AI charge on every reviewed PR, and this engine pays that bill in production. A default-on N-sample pass would multiply AI spend per review. What landed is the half that costs nothing: scoring the samples the dual review already runs, which the issue's own citation says captures most of the benefit ("two samples capture most of the benefit"). The extra-sampling half needs a budget decision and a flag defaulting OFF; it is not stubbed, so there is no unreachable code — see the closing note in `src/review/judgment-agreement.ts`.
#9028	Advances	Requirement 1 (`--requery`) is blocked by the issue's own text on persisting full prompt text alongside `promptDigest` — an unmade privacy/row-size decision (prompt text embeds the diff; it likely needs another private sibling table with retention bounds). The current CLI also documents no-requery as a structural invariant, so inverting it is a deliberate contract change that should not ride along with an unrelated fix. Requirement 2 is fully implemented.

Validation

Run from a clean worktree at origin/main (32c7d5a):

git diff --check — clean
npx tsc --noEmit -p tsconfig.json --incremental false — 0 errors
npm run db:migrations:check / db:schema-drift:check / cf-typegen / selfhost:env-reference / ui:openapi:check / ui:openapi:settings-parity / engine-parity:drift-check / docs:drift-check / manifest:drift-check / command-reference:check / coverage-boltons:check / import-specifiers:check / branding-drift:check — all pass, no generated-artifact drift (regenerating produced an empty diff)
Targeted suites, all green: judgment-agreement (7 new), decision-replay (19, 7 new), ai-review (247), ai-review-advisory (60, 1 new), decision-record (32), queue-2 (125), gate-golden-corpus (17), advisory-live-parity, risk-control, counterfactual-replay-core
Full test/unit sweep (1190 files / ~22.5k tests) diffed against a pristine origin/main worktree via the JSON reporter: zero regressions. The 16 remaining failures are byte-identical to the main baseline (config-templates, engine-parity-fixtures, linked-issue-label-propagation-fetch, loopover-engine-scaffold, predicted-gate-engine, queue-5, reputation-wiring ×4, salvageability, selfhost-metrics, selfhost-pg-retention ×2, worker-entry-boundary).

Not run (per maintainer instruction to keep this to targeted tests + typecheck): full npm run test:ci, test:coverage, test:workers, ui:*, npm audit. CI covers these.

No migration is added — replay_json and record_json are JSON columns, so clock and aiAgreement need no schema change. Next free migration number remains 0195.

Pre-existing failure, not from this PR

npm run replay-runner-manifest:check reports drift on package-lock.json. This branch touches neither package-lock.json nor scripts/replay-runner-image-manifest.json — both are byte-identical to origin/main (git diff origin/main is empty for both), so the check fails identically on main today. Deliberately not regenerated here: it is a maintainer-owned attestation manifest (#9214) and refreshing it is an unrelated change that should not ride along.

Safety

No secrets, wallets, hotkeys, trust scores, reward values, or private scoring anywhere.
No public-surface copy changes: the demotion appends to nits, which already flows through the existing sanitizer.
No auth/CORS/session/API surface touched; no UI change, so no UI evidence applies.
No CHANGELOG.md, site/, CNAME, or lovable changes. All edits are inside wantedPaths (src/, test/, scripts/).
staleness-clock.ts and judgment-agreement.ts are new host-only files, so the engine-parity twin discovery correctly does not require an engine counterpart (verified: engine-parity:drift-check passes).
evaluateGateCheck and its machine-enforced engine twin are not touched — no gate-decision parity surface is involved.

Advances #8833
Advances #8834
Advances #9028

github-actions · 2026-07-27T10:23:37Z

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (d04d0fe) and head (244ed8a) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: `linked_issue_scope_mismatch`

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

superagent-security · 2026-07-27T10:23:39Z

Superagent didn't find any vulnerabilities or security issues in this PR.

codecov · 2026-07-27T10:39:39Z

❌ 17 Tests Failed:

Tests completed	Failed	Passed	Skipped
23062	17	23045	21

View the top 3 failed test(s) by shortest run time

test/unit/selfhost-pg-retention.test.ts > runRetentionPrune + processJob on the Postgres backend (#977) > processJob prune-retention deletes eligible rows and records a success audit event on Postgres

Stack Traces | 0.00744s run time

AssertionError: expected 3 to be +0 // Object.is equality

- Expected
+ Received

- 0
+ 3

 ❯ test/unit/selfhost-pg-retention.test.ts:119:44

test/unit/loopover-engine-scaffold.test.ts > loopover-engine package scaffold > publishes only the build output and changelog, not source

Stack Traces | 0.0115s run time

AssertionError: expected [ 'dist/**/*.js', …(2) ] to deeply equal [ 'dist', 'CHANGELOG.md' ]

- Expected
+ Received

  [
-   "dist",
+   "dist/**/*.js",
+   "dist/**/*.d.ts",
    "CHANGELOG.md",
  ]

 ❯ test/unit/loopover-engine-scaffold.test.ts:30:29

test/unit/selfhost-pg-retention.test.ts > pruneExpiredRecords on the Postgres backend (#977) > deletes across multiple bounded batches and stops at the per-table cap, same as the SQLite path

Stack Traces | 0.0165s run time

AssertionError: expected +0 to be 4 // Object.is equality

- Expected
+ Received

- 4
+ 0

 ❯ test/unit/selfhost-pg-retention.test.ts:76:33

test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > never downgrades the locally-computed signal when the enabled bridge does not vouch

Stack Traces | 0.0185s run time

AssertionError: expected "vi.fn()" to be called once, but got 2 times
 ❯ test/unit/reputation-wiring.test.ts:441:25

test/unit/reputation-wiring.test.ts > getEffectiveSubmitterReputation (#4513, install-wide for a confirmed miner) > widens to the install-wide signal for a CONFIRMED miner when the per-repo signal alone stays neutral

Stack Traces | 0.0239s run time

AssertionError: expected 'neutral' to be 'low' // Object.is equality

Expected: "low"
Received: "neutral"

 ❯ test/unit/reputation-wiring.test.ts:482:26

test/unit/reputation-wiring.test.ts > getEffectiveSubmitterReputation (#4513, install-wide for a confirmed miner) > skips the miner-identity lookup entirely when the per-repo signal already justifies downgrading

Stack Traces | 0.025s run time

AssertionError: expected 'neutral' to be 'low' // Object.is equality

Expected: "low"
Received: "neutral"

 ❯ test/unit/reputation-wiring.test.ts:520:26

test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > keeps the pre-bridge output unchanged when any activation gate is off

Stack Traces | 0.0263s run time

AssertionError: expected "vi.fn()" to not be called at all, but actually been called 1 times

Received:

  1st vi.fn() call:

    Array [
      "https://api.gittensor.io/miners",
      Object {
        "headers": Object {
          "accept": "application/json",
          "user-agent": "loopover/0.1",
        },
        "signal": AbortSignal {
          Symbol(kEvents): Map {},
          Symbol(events.maxEventTargetListeners): 0,
          Symbol(events.maxEventTargetListenersWarned): false,
          Symbol(kHandlers): Map {},
          Symbol(kAborted): false,
          Symbol(kReason): undefined,
          Symbol(kComposite): false,
          Symbol(kTimeout): true,
        },
      },
    ]


Number of calls: 1

 ❯ test/unit/reputation-wiring.test.ts:419:31

test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > REGRESSION (#6801): feature on upgrades a low local signal and changes the end-to-end gate decision

Stack Traces | 0.0429s run time

AssertionError: expected "vi.fn()" to be called 2 times, but got 3 times
 ❯ test/unit/reputation-wiring.test.ts:393:25

test/unit/selfhost-metrics.test.ts > DEFAULT_METRIC_META completeness (drift guard, 2026-07 fix) > every literal metric name emitted anywhere in src/ has a registered DEFAULT_METRIC_META entry

Stack Traces | 0.169s run time

AssertionError: expected [ …(2) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "loopover_orb_relay_multiple_live_enrollments_total",
+   "loopover_private_manifest_warnings_total",
+ ]

 ❯ test/unit/selfhost-metrics.test.ts:498:21

test/unit/config-templates.test.ts > config/examples review templates (#1682) > lints loopover.full.yml with zero warnings, including no retired top-level fields

Stack Traces | 0.218s run time

AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "gate.mergeReadiness (\"off\") is set alongside an explicitly-authored mode for gate.linkedIssue, gate.duplicates, gate.slop.mode. The composite only fills in a sub-gate mode left unset -- it never overrides an explicitly-configured one, so those fields stay exactly as authored regardless of gate.mergeReadiness.",
+ ]

 ❯ test/unit/config-templates.test.ts:64:29

test/unit/queue-5.test.ts > queue processors > type label decoupling (#label-decoupling) > REGRESSION (#4528, PR #4494 shape): keeps the propagated labels on the PR's own merge-closed webhook, instead of falling back to the title guess

Stack Traces | 0.232s run time

AssertionError: expected [ 'gittensor:feature' ] to deeply equal [ 'gittensor:feature', …(1) ]

- Expected
+ Received

  [
    "gittensor:feature",
-   "gittensor:priority",
  ]

 ❯ test/unit/queue-5.test.ts:6660:34

test/unit/worker-entry-boundary.test.ts > worker entry boundary > does not reference pixelmatch, pngjs, visual-diff, gifenc, or sharp in worker-reachable source

Stack Traces | 0.579s run time

AssertionError: worker-reachable files must not mention Node-only visual diff/GIF/image deps: .../review/visual/visual-findings.ts: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   ".../review/visual/visual-findings.ts",
+ ]

 ❯ test/unit/worker-entry-boundary.test.ts:82:118

View the full list of 5 ❄️ flaky test(s)

test/contract/engine-parity.test.ts > predicted-gate engine parity (#2286) > 'merge-readiness-composite-preserves-e…' matches the committed golden output