Skip to content

feat(calibration): implicit-confirmation overrides for non-AI rules on an owner terminal close - #8785

Merged
JSONbored merged 2 commits into
mainfrom
feat/implicit-terminal-confirmations
Jul 26, 2026
Merged

feat(calibration): implicit-confirmation overrides for non-AI rules on an owner terminal close#8785
JSONbored merged 2 commits into
mainfrom
feat/implicit-terminal-confirmations

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

The confirmation-asymmetry fix from adversarial-audit epic #8757. Only the two AI-judgment codes ever received "confirmed" HumanOverrideEvents (#8123); every other rule accumulated reversals exclusively, structurally biasing computeRulePrecision toward 0 for them as data grew — silently corrupting the backtest_regression gate, the knob-loosening ladders, and the public precision surface regardless of the rules' real accuracy.

Changes

  • src/review/outcomes-wire.ts — new recordImplicitTerminalConfirmations, called from the same owner-close-without-merge branch calibration: record a 'confirmed' HumanOverrideEvent when an owner closes an aiReviewLowConfidenceHold PR #8123 already uses (the one human disposition consistent with an adverse finding's verdict). For every non-AI, non-score rule that fired against the target (30-day lookback), records a "confirmed" override tagged metadata.basis = "implicit_terminal_disposition" so downstream consumers can weight or exclude implicit evidence separately from calibration: record a 'confirmed' HumanOverrideEvent when an owner closes an aiReviewLowConfidenceHold PR #8123's explicit kind.
  • Idempotent by construction: any existing override (either verdict) for the (ruleId, target) pair suppresses the write — a pair stays single-verdict, so a reopen-recorded reversal is never followed by a contradicting implicit confirmation, and a second owner close records nothing new.
  • Excluded by design (documented at the function): the AI codes (their explicit writer is byte-identical), the score signals (slop_gate_score/quality_gate_score fire on PASS outcomes too, so close-confirmation is incoherent for them — their ground truth is threshold replays), and merge-despite-advisory-finding (weak evidence against a finding, a reversal-shaped signal deserving its own design).
  • src/rules/advisory.ts drift fixbacktest_regression (Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138) was added to the gate switch but never to CONFIGURED_GATE_BLOCKER_SIGNAL_CODES, so its reversals silently never recorded — exactly the drift the list's own doc comment promises against. Added, with the drift called out inline.

Test plan

  • npm run typecheck — clean
  • outcomes-wire.test.ts: 99/99, including 9 new tests — basis-tagged confirmation, linked_issue_scope_mismatch's first-ever confirmed side, both idempotence directions, score-code exclusion, AI-writer isolation (exactly one confirmation, no basis tag), the backtest_regression drift fix, contributor-close guard, and the store-construction-throw catch path
  • 222 tests green across the recorder/rules adjacent suites
  • Changed line ranges: 0 uncovered statements, 0 uncovered branches (istanbul JSON, exact diff hunks)

Independent of #8784 (different regions of advisory.ts; the new tests seed events directly) — the two merge in either order.

Closes #8761

…n an owner terminal close (#8761)

Only the two AI-judgment codes ever received "confirmed" HumanOverrideEvents
(#8123); every other rule accumulated reversals exclusively, structurally
biasing computeRulePrecision toward 0 for them as data grew — corrupting the
backtest_regression gate, the knob ladders, and the public precision surface
regardless of the rules' real accuracy.

- New recordImplicitTerminalConfirmations in outcomes-wire.ts: the repo OWNER
  closing a PR without merging (the one human disposition consistent with an
  adverse finding's verdict — the same signal #8123 keys on) records a
  "confirmed" override for every non-AI, non-score rule that fired against the
  target, tagged metadata.basis = "implicit_terminal_disposition" so consumers
  can weight or exclude implicit evidence.
- Idempotent by construction: any existing override (either verdict) for the
  (ruleId, target) pair suppresses the write — a pair stays single-verdict, so
  a reopen-recorded reversal is never contradicted.
- Excluded by design: AI codes (their explicit writer is untouched), the score
  signals (they fire on pass outcomes too, so close-confirmation is incoherent
  for them), and merge-despite-advisory (reversal-shaped evidence, own issue).
- Drift fix: backtest_regression (#8138) was in the gate switch but missing
  from CONFIGURED_GATE_BLOCKER_SIGNAL_CODES, so its reversals silently never
  recorded — added, with the drift called out at the list.
@JSONbored JSONbored self-assigned this Jul 26, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 26, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui e2f12d2 Commit Preview URL

Branch Preview URL
Jul 26 2026, 05:45 AM

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@github-actions

Copy link
Copy Markdown
Contributor

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (476b98d) and head (e2f12d2) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

@JSONbored
JSONbored merged commit f4c946b into main Jul 26, 2026
6 of 7 checks passed
@JSONbored
JSONbored deleted the feat/implicit-terminal-confirmations branch July 26, 2026 05:56
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 26, 2026
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
21204 3 21201 21
View the top 3 failed test(s) by shortest run time
test/unit/check-engine-parity-script.test.ts > check-engine-parity script > runEngineParityMain returns 0 for the real monorepo workspace
Stack Traces | 0.038s run time
AssertionError: expected 1 to be +0 // Object.is equality

- Expected
+ Received

- 0
+ 1

 ❯ test/unit/check-engine-parity-script.test.ts:798:48
test/unit/check-engine-parity-script.test.ts > check-engine-parity script > engine version skew > uses default version readers against the real monorepo workspace
Stack Traces | 0.0583s run time
AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "Gate-decision logic change in src/rules/advisory.ts requires either:
+   • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or
+   • a version bump in packages/loopover-engine/package.json (currently 3.14.1 vs base 3.14.1).",
+ ]

 ❯ test/unit/check-engine-parity-script.test.ts:785:31
test/unit/check-engine-parity-script.test.ts > check-engine-parity script > prints a clean summary and exits 0 for the real repo state when run as a subprocess
Stack Traces | 0.84s run time
Error: Command failed: .../loopover/node_modules/.bin/tsx scripts/check-engine-parity.ts
Engine-parity check found 1 issue(s):
Gate-decision logic change in src/rules/advisory.ts requires either:
  • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or
  • a version bump in packages/loopover-engine/package.json (currently 3.14.1 vs base 3.14.1).

 ❯ test/unit/check-engine-parity-script.test.ts:803:20

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { status: 1, signal: null, output: [ null, '', 'Engine-parity check found 1 issue(s):\nGate-decision logic change in src/rules/advisory.ts requires either:\n  • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or\n  • a version bump in packages/loopover-engine/package.json (currently 3.14.1 vs base 3.14.1).\n' ], pid: 7190, stdout: '', stderr: 'Engine-parity check found 1 issue(s):\nGate-decision logic change in src/rules/advisory.ts requires either:\n  • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or\n  • a version bump in packages/loopover-engine/package.json (currently 3.14.1 vs base 3.14.1).\n' }

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

calibration: implicit-confirmation override signals for non-AI-judgment rules (merge-without-reversal => confirmed)

1 participant