feat(calibration): implicit-confirmation overrides for non-AI rules on an owner terminal close - #8785
Merged
Merged
Conversation
…n an owner terminal close (#8761) Only the two AI-judgment codes ever received "confirmed" HumanOverrideEvents (#8123); every other rule accumulated reversals exclusively, structurally biasing computeRulePrecision toward 0 for them as data grew — corrupting the backtest_regression gate, the knob ladders, and the public precision surface regardless of the rules' real accuracy. - New recordImplicitTerminalConfirmations in outcomes-wire.ts: the repo OWNER closing a PR without merging (the one human disposition consistent with an adverse finding's verdict — the same signal #8123 keys on) records a "confirmed" override for every non-AI, non-score rule that fired against the target, tagged metadata.basis = "implicit_terminal_disposition" so consumers can weight or exclude implicit evidence. - Idempotent by construction: any existing override (either verdict) for the (ruleId, target) pair suppresses the write — a pair stays single-verdict, so a reopen-recorded reversal is never contradicted. - Excluded by design: AI codes (their explicit writer is untouched), the score signals (they fire on pass outcomes too, so close-confirmation is incoherent for them), and merge-despite-advisory (reversal-shaped evidence, own issue). - Drift fix: backtest_regression (#8138) was in the gate switch but missing from CONFIGURED_GATE_BLOCKER_SIGNAL_CODES, so its reversals silently never recorded — added, with the drift called out at the list.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | e2f12d2 | Commit Preview URL Branch Preview URL |
Jul 26 2026, 05:45 AM |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
❌ 3 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The confirmation-asymmetry fix from adversarial-audit epic #8757. Only the two AI-judgment codes ever received
"confirmed"HumanOverrideEvents (#8123); every other rule accumulated reversals exclusively, structurally biasingcomputeRulePrecisiontoward 0 for them as data grew — silently corrupting thebacktest_regressiongate, the knob-loosening ladders, and the public precision surface regardless of the rules' real accuracy.Changes
src/review/outcomes-wire.ts— newrecordImplicitTerminalConfirmations, called from the same owner-close-without-merge branch calibration: record a 'confirmed' HumanOverrideEvent when an owner closes an aiReviewLowConfidenceHold PR #8123 already uses (the one human disposition consistent with an adverse finding's verdict). For every non-AI, non-score rule that fired against the target (30-day lookback), records a"confirmed"override taggedmetadata.basis = "implicit_terminal_disposition"so downstream consumers can weight or exclude implicit evidence separately from calibration: record a 'confirmed' HumanOverrideEvent when an owner closes an aiReviewLowConfidenceHold PR #8123's explicit kind.slop_gate_score/quality_gate_scorefire on PASS outcomes too, so close-confirmation is incoherent for them — their ground truth is threshold replays), and merge-despite-advisory-finding (weak evidence against a finding, a reversal-shaped signal deserving its own design).src/rules/advisory.tsdrift fix —backtest_regression(Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138) was added to the gate switch but never toCONFIGURED_GATE_BLOCKER_SIGNAL_CODES, so its reversals silently never recorded — exactly the drift the list's own doc comment promises against. Added, with the drift called out inline.Test plan
npm run typecheck— cleanoutcomes-wire.test.ts: 99/99, including 9 new tests — basis-tagged confirmation, linked_issue_scope_mismatch's first-ever confirmed side, both idempotence directions, score-code exclusion, AI-writer isolation (exactly one confirmation, no basis tag), the backtest_regression drift fix, contributor-close guard, and the store-construction-throw catch pathIndependent of #8784 (different regions of
advisory.ts; the new tests seed events directly) — the two merge in either order.Closes #8761