feat(calibration): record advisory-mode rule-fired signals too, tagged with the resolved gate mode - #8784
Merged
Merged
Conversation
…d with the resolved gate mode (#8760) recordConfiguredGateBlockerSignals only recorded findings whose gate RESOLVED to "block" — but every gate defaults to advisory/off, so the majority of real traffic (advisory findings a human actually saw) recorded nothing, and every downstream backtest corpus, reliability curve, and knob-loosening decision was built from the biased block-mode-only minority. - isConfiguredGateBlocker is now a =="block" projection of a new resolveConfiguredGateMode(finding, policy) -> "block"|"advisory"|"off" — one source of truth, so recorder and gate can never drift; every mode default is byte-identical to the old switch. - The recorder captures every non-"off" finding with metadata.gateMode so consumers can segment bot-actioned evidence from shown-to-a-human evidence. The advisory_only sub-floor AI demotion resolves "advisory" (a real finding a human saw), never "off". manifest_linked_issue_required composes the stricter of its two gates. - Consumer-segmentation audit (in the recorder's doc comment): every current consumer is decided-case-driven (corpus pairing drops unpaired fires; the repeat alarm uses its own composite-ruleId namespace), so advisory fires sit inert until #8761's implicit confirmations start pairing them.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 2aebb5e | Commit Preview URL Branch Preview URL |
Jul 26 2026, 05:36 AM |
❌ 3 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The corpus-starvation fix from adversarial-audit epic #8757.
recordConfiguredGateBlockerSignalsonly recorded findings whose gate resolved to"block"— but every gate defaults to advisory/off, so the majority of real traffic (advisory findings a human actually saw) recorded nothing, and every downstream backtest corpus, reliability curve, and knob-loosening decision was built from the biased block-mode-only minority sample.Changes
src/rules/advisory.ts—isConfiguredGateBlockeris now a=== "block"projection of a newresolveConfiguredGateMode(finding, policy) → "block" | "advisory" | "off": one source of truth, so the recorder and the gate can never drift. Every mode default is byte-identical to the old switch (verified by the 1,583 tests across the 20 gate/signal-adjacent suites). The recorder captures every non-"off"finding, taggedmetadata.gateMode, so consumers can segment bot-actioned evidence from shown-to-a-human evidence. Notable resolutions: theadvisory_onlysub-floor AI demotion resolves"advisory"(a real finding a human saw, worth corpus evidence — never"off");manifest_linked_issue_requiredcomposes the stricter of its two gates (block > advisory > off), preserving the original blocks-when-either-blocks semantics exactly.buildBacktestCorpusdrops unpaired fires;computeRulePrecision, the reliability curve, and the knob evaluators run over that decided corpus; the rule-repeat alarm reads its own separately-namespaced composite ruleId and never sees these events. Advisory fires therefore sit inert until calibration: implicit-confirmation override signals for non-AI-judgment rules (merge-without-reversal => confirmed) #8761's implicit confirmations start pairing them (which carry their ownbasistag for weighting). No existing metric shifts from this change alone..catch(() => undefined)).linked_issue_scope_mismatchstays excluded (calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 owns it);secret_leakraw-context exclusion unchanged (calibration: capture bounded raw context for the remaining isConfiguredGateBlocker codes, excluding secret_leak #8130) — it gains only thegateModetag.Test plan
npm run typecheck— cleanadvisory_onlydemotion tag, and the stricter-of-two-gates compositionevaluateGateCheck/ the signal surfaceCloses #8760