feat(orb): persist the AI judgment's confidence and prompt commitment in the decision record - #8847
Merged
Merged
Conversation
… in the decision record When an AI-judgment finding (ai_consensus_defect / ai_review_split) shaped a decision, the decision record now carries the finding's calibrated confidence and a digest of the prompt template commitment (REVIEW_PROMPT_VERSION + REVIEW_SYSTEM_PROMPT) -- the fields that join every decision to the risk-control calibration set (#8835) and complete #8842's tracked follow-up. Advances #8834 (the persistence half its issue text names as the core deliverable; the flag-gated rotated-exemplar N-run variant remains -- the existing dual-model consensus already IS a two-sample agreement signal, so the extra runs are a cost knob, not the prerequisite). - DecisionRecord schema v2: + aiConfidence (null when no AI judgment contributed); normalized once at the builder like its siblings - processors: source the confidence from the gate's own AI-judgment blocker; modelId deliberately stays null at this site -- the finding does not carry which concrete models ran, and a guess would be worse than nothing (reviewDiagnostics holds per-run identities) - render: the model line gains the confidence when present
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
loopover-orb
Bot
added
the
gittensor:bug
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8847 +/- ##
==========================================
- Coverage 93.84% 92.30% -1.54%
==========================================
Files 803 803
Lines 80065 80069 +4
Branches 24267 24270 +3
==========================================
- Hits 75138 73909 -1229
- Misses 3562 5096 +1534
+ Partials 1365 1064 -301
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When an AI-judgment finding (
ai_consensus_defect/ai_review_split) shaped a decision, the decision record now carries the finding's calibrated confidence and the prompt-template commitment digest (REVIEW_PROMPT_VERSION+REVIEW_SYSTEM_PROMPT). Advances #8834 and completes #8842's tracked follow-up.Why
#8835's risk-control thresholds calibrate over (confidence, adjudicated-correctness) pairs — so every decision must persist its confidence at decision time, joined to the same record the audit labels reference. The prompt digest makes 'which template judged this' a content-addressed fact, which the golden corpus's future AI tier and the replay harness both key on.
Scope note
The rotated-exemplar N-run consistency variant from #8834's text stays open on the issue as a flag-gated cost knob: the dual-model consensus already is a two-sample agreement signal (min-confidence + split→hold), so the extra sampling improves the signal but is not the prerequisite for calibration — persistence is, and this delivers it.
modelIddeliberately stays null at this site: the finding doesn't carry which concrete models ran, and recording a guess would be worse than nothing (reviewDiagnosticsholds per-run identities).Verification
TSC clean; 537 tests green (decision-record, queue, queue-2, ai-review) including: normalization arms (undefined→null, explicit 0 honored), render with/without confidence, and an end-to-end queue assertion that the consensus-defect one-shot close persists
schemaVersion: 2,aiConfidence: 0.3, and a prompt digest. Changed-line coverage: 0 uncovered statements/branches.