feat(review): rotated-exemplar self-consistency sampling behind AI_REVIEW_SELF_CONSISTENCY_RUNS - #9591
Merged
Merged
Conversation
…VIEW_SELF_CONSISTENCY_RUNS The paid half of #8834. The free half (judgment-agreement.ts) already scores inter-run agreement over whatever stances a review produces; this adds the deliberate extra stances: flag-gated re-runs of the SAME judge with deterministically rotated few-shot exemplar windows ('simulated annotators', Trust or Escalate, ICLR 2025), folded into the recorded confidence through the existing scoreJudgmentAgreement axis -- never into split detection. Decisions the free half's closing comment deferred, now made: - Flag: AI_REVIEW_SELF_CONSISTENCY_RUNS = TOTAL evaluations per review. Unset/0/garbage = OFF (today's behavior byte-identical); values clamp into {2,3} -- one run measures nothing, benefit saturates by three. - Budget: extras ride the daily neuron budget for real -- each records a status-ok ai_usage_events row (the same sum the quota gate reads) with selfConsistency metadata. An underfunded plan DEGRADES to fewer runs and the score records as uncorroborated; nothing is ever fabricated. - Rotation: window = f(repo#pr, runIndex) via FNV-1a, so replay can reconstruct what each run saw and re-reviews sample the same rotation instead of measuring prompt-shuffle variance. Exemplars are synthetic, versioned (JUDGE_EXEMPLAR_SET_VERSION), balanced 3 defect / 3 clean. - The rotated window rides the SYSTEM turn itself: the systemAppend parameter is a self-host-CLI-only transport (selfHostCliSystemAppend drops it for every other provider), so appending there would have made rotation silently provider-dependent. - Extras are skipped when the primary pass produced no usable stance: corroborating a non-verdict is pure spend. A failed extra run records its spend but contributes no stance; a dissenting one lands measurably in decision_records' aiAgreement. Closes #8834
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-28 21:09:42 UTC
Review summary Nits — 2 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 920358a | Commit Preview URL Branch Preview URL |
Jul 28 2026, 08:51 PM |
Bundle ReportChanges will decrease total bundle size by 4.62kB (-0.06%) ⬇️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9591 +/- ##
==========================================
- Coverage 89.84% 89.03% -0.81%
==========================================
Files 879 880 +1
Lines 111064 111106 +42
Branches 26439 26447 +8
==========================================
- Hits 99780 98925 -855
- Misses 9992 11155 +1163
+ Partials 1292 1026 -266
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The paid half of #8834. The free half (
judgment-agreement.ts, already merged) scores inter-run agreement over whatever stances a review happens to produce. This PR adds the deliberate extra stances: flag-gated re-runs of the same judge with deterministically rotated few-shot exemplar windows ("simulated annotators", Trust or Escalate, ICLR 2025), folded into the recorded confidence through the existingscoreJudgmentAgreementaxis — never into split detection, which still readsreviewerVotesalone.Decisions (the ones the free half's closing comment deferred)
AI_REVIEW_SELF_CONSISTENCY_RUNS= TOTAL evaluations per review. Unset/0/garbage ⇒ OFF, byte-identical to today. Explicit values clamp into{2,3}: one total run measures nothing, and the literature's benefit saturates by three.okai_usage_eventsrow (the exact sum the quota gate reads) withselfConsistencymetadata. An underfunded plan degrades to fewer runs and the score records as uncorroborated — a decision that could not afford corroboration is recorded as exactly that, never a fabricated score.f(repo#pr, runIndex)via FNV-1a over a versioned synthetic exemplar set (JUDGE_EXEMPLAR_SET_VERSION, balanced 3 defect / 3 clean, never drawn from real PRs). Deterministic twice over: replay can reconstruct what each run saw, and re-reviews sample the same rotation instead of secretly measuring prompt-shuffle variance.systemAppendparameter turns out to be a self-host-CLI-only transport (selfHostCliSystemAppenddrops it for every other provider), so appending there would have made rotation silently provider-dependent.Tests (21, all passing; new module 100% lines and branches)
decision_records'aiAgreement(0.75 over 4 stances); a failed extra records spend but fabricates no stance (sampleCount stays 2, agreement 1); no usable primary stance ⇒ zero extra spend.Closes #8834