Skip to content

fix(ai-review): scope the incoherent-diff bail to mechanical breakage and reclassify evidence-bearing bails - #8796

Merged
JSONbored merged 2 commits into
mainfrom
fix/incoherent-diff-bail-scope
Jul 26, 2026
Merged

fix(ai-review): scope the incoherent-diff bail to mechanical breakage and reclassify evidence-bearing bails#8796
JSONbored merged 2 commits into
mainfrom
fix/incoherent-diff-bail-scope

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

The root-cause fix for the recurring inconclusive-review incident (PRs #8735/#8737; siblings #8794 and #8795 fixed the fallback and copy/classification layers). The bail instruction's "does not cohere with the PR title/description" clause triggered on a valid diff whose title undersold it — title-vs-diff scope mismatch is a reviewable quality signal, not evidence of a stale diff.

Changes

  • Prompt — the FAIL-CLOSED bail now targets only mechanically broken diffs (empty / truncated mid-hunk / garbled / contradicting the PR's own changed-file list), with an explicit instruction that a readable diff with scope drift gets reviewed and the mismatch noted, never bailed on.
  • Parse-side reclassification — a bail arriving with a substantive valueAssessment rationale (valid magnitude + ≥40 trimmed chars; the confirmed live case carried ~300 chars naming exact call sites) reclassifies into a usable review whose assessment is the fixed public-safe SCOPE_MISMATCH_ASSESSMENT — model text travels only through the existing valueAssessment channel. A bare bail stays null and keeps the deliberate-bail retry break (fix(review): short-circuit the retry loop on a deliberate INCOHERENT_DIFF_ASSESSMENT bail #7518 behavior pinned by the existing #ops-review-burst tests, untouched).
  • isIncoherentDiffBail mirrors the reclassification exactly, including toValueAssessment's magnitude validation, honoring its documented "can never disagree with parseModelReview" contract — with a test pinning agreement on the invalid-magnitude edge.

Test plan

  • npm run typecheck — clean
  • 4 new tests: reclassification (fixed assessment, confidence + rationale preserved), short-rationale bare bail, invalid-magnitude mirror agreement, bail-false for reclassifiable text
  • 510 tests green across the five ai-review/selfhost suites; existing bare-bail retry-break tests unchanged
  • Changed hunks: 0 uncovered statements/branches

Closes #8789

JSONbored and others added 2 commits July 25, 2026 23:25
… and reclassify evidence-bearing bails (#8789)

Confirmed live 2026-07-26 (PR #8735): the bail instruction's "does not cohere
with the PR title/description" clause made the reviewer bail on a VALID diff
whose title merely undersold it — the model's own rationale named the exact
production change it had just read — collapsing a reviewable PR into an
inconclusive manual hold once the fallback also failed.

- Prompt: the bail now fires only on a BROKEN diff (empty, truncated,
  garbled, or contradicting the PR's own file list). A readable diff whose
  scope differs from the description is explicitly NOT a bail — review it and
  note the mismatch in the assessment.
- Parse: a bail that arrives WITH a substantive valueAssessment rationale
  (valid magnitude, >= 40 trimmed chars — the model demonstrably read the
  diff) reclassifies to a usable scope-observation review with a fixed
  public-safe assessment (SCOPE_MISMATCH_ASSESSMENT); model text travels only
  through the existing valueAssessment channel. A bare bail stays null.
- isIncoherentDiffBail mirrors the reclassification (including
  toValueAssessment's magnitude validation) so the retry-loop break can never
  disagree with what parseModelReview actually did — its documented contract.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 26, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui 43c8379 Commit Preview URL

Branch Preview URL
Jul 26 2026, 06:28 AM

@JSONbored JSONbored self-assigned this Jul 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (a8a0912) and head (43c8379) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

@JSONbored
JSONbored merged commit 7544cd3 into main Jul 26, 2026
7 checks passed
@JSONbored
JSONbored deleted the fix/incoherent-diff-bail-scope branch July 26, 2026 06:30
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.26%. Comparing base (a8a0912) to head (43c8379).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8796      +/-   ##
==========================================
- Coverage   93.81%   92.26%   -1.55%     
==========================================
  Files         797      797              
  Lines       79584    79596      +12     
  Branches    24118    24122       +4     
==========================================
- Hits        74662    73441    -1221     
- Misses       3556     5090    +1534     
+ Partials     1366     1065     -301     
Flag Coverage Δ
backend 92.93% <100.00%> (-2.16%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/services/ai-review.ts 97.13% <100.00%> (+0.05%) ⬆️

... and 3 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ai-review): the incoherent-diff bail misfires on title-vs-diff scope mismatch, turning reviewable PRs into inconclusive holds

1 participant