Skip to content

orb(ai-review): the model can review a partial diff while believing it is complete — three unmarked truncation paths #9075

Description

@JSONbored

The diff builder correctly emits in-band truncation markers (### …diff truncated (N files total), … (N lower-signal hunk(s) dropped)src/review/review-diff.ts ~126, ~156). Three paths bypass them entirely, so the reviewer renders a confident verdict on silently incomplete input.

1. Pre-filter file deletion. filterReviewFilesForAi removes whole files before the builder runs (src/queue/ai-review-orchestration.ts ~540-543), and the N files total in the marker is the post-filter count. The model is never told and cannot infer it.

2. Context-budget break. selectContextSectionsWithinBudget (src/services/ai-review.ts ~1002-1017) breaks on the first oversized section, dropping every lower-priority one with no marker at all. With grounding at ~96k plus diff+body at ~122k, roughly 22k remains — a 25k RAG section drops impactMap, enrichment, cultureProfile, and testEvidence. That last one is ~200 chars and is not model context but a deterministic classifier fact ("this PR has NO test-path changes"). A large PR is exactly where the reviewer most needs it, and it is the first thing lost.

3. Linked-issue satisfaction re-slice. src/services/linked-issue-satisfaction.ts ~107-108 re-slices the already-truncated diff to 60k under the ambiguous header "Unified diff (truncated if large):" — a hedge, not a fact.

Why (3) is the sharpest

That module's own header claims "NO gate wiring, NO disposition change… advisory-only either way"that is stale. Under linkedIssueSatisfactionGateMode: "block" (which we run), an unaddressed verdict pushes a critical-path finding with the public text "this PR does not appear to satisfy its linked issue's scope."

So a PR whose issue-satisfying change sits in a lower-signal hunk, a lower-priority file, or past char 60,000 receives a confident public "you didn't fix the issue" computed from a window that never contained the fix. The 0.5 confidence floor guards against low-confidence hallucination, not against a high-confidence verdict on truncated input.

Fix

  1. Pass the pre-filter file count into the truncation marker, and state explicitly when files were filtered out.
  2. continue instead of break for sections under a small threshold (~2k), or hoist testEvidence out of the budgeted set entirely — it is a fact, not context, and costs nothing.
  3. Correct the linked-issue-satisfaction header, and when the diff hits MAX_DIFF_CHARS or contains the builder's truncation markers, suppress unaddressed (degrade to partial). Absence of evidence in a truncated window is not evidence of absence — the exact Review-input truncation causes false 'missing screenshot evidence' closes #8961 reasoning already applied to PR bodies elsewhere in this file.

Refs #8961.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions