Review agent: detect and flag scope expansion after prior approval

[fullsend-ai-retro]

What happened

On PR #742, the review agent approved at iteration 4 (SHA 39bc6cb). The author then pushed new commits that introduced undocumented behavioral changes to fullsend.yaml — removing auto-triage on issue open/edit and deleting the entire dispatch-prioritize job (55 lines). The review agent correctly escalated to request-changes in iteration 6, but its findings were presented as ordinary review feedback with no indication that this was a regression from a previously-approved state. The human reviewer (ralphbean) had to independently notice the scope creep. This is related to #849 (strategic fitness dimension).

What could go better

When the review agent has previously approved a PR and then reviews a new push, it should compare the new diff against the previously-approved diff and explicitly flag any changes that were not present in the approved version. This is especially important for changes that go beyond the PR's stated scope (issue title says 'rename commands' but the diff deletes entire jobs). Confidence: medium — the detection is straightforward (diff comparison), but determining whether new changes are 'in scope' requires judgment that may produce false positives on legitimate iterative PRs.

Proposed change

In the review agent definition or the pr-review skill, add awareness of prior review outcomes. The pre-review script already has access to prior reviews via the GitHub API. When the agent detects it previously approved the same PR, it should: (1) fetch the diff at the approved SHA, (2) compare it to the current diff, (3) highlight files or hunks that are new since approval, and (4) apply extra scrutiny to those changes — specifically checking whether they align with the PR description and originating issue. New deletions of entire code blocks or jobs should be flagged as potential scope expansion. This could be implemented as an additional step in pre-review.sh that writes a scope-delta.md file into the agent's working directory.

Validation criteria

On a future PR where the review agent approves and a subsequent push introduces changes outside the original scope, the review comment should explicitly call out 'these changes were introduced after a prior approval' and flag them for extra scrutiny. Measure: within the next 10 PRs that receive a post-approval push, the review agent should correctly identify scope-expanding changes in its review comment at least 80% of the time without significant false positives on legitimate refinements.

---

Generated by retro agent from #742

[github-actions]

fullsend triage is working on this — view logs

[fullsend-ai-triage]

Triage Summary

Severity: Medium | Category: Enhancement

Problem

The review agent reviews each push independently with no awareness of its own prior review outcomes. When it has previously approved a PR and a subsequent push introduces out-of-scope changes, the agent treats everything as a fresh review — it does not flag that the scope has expanded since approval.

Root Cause

This is by design: the review agent operates under a zero-trust model where each review is fully independent. The agent definition (agents/review.md), the pr-review skill, pre-review.sh, and post-review.sh all lack mechanisms to fetch or compare against prior review state. The input contract provides only GITHUB_PR_URL, GITHUB_ISSUE_URL, and FULLSEND_OUTPUT_DIR — no historical context.

Recommended Approach

1. pre-review.sh: Add a step to query the GitHub API for prior reviews by the review agent on the same PR. When a prior approval exists, fetch the diff at the approved SHA and compute a scope delta against the current diff.
2. Scope delta file: Write a scope-delta.md into the agent's working directory listing files/hunks that are new since the last approval.
3. agents/review.md: Add instructions for the agent to consume scope-delta.md when present and apply extra scrutiny to new-since-approval changes — particularly deletions of entire code blocks and changes that don't align with the PR description.
4. Design consideration: This augments rather than replaces zero-trust. The delta is informational context about what changed, not a trust signal from the prior approval.

Related: #849 (strategic fitness dimension) covers a complementary but independent concern.

Proposed Test Case

Scenario: Post-approval scope expansion detection
  Given a PR with a prior 'approve' review at SHA A
  And new commits pushed producing SHA B
  And SHA B introduces changes outside the PR's stated scope
    (e.g., PR title is 'fix typo' but new commits delete a CI job)
  When the pre-review pipeline runs
  Then scope-delta.md is generated listing out-of-scope files/hunks
  And the review agent's output explicitly references post-approval scope expansion

  Edge cases:
  - Legitimate iterative refinements → no false positive
  - PRs with no prior approval → delta check skipped entirely
  - Multiple prior reviews → use the most recent approval SHA