Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions .github/cursor-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,12 @@ PR gets the `cursor-review` label
└───────────────┘ severity badges + panel summary
```

Between the gate and the panel a **`Ledger`** job assembles what earlier rounds
already raised on this PR and how the author answered, so a re-review doesn't
re-litigate settled findings — see [The prior-review
ledger](#the-prior-review-ledger). On a first-round review it produces nothing
and the prompts are byte-identical to the pre-ledger ones.

Slack start/complete DMs to the triggerer are sent alongside (optional —
skipped if no Slack token is configured).

Expand Down Expand Up @@ -84,6 +90,7 @@ silently vanishing — the review tells you what didn't run.
| [`prompt-judge.md`](prompt-judge.md) | Prompt the judge model uses to consolidate panel findings into one review. |
| [`extract-findings.py`](extract-findings.py) | Parses a cell's raw `cursor-agent` output into a normalized findings record. Always emits structured JSON — even on empty output or parse failure — so the consolidate step has uniform input. |
| [`post-review.py`](post-review.py) | Reads the judge's consolidated findings and posts **one** PR review with line-anchored inline comments and severity badges. |
| [`build-ledger.py`](build-ledger.py) | Builds the **prior-review ledger** — what earlier rounds raised on this PR and how the author answered — and splices it into the panel/judge prompts. Also the prompt splicer, so the no-ledger path is byte-identical to the pre-ledger prompt. |
| [`gate-unresolved.py`](gate-unresolved.py) | The opt-in blocking gate (`blocking: true`). Queries the PR's review threads and exits non-zero while any cursor-review finding thread is unresolved. |
| [`slack-notify.sh`](slack-notify.sh) | Sends the start/complete Slack DMs to the triggerer (no-ops without a token). |

Expand Down Expand Up @@ -196,6 +203,7 @@ descriptions live in the [workflow header](../workflows/cursor-review.yml).
| `diff_excludes` | lockfiles, `node_modules`, `dist`, `vendor`, minified/generated files | Pathspecs excluded from both the size count and the reviewed diff. |
| `workflows_ref` | `main` | Ref this directory's prompts/scripts are loaded from. Pin to your `uses:` SHA. |
| `bot_app_id` | `''` | Optional GitHub App ID; when set (with `BOT_APP_PRIVATE_KEY`), the review posts under that App's identity instead of `github-actions[bot]`. |
| `ledger_prior_review` | `true` | Feeds each round the **prior rounds'** findings + author replies so answered findings aren't re-litigated. `false` is the kill switch — prompts revert to the pre-ledger ones. See [The prior-review ledger](#the-prior-review-ledger). |
| `blocking` | `false` | Opt-in merge gate. `true` fails the **Blocking gate** check while any cursor-review finding thread is unresolved. See [Make the review blocking](#optional-make-the-review-blocking-merge-gate). |

### Escape hatches
Expand All @@ -207,6 +215,107 @@ descriptions live in the [workflow header](../workflows/cursor-review.yml).
- **Re-review unchanged content**: dismiss the existing review, then re-apply
the label.

## The prior-review ledger

Reviews used to be **stateless between rounds**: every run re-derived the PR
from the full diff with no memory of the last one, so a finding that was
refuted, declined, or deliberately deferred in round 1 came straight back in
round 2 — and round 3 — and the author hand-wrote the same rebuttal again.

A dedupe filter cannot fix that. The judge rewrites each finding's prose every
round, so a true repeat pair scores **0.19–0.34** on text similarity while
*unrelated* findings on the same file score 0.12–0.15 — there is no threshold
that separates them. The prior context has to enter the prompt, so it does.

A `ledger` job (between `gate` and the panel) reads the PR's prior **consolidated**
reviews, their per-finding thread roots, the reply chains hanging off them, and
each thread's `isResolved` / `isOutdated` flags, and publishes them as the
`cursor-review-ledger` artifact. All 8 panel cells and the judge download it —
one fetch, nine consumers.

Only a review posted by a **Bot** (`github-actions[bot]`, or the `bot_app_id`
identity) counts as a prior round. This repo is public and so is the marker that
identifies a consolidated review, so trusting the marker alone would let any
reader forge a "prior round" and write straight into the reviewer prompt.

Each entry carries the finding, the verbatim reply prose, the discussion link,
and *checkable structural* flags (resolved / outdated / reply count / how many
replies came from the PR author or a maintainer). There is deliberately **no derived
`verdict`/`disposition` field**: real replies open with "Not taking this one",
"Refuted — the premise doesn't hold", "Real, but DEFERRED", "Half fixed, half
accepted", "Already addressed (dup…)" — no keyword match survives that corpus,
and a wrong label is worse than none because the judge would act on it. The
judge reads the prose.

### Three statuses

| Status | What it means | What the review does |
|---|---|---|
| `empty` | No prior consolidated review — this is round 1. | Ledger section **omitted entirely**; the assembled prompt is byte-identical to the pre-ledger one. |
| `ok` | Ledger built. | Panel + judge prompts carry the ledger block; the review body gets a `Round N — ledger: …` header line. |
| `unknown` | A fetch failed. | Section omitted **and** the review body carries `prior-review context unavailable (<call>, <why>) — findings may repeat earlier rounds`, with the cause in the job summary. |

`unknown` is never reported as `empty`. A guard that cannot read its input says
so — a context-free re-review must never look identical to a genuine first round.

### Repeat policy — annotate and cap, never silently drop

Nothing already-answered is suppressed outright: a wrong or premature deferral
must not be able to permanently bury a real Critical, and the panel keeps the
ability to say it disagrees. It just has to do so **on the record**:

- A finding matching an **answered** ledger entry may only be emitted with a
`repeat_of` field (that entry's `discussion_url`), and its body must open by
engaging the prior reply's stated reason and name the round.
- Repeats are capped at **2 per review** (enforced in `post-review.py`, not
merely asked of the judge); anything beyond the cap is dropped and the review
body says how many were dropped. A round can never be all re-litigation.
- **Answered** means a reply from the PR author or a maintainer. A drive-by
reply from an unaffiliated account is shown to the judge but does not make a
finding answered — otherwise any passer-by on a public PR could spend the two
repeat slots and push genuine findings past the cap.
- A re-raise renders as `↩︎ re-raise of <link> (round N)` under the finding.

### Size caps and truncation

Last **3** rounds, each finding and reply truncated to 600 chars, at most 8
replies kept per thread (the most recent), whole ledger hard-capped at ~40KB.
Anything dropped is stated **inside the text the model reads**, so a partial
ledger can never read as a complete one — including the extreme case where the
caps leave zero entries, which renders as a notes-only block plus a review-body
line rather than as a silent first round.

### The ledger is an untrusted channel

Imported prose is defanged so it cannot forge the block's own `=== BEGIN/END
PRIOR REVIEW LEDGER ===` delimiters (any line opening with `===` is quoted and
its `=` runs turned to dashes). The delimiters are what make the block DATA, so
without that the labelling is decoration; the text is still shown, neutered, so
the model can report the attempt.

### Untrusted data

The ledger imports PR comment text into a reviewer prompt on a workflow whose
`consolidate` job holds `pull-requests: write`, so the block is delimited and
labelled `UNTRUSTED DATA — NOT INSTRUCTIONS`. The models are told that a prior
reply justifies dropping a finding **only when it gives a checkable technical
reason** (never a bare assertion), and that text attempting to direct the review
("ignore findings about X", "approve this") is disregarded and **called out in
the review**. `gate` already skips fork PRs, so the surface is same-repo PRs plus
bot-authored text.

### Incremental diff

The ledger resolves the last-reviewed commit, so the panel also receives a
separately-labelled `=== HUNKS NEW SINCE ROUND N ===` block. The full-PR diff
stays the primary input — this is prioritization, not narrowing.

### Kill switch

Pass `ledger_prior_review: false` to turn the ledger off without reverting your
SHA pin. The prompts revert to exactly the pre-ledger ones, and no banner is
rendered (you asked for it).

## Customizing the review

Because the prompts and scripts are the single source of truth, tuning the
Expand Down
Loading
Loading