Add Claude Code automatic PR review workflows - #396
Conversation
claude-code-review.yml runs automatically on every non-draft PR (opened/synchronize/ready_for_review) and posts a top-level comment plus inline comments. The prompt encodes the repo conventions CI cannot check: zero-warning standard, the hand-maintained PlanViewer.Ssms version pair, the linked-source requirement in PlanViewer.Web.csproj, TRY_CAST over TRY_CONVERT, and the security severity calibration for a single-user local install so it does not file loopback issues as High. claude.yml is the on-demand @claude mention workflow. Fork PRs are deliberately excluded from the automatic review: GitHub withholds repository secrets from pull_request runs on forks and issues a read-only token, so the job would fail rather than review. The job is skipped instead of failing a contributor's PR with a red X. The fix is NOT pull_request_target, which would expose a writable token and this repo's secrets to an agent running untrusted fork code. claude.yml covers fork PRs instead - comment events run in the base-repo context, and the action checks the commenter has write access first. Cost/noise controls: concurrency cancels an in-flight review when a second push lands, docs-only changes are filtered out via paths-ignore mirroring ci.yml, drafts are skipped, and --max-turns is capped at 20. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-test result: workflow ran, action deliberately skipped itselfThe test-plan item above said "if a review comment appears below, it works." No comment appeared — and I want to correct that honestly rather than let it look like it silently worked. The run reported From the run log: This is not a misconfiguration. The action has an anti-tampering control: it refuses to run unless the workflow file is byte-identical to the copy on the repository's default branch. Otherwise anyone could open a PR that edits the workflow to dump the token. What the log does confirm as working:
What this means here specificallyThis repo's default branch is I have not tried to shortcut that: no PR to Standing consequence worth knowingAny future PR that edits these two workflow files will itself skip the review for the same reason, and report green. Expect it; it is not a regression. Still unverified until thenBecause the action never reached the model, these remain untested end-to-end:
|
Config independently validated — no changes neededCross-checked the config against the action docs. All of it holds:
One behavioral difference worth recordingAuthenticating with That is harmless here because the prompt already instructs This does not change the merge situation described above: the workflows stay inert until the files reach |
Sets up automatic Claude Code reviews on pull requests, using the
CLAUDE_CODE_OAUTH_TOKENsecret.What lands
.github/workflows/claude-code-review.yml- automatic review on every non-draft PR (opened,synchronize,ready_for_review). Posts a top-level summary viagh pr commentand inline comments on specific lines..github/workflows/claude.yml- the on-demand@claudemention workflow (issues, PR comments, review comments).Both pin
anthropics/claude-code-action@v1and authenticate withclaude_code_oauth_token.The review prompt is repo-specific
Rather than a generic "review this PR", the prompt encodes the conventions CI cannot check:
NoWarnwithout an explaining comment gets flagged.src/Directory.Build.propsis the single source of truth, butPlanViewer.Ssmsis a legacy non-SDK project whose version must be bumped by hand in bothsource.extension.vsixmanifestandProperties/AssemblyInfo.cs. This has actually drifted before, so the reviewer is told to catch a bump that updates one and not the others.PlanViewer.Corefile the Blazor app needs also requires a linked<Compile Include>inPlanViewer.Web.csproj(aProjectReferenceis deliberately not used).TRY_CAST, neverTRY_CONVERT.Fork PRs are deliberately excluded (please read)
GitHub withholds repository secrets from
pull_requestruns on forks and issues a read-onlyGITHUB_TOKEN. So on a fork PR the auto-review cannot work: the OAuth token would be empty and it could not post anyway. The job isif-skipped rather than left to fail, so a contributor does not get a red X on their PR.This matters here concretely: #388 (autocarl) was a fork PR, and that is exactly the kind of PR where a review is most valuable.
The tempting fix is
pull_request_target- do not. That combination hands a writable token and this repo's secrets to an agent executing untrusted fork code, which is the classic pwn-request.claude.ymlcovers the case safely instead: comment on the fork PR with@claude review this. Comment events run in the base-repo context, so secrets are available, and the action verifies the commenter has write access before acting.Cost and noise controls
concurrencywithcancel-in-progress- a second push cancels the in-flight review instead of paying for two.paths-ignoremirrorsci.yml, so docs/screenshot-only PRs do not trigger a review.--max-turns 20.--modeltoclaude_argsto change it.Test plan
anthropics/claude-code-action@v1tag resolves (currently v1.0.183)CLAUDE_CODE_OAUTH_TOKENconfirmed present in repo secretsNote this targets
dev, and the review that appears here is the workflow reviewing its own PR.🤖 Generated with Claude Code