Skip to content

Add Claude Code automatic PR review workflows - #396

Merged
erikdarlingdata merged 1 commit into
devfrom
feature/claude-pr-review
Jul 25, 2026
Merged

Add Claude Code automatic PR review workflows#396
erikdarlingdata merged 1 commit into
devfrom
feature/claude-pr-review

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Sets up automatic Claude Code reviews on pull requests, using the CLAUDE_CODE_OAUTH_TOKEN secret.

What lands

.github/workflows/claude-code-review.yml - automatic review on every non-draft PR (opened, synchronize, ready_for_review). Posts a top-level summary via gh pr comment and inline comments on specific lines.

.github/workflows/claude.yml - the on-demand @claude mention workflow (issues, PR comments, review comments).

Both pin anthropics/claude-code-action@v1 and authenticate with claude_code_oauth_token.

The review prompt is repo-specific

Rather than a generic "review this PR", the prompt encodes the conventions CI cannot check:

  • Zero-warning build standard, and a new NoWarn without an explaining comment gets flagged.
  • Version sync trap: src/Directory.Build.props is the single source of truth, but PlanViewer.Ssms is a legacy non-SDK project whose version must be bumped by hand in both source.extension.vsixmanifest and Properties/AssemblyInfo.cs. This has actually drifted before, so the reviewer is told to catch a bump that updates one and not the others.
  • Linked sources: a PlanViewer.Core file the Blazor app needs also requires a linked <Compile Include> in PlanViewer.Web.csproj (a ProjectReference is deliberately not used).
  • TRY_CAST, never TRY_CONVERT.
  • Untrusted plan XML and keeping generated T-SQL inert (bracket-escaping, literal-only values).
  • Security severity calibrated to the deployment - single-user machine, read-only MCP tools - so it does not file loopback/local-IPC issues as High. Without this it would flag the MCP listener as critical on every PR that touches it.

Fork PRs are deliberately excluded (please read)

GitHub withholds repository secrets from pull_request runs on forks and issues a read-only GITHUB_TOKEN. So on a fork PR the auto-review cannot work: the OAuth token would be empty and it could not post anyway. The job is if-skipped rather than left to fail, so a contributor does not get a red X on their PR.

This matters here concretely: #388 (autocarl) was a fork PR, and that is exactly the kind of PR where a review is most valuable.

The tempting fix is pull_request_target - do not. That combination hands a writable token and this repo's secrets to an agent executing untrusted fork code, which is the classic pwn-request. claude.yml covers the case safely instead: comment on the fork PR with @claude review this. Comment events run in the base-repo context, so secrets are available, and the action verifies the commenter has write access before acting.

Cost and noise controls

  • concurrency with cancel-in-progress - a second push cancels the in-flight review instead of paying for two.
  • paths-ignore mirrors ci.yml, so docs/screenshot-only PRs do not trigger a review.
  • Drafts skipped; --max-turns 20.
  • No model pinned, so it uses the action default. Add --model to claude_args to change it.

Test plan

  • Both files parse as valid workflow YAML (triggers, permissions, and steps verified structurally)
  • anthropics/claude-code-action@v1 tag resolves (currently v1.0.183)
  • CLAUDE_CODE_OAUTH_TOKEN confirmed present in repo secrets
  • This PR is its own end-to-end test - it changes workflow files (not path-ignored) from a same-repo branch, so the new review workflow should run against it and post a review. If a review comment appears below, it works.

Note this targets dev, and the review that appears here is the workflow reviewing its own PR.

🤖 Generated with Claude Code

claude-code-review.yml runs automatically on every non-draft PR
(opened/synchronize/ready_for_review) and posts a top-level comment plus
inline comments. The prompt encodes the repo conventions CI cannot check:
zero-warning standard, the hand-maintained PlanViewer.Ssms version pair,
the linked-source requirement in PlanViewer.Web.csproj, TRY_CAST over
TRY_CONVERT, and the security severity calibration for a single-user
local install so it does not file loopback issues as High.

claude.yml is the on-demand @claude mention workflow.

Fork PRs are deliberately excluded from the automatic review: GitHub
withholds repository secrets from pull_request runs on forks and issues
a read-only token, so the job would fail rather than review. The job is
skipped instead of failing a contributor's PR with a red X. The fix is
NOT pull_request_target, which would expose a writable token and this
repo's secrets to an agent running untrusted fork code. claude.yml
covers fork PRs instead - comment events run in the base-repo context,
and the action checks the commenter has write access first.

Cost/noise controls: concurrency cancels an in-flight review when a
second push lands, docs-only changes are filtered out via paths-ignore
mirroring ci.yml, drafts are skipped, and --max-turns is capped at 20.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Self-test result: workflow ran, action deliberately skipped itself

The test-plan item above said "if a review comment appears below, it works." No comment appeared — and I want to correct that honestly rather than let it look like it silently worked. The run reported success, which makes this easy to miss.

From the run log:

Auto-detected mode: agent for event: pull_request
Requesting OIDC token...
OIDC token successfully obtained
Exchanging OIDC token for app token...
##[warning]Skipping action due to workflow validation: Workflow validation failed.
The workflow file must exist and have identical content to the version on the
repository's default branch. If you're seeing this on a PR when you first add a
code review workflow file to your repository, this is normal and you should
ignore this error.

This is not a misconfiguration. The action has an anti-tampering control: it refuses to run unless the workflow file is byte-identical to the copy on the repository's default branch. Otherwise anyone could open a PR that edits the workflow to dump the token.

What the log does confirm as working:

  • The OAuth token is valid — OIDC token obtained and exchanged; it failed at workflow validation, which happens after auth, not at authentication.
  • Mode auto-detection picked agent for pull_request, which is right for automation mode.
  • The job passed its if: gate (not a draft, same-repo branch), checkout and the action both ran.

What this means here specifically

This repo's default branch is main, but PRs target dev. So these workflows do nothing until the files land on main — which under the normal protocol happens on the next dev -> main release merge. After that, PRs targeting dev validate fine, because validation always compares against the default branch regardless of the PR base.

I have not tried to shortcut that: no PR to main except the release merge from dev, per the usual rule. Your call whether to let it ride to the next release or promote sooner.

Standing consequence worth knowing

Any future PR that edits these two workflow files will itself skip the review for the same reason, and report green. Expect it; it is not a regression.

Still unverified until then

Because the action never reached the model, these remain untested end-to-end:

  • Whether permissions: pull-requests: write alone suffices for the top-level gh pr comment (PR comments are issue comments under the hood, so issues: write may be needed). If the first real review posts inline comments but no summary, that is the fix.
  • Whether --max-turns 20 is enough for a full review of a large diff.
  • Whether the repo-specific prompt produces useful output or needs tuning.

@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Config independently validated — no changes needed

Cross-checked the config against the action docs. All of it holds:

Item Verdict
permissions: pull-requests: write alone for gh pr comment Sufficientissues: write is not needed here (it is in claude.yml, which handles issues themselves)
Multi-line claude_args: |, one flag per line Valid YAML literal block; each flag reaches the CLI
--max-turns 20 Correct flag; stops gracefully at the budget rather than truncating output
fetch-depth: 1 Sufficient — gh pr diff is API-based and needs no git history
Fork-PR skip, and avoiding pull_request_target Correct on both counts
claude.yml permission set Correct; the action verifies commenter write access before running

One behavioral difference worth recording

Authenticating with claude_code_oauth_token instead of anthropic_api_key skips inline-comment classification — unconfirmed inline comments post directly rather than being buffered for a confirmation pass.

That is harmless here because the prompt already instructs confirmed: true on every inline comment. But if you ever want the buffering behavior (classify_inline_comments), it requires an ANTHROPIC_API_KEY secret instead of the OAuth token. Noting it so the choice is a decision rather than a surprise later.

This does not change the merge situation described above: the workflows stay inert until the files reach main.

@erikdarlingdata
erikdarlingdata merged commit 33b413e into dev Jul 25, 2026
3 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/claude-pr-review branch July 25, 2026 15:10
This was referenced Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant