fix(workflows): skip approved PRs in review reminders#3295
Conversation
Clarify the assign-reviews prompt so reviewer reminder comments skip PRs that already have a human approval, even if another reviewer is still requested. Mirror the same wording in the example workflow template. Co-authored-by: openhands <openhands@all-hands.dev>
github-actions
Bot
added
the
release-note-required
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED Behavioral default changes detectedThese public
|
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Replace the broader assign-reviews prompt rewrite with a small note that reviewer reminders only apply when no human reviewer has already approved the PR. Co-authored-by: openhands <openhands@all-hands.dev>
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
🟢 Taste Rating: Good taste — this is a small, targeted prompt clarification mirrored in the example workflow. No issues found.
[RISK ASSESSMENT]
- [Overall PR]
⚠️ Risk Assessment: 🟢 LOW
Prompt-only workflow/example update; no runtime code, dependency, or API impact.
VERDICT:
✅ Worth merging
This review was generated by an AI agent (OpenHands) on behalf of the user.
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS
Verified the workflow prompt change with the actual OpenHands workflow runner against real GitHub PR data; human-approved PRs are now skipped while bot-only approvals are still ignored.
Does this PR achieve its stated goal?
Yes. The PR set out to prevent the Assign Reviews workflow from posting Need Reviewer Action reminders on PRs that already have a human approval, without treating bot approvals as human approvals. Running the workflow's actual agent_script.py against real GitHub PRs showed PR #3237 changes from BASE_WAITING_FOR_REVIEW=YES to PR_WAITING_FOR_REVIEW=NO because human reviewer enyst approved it, while bot-only-approved PR #3252 remains PR_WAITING_FOR_REVIEW=YES because the only approval is from all-hands-bot.
| Phase | Result |
|---|---|
| Environment Setup | ✅ uv run created/used the project environment; SDK workflow runner imports succeeded; changed YAML parsed. |
| CI Status | cleanup-on-approval), 4 in progress. |
| Functional Verification | ✅ Actual workflow agent runner produced the expected before/after decisions on real PR data. |
Functional Verification
Test 1: Human-approved PR is no longer considered waiting for review
Step 1 — Reproduce / establish baseline (without the fix):
Ran a base-vs-PR prompt comparison against the workflow YAML:
base: there are no open review comments or change requests
base_human_approval_skip_present=False
pr: there are no open review comments or change requests, and no human reviewer's latest submitted review is APPROVED; ignore approvals from automation/bot accounts for this check
pr_human_approval_skip_present=True
This shows the base workflow prompt did not instruct the agent to skip a PR with a latest human approval.
Step 2 — Apply the PR's changes:
Used the checked-out PR branch (56905bd9396811e41fe13eb4cd8fc164853b6fe8) and ran the same workflow entry point used by the GitHub Action, with a read-only QA prompt:
OPENHANDS_SUPPRESS_BANNER=1 GITHUB_TOKEN=*** LLM_API_KEY=*** LLM_MODEL=litellm_proxy/openai/gpt-5.5 LLM_BASE_URL=https://llm-proxy.app.all-hands.dev PROMPT_STRING="$(cat /tmp/qa_assign_reviews_prompt.txt)" uv run python /tmp/agent_script.py > /tmp/qa_agent_run.log 2>&1
Step 3 — Re-run with the fix in place:
The runner queried real PR #3237 and produced:
reviewDecision: APPROVED
reviewThreads: nodes=[]
reviews: all-hands-bot APPROVED/COMMENTED, enyst APPROVED
BASE_WAITING_FOR_REVIEW=YES
PR_WAITING_FOR_REVIEW=NO
EVIDENCE=There are no review threads or change-request reviews, and human reviewer enyst's latest submitted review is APPROVED.
This confirms the changed prompt makes the actual workflow runner skip a PR that the old waiting-for-review criterion would still treat as waiting.
Test 2: Bot-only approvals remain ignored
Step 1 — Reproduce / establish baseline (without the fix):
Found a real open PR with only a bot approval:
PR #3252: feat(sdk): allow custom FileStore injection
reviewDecision: APPROVED
botApprovals: ["all-hands-bot"]
latestReviews: all-hands-bot APPROVED
This is the edge case from the PR description: GitHub may report an approval, but it should not count as a human approval.
Step 2 — Apply the PR's changes:
Ran the same workflow runner against PR #3252 with the PR prompt variant.
Step 3 — Re-run with the fix in place:
The runner produced:
reviewThreads: nodes=[]
reviews: all-hands-bot APPROVED
BASE_WAITING_FOR_REVIEW=YES
PR_WAITING_FOR_REVIEW=YES
EVIDENCE=There are no review threads/change requests, and the only submitted approval is from all-hands-bot, with no human reviewer latest APPROVED.
This confirms automation/bot approvals are ignored by the new human-approval check.
Test 3: Changed workflow files remain parseable
Ran YAML parsing against both changed workflow files:
.github/workflows/assign-reviews.yml: YAML parsed; PROMPT_STRING length=6327; human-approval skip present=True
examples/03_github_workflows/01_basic_action/assign-reviews.yml: YAML parsed; PROMPT_STRING length=3728; human-approval skip present=True
This confirms the folded multiline prompt still loads as valid YAML in both the real workflow and mirrored example.
Issues Found
None.
This QA review was created by an AI agent (OpenHands) on behalf of the user.
2 participants
Summary
Why
PR #3237 already had a human approval, but the scheduled Assign Reviews workflow still posted a reviewer reminder because the prompt did not explicitly tell the agent to skip PRs that were already approved by a human reviewer.
Validation
uv run pre-commit run --files .github/workflows/assign-reviews.yml examples/03_github_workflows/01_basic_action/assign-reviews.ymlThis PR was created by an AI agent (OpenHands) on behalf of the user.
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:56905bd-pythonRun
All tags pushed for this build
About Multi-Architecture Support
56905bd-python) is a multi-arch manifest supporting both amd64 and arm6456905bd-python-amd64) are also available if needed