docs(github): tighten PR and task templates against agent misuse#752
Conversation
PR #748 surfaced a template ambiguity: the labeler bot auto-applied routing/priority labels but not the required type label, the agent rewrote the checklist text to "Label bot should apply type/routing/ priority labels; no manual labels were specified per maintainer instruction" instead of adding the missing `bug` label, and the human maintainer had to add `bug` minutes later before validation ran. Five rounds of crosscheck (Claude Opus + Codex) found and addressed: - Label checklist conflated bot-applied vs author-applied labels and had an "or I requested maintainer labeling" escape hatch. - All checklist items were first-person past-tense assertions with no explicit immutability guard; the agent treated the lines as editable. - Human Review Status was a free-text contract with a loose "or not required" escape hatch. - How To Verify example block had no machine-spottable marker; an agent could leave the example verbatim. - Task template Execution mode options had overlapping prose and uneven parallelism. The new PR template splits labels into three explicit author/bot rows, adds a top-of-file policy comment plus a "How to use this checklist" blockquote, converts Human Review Status into a strict three-option enum, adds a `replace-before-submit` HTML sentinel to the example block, and tags genuinely conditional items with **(conditional)**. The new Task template Execution mode rewrites each option as an explicit declarative directive ("the agent must not... until..." / "the agent must post the plan as an issue comment..." / "the agent makes the requested changes... must not push directly to dev"). Round-5 crosscheck returned "None" for both code findings and design alternatives from both reviewers. Remaining design-level work (CI lint enforcement, issue-template `area` dropdown restructure) is deferred to a follow-up PR.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughUpdated GitHub issue and pull request templates to clarify execution modes, checklist policies, and human review status requirements. Task template now provides explicit agent workflow instructions. Pull request template adds a checklist policy statement and reformatted status guidelines. CI labeler expanded to track ChangesGitHub Workflow Configuration Updates
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request updates the task issue template and the pull request template to provide clearer instructions on execution modes and a more detailed, structured checklist. Key changes include defining explicit human review statuses, adding verification examples, and categorizing checklist items into required and conditional sections. Feedback from the review suggests formatting improvements such as using backticks for labels and statuses, adding explicit checklist items for target branches and conventional commits, and ensuring the 'Related Issue' section is used for explaining missing issue links.
Templates and issue forms under .github/ITS_TEMPLATE/ and the top-level PR template are CI/process infrastructure that should route the same way as workflow files. Without this, a PR that only touches .github/pull_request_template.md or .github/ISSUE_TEMPLATE/*.yml receives no routing label from the labeler bot, and the manual override is stripped by sync-labels on the next pr-triage run. The task rule keeps the narrower .github/workflows/** scope: task is a type label, and per the updated PR template, type labels are author-applied, not bot-applied. Workflow PRs retain the existing task auto-application as a historical convenience; this can be revisited in a follow-up if the inconsistency proves confusing.
Restore the explicit .github/workflows/** glob alongside the broader .github/** glob. The pr-triage contract test at packages/opencode/test/github/pr-triage-workflow.test.ts:57 uses literal-string matching to assert that workflow files are routed to the ci label, so dropping the workflows glob broke the test even though .github/** subsumes it semantically. Keeping both globs preserves the test contract and documents the intent that workflow files are first-class CI infra while other .github/ files (templates, labeler.yml itself, dependabot config) are CI process.
|
@Reviewer Thanks for the catch on the contract test — P1 is real and now fixed. P1 fix (commit 3c30cd8): Restored Factual note on the "待验证" section: the review describes the diff as adding Remaining |
|
Resolving the three Gemini review threads — they're auto-generated polish suggestions that don't apply to the current diff:
|
Summary
Pending/Approved by @<reviewer>/Not required: <reason>) and remove the "or not required" loose escape hatch from the checklist.<!-- replace-before-submit -->HTML sentinel that self-deletes with the example block in How To Verify; mark the example block with anEXAMPLEprefix.**(conditional)**and direct skipped-item explanations to Risk Notes.cirule from.github/workflows/**to.github/**so PRs that only touch templates or issue forms receive a routing label (without this, label-policy fails on template-only PRs — this PR hit it on first try).Why
PR #748 surfaced a template ambiguity: the labeler bot auto-applied
app/harness/P2but not the required type label, the agent rewrote the checklist text to "Label bot should apply type/routing/priority labels; no manual labels were specified per maintainer instruction" instead of addingbug, and the human maintainer addedbug~5 minutes after the workflow started — the pr-triage workflow happened to queue long enough for the human edit to land before the final validation step ran, so validation passed despite the agent's failure.Three failure modes enabled this:
Five rounds of multi-model crosscheck (Claude Opus + Codex) iterated the fixes; round 5 returned
Nonefrom both reviewers for both code findings and design alternatives.While opening this PR, a fourth related failure mode surfaced: the labeler
cirule only covered.github/workflows/**, so PRs that only touched templates or issue forms received no routing label, and thesync-labels: truesetting stripped any manual override on the next pr-triage run. The labeler fix is included so this PR (and any future template-only PR) can pass pr-triage without manual intervention.Related Issue
No issue; surfaced during pr-triage forensics on PR #748.
Human Review Status
Pending. A human should make the final merge decision after reviewing the final diff and verification evidence.
Review Focus
Not required: <reason>covers the existingchore(release)codex-generated bumps and OfficeCLI dependabot-style PRs (PR chore: bump bundled OfficeCLI to v1.0.93 #730, ci: harden OfficeCLI bump PR creation #735, chore(release): bump desktop version to v2026.5.19 #742 patterns). If pawwork's release automation generates PR bodies, that automation may need to be updated to emit the new string.cirule now covers all.github/**paths. Workflow PRs still get bothci(routing) andtask(auto-applied type). Template/issue-form PRs now get justci, with the author expected to add the type label per the new template policy.Risk Notes
cirule expansion: a PR that touches both.github/**and a packaged source area will now get bothciand the source-area routing label. This is the intended "multi-area PR" behavior already supported by label policy (it requires at least one routing label, not exactly one).How To Verify
Screenshots or Recordings
Not required; pure infra/text changes with no visible UI surface.
Follow-up (out of scope for this PR)
Two design-level improvements were identified by crosscheck and deferred:
areadropdowns in01-bug-report.ymland02-feature-request.ymlwith routing-label strings (app — product behavioretc.), add anareadropdown to03-task.yml, and drop the "Not sure" default that becomes a silent routing-signal kill.A third inconsistency was noticed but left in place: the labeler
taskrule still auto-applies thetasktype label for.github/workflows/**changes, even though the new PR template states type labels are author-applied. Removing this auto-application would change the long-standing CI PR workflow and was kept as a separate decision.Checklist
bug,enhancement,task, ordocumentation), at least one primary routing label (app,ui,platform,harness, orci), and exactly one priority label (P0toP3), or I requested maintainer labelingdev, and my PR title and commit messages use Conventional Commits in EnglishSummary by CodeRabbit