Skip to content

docs(github): tighten PR and task templates against agent misuse#752

Merged
Astro-Han merged 3 commits into
devfrom
claude/improve-templates
May 19, 2026
Merged

docs(github): tighten PR and task templates against agent misuse#752
Astro-Han merged 3 commits into
devfrom
claude/improve-templates

Conversation

@Astro-Han
Copy link
Copy Markdown
Owner

@Astro-Han Astro-Han commented May 19, 2026

Summary

  • Rewrite the PR template's label checklist into three explicit rows (type / routing / priority) that distinguish author-applied vs bot-applied labels, with explicit "tick after the bot runs" guidance.
  • Add a top-of-file HTML policy comment plus a "How to use this checklist" blockquote that mark the checklist text as immutable.
  • Convert the Human Review Status section into a strict three-option enum (Pending / Approved by @<reviewer> / Not required: <reason>) and remove the "or not required" loose escape hatch from the checklist.
  • Add a <!-- replace-before-submit --> HTML sentinel that self-deletes with the example block in How To Verify; mark the example block with an EXAMPLE prefix.
  • Tag the three genuinely conditional checklist items with **(conditional)** and direct skipped-item explanations to Risk Notes.
  • Rewrite the Task issue template's Execution mode dropdown so each option states declaratively what the agent must do next.
  • Extend the labeler ci rule from .github/workflows/** to .github/** so PRs that only touch templates or issue forms receive a routing label (without this, label-policy fails on template-only PRs — this PR hit it on first try).

Why

PR #748 surfaced a template ambiguity: the labeler bot auto-applied app/harness/P2 but not the required type label, the agent rewrote the checklist text to "Label bot should apply type/routing/priority labels; no manual labels were specified per maintainer instruction" instead of adding bug, and the human maintainer added bug ~5 minutes after the workflow started — the pr-triage workflow happened to queue long enough for the human edit to land before the final validation step ran, so validation passed despite the agent's failure.

Three failure modes enabled this:

  1. The single label checklist item conflated bot-applied routing/priority labels with author-applied type labels.
  2. The "or I requested maintainer labeling" escape hatch let the agent tick the box by treating the labeler bot as the maintainer.
  3. No instruction told the agent the checklist text itself was immutable.

Five rounds of multi-model crosscheck (Claude Opus + Codex) iterated the fixes; round 5 returned None from both reviewers for both code findings and design alternatives.

While opening this PR, a fourth related failure mode surfaced: the labeler ci rule only covered .github/workflows/**, so PRs that only touched templates or issue forms received no routing label, and the sync-labels: true setting stripped any manual override on the next pr-triage run. The labeler fix is included so this PR (and any future template-only PR) can pass pr-triage without manual intervention.

Related Issue

No issue; surfaced during pr-triage forensics on PR #748.

Human Review Status

Pending. A human should make the final merge decision after reviewing the final diff and verification evidence.

Review Focus

  • Confirm the new PR template flow (author returns to PR after open to confirm bot-applied labels before ticking routing/priority boxes) is workable in practice — typical labeler-bot latency should support it.
  • Confirm Not required: <reason> covers the existing chore(release) codex-generated bumps and OfficeCLI dependabot-style PRs (PR chore: bump bundled OfficeCLI to v1.0.93 #730, ci: harden OfficeCLI bump PR creation #735, chore(release): bump desktop version to v2026.5.19 #742 patterns). If pawwork's release automation generates PR bodies, that automation may need to be updated to emit the new string.
  • Sanity-check the Task template's Execution mode wording does not break existing agent task-issue workflows.
  • The labeler ci rule now covers all .github/** paths. Workflow PRs still get both ci (routing) and task (auto-applied type). Template/issue-form PRs now get just ci, with the author expected to add the type label per the new template policy.

Risk Notes

  • Existing open PRs that already used the old template are unaffected; the template only applies to new PR descriptions opened after merge.
  • The "Not required: " path replaces the older loose "or not required" escape hatch. Bot-generated release PRs must now include this string.
  • The labeler ci rule expansion: a PR that touches both .github/** and a packaged source area will now get both ci and the source-area routing label. This is the intended "multi-area PR" behavior already supported by label policy (it requires at least one routing label, not exactly one).
  • No platform, migration, dependency, credential, generated-file, or deletion-behavior impact.

How To Verify

YAML parse (03-task.yml): ok
  python3 -c "import yaml; yaml.safe_load(open('.github/ISSUE_TEMPLATE/03-task.yml'))"

YAML parse (labeler.yml): ok
  python3 -c "import yaml; yaml.safe_load(open('.github/labeler.yml'))"

Crosscheck convergence (templates): 5 rounds (Claude Opus + Codex). Round 5 both reviewers returned None for code findings and None for design alternatives.

Live verify (labeler rule): first pr-triage run on this PR failed with no routing label, confirming the bug; after pushing the labeler.yml fix, the synchronize-triggered pr-triage run should pass with ci + documentation + P2 labels.

Diff scope: 3 files. No workflow, policy-script, or other config changes.

Screenshots or Recordings

Not required; pure infra/text changes with no visible UI surface.

Follow-up (out of scope for this PR)

Two design-level improvements were identified by crosscheck and deferred:

  1. CI lint that diffs PR-body checklist against the template (mechanically enforces the immutability rule that this PR can only request socially).
  2. Issue-template restructure: replace prose area dropdowns in 01-bug-report.yml and 02-feature-request.yml with routing-label strings (app — product behavior etc.), add an area dropdown to 03-task.yml, and drop the "Not sure" default that becomes a silent routing-signal kill.

A third inconsistency was noticed but left in place: the labeler task rule still auto-applies the task type label for .github/workflows/** changes, even though the new PR template states type labels are author-applied. Removing this auto-application would change the long-standing CI PR workflow and was kept as a separate decision.

Checklist

  • Human review status is stated above as pending, approved, or not required
  • I linked the related issue, or stated why there is no issue
  • This PR has exactly one type label (bug, enhancement, task, or documentation), at least one primary routing label (app, ui, platform, harness, or ci), and exactly one priority label (P0 to P3), or I requested maintainer labeling
  • I described the review focus and any meaningful risks
  • I listed the relevant verification steps and the key result for each
  • I did not introduce unrelated refactors, dependencies, generated files, or file changes beyond the stated scope
  • I manually checked visible UI or copy changes when needed, with screenshots or recordings
  • I considered macOS and Windows impact for platform, packaging, updater, signing, paths, shell, or permissions changes
  • I called out docs, release notes, dependencies, permissions, credentials, deletion behavior, generated content, or local file changes when relevant
  • I reviewed the final diff for unrelated changes and suspicious dependency changes
  • I am targeting dev, and my PR title and commit messages use Conventional Commits in English

Summary by CodeRabbit

  • Chores
    • Updated Task issue template execution mode options with clearer, instruction-based labels
    • Expanded GitHub Actions labeler configuration to include additional file patterns
    • Enhanced pull request template with improved checklist formatting and human review status guidelines

Review Change Stack

PR #748 surfaced a template ambiguity: the labeler bot auto-applied
routing/priority labels but not the required type label, the agent
rewrote the checklist text to "Label bot should apply type/routing/
priority labels; no manual labels were specified per maintainer
instruction" instead of adding the missing `bug` label, and the human
maintainer had to add `bug` minutes later before validation ran.

Five rounds of crosscheck (Claude Opus + Codex) found and addressed:
- Label checklist conflated bot-applied vs author-applied labels and
  had an "or I requested maintainer labeling" escape hatch.
- All checklist items were first-person past-tense assertions with no
  explicit immutability guard; the agent treated the lines as editable.
- Human Review Status was a free-text contract with a loose
  "or not required" escape hatch.
- How To Verify example block had no machine-spottable marker; an
  agent could leave the example verbatim.
- Task template Execution mode options had overlapping prose and
  uneven parallelism.

The new PR template splits labels into three explicit author/bot rows,
adds a top-of-file policy comment plus a "How to use this checklist"
blockquote, converts Human Review Status into a strict three-option
enum, adds a `replace-before-submit` HTML sentinel to the example
block, and tags genuinely conditional items with **(conditional)**.

The new Task template Execution mode rewrites each option as an
explicit declarative directive ("the agent must not... until..." /
"the agent must post the plan as an issue comment..." / "the agent
makes the requested changes... must not push directly to dev").

Round-5 crosscheck returned "None" for both code findings and design
alternatives from both reviewers. Remaining design-level work (CI
lint enforcement, issue-template `area` dropdown restructure) is
deferred to a follow-up PR.
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 19, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 36cd6b1b-6113-4558-a342-54549eebb992

📥 Commits

Reviewing files that changed from the base of the PR and between e547fbc and 3c30cd8.

📒 Files selected for processing (3)
  • .github/ISSUE_TEMPLATE/03-task.yml
  • .github/labeler.yml
  • .github/pull_request_template.md

📝 Walkthrough

Walkthrough

Updated GitHub issue and pull request templates to clarify execution modes, checklist policies, and human review status requirements. Task template now provides explicit agent workflow instructions. Pull request template adds a checklist policy statement and reformatted status guidelines. CI labeler expanded to track .github/** changes.

Changes

GitHub Workflow Configuration Updates

Layer / File(s) Summary
Task Execution Mode Workflow
.github/ISSUE_TEMPLATE/03-task.yml
Task template's "Execution mode" dropdown replaced with explicit agent workflow descriptions clarifying code-writing authority, PR opening permissions, and approval requirements across multiple modes.
Pull Request Template Checklist and Review Policy
.github/pull_request_template.md
Added top-level checklist policy; replaced generic "Human Review Status" guidance with explicit status line format requirements; reworked the checklist section with usage guidance and constrained, conditional items including label and status confirmation.
CI Labeler Configuration
.github/labeler.yml
Expanded ci labeler's changed-files patterns to include .github/** glob, ensuring GitHub template and workflow edits are properly routed.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

  • Astro-Han/pawwork#345: Both PRs update the GitHub issue/pull-request templates—most directly the shared .github/pull_request_template.md checklist/human-review guidance.
  • Astro-Han/pawwork#368: The main PR updates the existing Task issue template's execution_mode dropdown text and option labels, building on the retrieved PR that introduced the same form.
  • Astro-Han/pawwork#717: Both PRs modify .github/labeler.yml to change how GitHub auto-applies labels based on .github/**/workflow file paths.

Suggested labels

task

Poem

🐰 A rabbit hops through GitHub's halls,
Updating templates, heeding the calls,
"Tell agents clear what they can do,"
Task modes now shine, checklist anew,
Labels and routes in .github flow true! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: tightening PR and task templates to prevent agent misuse, which aligns directly with the primary objectives of the changeset.
Description check ✅ Passed The description comprehensively covers all required template sections: Summary, Why, Related Issue, Human Review Status, Review Focus, Risk Notes, How To Verify, Screenshots/Recordings, and Checklist with all items completed.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/improve-templates

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added the P2 Medium priority label May 19, 2026
Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested priority: P2 (includes non-doc, non-test paths outside the low-risk bucket).

P1/P0 are reserved for maintainer confirmation. Please relabel manually if this is a release blocker, security issue, data-loss risk, or updater/runtime failure.

@Astro-Han Astro-Han added documentation Improvements or additions to documentation ci Continuous integration / GitHub Actions labels May 19, 2026
@Astro-Han Astro-Han closed this May 19, 2026
@Astro-Han Astro-Han reopened this May 19, 2026
@github-actions github-actions Bot removed the ci Continuous integration / GitHub Actions label May 19, 2026
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the task issue template and the pull request template to provide clearer instructions on execution modes and a more detailed, structured checklist. Key changes include defining explicit human review statuses, adding verification examples, and categorizing checklist items into required and conditional sections. Feedback from the review suggests formatting improvements such as using backticks for labels and statuses, adding explicit checklist items for target branches and conventional commits, and ensuring the 'Related Issue' section is used for explaining missing issue links.

Comment thread .github/pull_request_template.md
Comment thread .github/pull_request_template.md
Comment thread .github/pull_request_template.md
Templates and issue forms under .github/ITS_TEMPLATE/ and the top-level
PR template are CI/process infrastructure that should route the same
way as workflow files. Without this, a PR that only touches
.github/pull_request_template.md or .github/ISSUE_TEMPLATE/*.yml
receives no routing label from the labeler bot, and the manual
override is stripped by sync-labels on the next pr-triage run.

The task rule keeps the narrower .github/workflows/** scope: task is
a type label, and per the updated PR template, type labels are
author-applied, not bot-applied. Workflow PRs retain the existing
task auto-application as a historical convenience; this can be
revisited in a follow-up if the inconsistency proves confusing.
@Astro-Han Astro-Han added the ci Continuous integration / GitHub Actions label May 19, 2026
Restore the explicit .github/workflows/** glob alongside the broader
.github/** glob. The pr-triage contract test at
packages/opencode/test/github/pr-triage-workflow.test.ts:57 uses
literal-string matching to assert that workflow files are routed to
the ci label, so dropping the workflows glob broke the test even
though .github/** subsumes it semantically.

Keeping both globs preserves the test contract and documents the
intent that workflow files are first-class CI infra while other
.github/ files (templates, labeler.yml itself, dependabot config)
are CI process.
@github-actions github-actions Bot removed the ci Continuous integration / GitHub Actions label May 19, 2026
@Astro-Han Astro-Han added the ci Continuous integration / GitHub Actions label May 19, 2026
@Astro-Han
Copy link
Copy Markdown
Owner Author

@Reviewer Thanks for the catch on the contract test — P1 is real and now fixed.

P1 fix (commit 3c30cd8): Restored .github/workflows/** as an explicit glob in the ci rule alongside the broader .github/**. The contract test at packages/opencode/test/github/pr-triage-workflow.test.ts:57 does literal-string matching (labelerLabelsForGlob returns labels whose glob list contains the exact string .github/workflows/**), so collapsing the two globs broke the test even though .github/** subsumes the workflows path semantically. Both globs are kept now — workflows match either, non-workflow .github/ files match only the broader one. CI re-run on the new head: all unit jobs green, including unit-opencode (run 26079323054).

Factual note on the "待验证" section: the review describes the diff as adding !.github/workflows/dependabot-auto-merge.yml as an exclusion under the ci rule. That exclusion is not in this PR's diff — git diff dev..HEAD -- .github/labeler.yml is a single-line change (and now, with the followup commit, a two-line change to add the workflows glob back). No exclusion rules were added or removed. If the reviewer wants the dependabot auto-merge workflow excluded from ci routing, that's a reasonable follow-up but out of scope here.

Remaining pr-triage red status: this is a chicken-and-egg of pull_request_target — labeler runs with the base SHA's (dev's) labeler.yml, which still has only .github/workflows/** under ci. Combined with sync-labels: true, any manual ci label gets stripped on the next pr-triage run. The PR is unable to fix its own pr-triage; once merged, future template-only PRs route correctly. I've re-added the ci label manually so the policy state is correct on the issue/PR list, even though the check itself stays red.

@Astro-Han
Copy link
Copy Markdown
Owner Author

Resolving the three Gemini review threads — they're auto-generated polish suggestions that don't apply to the current diff:

  • L57 'labels should be in backticks': the labels are already in backticks (bug, enhancement, ..., P0-P3, app, etc.). The suggested 'external contributors who can't add labels' escape hatch is the same 'or I requested maintainer labeling' pattern that PR fix(session): clear stale pending interactions #748 exploited; intentionally removed. The dev-target + Conventional Commits item is the last checklist row (L67).
  • L58 'status values should be in backticks': they already are (Pending, Approved by @<reviewer>, Not required: <reason>).
  • L59 'state why there is no issue in Related Issue section, not Summary': pure stylistic preference, both placements are fine. Keeping current wording.

@Astro-Han Astro-Han merged commit a09bca0 into dev May 19, 2026
26 of 27 checks passed
@Astro-Han Astro-Han deleted the claude/improve-templates branch June 2, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Continuous integration / GitHub Actions documentation Improvements or additions to documentation P2 Medium priority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant