Add manual acceptance tests workflow and PR template#40
Merged
Conversation
Copilot created this pull request from a session on behalf of
pmcelhaney
June 8, 2026 23:22
View session
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+36
to
+102
| name: Validate manual acceptance tests | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Decide whether to enforce | ||
| id: decide | ||
| env: | ||
| PR_BRANCH: ${{ github.event.pull_request.head.ref }} | ||
| PR_BODY: ${{ github.event.pull_request.body }} | ||
| PR_LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }} | ||
| run: | | ||
| if python3 - <<'PY' | ||
| import json, os, sys | ||
| try: | ||
| labels = json.loads(os.environ.get('PR_LABELS', '[]')) | ||
| sys.exit(0 if 'design' in labels else 1) | ||
| except Exception as e: | ||
| print(f'Error parsing PR labels: {e}', file=sys.stderr) | ||
| sys.exit(1) | ||
| PY | ||
| then | ||
| echo "enforce=false" >> "$GITHUB_OUTPUT" | ||
| elif [[ "$PR_BRANCH" == copilot* ]] || grep -iq "## manual acceptance tests" <<<"$PR_BODY"; then | ||
| echo "enforce=true" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "enforce=false" >> "$GITHUB_OUTPUT" | ||
| fi | ||
| - name: Skip (no enforcement needed) | ||
| if: steps.decide.outputs.enforce == 'false' | ||
| run: echo "Skipping manual acceptance test validation (design PR, not a Copilot PR, or no section present)" | ||
|
|
||
| - name: Validate PR body | ||
| if: steps.decide.outputs.enforce == 'true' | ||
| env: | ||
| PR_BODY: ${{ github.event.pull_request.body }} | ||
| run: | | ||
| python <<'PY' | ||
| import os | ||
| import re | ||
| import sys | ||
|
|
||
| body = os.environ.get("PR_BODY") or "" | ||
|
|
||
| match = re.search( | ||
| r"(?ims)^##\s*manual acceptance tests\s*\n(.*?)(?=^##\s|\Z)", | ||
| body, | ||
| ) | ||
|
|
||
| if not match: | ||
| print('Missing required section: "## Manual acceptance tests"') | ||
| sys.exit(1) | ||
|
|
||
| section = match.group(1) | ||
|
|
||
| checked = re.findall(r"(?mi)^\s*-\s\[x\]\s+.+$", section) | ||
| unchecked = re.findall(r"(?m)^\s*-\s\[\s\]\s+.+$", section) | ||
|
|
||
| if unchecked: | ||
| print(f"Found {len(unchecked)} unchecked manual acceptance test box(es). All boxes must be checked before merge.") | ||
| sys.exit(1) | ||
|
|
||
| if len(checked) < 1: | ||
| print("Expected at least one checked manual acceptance test box.") | ||
| sys.exit(1) | ||
|
|
||
| print(f"OK: found {len(checked)} checked manual acceptance test box(es) and no unchecked boxes.") | ||
| PY |
pmcelhaney
approved these changes
Jun 8, 2026
There was a problem hiding this comment.
Pull request overview
Adds repository-level guardrails for Copilot-authored PRs by requiring a “Manual acceptance tests” checklist to be completed before merge, along with supporting documentation and a PR template.
Changes:
- Added a GitHub Actions workflow to validate the “## Manual acceptance tests” section on Copilot PRs (with a
designlabel exemption). - Added a pull request template that includes the required acceptance-test section placeholder.
- Updated
.github/copilot-instructions.mdto describe the manual acceptance test expectations.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| .github/workflows/manual-acceptance-tests.yaml | Adds PR-body validation workflow for the manual acceptance test checklist. |
| .github/pull_request_template.md | Introduces a PR template that includes the required acceptance tests section. |
| .github/copilot-instructions.md | Documents the manual acceptance test checklist requirement for PR descriptions. |
Comment on lines
+28
to
+33
| name: Manual acceptance tests | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [opened, edited, synchronize, reopened] | ||
|
|
| env: | ||
| PR_BODY: ${{ github.event.pull_request.body }} | ||
| run: | | ||
| python <<'PY' |
Comment on lines
+45
to
+46
| - Cover the main success path, at least one edge case, and one regression check where applicable. | ||
| - Exception: if a PR only adds files under `.github/issue-proposals/`, this section may be omitted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the manual acceptance tests enforcement from
counterfact/api-simulator: a CI workflow that blocks merge on Copilot PRs until all acceptance test checkboxes are checked, a PR template with the required section, and matching instructions incopilot-instructions.md.Original Prompt
The counterfact/api-simulator repo has a workflow that tells Copilot to add manual acceptance tests to a pull request. It uses copilot-instructions.md, a pull request template, and a CI workflow. Copy those elements to this repo and create a PR.
Manual acceptance tests
copilot/*branch shows the PR template with the## Manual acceptance testsplaceholder sectioncopilotand has no## Manual acceptance testssection in the body- [ ]) in the section- [x]) with none uncheckeddesignlabel passes the check regardless of branch name or body contentTasks
## Manual acceptance testssection to.github/copilot-instructions.mdclarifying that Copilot writes unchecked boxes and the reviewer checks each after verifying behavior.github/pull_request_template.mdwith Summary, Original Prompt, Manual acceptance tests, and Tasks sections.github/workflows/manual-acceptance-tests.yamlenforcing the checklist oncopilot/*branches or any PR containing the section;design-labelled PRs are exempt