chore: add mutation testing with Stryker (per-package CI gates) by iliassjabali · Pull Request #49 · agents-oss/agentspec

iliassjabali · 2026-04-12T05:58:33Z

Summary

Adds mutation testing and coverage reporting across all 5 packages, with three layers of feedback on every PR.

What you see on PRs

1. Quality Report comment -- one sticky comment showing coverage % and mutation score per package, with deltas vs main:

Package	Line Coverage	Mutation Score
adapter-claude	83.9% (+1.2)	50.8% (+0.5)
sdk	67.2%	54.3% (-0.3)
...	...	...

2. Inline annotations -- surviving mutants appear as warnings directly on the PR diff, on the exact line Stryker mutated. No need to download the HTML report for the common case.

3. Mutation gate -- merge is blocked if any package's mutation score drops below its threshold.

What happens post-merge

quality-baseline.json is updated with the latest scores (used for deltas on future PRs).
If a package's score exceeds its threshold by >5 points, a bot PR is opened to bump the threshold automatically.

What was added (no production code changes)

stryker.config.mjs -- single root config with per-package threshold map
scripts/mutation.mjs -- wrapper: pnpm mutation adapter-claude or pnpm mutation for all
scripts/annotate-survivors.mjs -- emits GitHub warning annotations for surviving mutants
scripts/auto-ratchet.mjs -- bumps thresholds when scores improve by >5 points
quality-baseline.json -- last-known scores on main
.github/workflows/mutation.yml -- Quality workflow (coverage + mutation + PR comment)
.github/workflows/auto-ratchet.yml -- post-merge baseline update + threshold ratchet
packages/*/vitest.config.ts -- explicit configs for packages that lacked them
packages/cli/vitest.stryker.config.ts -- excludes cli.test.ts (spawns dist binary, can't see mutations)
docs/guides/mutation-testing.md -- user guide

Pilot scores (local run on main)

Package	Threshold	Pilot score
adapter-claude	45	49.57%
sdk	49	54.31%
mcp-server	26	31.31%
cli	45	49.50%
sidecar	46	51.43%

Test plan

Local pilot per package
Gate mechanism verified (score < threshold exits non-zero)
CI: all 5 quality jobs produce scores and a unified PR comment
CI: inline annotations appear on PR diff for surviving mutants
Post-merge: baseline update and auto-ratchet trigger

Stryker CLI does not support nested flags like --thresholds.break or --vitest.configFile. Move all per-package overrides into stryker.config.mjs which reads STRYKER_PKG and STRYKER_BREAK env vars. The npm scripts set these env vars instead of passing CLI flags.

- docs/mutation-testing.md -> docs/guides/mutation-testing.md - docs/superpowers/specs/... -> docs/decisions/2026-04-12-mutation-testing.md - docs/superpowers/plans/... -> docs/decisions/2026-04-12-mutation-testing-plan.md - Remove docs/superpowers/ (skill framework artifact, not project structure)

Thresholds now live in the THRESHOLDS map in stryker.config.mjs. scripts/mutation.mjs is a thin wrapper that takes a package name from argv, sets STRYKER_PKG, and execs stryker run. Usage: pnpm mutation adapter-claude (or pnpm mutation for all).

Adds a coverage matrix job to ci.yml that runs vitest --coverage per package on PRs and uses vitest-coverage-report-action to post a sticky comment with overall %, per-file breakdown, and diff coverage on changed lines. Requires pull-requests: write permission.

github-actions · 2026-04-12T06:30:45Z

Coverage Report for sdk

Status	Category	Percentage	Covered / Total
🔵	Lines	95.59%	2627 / 2748
🔵	Statements	95.59%	2627 / 2748
🔵	Functions	96.11%	99 / 103
🔵	Branches	87.52%	540 / 617

File Coverage

No changed files found.

Generated in workflow #113 for commit 7ff1b1a by the Vitest Coverage Report Action

github-actions · 2026-04-12T06:30:47Z

Coverage Report for adapter-claude

Status	Category	Percentage	Covered / Total
🔵	Lines	83.93%	162 / 193
🔵	Statements	83.93%	162 / 193
🔵	Functions	90%	9 / 10
🔵	Branches	88.23%	45 / 51

File Coverage

No changed files found.

Generated in workflow #113 for commit 7ff1b1a by the Vitest Coverage Report Action

github-actions · 2026-04-12T06:30:55Z

Coverage Report for sidecar

Status	Category	Percentage	Covered / Total
🔵	Lines	78.98%	1248 / 1580
🔵	Statements	78.98%	1248 / 1580
🔵	Functions	85.24%	52 / 61
🔵	Branches	80.56%	344 / 427

File Coverage

No changed files found.

Generated in workflow #113 for commit 7ff1b1a by the Vitest Coverage Report Action

github-actions · 2026-04-12T06:30:56Z

Coverage Report for cli

Status	Category	Percentage	Covered / Total
🔵	Lines	85.44%	2255 / 2639
🔵	Statements	85.44%	2255 / 2639
🔵	Functions	86.15%	112 / 130
🔵	Branches	81.84%	613 / 749

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/cli/vitest.stryker.config.ts	0%	0%	0%	0%	1-14

Generated in workflow #113 for commit 7ff1b1a by the Vitest Coverage Report Action

…utput

…mment Removes the separate coverage matrix from ci.yml. The mutation.yml workflow (renamed to Quality) now runs both vitest --coverage and Stryker per package in one job, then a final comment job posts a single sticky PR comment with a table of coverage % and mutation % per package.

github-actions · 2026-04-12T06:48:00Z

Quality Report

Package	Line Coverage	Mutation Score
adapter-claude	83.93%	50.00%
cli	85.44%	49.40%
mcp-server	N/A	31.31%
sdk	95.59%	53.40%
sidecar	78.98%	51.27%

Coverage = % of lines your tests touch. Mutation = % of code changes your tests catch.
Deltas shown vs main baseline. Mutation reports available as workflow artifacts.

…NSI output

- Surviving mutants now appear as warning annotations inline on the PR diff, on the exact line Stryker mutated. - PR comment shows deltas vs main baseline (e.g. 54.3% (+2.1)). Baseline stored in quality-baseline.json, updated post-merge. - Auto-ratchet workflow runs on push to main. If a package's mutation score exceeds its threshold by >5 points, it opens a PR to bump the threshold in stryker.config.mjs.

Copilot

Pull request overview

Adds mutation testing (Stryker) and PR-facing quality reporting across the monorepo, with per-package configuration and CI workflows intended to annotate survivors and gate merges.

Changes:

Introduces a root Stryker config + scripts to run mutation tests per package, annotate surviving mutants, and auto-ratchet thresholds.
Adds CI workflows to run coverage + mutation per package and post a sticky “Quality Report” comment on PRs.
Adds/standardizes Vitest configs for packages that lacked explicit configuration, plus docs and baseline score storage.

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`stryker.config.mjs`	Root Stryker config with per-package thresholds and Vitest runner wiring.
`scripts/mutation.mjs`	CLI wrapper to run mutation testing sequentially per package.
`scripts/auto-ratchet.mjs`	Parses reports and bumps break thresholds when scores improve enough.
`scripts/annotate-survivors.mjs`	Emits GitHub Actions warning annotations for surviving mutants.
`quality-baseline.json`	Stores baseline coverage/mutation scores used for deltas in PR comments.
`package.json`	Adds Stryker dependencies and a `pnpm mutation` script entrypoint.
`pnpm-lock.yaml`	Locks Stryker and transitive dependencies.
`packages/adapter-claude/vitest.config.ts`	Adds explicit Vitest config for adapter-claude.
`packages/cli/vitest.config.ts`	Adds explicit Vitest config for cli.
`packages/cli/vitest.stryker.config.ts`	Adjusts cli tests under Stryker (excludes `cli.test.ts`).
`packages/mcp-server/vitest.config.ts`	Adds explicit Vitest config for mcp-server (incl. timeout).
`docs/guides/mutation-testing.md`	Documents running mutation tests, CI behavior, and thresholds.
`.gitignore`	Ignores Stryker temp dir and mutation reports.
`.github/workflows/mutation.yml`	PR workflow for coverage + mutation runs, annotations, and PR comment.
`.github/workflows/auto-ratchet.yml`	Post-merge workflow to update baseline and open ratchet PRs.
`.github/workflows/ci.yml`	Minor comment punctuation change.

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/mutation.yml

.github/workflows/auto-ratchet.yml

scripts/annotate-survivors.mjs

.github/workflows/mutation.yml

1. mutation.yml: Stryker || true was swallowing the gate exit code. Now captures pass/fail, runs all post-steps with if: always(), then fails the job at the end via an 'Enforce mutation threshold' step. 2. auto-ratchet.yml: no longer pushes directly to main. Both baseline- only and threshold-bump updates now go through a PR. 3. auto-ratchet.yml: branch name includes short SHA to avoid collision when two merges happen on the same day. 4. auto-ratchet.yml: imports PACKAGES from stryker.config.mjs instead of duplicating the package list inline. 5. stryker.config.mjs: replaced em dash with double hyphen in comment.

1. mutation.yml: replace || true on coverage step with continue-on-error so test failures are visible in the UI but don't block post-steps. 2. mutation.yml: add issues: write permission for the PR comment API calls (listComments/createComment/updateComment). 3. mutation.yml: skip PR comment job on fork PRs where GITHUB_TOKEN is read-only (guard via head.repo.full_name == github.repository). 4. auto-ratchet.yml: merge with existing baseline instead of overwriting. Coverage fields are preserved from the previous baseline when no new coverage file is available (this workflow runs mutation, not coverage). 5. annotate-survivors.mjs: escape %, newlines, colons, and commas per GitHub Actions annotation spec to prevent truncated/broken output.

iliassjabali added 20 commits April 12, 2026 01:59

chore: ignore .worktrees/ directory

d941b66

docs: add mutation testing design spec

9e3e1d7

docs: add mutation testing implementation plan

6740326

chore: add stryker mutation testing dev deps

89c50ba

chore: add explicit vitest configs to adapter-claude, mcp-server, cli

c09f010

chore(cli): add stryker-only vitest config that excludes cli.test.ts

54cf8f4

chore: add root stryker config with shared defaults

bc4dadd

chore: add per-package mutation scripts to root package.json

fffbb99

chore: ignore .stryker-tmp and reports/mutation directories

8c96bca

fix: add vitest dir option to stryker config for correct test discovery

19e5af5

chore: set mutation thresholds from pilot scores (actual - 5)

dc73429

docs: record pilot mutation scores in design spec

70ba662

ci: add mutation testing workflow with per-package matrix

5b70cc5

docs: add mutation testing user guide

6973315

fix(ci): specify pnpm version 10 in mutation workflow

41f85b1

docs: simplify mutation testing to a single guide, remove decision docs

270801f

iliassjabali added 2 commits April 12, 2026 07:39

fix(ci): tolerate test failures in coverage step, skip report if no o…

7ff1b1a

…utput

iliassjabali added 2 commits April 12, 2026 07:57

fix(ci): extract mutation score from JSON report instead of parsing A…

ba4fb43

…NSI output

iliassjabali requested a review from Copilot April 12, 2026 07:28

iliassjabali requested a review from skokaina April 12, 2026 07:29

Copilot started reviewing on behalf of iliassjabali April 12, 2026 07:29 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

iliassjabali added 3 commits April 12, 2026 13:48

Merge branch 'main' into chore/mutation-testing

cfc9a46

iliassjabali self-assigned this Apr 12, 2026

Merge branch 'main' into chore/mutation-testing

708019e

iliassjabali removed their assignment Apr 12, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add mutation testing with Stryker (per-package CI gates)#49

chore: add mutation testing with Stryker (per-package CI gates)#49
iliassjabali wants to merge 28 commits intomainfrom
chore/mutation-testing

iliassjabali commented Apr 12, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 12, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 12, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 12, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 12, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

iliassjabali commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What you see on PRs

What happens post-merge

What was added (no production code changes)

Pilot scores (local run on main)

Test plan

Uh oh!

github-actions bot commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for sdk

Uh oh!

github-actions bot commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for adapter-claude

Uh oh!

github-actions bot commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for sidecar

Uh oh!

github-actions bot commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for cli

Uh oh!

github-actions bot commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Quality Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

iliassjabali commented Apr 12, 2026 •

edited

Loading

github-actions bot commented Apr 12, 2026 •

edited

Loading

github-actions bot commented Apr 12, 2026 •

edited

Loading

github-actions bot commented Apr 12, 2026 •

edited

Loading

github-actions bot commented Apr 12, 2026 •

edited

Loading

github-actions bot commented Apr 12, 2026 •

edited

Loading