Gate pull requests on the mutation score of what they changed - #316
Merged
Conversation
Reefact
force-pushed
the
claude/automated-mutation-testing-4okfe0
branch
from
July 27, 2026 13:02
ec778c8 to
bb24ecb
Compare
Records the decision to require a mutation-score threshold on every pull request, measured over the mutants of the files it changed, for the five libraries the repository ships — enforced by two independent gates, split along the boundary JustDummies is expected to leave through. The context section holds what measurement established before the decision: Stryker's default VSTest runner reports every mutant as survived on this xUnit v3 test bed, its Microsoft Testing Platform runner works but is preview, that runner's coverage-based selection misclassifies killed mutants as uncovered, and the cost is one test-suite run per mutant. That cost is also why one library ships without a bar: JustDummies' sweep runs well past an hour, no score was ever measured for it, and the follow-up actions name the weekly sweep as what must supply one. Drafted as Proposed; the maintainer decides whether it is accepted.
Adds two mutation workflows running Stryker.NET: `mutation` over the three FirstClassErrors libraries, `justdummies-mutation` over the two JustDummies packages. On a pull request each matrix leg mutates only the files the diff touched — `--since` against the fork point resolved with `git merge-base` — and fails when the score falls under that library's threshold. Each workflow aggregates its matrix into one stable check name to mark as required on `main`; a weekly schedule sweeps every mutant, advisory by construction. Both paths render the surviving mutants into the run summary. The two workflows are deliberate near-duplicates rather than one shared matrix. JustDummies already holds no reference to FirstClassErrors and is headed for a repository of its own, so the split keeps that move a file move, and it lets the two quality bars sit at different heights. Two settings are load-bearing rather than tuning, and both were established by measurement. `test-runner: mtp` is required: Stryker's default VSTest runner cannot activate a mutant in an xUnit v3 test project, so it runs to completion and reports every mutant as survived — including mutants that break the suite when the same edit is applied by hand. `coverage-analysis: off` is for accuracy rather than speed: the Microsoft Testing Platform runner's coverage selection still classifies killed mutants as uncovered, which understates the score. Four of the five thresholds are derived from that library's measured full-sweep score, so those gates start green and only ever ratchet up. JustDummies is the exception: a few thousand mutants over a heavy suite put its sweep well past an hour and no score was measured for it, so rather than invent a bar its score gate is left off — its leg still runs, still fails on a broken build or a failing suite, and still lists its survivors, but does not yet refuse a pull request over a score. The weekly sweep is what supplies the missing figure. The engine is pinned in a tool manifest: a newer Stryker invents new mutants and would move every score on its own. See ADR-0043 for the decision.
Adds the bilingual reference pages for `mutation` and `justdummies-mutation`, indexes both in the workflow reference, and points CLAUDE.md's build & test section at them. The `mutation` page carries the shared mechanics; the JustDummies page carries what is specific to it and defers to the other rather than duplicating prose, plus the checklist for the day JustDummies moves to its own repository and the reason its own score gate is still off. Between them the pages record what measurement established, and what a reader would otherwise re-discover the hard way: why the Microsoft Testing Platform runner is mandatory rather than preferred, why coverage-based selection is off, that `--since` selects per changed file and rejects `HEAD` as a target, that the CI warning ratchet turns out not to reach Stryker's mutant compilation, where the per-library thresholds come from, and how to answer an equivalent mutant with a `// Stryker disable` comment rather than a lowered threshold.
Reefact
force-pushed
the
claude/automated-mutation-testing-4okfe0
branch
from
July 27, 2026 13:07
bb24ecb to
6db70e4
Compare
The scope excluded the `fce` command line, the documentation generator and the Roslyn analyzers on a cost argument: their suites drive Roslyn compilations and snapshot comparisons, so their mutants are the expensive ones. That argument does not survive contact with the weekly sweep, which is the run allowed to take as long as it takes — and the exclusion left a hole rather than a boundary, since a pull request touching only the analyzers crossed no mutation gate at all. Measured before making the change: Stryker runs on all three under the MTP runner, snapshot suites and process-spawning tests included, and they carry 3744 mutants between them — about as many again as the FirstClassErrors libraries already in scope. They ship with their score gate disabled, like JustDummies: their sweeps are too long to have been run interactively, so no score was measured and none was guessed. Their legs still run, still fail on a broken build or a failing suite, and still list survivors; the first weekly sweep supplies the four missing figures. What stays out now stays out for a reason other than cost: the `Usage` samples and the binder benchmarks are not shipped behaviour, and `GenDoc.Worker` is a process entry point no test exercises in process — the `floor` job of `ci` proves it end to end, and mutating it would only manufacture unkillable survivors. Both sweep caps go from 180 to 350 minutes, just under the six-hour ceiling GitHub imposes on a job. The weekly run is meant to take as long as it takes rather than be cut short: JustDummies alone runs well past an hour on four cores. See ADR-0043 for the decision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds automated mutation testing with Stryker.NET, as two workflows with two required checks:
mutationfor the FirstClassErrors libraries and tooling,justdummies-mutationfor the JustDummies packages. On a pull request each one mutates only the files the diff touched and fails under that project's threshold; a weekly schedule sweeps everything, advisory.The split into two workflows is deliberate: JustDummies is headed for a repository of its own, and this keeps that move a file move rather than a rewrite.
Type of change
Changes
.github/workflows/mutation.yml— mutation gate over every FirstClassErrors project whose code ships or runs: the three libraries plus the analyzers, the documentation generator and thefcecommand line. Required check:Mutation gate..github/workflows/justdummies-mutation.yml— the same forJustDummiesandJustDummies.Xunit. Required check:JustDummies mutation gate. Deliberate near-duplicate of the above; adiffbetween them shows only the cron slot, the matrix entries, the gate name and its error message.build/stryker/*.json— one Stryker configuration per project: project, test projects, runner mode, thresholds..config/dotnet-tools.json— pinsdotnet-stryker4.16.0, so CI and a local run use the same mutant generator..gitignore— ignoreStrykerOutput/from local runs.CLAUDE.mdpoints at them.Proposed) records the decision.Out of scope, for reasons other than cost: the
Usagesamples and the binder benchmarks (not shipped behaviour), andGenDoc.Worker— a process entry point no test exercises in process, proven end to end by thefloorjob ofci; mutating it would only manufacture survivors no test could kill.Two findings worth reading before reviewing the config
errorCode == null→!=inError.CreateSafeCode) fails 7 tests. Hence"test-runner": "mtp", which Stryker flags as preview."coverage-analysis": "off"is for accuracy, not speed. Under the MTP runner, coverage-based selection still classifies killed mutants as uncovered. Measured onError.cs: 75 % with selection on, 100 % with it off.Thresholds
Each bar was set from that project's measured full sweep, so those gates start green and only ratchet up. Four projects have no bar: their sweeps are too long to have been run interactively, so no score was measured and none was guessed.
breakFirstClassErrorsFirstClassErrors.RequestBinderJustDummies.XunitFirstClassErrors.TestingJustDummiesFirstClassErrors.GenDocFirstClassErrors.AnalyzersFirstClassErrors.CliThe four ungated legs still run, still fail on a broken build or a failing suite, and still list their survivors — they simply do not yet refuse a pull request over a score. The first weekly sweep supplies the missing figures; ADR-0043's follow-up and the workflow pages say so.
The two low scores are a real finding, not noise —
FirstClassErrors.TestingandJustDummies.Xunithave behaviours nothing asserts (for instance, invertingscope is not nullandTestState?.Result != FailedinReproducibleAttributeboth survive).Both sweep caps are 350 minutes, just under GitHub's six-hour job ceiling: the weekly run is meant to take as long as it takes.
Testing
dotnet build FirstClassErrors.slndotnet test FirstClassErrors.slnFirstClassErrors.Analyzers.UnitTests)No source code changes, so the suites were not run as such. They were exercised indirectly and repeatedly: every Stryker run performs an initial test run and aborts if it fails, and full sweeps of
FirstClassErrors,FirstClassErrors.Testing,FirstClassErrors.RequestBinderandJustDummies.Xunitall completed on this branch.What was verified directly, locally:
FirstClassErrors.Testing/Clock.csselects exactly its 4 mutants, scores 75 %, and exits 2 against the 80 threshold in force at the time;--sinceaccepts a commit SHA and rejectsHEAD, which is why the workflow resolvesgit merge-basefirst;GITHUB_ACTIONS=trueand unset, so no escape hatch was added toDirectory.Build.props;jqstep against real reports, in all three branches (nothing selected / all killed / survivors listed).On a GitHub runner, the previous revision's five legs all passed and the
--sincefilter behaved as designed (985 mutants Ignored. Reason: Removed by since filter,0 total mutants will be tested, exit 0, ~1 min 25 per leg). The three tooling legs added since have not yet run there.Documentation
doc/updateddoc/handwritten/for-users/README.fr.md) updated if user-facing behavior changedBoth new reference pages are bilingual (EN canonical + FR), indexed in the workflow reference in both languages. No user-facing library behaviour changed, so the user documentation is untouched.
Architecture decisions
Proposed: ADR-0043Related issues
None.
One thing needing your call before merge: branch protection. The workflows cannot make themselves mandatory —
Mutation gateandJustDummies mutation gatehave to be marked required onmain.