Skip to content

test: add PIT mutation testing with per-module score gates - #84

Merged
emaarco merged 4 commits into
mainfrom
emaarco/add-pitest-mutation-testing
Aug 14, 2026
Merged

test: add PIT mutation testing with per-module score gates#84
emaarco merged 4 commits into
mainfrom
emaarco/add-pitest-mutation-testing

Conversation

@emaarco

@emaarco emaarco commented Aug 14, 2026

Copy link
Copy Markdown
Member

What

Adds PIT mutation testing to complement the JaCoCo line-coverage gate. Line coverage proves a line ran; mutation testing proves a test would notice if its behaviour changed.

  • Wires the info.solidsoft.pitest plugin into the four logic-bearing modules (core, runtime, web, testing), following the existing JaCoCo wiring convention, with a hard per-module mutationThreshold gate.
  • Runs locally on demand (./gradlew pitest --no-configuration-cache) and as a separate opt-in CI workflow (nightly + manual dispatch), not on the PR path — PIT is too slow to gate every PR.
  • Suppresses Kotlin synthetic-mutant noise so the scores are honest: synthetic classes ($DefaultImpls, $Companion, $WhenMappings, kotlinx.serialization $$serializer, sortedBy $$inlined) via excludedClasses, and null/resource intrinsics + kotlin-logging calls via avoidCallsTo. This noise alone understated the raw scores by 6–15 points per module.
  • Strengthens the runtime, testing and web suites to close the real gaps PIT surfaced.

Mutation coverage / thresholds

Module Coverage Threshold
runtime 100% 95
testing 98% 95
web 95% 90
core 86% 80

Thresholds sit a few points below current coverage as a regression ratchet. web started at 28% — most of that was kotlin-logging call noise; the rest was a genuinely untested LibrarySourceProvider.

Bug found & fixed

Testing LibrarySourceProvider surfaced a real bug: loadProjectVersion() read the wrong properties key (projectVersion instead of version), so the playground's runtime-dependency snippet always rendered unknown. Fixed here.

Notes

  • Verified PIT works against the repo's JUnit 6.1.0 line (the pitest-junit5-plugin docs only list JUnit 5).
  • The pitest task is not configuration-cache compatible, so it runs with --no-configuration-cache; applying the plugin does not affect the normal cached build.
  • core is still climbing (survivors concentrated in the BPMN parsing/dialect adapters) — a follow-up can push it higher.
  • Rebased onto main after build: enforce Kotlin quality with ktlint + detekt guardrails #83 (ktlint guardrails); lintKotlin, detekt and build are green.

See ADR 019 for the full rationale.

Wire the info.solidsoft.pitest plugin into the four logic-bearing modules
(core, runtime, web, testing) following the existing JaCoCo convention, with
a hard per-module mutationThreshold gate. Runs locally via ./gradlew pitest
and as an opt-in nightly/manual CI workflow.

Suppress Kotlin synthetic-mutant noise (null intrinsics, serializers, inlined
comparators) via avoidCallsTo and excludedClasses, and strengthen the runtime
and testing suites to 100% / 98% mutation coverage.
Add kotlin-logging (io.github.oshai) to avoidCallsTo — the bulk of web's
survivors were logging-call noise — add real LibrarySourceProvider tests
plus response-DTO assertions, and raise the web mutationThreshold to 90.

Fix a bug the new tests surfaced: LibrarySourceProvider read the wrong
properties key (projectVersion instead of version), so the playground's
runtime-dependency snippet always rendered "unknown".
@emaarco
emaarco force-pushed the emaarco/add-pitest-mutation-testing branch from 8066590 to 86c50c7 Compare August 14, 2026 05:50
@github-actions

Copy link
Copy Markdown
Contributor

Test Coverage

Module Line Coverage
bpmn-to-code-core 97.6% (2317/2373 lines)
bpmn-to-code-gradle 25.7% (29/113 lines)
bpmn-to-code-maven 62.2% (28/45 lines)
bpmn-to-code-web 95.6% (196/205 lines)
bpmn-to-code-testing 97.2% (171/176 lines)
bpmn-to-code-runtime 99.0% (96/97 lines)

@emaarco
emaarco merged commit 008f480 into main Aug 14, 2026
5 of 6 checks passed
@emaarco emaarco self-assigned this Aug 14, 2026
@emaarco
emaarco deleted the emaarco/add-pitest-mutation-testing branch August 14, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant