Skip to content

Multiplatform mutation runtime and new operators - #14

Open
iho wants to merge 6 commits into
anschnapp:masterfrom
iho:master
Open

Multiplatform mutation runtime and new operators#14
iho wants to merge 6 commits into
anschnapp:masterfrom
iho:master

Conversation

@iho

@iho iho commented Aug 8, 2026

Copy link
Copy Markdown

Summary

Convert mutflow from JVM-only to a multiplatform mutation-testing framework targeting JVM, Kotlin/JS, Kotlin/WASM, and Kotlin/Native:

  • Multiplatform runtime & new operators — core, runtime, and annotations sources move to commonMain/commonTest, with expect/actual primitives (concurrent map, thread id) added to mutflow-core, a wasmJs target added across all KMP modules, and the compiler plugin wired into every compilation via the Gradle plugin. Adds operators previously marked planned in the catalog: ReferenceEquality (===/!==), Elvis/SafeCall (with block-origin detection for common IR), EmptyCollectionReturn, AssignConst (via a new assignment visitor path), and %*/xoror arithmetic/bitwise variants.
  • Kotlin/JS-Wasm Yarn lockfile — generated for reproducible builds.
  • Multiplatform binary inspector + HTML dashboardPlatformInspectorTest (commonTest) discovers every mutation point and classifies each point × variant as killed/survived, writing JSON per platform via PlatformInspectorActual.{jvm,js,wasm,native}; tools/mutflow-inspect/inspect-all.sh builds all targets, runs the inspector on each, and renders report-all.html. Also fixes ConstructorCallOperator to skip Regex(...) constructor calls (a null-deref that is a catchable NPE on JVM/JS but an uncatchable segfault on Kotlin/Native and Kotlin/Wasm).
  • Code-review fixesArgumentPropagationOperator derives value-arg indices from each parameter's actual kind (handles dispatch+extension receiver / context-parameter combos); inspect-all.sh guards the kill-rate division against zero variants; removes the JVM-only BinaryInspector.java/inspect.sh duplicate; moves the Native/Wasm segfault workaround into a documented unsafe-types set; stops running the baseline battery twice; native actual surfaces mkdir/open failures; WASM actual checks existsSync instead of swallowing all exceptions.
  • macosArm64 native target — declared alongside linuxX64() in every KMP module so the library and plugin build/test on Apple Silicon natively. The POSIX writeResultsFile moves to a shared nativeTest source set; mode_t uses .convert() (UShort on Darwin vs UInt on Linux); inspect-all.sh aggregates macosArm64 too, with the non-host native target skipped rather than failing; kotlin.native.ignoreDisabledTargets=true suppresses the target-disabled warning.

Test plan

  • ./gradlew build — full build across all modules and targets, including mutflow-test-kmp:allTests on JVM/JS/WASM — passes locally
  • mutflow-test-kmp:allTests on linuxX64 — passes locally (macosArm64 is skipped on a Linux host)
  • tools/mutflow-inspect/inspect-all.sh — renders the dashboard for all runnable targets (jvm/js/wasmJs/linuxX64 at 37/44 killed; macosArm64 shows "no results" on Linux)
  • CI on this PR

iho and others added 6 commits August 8, 2026 13:51
Convert mutflow from JVM-only to a multiplatform mutation-testing framework
targeting JVM, Kotlin/JS, Kotlin/WASM, and Kotlin/Native:
- Move core, runtime, and annotations sources to commonMain/commonTest
- Add expect/actual primitives (concurrent map, thread id) to mutflow-core
- Add wasmJs target across all KMP modules
- Wire the compiler plugin into every compilation via the Gradle plugin

Add mutation operators previously marked planned in the catalog:
- ReferenceEquality (=== <-> !==)
- Elvis (a ?: b -> a / b) and SafeCall (a?.b -> a!!.b), with block-origin
  detection for common IR
- EmptyCollectionReturn (collection return -> emptyList/emptySet/emptyMap)
- AssignConst (a = b -> a = default), via a new assignment visitor path
- % -> * and xor -> or arithmetic/bitwise variants

Add the mutflow-test-kmp module to prove injection works on all four targets.
Generated by the Kotlin Gradle plugin for the js()/wasmJs() targets added
in the multiplatform conversion. Kotlin/JS recommends committing this for
reproducible builds.
Add a multiplatform mutation inspector that runs the mutated KMP binary
against a test battery on every target (JVM, JS, WASM, Native) and
aggregates per-platform results into one HTML dashboard.

- PlatformInspectorTest (commonTest) discovers every mutation point and
  classifies each point x variant as killed/survived, writing JSON per
  platform via PlatformInspectorActual.{jvm,js,wasm,native}.
- tools/mutflow-inspect/inspect-all.sh builds all targets, runs the
  inspector on each, and renders report-all.html.
- Fix ConstructorCallOperator to skip Regex(...) constructor calls:
  replacing a Regex with null makes the subsequent .containsMatchIn() a
  null-deref that is a catchable NPE on JVM/JS but an uncatchable
  segfault on Kotlin/Native and Kotlin/Wasm, killing the whole test
  process.
- ArgumentPropagationOperator: derive value-arg indices from each
  parameter's actual kind instead of a single offset heuristic, so
  calls with both a dispatch and extension receiver, or context
  parameters, no longer mutate the wrong argument slot.
- inspect-all.sh: guard the kill-rate division so a platform with
  zero discovered variants doesn't crash the whole aggregation.
- Remove BinaryInspector.java/inspect.sh: a JVM-only duplicate of the
  battery/kill logic already covered by PlatformInspectorTest.kt +
  inspect-all.sh (which includes JVM as one of its four targets).
- ConstructorCallOperator: move the Native/Wasm uncatchable-segfault
  workaround into a documented, extensible set of unsafe types instead
  of a single inline FQN comparison.
- PlatformInspectorTest: stop running the baseline battery twice.
- PlatformInspectorActual (native): surface mkdir/open failures
  instead of silently skipping the write.
- PlatformInspectorActual (wasm): check existsSync before mkdirSync
  instead of catching and discarding every exception, so real mkdir
  failures are no longer masked as "already exists".
Declare macosArm64() alongside linuxX64() in every KMP module
(annotations, core, runtime, test-kmp) so the library and its mutation
compiler plugin also build and test on Apple Silicon natively, instead
of only cross-compiling on Linux CI.

For mutflow-test-kmp's binary inspector, split the native actuals: the
POSIX-based writeResultsFile now lives in a shared nativeTest source
set (used by both linuxX64 and macosArm64 via the default hierarchy
template), while currentPlatform() stays a per-leaf-target actual.
open()'s/mkdir()'s mode_t argument uses .convert() instead of a fixed
.toUInt(), since mode_t is UShort on Darwin but UInt on Linux.

inspect-all.sh now builds, runs, and aggregates the macosArm64 target
too; whichever native target doesn't match the current host is simply
skipped by Kotlin/Native rather than failing the run.

kotlin.native.ignoreDisabledTargets=true suppresses the resulting
"target disabled" warning now that two native targets are declared.
Add multiplatform binary inspector and HTML dashboard
@anschnapp

Copy link
Copy Markdown
Owner

Hi @iho, thank you very much for your contribution!

I tried to read and understand all the changes as well as I could. But I am
afraid it is quite an overwhelming amount of them. We have to separate them and
decide one by one how well each fits into the current project.

I hope it is not too disappointing that we cannot merge the whole PR.

In general, smaller PRs (one feature each) are much easier to review and
discuss. There is also a note in CONTRIBUTING.md about discussing things first.
At least for the native support, that would have saved us the overlap here: I
already had native support in the public kotlin-native branch, meant as an
ongoing branch until it is worth putting to master.

I would like to stick with the design in kotlin-native over the native support
in this PR. Two main reasons:

  • Tests on native use MutFlow.underTest { } in plain commonTest, exactly the
    same API as on the JVM, and a mutant counts as killed when the user's own
    tests fail. See example-native/. Your PlatformInspectorTest decides killed
    by comparing outputs against a baseline and never runs the user's tests, which
    answers a different question.
  • Production artifacts stay uninstrumented. The branch keeps a separate
    mutatedMain/mutatedTest compilation per target, so MutationRegistry
    never lands in a user's shipped binary. In this PR, isApplicable returns
    true for every KMP compilation, so it also ends up in main. If mutatedMain
    gave you trouble under KMP, I would honestly like to know what you ran into.

But there is a lot in this PR I would love to get into master:

IrTestCompiler is my favourite part. Loading the registrar through a
fabricated META-INF/services directory so it resolves off the test
classloader is a nice piece of work, and it fixes a real weakness: today,
testing an operator means appending to Calculator.kt and reading a mutation
count. Your 63 named assertions are a big improvement. As its own PR it can go
in almost as-is.

These operators, in one PR: reference equality, unary minus, bitwise,
increment, string method, collection method, primitive return, object return,
empty collection return. One thing to adjust: please keep them on the current
MutationOperator interface, without MutatorDescriptor. I would rather decide
what metadata operators carry (ids, groups, stability tiers) together with the
opt-in design, since what we want to configure determines what metadata we
actually need. Adding it first means guessing.

The operators I left out are either touching a deliberate decision (SafeCall
and Elvis: mutating ?. and ?: is intentionally out of scope, which is also
why NullSafetyTargetTest asserts no mutation points) or carrying crash risk
(ConstructorCall, ForceConditional, ReplaceNonVoidCall, RegexPattern).
Not a permanent no, just not in the first round.

A discussion ticket about opt-in operators. If we find a good way to make
operators selectable, most of the rest of yours could come in too, and that is
where the metadata question belongs.

And help checking the kotlin-native branch. It would be great if you could
run example-native/ with ./gradlew mutflowLinuxX64Test, read
DESIGN-KOTLIN-NATIVE.md, and tell me where it falls short for your use cases.
The dashboard idea is interesting, but it is a different model, so let us
discuss that one separately rather than port it directly.

It would be great to find a way to bring these things in together, it could give
mutflow a good push forward.

Let me know what you think. My suggestion would be to close this PR after some
discussion and bring the pieces in as smaller PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants