build: move test suites from the VSTest bridge to Microsoft Testing Platform

[jeremydmiller]

Closes #581

Follow-up to #577, which moved everything to xunit v3 but deliberately stayed on the VSTest bridge to keep that migration reviewable.

All nine test projects now drop Microsoft.NET.Test.Sdk and xunit.runner.visualstudio and run as their own MTP executables — no testhost indirection, faster startup, xunit v3's own reporting.

Three things the issue didn't anticipate

1. OutputType=Exe has to become explicit. Microsoft.NET.Test.Sdk was setting it. Remove that package and xunit.v3's targets fail the build outright:

error : xUnit.net v3 test projects must be executable
        (set project property '<OutputType>Exe</OutputType>')

2. Microsoft.Testing.Extensions.CodeCoverage is pinned to 18.0.6, not the latest 18.9.0. This is the interesting one. xunit.v3 3.2.2 is built against Microsoft.Testing.Platform 1.x — its packages are literally named xunit.v3.core.mtp-v1. CodeCoverage 18.1.0 and later moved to MTP 2.x. NuGet unifies the platform assembly up to 2.x while Microsoft.Testing.Platform.MSBuild 1.9.1 stays behind, and every single test run dies at startup:

Unhandled exception. System.TypeLoadException: Could not load type
'Microsoft.Testing.Platform.Extensions.TestHost.IDataConsumer' from assembly
'Microsoft.Testing.Platform, Version=2.3.0.0, ...'
   at Microsoft.Testing.Platform.MSBuild.MSBuildExtensions.AddMSBuild(...)

18.0.6 is the last release on MTP 1.x. The constraint and the symptom are both recorded in Directory.Packages.props so whoever bumps it next knows what they're looking at — the fix is to move forward only once xunit.v3 ships an mtp-v2 build.

3. The MTP properties belong in Directory.Build.props, not per project. TestingPlatformDotnetTestSupport changes how dotnet test invokes a project; a solution where only some projects answered to MTP would leave dotnet test on the .slnx in a mixed mode. Both properties are inert in projects that don't reference xunit.v3.

build/Build.cs needed no changes

The issue flagged Nuke's DotNetTest as a blocker. All seven calls work as-is under MTP — they only use SetProjectFile / SetConfiguration / EnableNoBuild / EnableNoRestore, none of which change shape. TestCodegenFSharp's .SetFramework("net9.0") works too:

> dotnet test .../CodegenTests.FSharp.csproj --framework net9.0 --configuration Debug --no-build --no-restore
  Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1 - CodegenTests.FSharp.dll (net9.0|arm64)

The VSTest-only arguments the issue worried about (--collect, --logger, --filter, --settings) aren't used anywhere in this repo.

Verification

./build.sh test end to end, exit 0. Every count matches the baseline recorded in #581, on every target framework:

Suite	net9.0	net10.0	Baseline
CoreTests	475	475	475
CodegenTests	419	419	419
CommandLineTests	295	295	295
EventTests	648	648	648
EventStoreTests	72	72	72
JasperFx.Aspire.Tests	—	51	51
CodegenTests.FSharp	1	—	1
JasperFx.SourceGenerator.Tests	19	—	19
JasperFx.Events.SourceGenerator.Tests	26	—	26

Failures still fail the build. A runner swap can silently break exit-code propagation and turn CI permanently green, so I checked it directly by adding a deliberately failing test:

Failed! - Failed: 1, Passed: 72, Skipped: 0, Total: 73 - EventStoreTests.dll (net9.0|arm64)
Nuke.Common.Tooling.ProcessException: Process 'dotnet' exited with code 1.
TestEventStore     Failed
NUKE EXIT=255

CollectionBehavior still honored. CodegenTests and CommandLineTests — the two suites carrying [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)] — pass unchanged. The attribute is an xunit-level concern, independent of the runner. Note that MTP runs the two target frameworks concurrently, which is safe here: they are separate processes with separate per-TFM output directories, so neither the process-wide statics nor the on-disk assembly compilation collide.

./build.sh smoke-test-commands also still passes.

One note on scope

The issue asked to swap coverlet.collector → Microsoft.Testing.Extensions.CodeCoverage, which is what this does. Worth flagging: no build target actually asks for coverage — there is no --collect anywhere in build/Build.cs or the workflow, so coverlet.collector was never invoked either. This is capability parity, not something in use. Say the word if you'd rather just drop the collector entirely.

🤖 Generated with Claude Code

[jeremydmiller]

Rebased onto main (now carrying #585–#588) and pushed two changes in response to the CI failure.

1. MTP was hiding the failure — fixed

The CI failure was undiagnosable, and that was this PR's fault. All the log said was:

error run failed: Tests failed: '.../TestResults/CommandLineTests_net9.0_x64.log'

MTP sends the test name, assertion, and stack trace to that file instead of stdout, and CI never publishes it — so a red build carried no indication of what broke. That is strictly worse than the VSTest bridge, which printed failures inline, and it would have made this migration a bad trade on its own.

Reproduced locally by deliberately failing a test, then fixed with <TestingPlatformShowTestsFailure>true</TestingPlatformShowTestsFailure> in Directory.Build.props. Failures now print inline, with a file(line) prefix that CI can annotate:

EventExtensionsTests.cs(32): error test failed:
  EventStoreTests.EventExtensionsTests.deliberately_failing_probe (12ms):
  Shouldly.ShouldAssertException : "actual"
      should be
      but was not
   at EventStoreTests.EventExtensionsTests.deliberately_failing_probe() in .../EventExtensionsTests.cs:line 32

Thanks to that property, if the flake below ever recurs the log will name it instead of pointing at a file nobody can read.

2. The actual failure was a pre-existing order-dependent test

CommandFactory prints Searching '<assembly>' for commands until _hasAppliedExtensions flips. That field is static, so it flips once per process and exactly one test in the assembly picks the banner up as a prefix — whichever runs first.

execute_single_command_synchronously and execute_single_command_asynchronously both asserted trimmed output with an exact ShouldBe, so whichever drew the short straw failed:

theOutput.ToString().Trim()
    should be
"Big is True, Number is 7"
    but was
"Searching 'JasperFx, Version=2.36.2.0, ...' for commands
Big is True, Number is 7"

This is not an MTP problem. It hit #587 first, which was still on the VSTest bridge, and that PR went green on a re-run of the same commit. MTP simply orders tests differently, which is why it resurfaced immediately after the runner change.

Both assertions are now ShouldEndWith. The banner is always a prefix — discovery runs before the command does — so the assertion is exactly as strong, just order-independent. It's the same class of bug as the AnsiConsole binding already documented a few lines up in that constructor, which #577 fixed for the same reason.

Happy to split that commit out into its own PR if you'd rather keep this one purely about the runner — it's a pre-existing bug on main, not something this PR introduced.

Verification after the rebase

./build.sh test green end to end, exit 0:

Suite	net9.0	net10.0
CoreTests	479 + 1 skipped	479 + 1 skipped
CodegenTests	419	419
CommandLineTests	295	295
EventTests	648	648
EventStoreTests	72	72
JasperFx.Aspire.Tests	—	51
CodegenTests.FSharp	1	—
JasperFx.SourceGenerator.Tests	19	—
JasperFx.Events.SourceGenerator.Tests	26	—

CoreTests is 480 rather than the 475 in the original description because #587 added 4 tests and #588 turned a commented-out //[Fact] into an explicit skip.