chore: migrate tests to xunit v3 - #577
Merged
Merged
Conversation
Package swap is a 4-line edit in Directory.Packages.props now that central
package management is in: xunit 2.9.3 -> xunit.v3 3.2.2,
xunit.runner.visualstudio 3.0.0 -> 3.1.5, Microsoft.NET.Test.Sdk 17.12.0 ->
18.8.1. Staying on the VSTest bridge deliberately, so `dotnet test`, the Nuke
DotNetTest targets and coverlet.collector all keep working untouched;
Microsoft Testing Platform is a separate follow-up.
Mechanical changes:
- Xunit.Abstractions no longer exists; ITestOutputHelper moved into Xunit,
which is already a global Using. 15 stray usings deleted, no other edits.
- DocSamples drops its test packages entirely rather than migrating. It has
zero [Fact]/[Theory] methods, holds compile-checked doc samples, and is not
in jasperfx.slnx or any Nuke target. xunit.v3 would also have forced
OutputType=Exe onto a Microsoft.NET.Sdk.Web project for no benefit.
- CodegenTests.MethodCallTester borrowed Xunit.Sdk.ErrorMessage as a return
type for its codegen probes via a stray `using Xunit.Sdk`. The tests only
assert the inferred type identity, so any reference type does; replaced with
a local one and dropped the accidental dependency on xunit internals.
Three real failures surfaced, none of them mechanical. All are latent problems
that xunit v2 happened to hide:
1. JasperFxOptions.HasReferenceToJasperFxTool loaded every referenced assembly
and read its attributes with no guard. v3 test projects are executables, so
Assembly.GetEntryAssembly() is now the test assembly rather than
testhost.exe, and the walk reached a project reference whose own dependency
is missing from the output -- FileNotFoundException out of host startup.
This is a product bug, not a test bug: any application whose assembly graph
contains an unresolvable reference (an optional dependency that was never
deployed, something trimmed) crashes the same way today. Detection is
best-effort, so unloadable assemblies are now skipped.
2. EventTests.SliceGroupTests self-stubs IProjectionStorage, which extends
IAsyncDisposable, and left DisposeAsync throwing NotImplementedException.
v2 never called it; v3 disposes test classes that implement
IAsyncDisposable, so all 16 tests in the class failed in teardown. The stub
is part of the storage contract, not xunit lifecycle -- made it a no-op.
3. CommandExecutorTester.run_an_async_command_that_fails asserted on output
captured via Console.SetOut, but CommandExecutor renders failures through
Spectre's AnsiConsole, which caches the writer it binds to on first use
process-wide. The test was therefore passing on test ordering alone
(Console.SetOut cannot redirect an already-pinned Spectre console) and the
happy-path tests passed only because they use plain Console.WriteLine. Now
binds AnsiConsole explicitly, so it no longer depends on order.
Also retires two dead CI environment variables. FRAMEWORK bound to no Nuke
parameter, and DISABLE_TEST_PARALLELIZATION was read by nothing at all -- no
test project, runner config, or build target. The two suites that genuinely must
not run concurrently already declare it in code via
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)], which
v3 still honors. CONFIGURATION stays; it does bind to the Nuke parameter.
Test counts match the v2 baseline exactly on both frameworks -- no discovery
loss: CoreTests 475, CodegenTests 419, CommandLineTests 295, EventTests 648,
EventStoreTests 72, CodegenTests.FSharp 1. Stable over three consecutive runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 29, 2026
jeremydmiller
added a commit
that referenced
this pull request
Jul 29, 2026
A clean rebuild emitted 504 xunit analyzer warnings, burying genuinely
interesting output. That 504 is 129 unique source locations - the rest is
per-TFM and per-node duplication - which made hand-fixing every one of
them practical, so nothing is suppressed wholesale.
xUnit1013 (26 locations) - triaged by hand, as the issue asked, and it
was worth it:
- ProjectionRebuildSelectionTests implemented IProjectionHost on itself.
Interface members must be public, so every one read as an
un-attributed test method. That is the same self-stub shape that broke
all 16 SliceGroupTests in #577. Extracted to RecordingProjectionHost.
- RetryBlockTests.retry_within_threshold was a commented-out //[Fact].
Now [Fact(Skip = "Unreliable in CI")] so it is visibly skipped rather
than silently nonexistent. CoreTests totals 476 with 1 skipped.
- Documentation samples on test classes (ResourceHostExtensionsTests,
UsingSourceWriter) moved to sibling non-test classes, which keeps the
#region snippet text byte-for-byte identical - one of those regions
includes its method signature.
- Reflection targets (EventExtensionsTests x2,
event_parameter_naming_convention) made private, with BindingFlags
added where the lookup needed it.
xUnit1026 (2) - one was a real gap. AggregateVersioningTests
set_version_single_stream_on_internal ran ten InlineData cases and
asserted nothing at all; 'expected' was unused because the assertion was
missing entirely. It only ever proved TrySetVersion() doesn't throw. Now
asserts GetVersion() == expected, and passes.
xUnit1031 (4) - StoreUriStampingObserverTests had two sync tests calling
.Wait() on tasks. Both are now async/await.
xUnit1012 (8) - theory parameters that take null are now nullable.
xUnit1051 (89) - TestContext.Current.CancellationToken threaded through.
One documented #pragma exception in TaskExtensionsTests: SetCanceled(ct)
records which token DID cancel a task rather than observing one, so
passing the test's uncancelled token would assert something false.
Verified: 504 -> 0 xunit warnings on a clean rebuild, and the CS warning
count is unchanged at exactly 1920, so nothing was traded away.
Test counts against the #581 baseline: CoreTests 475+1 skipped,
CodegenTests 419, CommandLineTests 295, EventTests 648, EventStoreTests
72, CodegenTests.FSharp 1, JasperFx.SourceGenerator.Tests 19,
JasperFx.Events.SourceGenerator.Tests 26. CoreTests and EventTests also
run green on net10.0.
Closes #582
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Jul 29, 2026
…latform Follow-up to #577, which moved everything to xunit v3 but deliberately stayed on the VSTest bridge to keep that migration reviewable. All nine test projects drop Microsoft.NET.Test.Sdk and xunit.runner.visualstudio and run as their own MTP executables: no testhost indirection, faster startup, and xunit v3's own reporting. Four things the issue didn't anticipate: 1. OutputType=Exe now has to be explicit. Microsoft.NET.Test.Sdk was setting it; without it, xunit.v3's targets fail the build outright ("xUnit.net v3 test projects must be executable"). 2. Microsoft.Testing.Extensions.CodeCoverage is pinned to 18.0.6, not the latest 18.9.0. xunit.v3 3.2.2 is built against Microsoft.Testing. Platform 1.x -- its packages are literally named xunit.v3.core.mtp-v1. CodeCoverage 18.1.0+ moved to MTP 2.x, and the mixed graph dies at startup on every single test run with TypeLoadException: Could not load type 'Microsoft.Testing.Platform.Extensions.TestHost.IDataConsumer' from assembly 'Microsoft.Testing.Platform, Version=2.3.0.0' 18.0.6 is the last release on MTP 1.x. The constraint is recorded in Directory.Packages.props with the symptom, so the next person to bump it knows what they're looking at. 3. TestingPlatformShowTestsFailure is required, and its absence is a trap. By default MTP prints only error run failed: Tests failed: '.../TestResults/EventStoreTests_net9.0_x64.log' and sends the test name, assertion and stack trace to that file. CI never publishes it, so a red build carries no indication of what broke -- strictly worse than the VSTest bridge, which printed failures inline. Caught by deliberately failing a test and reading the CI log. With the property set, failures print as EventExtensionsTests.cs(32): error test failed: EventStoreTests.EventExtensionsTests.deliberately_failing_probe: Shouldly.ShouldAssertException : "actual" should be "expected" 4. The MTP properties belong in Directory.Build.props rather than per project: TestingPlatformDotnetTestSupport changes how `dotnet test` invokes a project, so a solution where only some projects answered to MTP would leave `dotnet test` on the .slnx in a mixed mode. build/Build.cs needed no changes at all. All seven DotNetTest calls work as-is under MTP, including TestCodegenFSharp's .SetFramework("net9.0"). Verified `./build.sh test` end to end (exit 0), every count matching the current baseline on every target framework: CoreTests 480 (479 + 1 skipped) CodegenTests 419 CommandLineTests 295 EventTests 648 EventStoreTests 72 Aspire.Tests 51 (net10.0) CodegenTests.FSharp 1 SourceGenerator.Tests 19 Events.SourceGenerator.Tests 26 Also verified failures still fail the build, which a silent runner swap could quietly break: a deliberately failing test produced "Failed: 1, Passed: 72" and Nuke surfaced it as ProcessException, 'dotnet' exited with code 1. CodegenTests and CommandLineTests, the two suites carrying [assembly: CollectionBehavior(CollectionPerAssembly)], pass unchanged. Closes #581 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Jul 29, 2026
CommandFactory prints "Searching '<assembly>' for commands" until its
static _hasAppliedExtensions latch flips. Being static, it flips once per
process, so exactly one test in the assembly picks up that banner as a
prefix on its captured console output -- whichever test happens to run
first.
execute_single_command_synchronously and its async sibling both asserted
their trimmed output with an exact ShouldBe, so whichever one drew the
short straw failed:
theOutput.ToString().Trim()
should be
"Big is True, Number is 7"
but was
"Searching 'JasperFx, Version=2.36.2.0, ...' for commands
Big is True, Number is 7"
This turned CI red on #587 (which passed on re-run, same commit) and
again on #589. It is not an MTP problem -- #587 was still on the VSTest
bridge -- but MTP orders tests differently, which is why it resurfaced
immediately after the runner change.
The banner is always a prefix, since discovery runs before the command
does, so anchoring to the end of the output keeps the assertion exactly
as strong while making it order-independent.
This is the same class of bug as the AnsiConsole binding already
documented in this constructor, which #577 fixed for the same reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Jul 29, 2026
…latform (#589) * build: move test suites from the VSTest bridge to Microsoft Testing Platform Follow-up to #577, which moved everything to xunit v3 but deliberately stayed on the VSTest bridge to keep that migration reviewable. All nine test projects drop Microsoft.NET.Test.Sdk and xunit.runner.visualstudio and run as their own MTP executables: no testhost indirection, faster startup, and xunit v3's own reporting. Four things the issue didn't anticipate: 1. OutputType=Exe now has to be explicit. Microsoft.NET.Test.Sdk was setting it; without it, xunit.v3's targets fail the build outright ("xUnit.net v3 test projects must be executable"). 2. Microsoft.Testing.Extensions.CodeCoverage is pinned to 18.0.6, not the latest 18.9.0. xunit.v3 3.2.2 is built against Microsoft.Testing. Platform 1.x -- its packages are literally named xunit.v3.core.mtp-v1. CodeCoverage 18.1.0+ moved to MTP 2.x, and the mixed graph dies at startup on every single test run with TypeLoadException: Could not load type 'Microsoft.Testing.Platform.Extensions.TestHost.IDataConsumer' from assembly 'Microsoft.Testing.Platform, Version=2.3.0.0' 18.0.6 is the last release on MTP 1.x. The constraint is recorded in Directory.Packages.props with the symptom, so the next person to bump it knows what they're looking at. 3. TestingPlatformShowTestsFailure is required, and its absence is a trap. By default MTP prints only error run failed: Tests failed: '.../TestResults/EventStoreTests_net9.0_x64.log' and sends the test name, assertion and stack trace to that file. CI never publishes it, so a red build carries no indication of what broke -- strictly worse than the VSTest bridge, which printed failures inline. Caught by deliberately failing a test and reading the CI log. With the property set, failures print as EventExtensionsTests.cs(32): error test failed: EventStoreTests.EventExtensionsTests.deliberately_failing_probe: Shouldly.ShouldAssertException : "actual" should be "expected" 4. The MTP properties belong in Directory.Build.props rather than per project: TestingPlatformDotnetTestSupport changes how `dotnet test` invokes a project, so a solution where only some projects answered to MTP would leave `dotnet test` on the .slnx in a mixed mode. build/Build.cs needed no changes at all. All seven DotNetTest calls work as-is under MTP, including TestCodegenFSharp's .SetFramework("net9.0"). Verified `./build.sh test` end to end (exit 0), every count matching the current baseline on every target framework: CoreTests 480 (479 + 1 skipped) CodegenTests 419 CommandLineTests 295 EventTests 648 EventStoreTests 72 Aspire.Tests 51 (net10.0) CodegenTests.FSharp 1 SourceGenerator.Tests 19 Events.SourceGenerator.Tests 26 Also verified failures still fail the build, which a silent runner swap could quietly break: a deliberately failing test produced "Failed: 1, Passed: 72" and Nuke surfaced it as ProcessException, 'dotnet' exited with code 1. CodegenTests and CommandLineTests, the two suites carrying [assembly: CollectionBehavior(CollectionPerAssembly)], pass unchanged. Closes #581 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: stop two CommandExecutor tests depending on running first CommandFactory prints "Searching '<assembly>' for commands" until its static _hasAppliedExtensions latch flips. Being static, it flips once per process, so exactly one test in the assembly picks up that banner as a prefix on its captured console output -- whichever test happens to run first. execute_single_command_synchronously and its async sibling both asserted their trimmed output with an exact ShouldBe, so whichever one drew the short straw failed: theOutput.ToString().Trim() should be "Big is True, Number is 7" but was "Searching 'JasperFx, Version=2.36.2.0, ...' for commands Big is True, Number is 7" This turned CI red on #587 (which passed on re-run, same commit) and again on #589. It is not an MTP problem -- #587 was still on the VSTest bridge -- but MTP orders tests differently, which is why it resurfaced immediately after the runner change. The banner is always a prefix, since discovery runs before the command does, so anchoring to the end of the output keeps the assertion exactly as strong while making it order-independent. This is the same class of bug as the AnsiConsole binding already documented in this constructor, which #577 fixed for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #576. The package swap itself is now a 4-line edit in
Directory.Packages.props:xunit→xunit.v3xunit.runner.visualstudioMicrosoft.NET.Test.SdkStaying on the VSTest bridge deliberately —
dotnet test, the NukeDotNetTesttargets, andcoverlet.collectorall keep working untouched. Microsoft Testing Platform is a clean follow-up now that v3 is in.Mechanical changes
Xunit.Abstractionsis gone;ITestOutputHelpermoved intoXunit, already a globalUsing. 15 stray usings deleted, nothing else.[Fact]/[Theory]methods, holds compile-checked doc samples, and isn't injasperfx.slnxor any Nuke target.xunit.v3would also have forcedOutputType=Exeonto aMicrosoft.NET.Sdk.Webproject for nothing.MethodCallTesterborrowedXunit.Sdk.ErrorMessageas a return type for its codegen probes via a strayusing Xunit.Sdk. The tests only assert the inferred type identity, so any reference type does — replaced with a local one, dropping an accidental dependency on xunit internals.Zero
IAsyncLifetimeimplementations and zeroIClassFixture/ICollectionFixtureusages in the repo, so none of the usual v3 migration work applied.Three real failures — none mechanical
All three are latent problems v2 happened to hide. Worth reviewing individually:
1.
HasReferenceToJasperFxToolhad no guard — this is a product bug, not a test bug.It loads every referenced assembly and reads its attributes. v3 test projects are executables, so
Assembly.GetEntryAssembly()is now the test assembly instead oftesthost.exe, and the walk reached a project reference whose own dependency is missing from the output →FileNotFoundExceptionstraight out of host startup.This affects real consumers today: any application whose assembly graph contains an unresolvable reference — an optional dependency never deployed, something trimmed — crashes identically. Detection is best-effort, so unloadable assemblies are now skipped.
2.
SliceGroupTestsself-stubsIProjectionStorage, which extendsIAsyncDisposable, and leftDisposeAsyncthrowingNotImplementedException. v2 never called it; v3 disposes test classes implementingIAsyncDisposable, so all 16 tests in the class failed in teardown. The stub belongs to the storage contract, not xunit lifecycle — now a no-op.3.
run_an_async_command_that_failswas passing on test ordering alone. It asserts on output captured viaConsole.SetOut, butCommandExecutorrenders failures through Spectre'sAnsiConsole, which caches the writer it binds to on first use, process-wide.Console.SetOutcannot redirect an already-pinned Spectre console. The sibling happy-path tests passed only because they use plainConsole.WriteLine. Now bindsAnsiConsoleexplicitly, so it no longer depends on order.Dead CI config retired
FRAMEWORKbound to no Nuke parameter, andDISABLE_TEST_PARALLELIZATIONwas read by nothing — no test project, runner config, or build target. The two suites that genuinely must not run concurrently already declare it in code via[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)], which v3 still honors, so serialization is enforced where it can't be silently dropped.CONFIGURATIONstays — it does bind.Verification
Test counts match the v2 baseline exactly on both frameworks — no discovery loss:
Stable over three consecutive full runs (the order/parallelism risk of a v3 swap).
smoke-test-aotandsmoke-test-commandsalso pass.🤖 Generated with Claude Code