Skip to content

chore: migrate tests to xunit v3 - #577

Merged
jeremydmiller merged 1 commit into
mainfrom
chore/xunit-v3
Jul 29, 2026
Merged

chore: migrate tests to xunit v3#577
jeremydmiller merged 1 commit into
mainfrom
chore/xunit-v3

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Builds on #576. The package swap itself is now a 4-line edit in Directory.Packages.props:

Package From To
xunitxunit.v3 2.9.3 3.2.2
xunit.runner.visualstudio 3.0.0 3.1.5
Microsoft.NET.Test.Sdk 17.12.0 18.8.1

Staying on the VSTest bridge deliberately — dotnet test, the Nuke DotNetTest targets, and coverlet.collector all keep working untouched. Microsoft Testing Platform is a clean follow-up now that v3 is in.

Mechanical changes

  • Xunit.Abstractions is gone; ITestOutputHelper moved into Xunit, already a global Using. 15 stray usings deleted, nothing else.
  • DocSamples drops its test packages rather than migrating — zero [Fact]/[Theory] methods, holds compile-checked doc samples, and isn't in jasperfx.slnx or any Nuke target. xunit.v3 would also have forced OutputType=Exe onto a Microsoft.NET.Sdk.Web project for nothing.
  • MethodCallTester borrowed Xunit.Sdk.ErrorMessage as a return type for its codegen probes via a stray using Xunit.Sdk. The tests only assert the inferred type identity, so any reference type does — replaced with a local one, dropping an accidental dependency on xunit internals.

Zero IAsyncLifetime implementations and zero IClassFixture/ICollectionFixture usages in the repo, so none of the usual v3 migration work applied.

Three real failures — none mechanical

All three are latent problems v2 happened to hide. Worth reviewing individually:

1. HasReferenceToJasperFxTool had no guard — this is a product bug, not a test bug.

It loads every referenced assembly and reads its attributes. v3 test projects are executables, so Assembly.GetEntryAssembly() is now the test assembly instead of testhost.exe, and the walk reached a project reference whose own dependency is missing from the output → FileNotFoundException straight out of host startup.

This affects real consumers today: any application whose assembly graph contains an unresolvable reference — an optional dependency never deployed, something trimmed — crashes identically. Detection is best-effort, so unloadable assemblies are now skipped.

2. SliceGroupTests self-stubs IProjectionStorage, which extends IAsyncDisposable, and left DisposeAsync throwing NotImplementedException. v2 never called it; v3 disposes test classes implementing IAsyncDisposable, so all 16 tests in the class failed in teardown. The stub belongs to the storage contract, not xunit lifecycle — now a no-op.

3. run_an_async_command_that_fails was passing on test ordering alone. It asserts on output captured via Console.SetOut, but CommandExecutor renders failures through Spectre's AnsiConsole, which caches the writer it binds to on first use, process-wide. Console.SetOut cannot redirect an already-pinned Spectre console. The sibling happy-path tests passed only because they use plain Console.WriteLine. Now binds AnsiConsole explicitly, so it no longer depends on order.

Dead CI config retired

FRAMEWORK bound to no Nuke parameter, and DISABLE_TEST_PARALLELIZATION was read by nothing — no test project, runner config, or build target. The two suites that genuinely must not run concurrently already declare it in code via [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)], which v3 still honors, so serialization is enforced where it can't be silently dropped. CONFIGURATION stays — it does bind.

Verification

Test counts match the v2 baseline exactly on both frameworks — no discovery loss:

Suite net9.0 net10.0
CoreTests 475 475
CodegenTests 419 419
CommandLineTests 295 295
EventTests 648 648
EventStoreTests 72 72
CodegenTests.FSharp 1

Stable over three consecutive full runs (the order/parallelism risk of a v3 swap). smoke-test-aot and smoke-test-commands also pass.

🤖 Generated with Claude Code

Package swap is a 4-line edit in Directory.Packages.props now that central
package management is in: xunit 2.9.3 -> xunit.v3 3.2.2,
xunit.runner.visualstudio 3.0.0 -> 3.1.5, Microsoft.NET.Test.Sdk 17.12.0 ->
18.8.1. Staying on the VSTest bridge deliberately, so `dotnet test`, the Nuke
DotNetTest targets and coverlet.collector all keep working untouched;
Microsoft Testing Platform is a separate follow-up.

Mechanical changes:

  - Xunit.Abstractions no longer exists; ITestOutputHelper moved into Xunit,
    which is already a global Using. 15 stray usings deleted, no other edits.
  - DocSamples drops its test packages entirely rather than migrating. It has
    zero [Fact]/[Theory] methods, holds compile-checked doc samples, and is not
    in jasperfx.slnx or any Nuke target. xunit.v3 would also have forced
    OutputType=Exe onto a Microsoft.NET.Sdk.Web project for no benefit.
  - CodegenTests.MethodCallTester borrowed Xunit.Sdk.ErrorMessage as a return
    type for its codegen probes via a stray `using Xunit.Sdk`. The tests only
    assert the inferred type identity, so any reference type does; replaced with
    a local one and dropped the accidental dependency on xunit internals.

Three real failures surfaced, none of them mechanical. All are latent problems
that xunit v2 happened to hide:

  1. JasperFxOptions.HasReferenceToJasperFxTool loaded every referenced assembly
     and read its attributes with no guard. v3 test projects are executables, so
     Assembly.GetEntryAssembly() is now the test assembly rather than
     testhost.exe, and the walk reached a project reference whose own dependency
     is missing from the output -- FileNotFoundException out of host startup.
     This is a product bug, not a test bug: any application whose assembly graph
     contains an unresolvable reference (an optional dependency that was never
     deployed, something trimmed) crashes the same way today. Detection is
     best-effort, so unloadable assemblies are now skipped.

  2. EventTests.SliceGroupTests self-stubs IProjectionStorage, which extends
     IAsyncDisposable, and left DisposeAsync throwing NotImplementedException.
     v2 never called it; v3 disposes test classes that implement
     IAsyncDisposable, so all 16 tests in the class failed in teardown. The stub
     is part of the storage contract, not xunit lifecycle -- made it a no-op.

  3. CommandExecutorTester.run_an_async_command_that_fails asserted on output
     captured via Console.SetOut, but CommandExecutor renders failures through
     Spectre's AnsiConsole, which caches the writer it binds to on first use
     process-wide. The test was therefore passing on test ordering alone
     (Console.SetOut cannot redirect an already-pinned Spectre console) and the
     happy-path tests passed only because they use plain Console.WriteLine. Now
     binds AnsiConsole explicitly, so it no longer depends on order.

Also retires two dead CI environment variables. FRAMEWORK bound to no Nuke
parameter, and DISABLE_TEST_PARALLELIZATION was read by nothing at all -- no
test project, runner config, or build target. The two suites that genuinely must
not run concurrently already declare it in code via
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)], which
v3 still honors. CONFIGURATION stays; it does bind to the Nuke parameter.

Test counts match the v2 baseline exactly on both frameworks -- no discovery
loss: CoreTests 475, CodegenTests 419, CommandLineTests 295, EventTests 648,
EventStoreTests 72, CodegenTests.FSharp 1. Stable over three consecutive runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 82fe3e2 into main Jul 29, 2026
1 check passed
jeremydmiller added a commit that referenced this pull request Jul 29, 2026
A clean rebuild emitted 504 xunit analyzer warnings, burying genuinely
interesting output. That 504 is 129 unique source locations - the rest is
per-TFM and per-node duplication - which made hand-fixing every one of
them practical, so nothing is suppressed wholesale.

xUnit1013 (26 locations) - triaged by hand, as the issue asked, and it
was worth it:

  - ProjectionRebuildSelectionTests implemented IProjectionHost on itself.
    Interface members must be public, so every one read as an
    un-attributed test method. That is the same self-stub shape that broke
    all 16 SliceGroupTests in #577. Extracted to RecordingProjectionHost.
  - RetryBlockTests.retry_within_threshold was a commented-out //[Fact].
    Now [Fact(Skip = "Unreliable in CI")] so it is visibly skipped rather
    than silently nonexistent. CoreTests totals 476 with 1 skipped.
  - Documentation samples on test classes (ResourceHostExtensionsTests,
    UsingSourceWriter) moved to sibling non-test classes, which keeps the
    #region snippet text byte-for-byte identical - one of those regions
    includes its method signature.
  - Reflection targets (EventExtensionsTests x2,
    event_parameter_naming_convention) made private, with BindingFlags
    added where the lookup needed it.

xUnit1026 (2) - one was a real gap. AggregateVersioningTests
set_version_single_stream_on_internal ran ten InlineData cases and
asserted nothing at all; 'expected' was unused because the assertion was
missing entirely. It only ever proved TrySetVersion() doesn't throw. Now
asserts GetVersion() == expected, and passes.

xUnit1031 (4) - StoreUriStampingObserverTests had two sync tests calling
.Wait() on tasks. Both are now async/await.

xUnit1012 (8) - theory parameters that take null are now nullable.

xUnit1051 (89) - TestContext.Current.CancellationToken threaded through.
One documented #pragma exception in TaskExtensionsTests: SetCanceled(ct)
records which token DID cancel a task rather than observing one, so
passing the test's uncancelled token would assert something false.

Verified: 504 -> 0 xunit warnings on a clean rebuild, and the CS warning
count is unchanged at exactly 1920, so nothing was traded away.

Test counts against the #581 baseline: CoreTests 475+1 skipped,
CodegenTests 419, CommandLineTests 295, EventTests 648, EventStoreTests
72, CodegenTests.FSharp 1, JasperFx.SourceGenerator.Tests 19,
JasperFx.Events.SourceGenerator.Tests 26. CoreTests and EventTests also
run green on net10.0.

Closes #582

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeremydmiller added a commit that referenced this pull request Jul 29, 2026
…latform

Follow-up to #577, which moved everything to xunit v3 but deliberately
stayed on the VSTest bridge to keep that migration reviewable.

All nine test projects drop Microsoft.NET.Test.Sdk and
xunit.runner.visualstudio and run as their own MTP executables:
no testhost indirection, faster startup, and xunit v3's own reporting.

Four things the issue didn't anticipate:

1. OutputType=Exe now has to be explicit. Microsoft.NET.Test.Sdk was
   setting it; without it, xunit.v3's targets fail the build outright
   ("xUnit.net v3 test projects must be executable").

2. Microsoft.Testing.Extensions.CodeCoverage is pinned to 18.0.6, not
   the latest 18.9.0. xunit.v3 3.2.2 is built against Microsoft.Testing.
   Platform 1.x -- its packages are literally named xunit.v3.core.mtp-v1.
   CodeCoverage 18.1.0+ moved to MTP 2.x, and the mixed graph dies at
   startup on every single test run with

     TypeLoadException: Could not load type
     'Microsoft.Testing.Platform.Extensions.TestHost.IDataConsumer'
     from assembly 'Microsoft.Testing.Platform, Version=2.3.0.0'

   18.0.6 is the last release on MTP 1.x. The constraint is recorded in
   Directory.Packages.props with the symptom, so the next person to bump
   it knows what they're looking at.

3. TestingPlatformShowTestsFailure is required, and its absence is a
   trap. By default MTP prints only

     error run failed: Tests failed: '.../TestResults/EventStoreTests_net9.0_x64.log'

   and sends the test name, assertion and stack trace to that file. CI
   never publishes it, so a red build carries no indication of what
   broke -- strictly worse than the VSTest bridge, which printed
   failures inline. Caught by deliberately failing a test and reading
   the CI log. With the property set, failures print as

     EventExtensionsTests.cs(32): error test failed:
       EventStoreTests.EventExtensionsTests.deliberately_failing_probe:
       Shouldly.ShouldAssertException : "actual" should be "expected"

4. The MTP properties belong in Directory.Build.props rather than per
   project: TestingPlatformDotnetTestSupport changes how `dotnet test`
   invokes a project, so a solution where only some projects answered to
   MTP would leave `dotnet test` on the .slnx in a mixed mode.

build/Build.cs needed no changes at all. All seven DotNetTest calls work
as-is under MTP, including TestCodegenFSharp's .SetFramework("net9.0").

Verified `./build.sh test` end to end (exit 0), every count matching the
current baseline on every target framework:

  CoreTests      480 (479 + 1 skipped)   CodegenTests   419
  CommandLineTests 295                   EventTests     648
  EventStoreTests   72                   Aspire.Tests    51 (net10.0)
  CodegenTests.FSharp 1                  SourceGenerator.Tests 19
  Events.SourceGenerator.Tests 26

Also verified failures still fail the build, which a silent runner swap
could quietly break: a deliberately failing test produced
"Failed: 1, Passed: 72" and Nuke surfaced it as ProcessException,
'dotnet' exited with code 1. CodegenTests and CommandLineTests, the two
suites carrying [assembly: CollectionBehavior(CollectionPerAssembly)],
pass unchanged.

Closes #581

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeremydmiller added a commit that referenced this pull request Jul 29, 2026
CommandFactory prints "Searching '<assembly>' for commands" until its
static _hasAppliedExtensions latch flips. Being static, it flips once per
process, so exactly one test in the assembly picks up that banner as a
prefix on its captured console output -- whichever test happens to run
first.

execute_single_command_synchronously and its async sibling both asserted
their trimmed output with an exact ShouldBe, so whichever one drew the
short straw failed:

  theOutput.ToString().Trim()
      should be
  "Big is True, Number is 7"
      but was
  "Searching 'JasperFx, Version=2.36.2.0, ...' for commands
  Big is True, Number is 7"

This turned CI red on #587 (which passed on re-run, same commit) and
again on #589. It is not an MTP problem -- #587 was still on the VSTest
bridge -- but MTP orders tests differently, which is why it resurfaced
immediately after the runner change.

The banner is always a prefix, since discovery runs before the command
does, so anchoring to the end of the output keeps the assertion exactly
as strong while making it order-independent.

This is the same class of bug as the AnsiConsole binding already
documented in this constructor, which #577 fixed for the same reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jeremydmiller added a commit that referenced this pull request Jul 29, 2026
…latform (#589)

* build: move test suites from the VSTest bridge to Microsoft Testing Platform

Follow-up to #577, which moved everything to xunit v3 but deliberately
stayed on the VSTest bridge to keep that migration reviewable.

All nine test projects drop Microsoft.NET.Test.Sdk and
xunit.runner.visualstudio and run as their own MTP executables:
no testhost indirection, faster startup, and xunit v3's own reporting.

Four things the issue didn't anticipate:

1. OutputType=Exe now has to be explicit. Microsoft.NET.Test.Sdk was
   setting it; without it, xunit.v3's targets fail the build outright
   ("xUnit.net v3 test projects must be executable").

2. Microsoft.Testing.Extensions.CodeCoverage is pinned to 18.0.6, not
   the latest 18.9.0. xunit.v3 3.2.2 is built against Microsoft.Testing.
   Platform 1.x -- its packages are literally named xunit.v3.core.mtp-v1.
   CodeCoverage 18.1.0+ moved to MTP 2.x, and the mixed graph dies at
   startup on every single test run with

     TypeLoadException: Could not load type
     'Microsoft.Testing.Platform.Extensions.TestHost.IDataConsumer'
     from assembly 'Microsoft.Testing.Platform, Version=2.3.0.0'

   18.0.6 is the last release on MTP 1.x. The constraint is recorded in
   Directory.Packages.props with the symptom, so the next person to bump
   it knows what they're looking at.

3. TestingPlatformShowTestsFailure is required, and its absence is a
   trap. By default MTP prints only

     error run failed: Tests failed: '.../TestResults/EventStoreTests_net9.0_x64.log'

   and sends the test name, assertion and stack trace to that file. CI
   never publishes it, so a red build carries no indication of what
   broke -- strictly worse than the VSTest bridge, which printed
   failures inline. Caught by deliberately failing a test and reading
   the CI log. With the property set, failures print as

     EventExtensionsTests.cs(32): error test failed:
       EventStoreTests.EventExtensionsTests.deliberately_failing_probe:
       Shouldly.ShouldAssertException : "actual" should be "expected"

4. The MTP properties belong in Directory.Build.props rather than per
   project: TestingPlatformDotnetTestSupport changes how `dotnet test`
   invokes a project, so a solution where only some projects answered to
   MTP would leave `dotnet test` on the .slnx in a mixed mode.

build/Build.cs needed no changes at all. All seven DotNetTest calls work
as-is under MTP, including TestCodegenFSharp's .SetFramework("net9.0").

Verified `./build.sh test` end to end (exit 0), every count matching the
current baseline on every target framework:

  CoreTests      480 (479 + 1 skipped)   CodegenTests   419
  CommandLineTests 295                   EventTests     648
  EventStoreTests   72                   Aspire.Tests    51 (net10.0)
  CodegenTests.FSharp 1                  SourceGenerator.Tests 19
  Events.SourceGenerator.Tests 26

Also verified failures still fail the build, which a silent runner swap
could quietly break: a deliberately failing test produced
"Failed: 1, Passed: 72" and Nuke surfaced it as ProcessException,
'dotnet' exited with code 1. CodegenTests and CommandLineTests, the two
suites carrying [assembly: CollectionBehavior(CollectionPerAssembly)],
pass unchanged.

Closes #581

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test: stop two CommandExecutor tests depending on running first

CommandFactory prints "Searching '<assembly>' for commands" until its
static _hasAppliedExtensions latch flips. Being static, it flips once per
process, so exactly one test in the assembly picks up that banner as a
prefix on its captured console output -- whichever test happens to run
first.

execute_single_command_synchronously and its async sibling both asserted
their trimmed output with an exact ShouldBe, so whichever one drew the
short straw failed:

  theOutput.ToString().Trim()
      should be
  "Big is True, Number is 7"
      but was
  "Searching 'JasperFx, Version=2.36.2.0, ...' for commands
  Big is True, Number is 7"

This turned CI red on #587 (which passed on re-run, same commit) and
again on #589. It is not an MTP problem -- #587 was still on the VSTest
bridge -- but MTP orders tests differently, which is why it resurfaced
immediately after the runner change.

The banner is always a prefix, since discovery runs before the command
does, so anchoring to the end of the output keeps the assertion exactly
as strong while making it order-independent.

This is the same class of bug as the AnsiConsole binding already
documented in this constructor, which #577 fixed for the same reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant