Add the Dummies.Xunit companion package for reproducible xUnit v3 tests - #300
Merged
Conversation
The audit of 2026-07-20 asks for an explicit yes/no on the test-framework adapter that ADR-0006's follow-ups anticipated and the ADR-0026 rebase dropped. Two decisions, one ADR each: * ADR-0038 opens the ambient seed scope as a public, disposable handle carrying the replay instruction diagnostics will name. An adapter runs before and after a test, so it cannot use the delegate-scoped runner; a public handle keeps every adapter possible without privileged access. * ADR-0039 adapts Dummies to xUnit v3 only. The capability is "report the seed only when the test fails": v3 exposes a finished test's outcome in its documented extensibility, v2 exposes none from the equivalent hook and would require owning test discovery and execution. Refs: #226
Any.Reproducibly needs a delegate to wrap. A test-framework adapter observes a test through before/after hooks and has none, so it could not pin the ambient source at all: the disposable handle existed but was internal. UseSeed(int) exposes it. The second overload carries the instruction a generation-failure diagnostic must name: the default points at Any.Reproducibly(seed, ...), which is the wrong instruction for a run pinned from outside the test body, where replaying means changing what the adapter reads. The scope now carries that instruction rather than the source fixing it, so the guarantee the hint mechanism exists for -- never name a mechanism the reader does not use -- survives the third way of pinning the ambient source. Public rather than an internals grant to Dummies.Xunit, so any adapter is possible without a further change here (ADR-0038). The surface-parity guard excludes UseSeed the way it already excludes WithSeed and Reproducibly: none is a generator factory, and an AnyContext already is an explicit deterministic context. Refs: #226
[Reproducible] pins the ambient seed for a test and reports it only when the test fails, replacing the per-test Any.Reproducibly lambda on xUnit v3. Built on BeforeAfterTestAttribute rather than a Fact/Theory subclass: the hook is collected from the method, the class and the assembly and runs once per theory case, so one attribute covers a test, a class, a whole suite and every case of a theory -- without touching test discovery. Levels nest, and xUnit closes them in reverse, so the most specific declaration wins for the duration of the test and the outer ones are restored after it. The failure-only rule reads the finished test's outcome from the ambient test context, which xUnit v3 exposes and v2 does not; that asymmetry is why the package targets v3 only (ADR-0039). The seed is pinned through the public Any.UseSeed handle, so the package holds no privileged access. Targets netstandard2.0: xunit.v3.extensibility.core is itself a netstandard2.0 library, so nothing forces the adapter above the floor Dummies already keeps. Rides the dum train with Dummies. Tests cover pinning, a zero seed, the per-case seed, scope closing, the class-level and method-level precedence, the replay instruction reaching generation failures, and the reporting rule; a contract test fails if xUnit stops exposing a finished test's outcome. Refs: #226
Dummies.Xunit ships netstandard2.0, so a .NET Framework consumer loads that asset -- but the adapter suite only ever ran on net10, leaving the floor it claims to support unexercised. Import the shared Net472TestFloor plumbing and add the project to the framework-floor job, the same way Dummies.UnitTests already carries its own contract suite there (ADR-0022). Verified: the suite compiles on net472. Running it needs Windows -- there is no Mono on the Linux legs -- so execution is the CI job's to prove. Refs: #226
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Answers the open question the 2026-07-20 Dummies audit raised: the test-framework adapter that ADR-0006's follow-ups anticipated and the ADR-0026 rebase dropped. A new
Dummies.Xunitcompanion package supplies[Reproducible], which pins the ambient seed for a test and reports it only when the test fails — replacing the per-testAny.Reproducibly(() => { ... })lambda on xUnit v3. Getting there needed one addition toDummiesitself: a public handle for pinning the ambient seed, since the delegate-scoped runner cannot serve a caller that observes a test from before/after hooks.Type of change
Changes
Any.UseSeed(int)/Any.UseSeed(int, string)— the ambient seed scope, previously internal, exposed as a public disposable handle. The second overload carries the replay instruction a generation-failure diagnostic names: the default points atAny.Reproducibly(seed, ...), which is the wrong instruction for a run pinned from outside the test body. The scope now carries that instruction rather than the source fixing it. Public rather than anInternalsVisibleTogrant, so any adapter — including one for another framework — needs no further change toDummies.Dummies.Xunit— new netstandard2.0 package exposingReproducibleAttribute, built onBeforeAfterTestAttributerather than aFact/Theorysubclass. The hook is collected from the method, the class and the assembly and runs once per theory case, so one attribute covers a test, a class, a whole suite and every case of a theory — without touching test discovery.Dummies.Xunit.UnitTests— 14 tests, including the architecture guard that the package references no FirstClassErrors assembly.Dummies.Xunitrides the existingdumtrain intools/packaging/pack.sh; both projects are wired into the solution.Dummies.Xunit.UnitTestsadded to theframework-floorjob.ArbitraryTestValues(EN + FR), theDummiesreadme, and a newDummies.Xunitreadme.Why xUnit v3 only
The capability is "report the seed only when the test fails", which requires observing a finished test's outcome. Verified empirically against both majors:
BeforeAfterTestAttribute.After(MethodInfo, IXunitTest), withTestContext.Current.TestStatecarryingResult,FailureCauseand the exception detail, plusTestOutputHelper.xunit.coreshows onlyBefore(MethodInfo)/After(MethodInfo), and noTestContexttype in the assembly. The outcome is unreachable from the equivalent hook; expressing the rule there means owning test-case discovery and execution against semi-internal surface.Users of v2, NUnit and MSTest lose nothing:
Any.Reproducibly(...)is unaffected and remains the portable form.Testing
dotnet build FirstClassErrors.sln— succeeds, no warningsdotnet test FirstClassErrors.sln— 11 suites, 1559 tests, 0 failuresFirstClassErrors.Analyzers.UnitTests) — 132 passed, as part of the solution runDesign assumptions were verified by execution in throwaway spikes before implementation, not assumed from documentation: that an
AsyncLocalwritten inBeforereaches sync bodies, async bodies afterTask.Yield(), and each theory case; that it is still readable inAfter; thatTestState.ResultreportsFailedwithFailureCause = Assertionon a failing test; that hooks are collected at method, class and assembly level; and thatAfterruns in exact reverse order ofBefore.Not run here: the net472 leg of
Dummies.Xunit.UnitTestscompiles (dotnet build -p:EnableNet472Floor=true -f net472) but could not be executed — there is no Mono on the Linux legs, which is the documented reason that job is Windows-only. Execution is theframework-floorjob's to prove.One behaviour is covered by a seam rather than end to end: a test that genuinely fails would fail the suite, so the "report only on failure" rule is decided by an internal
ReportFor(bool, int)proved directly, paired with a contract test whose assertion lives in an after-hook and fails if xUnit stops exposing a finished test's outcome.Documentation
doc/updateddoc/handwritten/for-users/ArbitraryTestValues.{en,fr}.mdgained the scope form and the[Reproducible]section in lockstep;Dummies/README.nuget.mdmentionsUseSeed;Dummies.Xunit/README.nuget.mdis new. Both ADRs ship EN + FR.Architecture decisions
Proposed: ADR-0038 and ADR-0039Two ADRs rather than one, because the decisions have independent lifetimes: the public seed scope stands even if the xUnit adapter were dropped, and it serves any framework.
Related issues
Refs #226 — this completes the
Dummies.Xunitcompanion-package item; the tracking issue stays open for its other candidates.Generated by Claude Code