Testing

[delatrie]

Context

This PR sets up a testing infrastructure for allure-csharp packages. It also includes a set of Allure.NUnit tests to verify the infrastructure.

FAQ

How to create a test project?

Use Allure.NUnit.Tests as a reference.

1. Use SamplePackageReference items to add package references to all samples.
2. Use a ProjectUnderTest item to link samples to the target project/package.
3. Add a project reference to tests/Allure.Testing/Allure.Testing.csproj.
4. Add a project reference to build/Allure.Build.SourceGenerators/Allure.Build.SourceGenerators.csproj with OutputItemType="Analyzer" and ReferenceOutputAssembly="false".
5. Use the ALLURE_TEST_PARALLEL symbol with the conditional compilation directives to enable the parallel execution of the test. See ./tests/Allure.NUnit.Tests/GlobalSetup.cs for an example. See How to run the tests in parallel? below for more details.

How to create and use a single-file sample?

1. Create a sample file in the ./Samples/ directory of the project. Write the code of the sample.
2. In the test, pass Allure.Testing.AllureSampleRegistry.<Sample ID> to Allure.Testing.AllureSampleRunner.RunAsync. Sample ID is the name of the file without extension.
3. Assert the results.

How to create and use a multi-file sample?

1. Create a folder inside ./Samples/. Put the files inside the folder.
2. In the test, pass Allure.Testing.AllureSampleRegistry.<Sample ID> to Allure.Testing.AllureSampleRunner.RunAsync. Sample ID is the name of the folder you've created at step 1.
3. Assert the results.

How to run a test project?

Use dotnet run --project ./tests/<project-name>.

How to test against a NuGet package?

By default, the samples use project references to access the tested functionality. If you would like to use package references instead, use the Allure_TestTargetVersion MSBuild property:

dotnet run --project ./tests/<project-name> -p:Allure_TestTargetVersion=<version>

This will run the tests against a published NuGet package. To test a dev version, set the property to SNAPSHOT:

dotnet run --project ./tests/<project-name> -p:Allure_TestTargetVersion=SNAPSHOT

This will run the tests against the current SNAPSHOT version of the package in the local repository (./artifacts/package/release by default). Don't forget to run dotnet pack --configuration Release to create or update the packages first. If you've just updated the packages, you might need to clear the cache before running the test:

rm -rf ./packages/allure*

What is pre-run flow?

Pre-run flow separates the samples run from the tests run. First, all the samples are run together. Then the tests verify the results. This greatly reduces the total time of test runs.

The easiest way to utilize the flow is to set the Allure_PreRunTestSamples MSBuild property to true:

dotnet run --project ./tests/<project-name> -p:Allure_PreRunTestSamples=true

The command will update and run all the test project's samples. It will then run the project itself. Allure.Testing.AllureSampleRunner detects this setup. Instead of executing the sample, it will read the already available results and return them.

How to test against the dev version of a NuGet package with pre-run flow?

First, build and pack the packages:

dotnet pack --configuration Publish

Next, make sure the packages cache is clean:

rm -rf ./packages/allure*

Finally, run the test project with both Allure_PreRunTestSamples and Allure_TestTargetVersion set:

dotnet run --project ./tests/Allure.NUnit.Tests/ -p:Allure_PreRunTestSamples=true -p:Allure_TestTargetVersion=SNAPSHOT

How to run the tests in parallel?

By default, all tests are sequential. This is because the tests issue dotnet test commands against sample projects with shared project references (mainly, to Allure.Net.Commons). This can cause conflicts between parallel builds of the same dependency project.

To run the tests in parallel, use one of the following options:

1. Target a NuGet package with Allure_TestTargetVersion (see How to test against a NuGet package? above). In such a case, the samples will use package references instead of project references. No project references means no shared dependency projects to build.
2. Use pre-run flow. If Allure_PreRunTestSamples is set to true during the test project build (which also happens on dotnet run), the test project will assume no dotnet processes need to be run as the results must be available when the tests run. This doesn't give much, though, since in pre-run flow, the tests only run assertions, which are very fast even when run sequentially.

Note

In order for this to work, the test project must enable the parallel execution if and only if the ALLURE_TEST_PARALLEL symbol is defined. This is how ./tests/Allure.NUnit.Tests does this (./tests/Allure.NUnit.Tests/GlobalUsing.cs)

#if ALLURE_TEST_PARALLEL

[assembly: ParallelLimiter<DotnetParallelLimit>]

#else

[assembly: NotInParallel(["Allure.NUnit", "Allure.Net.Commons"])]

#endif

How to build the samples of a test project without building the project?

dotnet -t:Allure_BuildTestSample ./tests/<project-name>

How to clean the sample projects?

dotnet clean

or

dotnet -t:Allure_CleanTestSample

How to clean the sample solutions and their output?

dotnet -t:Allure_DeleteTestSample

The testing scheme

Here is the testing scheme that includes the testing infrastructure, a test project, a target project/package, a set of external dependencies, and temporary artefacts generated by the infrastructure.

Sample projects are fully decoupled from tests. They have their own dependencies, build configurations, target frameworks, etc.

The key elements

• GenerateSampleSolution - an MSBuild task that creates or updates a sample solution for a test project. It only writes files that have been changed. This keeps incremental builds.
• Source generators: the and classes in ./build/Allure.Build.SourceGenerators, which are Roslyn source generators. Their job is to keep the sample registry up to date with the actual set of samples in the project.
• AllureSampleRegistry - a class generated by a source generator. It exposes a set of static properties, each representing a sample project that can be run. The job of the class is to make sample execution type safe.
• AllureSampleRunner - a class defined in ./tests/Allure.Testing. Contains a couple of RunAsync overloads for executing test samples. In the pre-run mode, it only reads the result directory.

The flow

There are two types of flow: on-demand and pre-run, each consisting of four phases. Each phase can be run manually. Some are run automatically.

Phase	Description	When run automatically	How to run manually
Generate	During this phase, the sample solution is created or updated. If it already exists, only modified files are updated to support incremental builds.	On test project build.	dotnet msbuild -t:Allure_GenerateTestSamples
Build	In this phase, the sample project or entire solution is built against a selected configuration and framework.	Right after "Generate," if Allure_PreBuildTestSamples is true. Right before "Run" is requested by a test if the sample project is out of date.	dotnet msbuild -t:Allure_BuildTestSamples
Run	In this phase, the sample project is run. The result files are generated in the configured directory.	When requested by a test in on-demand mode.	dotnet msbuild -t:Allure_RunTestSamples
Assert	In this phase, the generated Allure results are validated by the tests.	When a test is run.	dotnet run --project <Testing Project>

On-demand flow

In the on-demand flow, sample projects run only when explicitly requested by a test. It may slightly vary, but it typically looks like this:

1. A developer runs dotnet run --project ./tests/Allure.NUnit.Tests.
2. A build is performed. The sample projects are updated during the build if necessary.
3. A test starts and asks the runner to execute the sample. The runner invokes dotnet test against the sample project.
4. The sample project is restored and built by dotnet test.
5. The tests in the sample projects are run.
6. The runner collects the result files and returns them to the test.
7. The test asserts the values.

This flow allows a single test to run more quickly. On the other hand, running a bunch of tests may be way too slow as each requires spinning a new dotnet test process. Additionally, the tests can't run in parallel because separate dotnet test processes may conflict when trying to rebuild common dependencies.

For such scenarios, a pre-run flow was designed.

Pre-run flow

Pre-run flow implies running all the samples in advance. The tests are only asserting the already prepared results. This can be performed during the development, but the primary goal of this flow is to speed up the CI pipeline. The flow looks like this:

1. A developer runs dotnet run --project Allure.NUnit.Tests -p:Allure_PreRunTestSamples=true
2. A build is performed. The sample solution is updated, restored, built, and tested during the build.
3. A test starts and asks the runner to execute the sample. The runner skips dotnet test. Instead, it reads the default results directory. If it exists and contains result files, the results are returned to the test. Otherwise, the runner throws an error.
4. The test asserts the values.

In a CI pipeline, a more expressive version can be used to test against the default configuration and .NET version:

1. dotnet restore
2. dotnet build --no-restore -p:Allure_PreRunTestingFlow=true -p:Allure_PreBuildTestSamples=true
3. dotnet msbuild -t:Allure_RunTestSamples
4. dotnet run --project ./tests/Allure.NUnit.Tests --no-restore --no-build

Testing packed projects

The same set of tests defined in a test project can be run against a NuGet package (either published or dev version).

To test against a dev version, it needs to be built and packed first:

dotnet pack --configuration Release

Next, make sure an old version is not cached:

rm -rf ./packages/allure*

Now, run the tests against the dev (snapshot) version:

dotnet run --project ./tests/Allure.NUnit.Tests -p:Allure_TestTargetVersion=SNAPSHOT

To test against a published version, set Allure_TestTargetVersion to this version:

dotnet run --project ./tests/Allure.NUnit.Tests -p:Allure_TestTargetVersion=2.14.1

Test projects that target NuGet packages will run their tests in parallel (if they followed the recommendations from the FAQ, see How to run the tests in parallel?.

Sample lifecycle targets

Use the following MSBuild targets to control the sample lifecycle:

• Allure_GenerateTestSamples - creates/updates the sample solutions.
• Allure_BuildTestSamples - restores and builds the sample solutions.
• Allure_RunTestSamples - executed the tests defined by projects of the sample solutions.
• Allure_CleanTestSamples - runs the Clean target against the sample solutions. This also cleans the default results directory.
• Allure_DeleteTestSamples - deletes sample solutions with all intermediate and output directories of their projects.

MSBuild properties and items

The following items need to be defined for each testing project:

• SamplePackageReference: each item will be converted to a PackageVersion item in the sample solution's Directory.Packages.props. Additionally, each item will be converted to a PackageReference for every sample project, unless Optional is set to true.
  Supported metadata:
  • Version: a version of the package to refer to
  • Optional: if set, the package will only be added to Directory.Packages.props. It won't be added to a sample project's package references.
• ProjectUnderTest: each item will be converted to a ProjectReference of a PackageReference for all sample projects, depending on the value of the Allure_TestTargetVersion property:
  • If Allure_TestTargetVersion is unset, ProjectUnderTest becomes ProjectReference.
  • Otherwise, ProjectUnderTest becomes PackageReference with Version set to Allure_TestTargetVersion. A special value SNAPSHOT indicates the current version ($(Version)).
    The motivation is to allow testing against a packed version of the target project (including already published versions).

The following properties can be used to tune the testing process:

• Allure_SampleSupportedTargetFrameworks: the list of target frameworks the sample projects can be compiled against. By default, the list includes one current TFM (as of today, it's net8.0). The motivation is to allow testing across multiple .NET versions without creating extra jobs.
  This property only affects the generation phase.
• Allure_PreBuildTestSamples: if set to true, the sample solution will be restored and built right after the generation.
  This property only affects the generation phase.
• Allure_PreRunTestSamples: if set to true, the sample solution will be restored, built, and tested right after the generation (which happens automatically during the test project's build). Additionally, it sets Allure_PreRunTestingFlow to true unless it's defined at the CLI level or redefined at the project level.
  This property only affects the generation phase.
• Allure_PreRunTestingFlow: if set to true, enables the pre-run flow. This property affects the generation and execution phases.
• Allure_SampleSelectedTargetFramework: a specific TFM to run the samples against. If not set, the samples are run against each TFM in the Allure_SampleSupportedTargetFrameworks list.
  This property only affects the execution phase.
• Allure_SampleConfiguration: a specific configuration used to build the sample projects.
  This property only affects the build phase.
• Allure_TestTargetVersion: if set, the sample projects reference the target as a NuGet package of the specified version. A special value SNAPSHOT indicated the current development version. In such a case, the package must exist in the local repository pointed by Allure_LocalNugetRepository before the sample solution is built. If unset, the sample projects reference the target via ProjectReference items.

The following properties affect the testing but have default values, and it's unlikely someone needs to change them:

• Allure_LocalNugetRepository: a path to the local NuGet repository that contains unpublished packages (like SNAPSHOT packages).
• Allure_PackageCacheDirectory: a path to the package cache directory. By default, it's ./packages/. The motivation is to ensure SNAPSHOT packages don't pollute the system-wide cache.
• Allure_SampleSolutionName: a name of the sample solution. By default, it's the test project's name, suffixed with .Samples.
• Allure_SampleSolutionDir: a directory of the sample solution. By default, it's ./artifacts/samples/<test project>.
• Allure_SampleResultsDirectoryFormat: a format string that has one {0} placeholder for the sample ID. When the ID is inserted, it must give the absolute path to a directory. This directory is where the sample runner looks for Allure result files during the pre-run flow.

AllureSample items

AllureSample items is that drive the generation of sample projects. Each test project generates exactly one sample solution, which contains multiple sample projects. Each sample project has one or more files, each represented by an AllureSample item of the test project.

By default, the items include Samples/**/* files inside a test project. The top-level entries in the Samples directory become sample projects. If the entry is a file, it will be copied right into the sample project directory. If the entry is a directory, all files in that directory will be copied to the sample project directory, preserving relative paths.

The name of the sample project is $(Allure_SampleSolutionName).$(ProjectSuffix), where ProjectSuffix is the name of the corresponding top-level entry in ./Samples/ without extension. For example, ProjectSuffix of Samples/Foo.cs is Foo, while ProjectSuffix of Samples/Bar/Baz.cs is Bar.

To overwrite ProjectSuffix, use the Update attribute on the item:

<ItemGroup>
  <AllureSample Update="Samples/MySample.cs" ProjectSuffix="MySuffix" />
</ItemGroup>

Closes #362.