Common attributes API

[delatrie]

Context

The PR introduces the following attributes to the Allure API for test authors:

Attribute	Effect	Targets	Notes
AllureAfter	Creates a tear down fixture from the method call.	Method.	Uses AspectInjector under the hood.
AllureAttachment	Creates an attachment from the method's return value.	Method.	Only supports string and byte[] return types. Uses AspectInjector under the hood.
AllureBddHierarchy	Adds the epic, feature, and story labels to test results at once.	Method, class/struct, interface.	story or epic and story can be omitted
AllureBefore	Creates a setup fixture from the method or constructor call.	Method, constructor.	Uses AspectInjector under the hood.
AllureDescription	Sets a description of test results.	Method, class/struct, interface.	Can be applied multiple times. By default, the most specific one wins. If Append is set, all texts are joined with \n\n.
AllureDescriptionHtml	Sets a description of test results in raw HTML.	Method, class/struct, interface.	Can be applied multiple times. By default, the most specific one wins. If Append is set, all texts are joined together with no separator.
AllureEpic	Adds the epic label to test results.	Method, class/struct, interface.	Discards the default BDD hierarchy.
AllureFeature	Adds the feature label to test results.	Method, class/struct, interface.	Discards the default BDD hierarchy.
AllureId	Applies the ALLURE_ID label to test results.	Method.	-
AllureIssue	Adds an issue link to test results	Method, class/struct, interface.	If a short ID is used instead of a full URL, a link template must exist in the config.
AllureLabel	Adds a custom label to test results.	Method, class/struct, interface.	-
AllureLink	Adds a link to test results.	Method, class/struct, interface.	If a short ID is used instead of a full URL, a link template must exist in the config.
AllureMeta	Applies a custom set of attributes to test results.	Method, class/struct, interface.	See #406.
AllureName	Sets a display name of test results.	Method, class/struct, interface.	When applied to a test class, affects the value of the default suite label created from this class.
AllureOwner	Applies the owner label to test results.	Method, class/struct, interface.	-
AllureParameter	Affects how method arguments are converted to Allure parameters.	Parameter.	Can be used to preclude parameters from the report, see #482.
AllureParentSuite	Applies the parentSuite label to test results.	Method, class/struct, interface.	Discards the default suite hierarchy.
AllureSeverity	Applies the severity label to test results.	Method, class/struct, interface.	-
AllureStep	Creates Allure steps from method calls.	Method.	Uses AspectInjector under the hood.
AllureStory	Applies the story label to test results.	Method, class/struct, interface.	Discards the default BDD hierarchy.
AllureSubSuite	Applies the subSuite label to test results.	Method, class/struct, interface.	Discards the default suite hierarchy.
AllureSuite	Applies the suite label to test results.	Method, class/struct, interface.	Discards the default suite hierarchy.
AllureSuiteHierarchy	Applies the parentSuite, suite, and subSuite labels at once.	Method, class/struct, interface.	subSuite or parentSuite and subSuite can be omitted
AllureTag	Applies the tag labels to test results.	Method, class/struct, interface.	-
AllureTmsItem	Applies a TMS item link to test results	Method, class/struct, interface.	If a short ID is used instead of a full URL, a link template must exist in the config.

Attribute targets

Most of the attributes can be applied to a method, a class/struct, or an interface, and are propagated down to test methods:

[AllureEpic("My epic")]
interface IAllureMetadata { }

[AllureFeature("My feature")]
class BaseClass { }

[AllureStory("My story")]
class MyTestClass : BaseClass, IAllureMetadata
{
    [Test]
    [AllureTag("foo")]
    public void MyTestMethod()
    {
    }
}

A test result of MyTestMethod receives labels from all four attributes: epic, feature, story, and tag.

This works in the same way for Xunit, which fixes #384.

The order of application

The following guarantees exist regarding the order in which the attributes are applied:

1. Interface attributes are applied before attributes of classes/structs.
2. Base class attributes are applied before attributes of derived classes.
3. Class/struct attributes are applied before attributes of methods.
4. Attributes of base methods (virtual or abstract) are applied before attributes of method overrides.

[AllureMeta]

A new [AllureMeta] attribute can be used to compose multiple attributes into a single one:

[AllureBddHierarchy("My epic", "My feature", "My story")]
[AllureTag("foo")]
[AllureOwner("John Doe")]
[AllureLink("https://allurereport.org")]
class MyAllureAttribute : AllureMetaAttribute { }

class MyTestClass
{
    [Test]
    [MyAllure]
    public void MyTestMethod()
    {
    }
}

A test result of MyTestMethod receives the labels and link defined by the attributes applied to MyAllureAttribute.

[AllureDescription] and [AllureDescriptionHtml]

These attributes set the description and descriptionHtml properties of test results accordingly. When applied on multiple levels (e.g., to a class and a method of the class), the most specific wins:

[AllureDescription("First description")]
class MyTestClass : BaseClass
{
    [Test]
    [AllureDescription("Second description")]
    public void MyTestMethod()
    {
        /*
        Second description
        */
    }

    [Test]
    public void MyOtherTestMethod()
    {
        /*
        First description
        */
    }
}

However, setting Append will join the descriptions together:

[AllureDescription("First description")]
class MyTestClass : BaseClass
{
    [Test]
    [AllureDescription("Second description", Append = true)]
    public void MyTestMethod()
    {
        /*
        First description

        Second description
        */
    }
}

Markdown descriptions ([AllureDescription]) are joined with a double newline (\n\n) to separate the paragraphs. HTML descriptions ([AllureDescriptionHtml]) are joined without a separator. Use block elements like <p> to keep the descriptions apart.

[AllureParameter]

A new attribute [AllureParameter] can now be applied to parameters of a test method:

class MyTestClass
{
    [Test]
    public void MyParameterizedTestMethod(
        [AllureParameter(Ignore = true)] IUserService users, // preclude from the test parameters in Allure
        [AllureParameter(Name = "User")] string userName, // change the name from "userName" to "User"
        [AllureParameter(Mode = ParameterMode.Masked)] string password, // mask the value with ***
        [AllureParameter(Excluded = true)] DateTime timestamp // don't use this argument to identify the test (e.g., because it changes from run to run)
    )
    {
    }
}

This allows to workaround #482.

Similarly, the attribute can be applied to parameters of a step method:

[AllureStep]
public void MyStep(
    [AllureParameter(Ignore = true)] IUserService users, // preclude from the step parameters in Allure
    [AllureParameter(Name = "User")] string userName, // change the name from "userName" to "User"
    [AllureParameter(Mode = ParameterMode.Masked)] string password // mask the value with ***
)
{
}

Note that Exclude has no effect on step parameters.

Other changes

• More tests for Allure.NUnit and Allure.Xunit. I will follow up with an optimization later that will allow samples to be nested with test classes, as having one Samples folder with billions of samples inside isn't very convenient.
• Links with no type are treated as having the link type when matching against the templates.
• Support fixture parameters by [AllureBefore] and [AllureAfter]
• Include type name in the constructor-based setup fixtures.
• Fix data race condition for tests that use a static lifecycle instance.
• Add RunInContextAsync to AllureLifecycle.
• NUnit's descriptions are ignored if the description has already been provided via the Allure API.
• Multiple NUnit descriptions are now joined with two newline characters (\n\n) to split markdown paragraphs.
• The application order of NUnit properties (Description, Author, and Category) is now from top to bottom (test fixture -> test -> test case). Previously, it was from bottom to top.

Known issues

[AllureAfter] on NUnit's [OneTimeTearDown]

The [AllureAfter] attribute applied to a [OneTimeTearDown] method of an NUnit test fixture won't emit the teardown fixture:

using Allure.Net.Commons.Attributes;
using NUnit.Framework;

class TestClass
{
    [Test]
    public void TestMethod() { }

    [AllureAfter]
    [OneTimeTearDown]
    public void Clean() { } // this fixture won't be present in the report
}

The reason is that when ITestAction.After is called on the test fixture level, the [OneTimeTearDown] method hasn't been called yet. If we emit the container from here, it will miss the fixture result.

Originally, we solved this by emitting the container from a custom aspect injected into Allure.NUnit.Attributes.AllureAfterAttribute. The new attribute doesn't have access to this aspect. Instead, an SDK event system should be introduced that allows integrations to run custom code on events such as fixture stop.

Closes #384.
Closes #406.
Closes #427.

Checklist

• Sign Allure CLA
• Provide unit tests