Setup/Cleanup and various improvement about Sequences

[DoubleNom]

One common need that has been observed using Frasy is that we sometime needs to have Setup/Cleanup for tests, or even the whole procedure.

To do so, we currently use the following work-around. Using Setup/Cleanup sequences and tests with proper order requirements.

Sequence("Setup", function()
    Requires(Sequence():ToBeFirst())
    Test("Setup", function())
end)

Sequence("Seq", function()
    Test("Setup", function() 
        Requires(Test():ToBeFirst())
    end)
    Test("T1", function() 
        Requires(Test("Setup"):ToBeBefore())
        Requires(Test("Cleanup"):ToBeAfter())
        --- My stuff
    end)
    Test("T2", function()
        Requires(Test("Setup"):ToBeBefore())
        Requires(Test("Cleanup"):ToBeAfter())
        --- My stuff
    end)
    Test("Cleanup", function() 
        --- 
    end)
    Test("T3", function()
        -- My Stuff
    end)
end)

Sequence("Cleanup", function()
    Requires(Sequence():ToBeLast())
    Test("Cleanup", function())
end)

While this works, it brings some issues with it.

1. Setup/Cleanup appear in log report.
   This is not a big issue, but it still increase the log report and its weird to see irrelevant data.

2. It can create a mess of order requirements
   The previous example show two issues. First, setup/cleanup consume the sequences edge requirement (first/last). This mean that if we had another sequence that required to be first, we will now need to use ToBeAfter for all others sequences. While this issue usually happens when you have sequences that requires a lot of ordering, reducing the risk of occurence would be better. Secondly, there is an issue on that example!! Acute reader might have realize that T3 might happen anywhere during the Seq sequence. Before, in-between or after the Setup/Cleanup. If we want to be sure that T3 does not happens in-between these sequence, we would either have to put it in a different sequence or use an order-requirement to be before setup or after cleanup.

Propositions

Replace Sequence by Group

This serve two purpose. First it change the name that does not properly carry information. While a Sequence does imply a combinaison, it also might imply that it is something that is ran.
In reality, we never run the content of a sequence. It is only there to make a group of tests, hence the new name.

But here is the other proposition, group can be a group of test, but also groups. So that in the previous example, we could have the following.

Group("G1", function()
    Group("G1-1")
        Test("Setup", function() 
            Requires(Test():ToBeFirst())
        end)
        Test("T1", function() 
            Requires(Test("Setup"):ToBeBefore())
            Requires(Test("Cleanup"):ToBeAfter())
            --- My stuff
        end)
        Test("T2", function()
            Requires(Test("Setup"):ToBeBefore())
            Requires(Test("Cleanup"):ToBeAfter())
            --- My stuff
        end)
        Test("Cleanup", function() 
            --- 
        end)
    end)
    Test("T3", function()
        -- My Stuff
    end)
end)

It is now implicit that T3 must be either run before or after Group G1-1, and no Order requirement were used.

Setup/Cleanup calls hidden from logs

This section is less clear and require discussion.
Reported issue from sam is that they want to be able to run setup, in which they check it went smoothly, and if not, prevent from running subsequent tests.
They want then this setup to be hidden from report.

You can also imagine case where setup checks nothing, and you want it hidden from reports.

So, a way to run Function, that are hidden from report could be interresting.
One could imagine Setup/Cleanup calls, that each group can have one of them.

Setup would allows to return either nothing, which imply setup was successful, or a more complex table telling if subsequent can run, and why if they cant.
Cleanup would be a basic function that return nothing.

Example

Setup(function()
    --- Global Setup
    --- Return nothing, always successful
end)

Group("G1", function()
    Group("G1-1")
        Setup(function()
            -- G1-1 Setup
            -- Complicated stuff done here
            -- Return something telling if successful
        end)
        Test("T1", function() 
            --- My stuff
        end)
        Test("T2", function()
            --- My stuff
        end)
        Cleanup(function() 
            --- G1-1 Cleanup
            --- Also complicated stuff done here
            --- No return
        end)
    end)
    Test("T3", function()
        -- My Stuff
    end)
end)

Cleanup(function()
    --- Global Cleanup
end)

One drawback from this method is that we would lose the time information for these setup/cleanup phase.