Filtering broken at nesting level > 2 ?

[marius311]

I'm not sure exactly what's happening here, but this seems broken with v0.3.1?

julia> module Foo

       using ReTest

       @testset "$N" for N = ["a", "b"]
           @testset "$M" for M = [1, 2]
               @testset "$P" for P = ["α", "β"]
                   @test true
               end
           end
       end

       end
Main.Foo

julia> Foo.retest("1", dry=true, verbose=3)
Main.Foo
1| a
2|   1
3|     α
3|     β
2|   2
1| b
2|   1
3|     α
3|     β
2|   2

If instead I only do 2 levels of nesting, the M=1 tests are instead correctly picked out.

[marius311]

Ahh I see, the M=2 is printed but its "empty" no tests get run. That kind of confused me and I expected to not see any M=2 in the list, so perhaps its better to hide empty testsets? Or maybe this is intended in which case please feel free to close.

[rfourquet]

Thanks for your report!

Yes this is a slightly tricky behavior, which is not explained in the docs besides vague statements specifying only which tests are guaranteed to run, and saying that which tests exactly are run might change.

So this is partly a documentation issue. What happens is that there are two phases:

1. filtering testsets
2. running selected testsets, while still filtering only "final" testsets, those which don't have nested testsets.

In phase 2), we redo some filtering because of two reasons:
a) it can happen that phase 1 was not able to give a definitive answer (e.g. in case where the description couldn't be fully known, when there are interpolated variables)
b) phase 1) selects testset at their "static" level, i.e. a @testset for ... is considered as one entity, and if it's selected, all of its iterations are selected. So we like to have more effective filtering in phase 2 for final testsets.

We can't filter out non-final testsets in phase 2, because we don't know yet whether some of their children will have to run.

I hope this helps. I should definitely put something along those lines in the documentation, and I also started exploring filtering testsets in phase 1 in a more fine-grained way, i.e. by unrolling the loops and selecting which iterations have to run. I'm not sure yet what the performance impact will be, but in some cases, it definitely is very useful to cut out the noise.

In the meantime, a good idea is to put heavy computations (the bulk of the @test ...) in final testsets, and to use non-final ones mainly as containers for other nested @testsets.