Test classes with compilation errors pass

[Ovid]

This is Test::Class::Moose version 0.91.

On a number of occasions we've found that our test classes have a syntax error and are skipped, but the test passes. Below is an example:

#!/usr/bin/env perl

use strict;
use warnings;

use Test::Class::Moose::Runner;

eval <<'END_CLASS';
package BadClass {
    use Test::Class::Moose;

    sub test_something {
        ok $undeclared_variable, 'You shall not pass';
    }
}
END_CLASS
eval <<'END_CLASS';
package GoodClass {
    use Test::Class::Moose;

    sub test_something_else {
        ok 1, 'OK, you can pass';
    }
}
END_CLASS

my $runner = Test::Class::Moose::Runner->new;
$runner->runtests;

And when you run that:

tcm.t ..
1..2
ok 1 - BadClass {
    1..0 # SKIP Skipping 'BadClass': no test methods found
}
ok 2 - GoodClass {
    1..1
    ok 1 - test_something_else {
        ok 1 - OK, you can pass
        1..1
    }
}
ok
All tests successful.
Files=1, Tests=2,  1 wallclock secs ( 0.01 usr  0.00 sys +  0.25 cusr  0.02 csys =  0.28 CPU)
Result: PASS

I suspect this may have the same underlying issue as #70.

[autarch]

Actually, in this particular case it's not really fixable in TCM. The fact that BadClass exists and is a subclass of TCM is an artifact of how Perl compiles stuff. The package and use statements are executed at compile time, before the compilation error from the undeclared var. There's nothing we can do about it.

However, what you should be doing is checking $@ after each eval.

So my question is: how are you loading your test classes? I tried putting each of these classes in a .pm file and loading them with TCM::Load. It dies with an error as I'd want.

[glauschwuffel]

How about adding this to the documentation?

[autarch]

@glauschwuffel What exactly do you think should be documented?

[glauschwuffel]

@autarch From Ovids text I understand that failed attempts to dynamically add test classes results in these classes being listed as test to be skipped. So maybe this small information could be added in a CAVEAT section or something like that.

[autarch]

@glauschwuffel If you wanted to make a PR for this that would be great.