You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The closer we get to addressing #171 the more difficult it becomes to test Checker. Some thoughts:
Unit tests: Our testsuite contains many unit tests that tightly check edge cases, which are specific to our current choice of parameters (specified in checker.yaml). Any change in the value of these parameters can make the testsuite fail, since the edge values change.
Property-based tests: Most property-based randomized tests should be able to pass even with a different choice of parameters, since these tests mostly describe general properties we expect to hold, that are mostly insensitive to the specific choice of parameters. However, now that we generate more code at build time, even property-based tests cannot always be shared. For example, different curves for the drift derivative (see #283) can be expected to have different properties.
Mutation testing: Apart from the challenge of keeping test coverage high while the codebase changes so fast, the results of mutation testing also become less useful:
We now rely on tezWrapper which is used meaningfully only in e2e tests. Mutating it thus does not yield meaningful results.
Several files have turned into jinja templates, which at the moment (reasonably) are not mutated.
Certainly Checker will be shipped with a set of default values and we need the testsuite to have sufficient coverage for the default configuration, but ideally we'd have a way to test a wider range of configurations. It is unclear how to achieve that at the moment, but it looks like we'll need to separate (a) general properties that we expect to hold independently of the choice of values in checker.yaml (assuming that they are reasonably set, of course) from (b) configuration-specific tests.
The closer we get to addressing #171 the more difficult it becomes to test Checker. Some thoughts:
Unit tests: Our testsuite contains many unit tests that tightly check edge cases, which are specific to our current choice of parameters (specified in
checker.yaml). Any change in the value of these parameters can make the testsuite fail, since the edge values change.Property-based tests: Most property-based randomized tests should be able to pass even with a different choice of parameters, since these tests mostly describe general properties we expect to hold, that are mostly insensitive to the specific choice of parameters. However, now that we generate more code at build time, even property-based tests cannot always be shared. For example, different curves for the drift derivative (see #283) can be expected to have different properties.
Mutation testing: Apart from the challenge of keeping test coverage high while the codebase changes so fast, the results of mutation testing also become less useful:
tezWrapperwhich is used meaningfully only in e2e tests. Mutating it thus does not yield meaningful results.Certainly Checker will be shipped with a set of default values and we need the testsuite to have sufficient coverage for the default configuration, but ideally we'd have a way to test a wider range of configurations. It is unclear how to achieve that at the moment, but it looks like we'll need to separate (a) general properties that we expect to hold independently of the choice of values in
checker.yaml(assuming that they are reasonably set, of course) from (b) configuration-specific tests.