Skip to content

Restrict access to "merge despite muted tests" feature #33

@jjohannes

Description

@jjohannes

Currently, when you mute tests on a PR, the checks that run the tests are ✅ despite the fact that tests were skipped. As GitHub does not know why these checks are ✅ it allows merging the PR independent of the PR owner and the merge rule configuration of the repository. This poses the risk of PRs being merged that have tests which are still muted.

The current mute feature primarily aims at two usage scenarios:

  1. Temporary muting: The user wants to focus on the real problems first. They mute the tests they know that are likely flaky to not distract them while working on the PR. Before they merge however, the re-activate the tests and rerun them – ideally with the reuse successful feature – until they pass. This scenario is shown in our announcement video.
  2. Ignore known flakiness that persists for a certain time period: In this scenario, certain tests constantly fail due to some environmental issue (such as outage of an external resource). In the project/team, there is agreement that merges can be performed despite these tests being muted. Currently, TestLens does not provide any features to control this further (e.g. only allow certain tests in this category). More features in this area of flaky test management are planned for the future.

For certain projects, scenario 2 is not applicable as they strictly need all tests to pass on PRs. In the current phase, these teams may still profit from scenario 1, but they need a feature that prevents accidental merges of PRs with muted tests.

The next step could be one of the following:

  • A - Add a global configuration options to disable the muting feature completely
  • B - Add a global configuration option to only enable the feature to certain users/groups (e.g. only repo maintainers)
  • C - Create an additional no tests were muted check (NOT run) that the TestLens App attaches to the PR if there are muted tests. Then the PR will not be completely ✅ but have one ❌ for a no tests were muted check. This gives more control to the repo maintainers, who can establish rules that allow or disallow merging of PRs where this check failed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions