🌱 Maintained Check: New probe: Do maintainers respond to public bug and security issues?

[AdamKorcz]

What kind of change does this PR introduce?

New probe

• PR title follows the guidelines defined in our pull request documentation

This PR adds new probes to the Maintained check that assess whether a project's maintainers respond respond to public issues with "bug" and "security" related labels. In essence, the core of this new probe is to assess how many of a project's issues labelled "bug" or "security" receive no maintainer response within the first 180 days after the label has been added.

This is a form of inactivity probe of a project and provides a signal about a project's practices with public "bug" and "security" issues. The probe aims to make it easy for projects to perform well in this probe and as it is now can be useful in assessing whether a project is very poor in responding to public issues. As such, the goal is to identify obvious bad behaviours instead of differentiating between fine-grained limits. Said otherwise, the probe sets a high bar for scoring low. If a project scores low, it can be a signal of poor maintenance. In my opinion it is a useful signal in and of itself as well as an indicator of potential underlying issues.

At a more technical level, the probe supports GitHub and Gitlab-hosted projects. The probe uses the existing issuesHandlers and fetches the issues of a repository, sorts the ones that are relevant for the probe - the ones that have had a "bug" and "security" label at some point - and then checks the history of each issue. Finally, the probe will do a relative scoring between all the issues that were labelled with a "bug" or "security" label at any point and the ones that have not had any maintainer activity within 180 days of the issue being labelled "bug" or "security".

Some example scenarios are:

1. A project has 10 issues with "bug" and "security" labels.
2. A maintainer has responded with comments to 4 of these within 50 days.
3. A maintainer has closed 4 of these within 100 days.
4. The remaining 2 issues have had the "bug" label for 5 years.

The project will score 10 because the maintainers have responded to 80% of the issues.

Another scenario:

1. A project has 10 issues with "bug" and "security" labels.
2. A maintainer has responded with comments on 4 of these within 100 days.
3. A maintainer has removed the "bug" from 2 of these within 100 days.
4. A maintainer has added a "triage" label to 1 of these within 100 days.
5. The remaining 3 issues have had the "security" label for 5 years.

The project will score 5 because maintainers have responded to 70% of these within 180 days.

In its current state, the probe does account for some edge cases such as:

1. If the issue gets closed and reopened - the probe will still check for any maintainer response and will only count the time it was open.
2. The probe shouldn't consider issues that have not been open for more than 180 days. In other words, it should count the time issues have had a "bug" or "security" label in an open state.
3. Different activity types. The probe should consider every possible maintainer activity.

That being said, there will likely be further edge cases that can be explored through usage.

Special notes for your reviewer

How this differs from the existing Maintained probes:

Currently, the Maintained check has four probes:

1. one that checks whether a project is archived
2. one that checks whether there has been recent commits
3. one that checks for activities on issues by project members
4. one that checks whether the repo was created recently.

The probe in this PR is related to probe 3 in that list, however has several differences: The existing probes checks for recent activity in issues. The project scores low if there is no recent activity. The probe in this PR checks whether the project maintainers respond to issues labelled "bug" or "security" within 180 days of the time the issue was assigned the label. As such, the probe in this PR does not check for recent activity from the date and time the user runs Scorecard; instead it considers the time between issues being labelled "bug" or "security" and 180 days after.

Does this PR introduce a user-facing change?

yes

Add new probe to Maintained check for whether maintainers respond to public bug and security issues

[codecov]

Codecov Report

❌ Patch coverage is 66.23877% with 263 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.46%. Comparing base (353ed60) to head (7502756).
⚠️ Report is 303 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4867      +/-   ##
==========================================
+ Coverage   66.80%   69.46%   +2.66%     
==========================================
  Files         230      256      +26     
  Lines       16602    16400     -202     
==========================================
+ Hits        11091    11393     +302     
+ Misses       4808     4116     -692     
- Partials      703      891     +188     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[spencerschrock]

This PR adds a new check for maintainer inactivity on public issues with "bug" and "security" related labels

This seems like a small tweak based on the existing issueActivityByProjectMember probe, where the key difference is the presence of a label, and perhaps a larger time frame.

Is this significant enough to warrant its own check vs. evaluating tweaking the existing Maintained?

[spencerschrock]

Another comment brought up during today's community meeting:

lots of projects have lots of slight variations in label names, e.g. we have kind/bug instead of just bug

[AdamKorcz]

Another comment brought up during today's community meeting:

lots of projects have lots of slight variations in label names, e.g. we have kind/bug instead of just bug

Thank you for the feedback. This is taken into account in the current version:

https://github.com/ossf/scorecard/pull/4867/files#diff-251cadad7d5fc85b28fb52385203719309773dbc2bc7bf8e8997f562eca9daf8R28-R34

From what I have found, those are by far the most popular ones, however, it will be very easy to include more label names.

[github-actions]

This pull request has been marked stale because it has been open for 10 days with no activity

[AdamKorcz]

Is this significant enough to warrant its own check vs. evaluating tweaking the existing Maintained?

IMO, this should definitely fit in the Maintained check. I think the biggest blocker for me to place it there is to update the evaluation. The scoring in this check could add a negative counter to the check: It could use more or less the same tiers but in a negative manner, so that it would reduce the score being created here:

scorecard/checks/evaluation/maintained.go

Lines 97 to 100 in 525a930

	return checker.CreateProportionalScoreResult(name, fmt.Sprintf(
	"%d commit(s) and %d issue activity found in the last %d days",
	commitsWithinThreshold, numberOfIssuesUpdatedWithinThreshold, lookBackDays),
	commitsWithinThreshold+numberOfIssuesUpdatedWithinThreshold, activityPerWeek*lookBackDays/daysInOneWeek)

[github-actions]

This pull request has been marked stale because it has been open for 10 days with no activity

[github-actions]

This pull request has been marked stale because it has been open for 10 days with no activity