Skip to content
This repository was archived by the owner on Jul 29, 2026. It is now read-only.

Fix scoring engine: check bucketing, path classification, blocker language, GitLab merge status - #36

Closed
JackSpiece wants to merge 3 commits into
mainfrom
fix/scoring-classification-and-check-buckets
Closed

Fix scoring engine: check bucketing, path classification, blocker language, GitLab merge status#36
JackSpiece wants to merge 3 commits into
mainfrom
fix/scoring-classification-and-check-buckets

Conversation

@JackSpiece

Copy link
Copy Markdown
Owner

Fixes correctness bugs in the triage engine that were producing wrong advice. No public API, dataclass shape, or CSV/render contract was changed, so render.py, cli.py, and the CSV columns are untouched.

What was wrong

1. summarize_checks silently dropped legacy commit statuses

GitHub's statusCheckRollup mixes CheckRun (status + conclusion) with StatusContext (state only). StatusContext entries matched no branch, so a repository whose CI reports through commit statuses (CircleCI, Jenkins, Buildkite, most non-Actions CI) was scored "no visible checks" while it was bright red. That is the single worst bug here: the tool told maintainers a broken PR was safe to review.

2. Cancelled runs were scored as failures

CANCELLED and STALE carry no verdict. Reporting them as failures made the tool tell maintainers to chase authors over runs that were superseded by a newer push or manually stopped.

3. Unknown conclusions bucketed as pending

Any conclusion the code did not recognise fell into pending, so choose_action returned "wait for CI" forever for runs that will never report again. Unknown/verdict-less conclusions now bucket to skipped.

4. ACTION_REQUIRED was treated as a failure

It means a human still has to approve or resume the run. It is pending, not broken.

5. An all-skipped or all-cancelled run read as "CI passed"

The no visible checks branch keyed off checks.total, which counted skipped runs. It now keys off passed + failed + pending.

6. File classification matched bare substrings

This is the bug with the widest blast radius, because code_files and test_files feed the code changed without tests (+10) and no test plan found (+8) flags.

Path Was classified Now
src/latest_release.py test (contains test) code
src/contest_manager.py test (contains test) code
src/license_checker.py doc (contains license) code
src/readme_generator.py doc (contains readme) code
src/generated_ids.py generated code
src/build/compiler.py generated code
deploy/main.tf uncounted code
Dockerfile, Makefile uncounted code
.github/workflows/*.yml uncounted code

Classification now anchors on whole path segments, basenames, stems, and extensions, and the four categories are mutually exclusive and evaluated generated > test > doc > code, so a file can no longer be counted as both a test and code. Build-output directory names (build, out, target, coverage, .next) only count at the repository root.

User-supplied test_hints / doc_hints / generated_hints from config deliberately keep their historical substring behaviour, so no existing .maintainer-radar.json changes meaning.

7. Bot comments were scored as maintainer feedback

Codecov posts "3 checks failed". Dependabot posts "this PR is blocked". Both matched BLOCKER_RE, added +25 risk, and forced the PR to "needs author follow-up" with the message "respond to unresolved maintainer feedback" — for a comment no maintainer wrote. Bots are now detected by login suffix ([bot], -bot, _bot), by a known-bot list, and by type / __typename / is_bot.

8. Pasted CI logs were scored as maintainer feedback

A maintainer quoting a failing log, or pasting it in a code fence, tripped the same +25 flag on the words in the log. Code fences, HTML comments, quoted (>) lines, and inline code are now stripped before matching.

9. GitLab ci_must_pass and not_approved mapped to CHANGES_REQUESTED

Neither means a human asked for changes. ci_must_pass is a pipeline gate; not_approved means review has not happened yet. Both were adding +25 risk and routing the merge request to "needs author follow-up" when the author owed nothing. Only discussions_not_resolved maps to CHANGES_REQUESTED now. Both conditions are still visible through mergeStateStatus: BLOCKED (+6) and the pipeline check.

10. GitLab manual-gate pipelines

blocked, canceling, and waiting_for_callback were missing from the pending set, so a manual gate fell through to an unrecognised conclusion instead of reporting as pending.

Deliberate non-changes

  • has_body = "body" in pr is unchanged, on purpose. An earlier read of this called it a bug — that was wrong. A shallow scan omits the body key entirely, while a real PR with an empty description has body: "". Treating an empty string as "no body" would waive the missing-test-plan penalty for authors who wrote nothing at all. The current check is correct.
  • .json is not in CODE_EXTENSIONS. Most JSON in a diff is fixtures, lockfiles, or generated schema. Counting it as code would fire code changed without tests on data-only PRs. Specific generated JSON names are already handled by GENERATED_BASENAMES.
  • CheckSummary and FileSummary field names are unchanged, because render.py and the CSV exporter consume them through __dict__.

Tests

24 new tests, 48 total. New classes: CheckRollupTests, FileClassificationTests, BlockerLanguageTests in tests/test_scoring.py, and GitlabMergeStatusTests in tests/test_normalize.py. Every bug above has a test that fails on main.

Also wraps a 116-character line in analyze_pr that exceeded the configured line-length = 110.

Still outstanding (not in this PR)

  • _render_markdown_row in render.py interpolates raw title_text into a Markdown table cell while the adjacent _markdown_pr_label correctly routes through markdown_cell(). A PR title containing | or a Markdown link breaks or injects into the generated report. Left alone because render.py and its 32 KB test file were out of scope for this change.
  • mypy is still continue-on-error: true in CI, so type regressions cannot fail a build.
  • ci.yml has no fail-fast: false, so one matrix leg failing hides the others.
  • No v0.21.0 tag exists despite __version__ = "0.21.0" and the README pinning @v0.21.0.
  • maintainer-radar is not published on PyPI, so the README's pip install maintainer-radar does not work.

- ci_must_pass and not_approved no longer map to CHANGES_REQUESTED. Both
  mean "nobody has reviewed/CI has not passed yet", not "a human asked for
  changes", so they were adding +25 risk and forcing "needs author
  follow-up" on merge requests where the author owed nothing. Both are
  still surfaced through mergeStateStatus (BLOCKED) and the pipeline check.
- Add blocked, canceling, and waiting_for_callback to the GitLab pending
  pipeline statuses so a manual gate is reported as pending instead of
  falling through to an unrecognised conclusion.
summarize_checks:
- Read legacy commit statuses (`state`) instead of silently dropping them.
  A repo whose CI reports through commit statuses scored as "no visible
  checks" while it was red.
- CANCELLED and STALE no longer count as failures; they carry no verdict.
- ACTION_REQUIRED is pending, not a failure.
- An unknown or verdict-less conclusion buckets to skipped instead of
  pending, so the tool stops advising "wait for CI" forever.
- Guard against non-dict rollup entries.

summarize_files:
- Replace substring hint matching with segment/basename/stem/extension
  anchoring. "src/latest_release.py" and "src/contest_manager.py" were
  counted as tests; "src/license_checker.py" was counted as documentation.
- Categories are now mutually exclusive and evaluated generated > test >
  doc > code, so one file can no longer be both a test and code.
- Extend CODE_EXTENSIONS and add CODE_BASENAMES so Terraform, workflow
  YAML, Dockerfiles, and Makefiles count as code.
- Root-only build-output segments, so "src/build/compiler.py" stays code.
- User-supplied config hints keep their historical substring behaviour.

Blocker language:
- Ignore comments and reviews from bot accounts. Coverage and CI bots post
  "3 checks failed" and "this PR is blocked", which was scored as
  maintainer blocker language and cost the author 25 risk.
- Strip code fences, HTML comments, quoted lines, and inline code before
  matching, so a pasted CI log is not read as maintainer feedback.

Risk:
- "no visible checks" now keys off conclusive checks rather than
  checks.total, so an all-skipped or all-cancelled run no longer reads as
  "CI passed".
- Wrap the 116-char author expression that was breaking the line limit.
24 new tests across three classes:

CheckRollupTests - legacy commit statuses (FAILURE/ERROR), cancelled and
skipped runs not reading as pass or fail, ACTION_REQUIRED as pending,
STARTUP_FAILURE as a failure, and unknown conclusions not stalling the
tool on "wait for CI" forever.

FileClassificationTests - src/latest_release.py and src/contest_manager.py
are code not tests; src/license_checker.py is code not documentation;
real test paths (tests/, .test.tsx, __tests__/, _spec.rb) still detected;
LICENSE, README.md, docs/*.rst still documentation; Terraform, Dockerfile
and workflow YAML count as code; root build/ is generated output but
src/build/ is source; src/generated_ids.py is not a generated file.

BlockerLanguageTests - bot comments by login suffix and by type are
ignored, quoted and fenced CI logs are stripped before matching, and a
genuine maintainer comment still trips the flag.

Total suite: 48 tests.
@JackSpiece

Copy link
Copy Markdown
Owner Author

Closing this because I am no longer pursuing this work. The branch and commit history are preserved.

@JackSpiece JackSpiece closed this Jul 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant