This repository was archived by the owner on Jul 29, 2026. It is now read-only.
Fix scoring engine: check bucketing, path classification, blocker language, GitLab merge status - #36
Closed
JackSpiece wants to merge 3 commits into
Closed
Conversation
- ci_must_pass and not_approved no longer map to CHANGES_REQUESTED. Both mean "nobody has reviewed/CI has not passed yet", not "a human asked for changes", so they were adding +25 risk and forcing "needs author follow-up" on merge requests where the author owed nothing. Both are still surfaced through mergeStateStatus (BLOCKED) and the pipeline check. - Add blocked, canceling, and waiting_for_callback to the GitLab pending pipeline statuses so a manual gate is reported as pending instead of falling through to an unrecognised conclusion.
summarize_checks: - Read legacy commit statuses (`state`) instead of silently dropping them. A repo whose CI reports through commit statuses scored as "no visible checks" while it was red. - CANCELLED and STALE no longer count as failures; they carry no verdict. - ACTION_REQUIRED is pending, not a failure. - An unknown or verdict-less conclusion buckets to skipped instead of pending, so the tool stops advising "wait for CI" forever. - Guard against non-dict rollup entries. summarize_files: - Replace substring hint matching with segment/basename/stem/extension anchoring. "src/latest_release.py" and "src/contest_manager.py" were counted as tests; "src/license_checker.py" was counted as documentation. - Categories are now mutually exclusive and evaluated generated > test > doc > code, so one file can no longer be both a test and code. - Extend CODE_EXTENSIONS and add CODE_BASENAMES so Terraform, workflow YAML, Dockerfiles, and Makefiles count as code. - Root-only build-output segments, so "src/build/compiler.py" stays code. - User-supplied config hints keep their historical substring behaviour. Blocker language: - Ignore comments and reviews from bot accounts. Coverage and CI bots post "3 checks failed" and "this PR is blocked", which was scored as maintainer blocker language and cost the author 25 risk. - Strip code fences, HTML comments, quoted lines, and inline code before matching, so a pasted CI log is not read as maintainer feedback. Risk: - "no visible checks" now keys off conclusive checks rather than checks.total, so an all-skipped or all-cancelled run no longer reads as "CI passed". - Wrap the 116-char author expression that was breaking the line limit.
24 new tests across three classes: CheckRollupTests - legacy commit statuses (FAILURE/ERROR), cancelled and skipped runs not reading as pass or fail, ACTION_REQUIRED as pending, STARTUP_FAILURE as a failure, and unknown conclusions not stalling the tool on "wait for CI" forever. FileClassificationTests - src/latest_release.py and src/contest_manager.py are code not tests; src/license_checker.py is code not documentation; real test paths (tests/, .test.tsx, __tests__/, _spec.rb) still detected; LICENSE, README.md, docs/*.rst still documentation; Terraform, Dockerfile and workflow YAML count as code; root build/ is generated output but src/build/ is source; src/generated_ids.py is not a generated file. BlockerLanguageTests - bot comments by login suffix and by type are ignored, quoted and fenced CI logs are stripped before matching, and a genuine maintainer comment still trips the flag. Total suite: 48 tests.
Owner
Author
|
Closing this because I am no longer pursuing this work. The branch and commit history are preserved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes correctness bugs in the triage engine that were producing wrong advice. No public API, dataclass shape, or CSV/render contract was changed, so
render.py,cli.py, and the CSV columns are untouched.What was wrong
1.
summarize_checkssilently dropped legacy commit statusesGitHub's
statusCheckRollupmixesCheckRun(status+conclusion) withStatusContext(stateonly).StatusContextentries matched no branch, so a repository whose CI reports through commit statuses (CircleCI, Jenkins, Buildkite, most non-Actions CI) was scored "no visible checks" while it was bright red. That is the single worst bug here: the tool told maintainers a broken PR was safe to review.2. Cancelled runs were scored as failures
CANCELLEDandSTALEcarry no verdict. Reporting them as failures made the tool tell maintainers to chase authors over runs that were superseded by a newer push or manually stopped.3. Unknown conclusions bucketed as pending
Any conclusion the code did not recognise fell into
pending, sochoose_actionreturned "wait for CI" forever for runs that will never report again. Unknown/verdict-less conclusions now bucket toskipped.4.
ACTION_REQUIREDwas treated as a failureIt means a human still has to approve or resume the run. It is pending, not broken.
5. An all-skipped or all-cancelled run read as "CI passed"
The
no visible checksbranch keyed offchecks.total, which counted skipped runs. It now keys offpassed + failed + pending.6. File classification matched bare substrings
This is the bug with the widest blast radius, because
code_filesandtest_filesfeed thecode changed without tests(+10) andno test plan found(+8) flags.src/latest_release.pytest)src/contest_manager.pytest)src/license_checker.pylicense)src/readme_generator.pyreadme)src/generated_ids.pysrc/build/compiler.pydeploy/main.tfDockerfile,Makefile.github/workflows/*.ymlClassification now anchors on whole path segments, basenames, stems, and extensions, and the four categories are mutually exclusive and evaluated generated > test > doc > code, so a file can no longer be counted as both a test and code. Build-output directory names (
build,out,target,coverage,.next) only count at the repository root.User-supplied
test_hints/doc_hints/generated_hintsfrom config deliberately keep their historical substring behaviour, so no existing.maintainer-radar.jsonchanges meaning.7. Bot comments were scored as maintainer feedback
Codecov posts "3 checks failed". Dependabot posts "this PR is blocked". Both matched
BLOCKER_RE, added +25 risk, and forced the PR to "needs author follow-up" with the message "respond to unresolved maintainer feedback" — for a comment no maintainer wrote. Bots are now detected by login suffix ([bot],-bot,_bot), by a known-bot list, and bytype/__typename/is_bot.8. Pasted CI logs were scored as maintainer feedback
A maintainer quoting a failing log, or pasting it in a code fence, tripped the same +25 flag on the words in the log. Code fences, HTML comments, quoted (
>) lines, and inline code are now stripped before matching.9. GitLab
ci_must_passandnot_approvedmapped toCHANGES_REQUESTEDNeither means a human asked for changes.
ci_must_passis a pipeline gate;not_approvedmeans review has not happened yet. Both were adding +25 risk and routing the merge request to "needs author follow-up" when the author owed nothing. Onlydiscussions_not_resolvedmaps toCHANGES_REQUESTEDnow. Both conditions are still visible throughmergeStateStatus: BLOCKED(+6) and the pipeline check.10. GitLab manual-gate pipelines
blocked,canceling, andwaiting_for_callbackwere missing from the pending set, so a manual gate fell through to an unrecognised conclusion instead of reporting as pending.Deliberate non-changes
has_body = "body" in pris unchanged, on purpose. An earlier read of this called it a bug — that was wrong. A shallow scan omits thebodykey entirely, while a real PR with an empty description hasbody: "". Treating an empty string as "no body" would waive the missing-test-plan penalty for authors who wrote nothing at all. The current check is correct..jsonis not inCODE_EXTENSIONS. Most JSON in a diff is fixtures, lockfiles, or generated schema. Counting it as code would firecode changed without testson data-only PRs. Specific generated JSON names are already handled byGENERATED_BASENAMES.CheckSummaryandFileSummaryfield names are unchanged, becauserender.pyand the CSV exporter consume them through__dict__.Tests
24 new tests, 48 total. New classes:
CheckRollupTests,FileClassificationTests,BlockerLanguageTestsintests/test_scoring.py, andGitlabMergeStatusTestsintests/test_normalize.py. Every bug above has a test that fails onmain.Also wraps a 116-character line in
analyze_prthat exceeded the configuredline-length = 110.Still outstanding (not in this PR)
_render_markdown_rowinrender.pyinterpolates rawtitle_textinto a Markdown table cell while the adjacent_markdown_pr_labelcorrectly routes throughmarkdown_cell(). A PR title containing|or a Markdown link breaks or injects into the generated report. Left alone becauserender.pyand its 32 KB test file were out of scope for this change.mypyis stillcontinue-on-error: truein CI, so type regressions cannot fail a build.ci.ymlhas nofail-fast: false, so one matrix leg failing hides the others.v0.21.0tag exists despite__version__ = "0.21.0"and the README pinning@v0.21.0.maintainer-radaris not published on PyPI, so the README'spip install maintainer-radardoes not work.