Remove confidence scores from visible output, fix CI lint cache - #134
Merged
Conversation
Confidence scores are no longer rendered in terminal output or published comments. They remain available in the JSON output and the hidden review envelope only. This simplifies the visible feedback by focusing on the verdict badge and explanation. - Remove confidence lines from terminal formatter output - Remove confidence lines from SCM markdown rendering (GitHub/GitLab) - Update documentation to reflect new behavior - Update tests and golden files to match changes
- Add assertion to TestTerminalFormatter to ensure confidence scores are not rendered in terminal output. - Update TestCarrierNotesReassemble to include confidence scores in test findings and verify they are preserved in the carrier envelope after reassembly.
Move from t.Fatalf to t.Errorf for better error reporting in concurrent and loop-based tests. Add explicit returns after fatal errors to ensure subsequent assertions are skipped. Consolidate modernize analyzers into golangci-lint configuration to unify local and CI checks. Remove the separate Makefile target and CI step.
A restored golangci-lint analysis cache loses staticcheck's fact that
(*testing.common).Fatal never returns. Every `if x == nil { t.Fatal(...) }`
followed by a dereference is then reported as SA5011, which is why CI kept
failing on code that lints clean locally. A cold run recomputes the fact and
reports nothing, including for a deliberately constructed probe.
- Pass skip-cache to golangci-lint-action so CI matches a clean local run.
- Lift max-same-issues and max-issues-per-linter, which defaulted to 3 and
turned one pattern into several rounds of CI failures.
- Correct the make lint comment, which had the cause backwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Those `return` statements after t.Fatal only existed to appease SA5011 findings that a cold golangci-lint run does not produce. With the cache issue fixed at its source, they are noise. The loop sites keep t.Errorf + continue: reporting every bad element instead of stopping at the first is worth having on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dgrieser
force-pushed
the
docs/remove-confidence-from-comments
branch
from
August 14, 2026 11:40
ecb96e3 to
53786db
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Confidence scores are no longer rendered in terminal output or in published
GitHub/GitLab comments. They remain available in the JSON output and in the
hidden review envelope, so nothing is lost for machine consumers — the visible
feedback just focuses on the verdict badge, the priority badge, and the
explanation.
Chasing the lint failures this surfaced turned up an unrelated CI problem, so
the lint setup is fixed here too.
Visible output
published summary comment.
published finding comments.
ConfidencePercent/ConfidenceLineininternal/scm/reviewmd,whose last callers are gone.
README.mdto state that confidence lives only in--output jsonand in the hidden review envelope.
Tests
TestTerminalFormatternow fails on anyconfidencesubstring in therendered buffer, before the golden compare, so a re-added render path gives a
named failure rather than an opaque golden diff. Golden file regenerated.
TestCarrierNotesReassembleassertsOverallConfidenceScoreand bothper-finding scores survive the carrier envelope round-trip. With the visible
bodies no longer carrying them, the envelope is the only thing keeping those
scores alive across a re-run.
Lint: cold cache in CI
CI kept failing with staticcheck SA5011 (
possible nil pointer dereference) ontest code that lints clean locally, three sites at a time, a different three
each run.
The cause is golangci-lint's analysis cache. SA5011 needs the fact that
(*testing.common).Fatalnever returns, which staticcheck derives by analyzingthe
testingpackage. When results are served from a partially populated cachethat fact is missing,
t.Fatallooks like an ordinary call, and everyif x == nil { t.Fatal(...) }followed by a dereference is reported. CI'sgolangci-lint-actionrestores a cache between runs and lands in exactly thatstate.
Reproduced with the official v2.12.2 binary: populating the cache with
--default=none --enable staticcheckand then running the full config reports14 issues in
internal/reviewalone; a cold run right after reports none. Apurpose-built probe confirms the cold result is the correct one — it is not
flagged cold, and it is precisely the shape CI was rejecting. Same binary
version, Go toolchain, and issue caps on both sides, so none of those are
involved.
skip-cache: truetogolangci-lint-actionso CI analyzes cold, likemake lintdoes.max-same-issuesandmax-issues-per-linter, which defaulted to 3 andturned one pattern into several rounds of CI failures instead of one list.
modernizelinter in.golangci.yml, delete themake modernizetarget, and drop the separate
go run …/modernizestep from CI. One lintinvocation now covers both locally and in CI.
make lintrunsgolangci-lint cache cleanfirst, for the same reasonskip-cacheis set.Two loop sites in
internal/git/history_test.goandcmd/nickpit/main_test.gokeep the
t.Errorf+continuethey picked up along the way: reporting everybad element instead of stopping at the first is worth having regardless.