Skip to content

Remove confidence scores from visible output, fix CI lint cache - #134

Merged
dgrieser merged 5 commits into
mainfrom
docs/remove-confidence-from-comments
Aug 14, 2026
Merged

Remove confidence scores from visible output, fix CI lint cache#134
dgrieser merged 5 commits into
mainfrom
docs/remove-confidence-from-comments

Conversation

@dgrieser

@dgrieser dgrieser commented Aug 14, 2026

Copy link
Copy Markdown
Owner

Confidence scores are no longer rendered in terminal output or in published
GitHub/GitLab comments. They remain available in the JSON output and in the
hidden review envelope, so nothing is lost for machine consumers — the visible
feedback just focuses on the verdict badge, the priority badge, and the
explanation.

Chasing the lint failures this surfaced turned up an unrelated CI problem, so
the lint setup is fixed here too.

Visible output

  • Drop the overall confidence line from the terminal summary and from the
    published summary comment.
  • Drop the per-finding confidence line from terminal findings and from
    published finding comments.
  • Delete ConfidencePercent / ConfidenceLine in internal/scm/reviewmd,
    whose last callers are gone.
  • Update README.md to state that confidence lives only in --output json
    and in the hidden review envelope.

Tests

  • TestTerminalFormatter now fails on any confidence substring in the
    rendered buffer, before the golden compare, so a re-added render path gives a
    named failure rather than an opaque golden diff. Golden file regenerated.
  • TestCarrierNotesReassemble asserts OverallConfidenceScore and both
    per-finding scores survive the carrier envelope round-trip. With the visible
    bodies no longer carrying them, the envelope is the only thing keeping those
    scores alive across a re-run.

Lint: cold cache in CI

CI kept failing with staticcheck SA5011 (possible nil pointer dereference) on
test code that lints clean locally, three sites at a time, a different three
each run.

The cause is golangci-lint's analysis cache. SA5011 needs the fact that
(*testing.common).Fatal never returns, which staticcheck derives by analyzing
the testing package. When results are served from a partially populated cache
that fact is missing, t.Fatal looks like an ordinary call, and every
if x == nil { t.Fatal(...) } followed by a dereference is reported. CI's
golangci-lint-action restores a cache between runs and lands in exactly that
state.

Reproduced with the official v2.12.2 binary: populating the cache with
--default=none --enable staticcheck and then running the full config reports
14 issues in internal/review alone; a cold run right after reports none. A
purpose-built probe confirms the cold result is the correct one — it is not
flagged cold, and it is precisely the shape CI was rejecting. Same binary
version, Go toolchain, and issue caps on both sides, so none of those are
involved.

  • Pass skip-cache: true to golangci-lint-action so CI analyzes cold, like
    make lint does.
  • Lift max-same-issues and max-issues-per-linter, which defaulted to 3 and
    turned one pattern into several rounds of CI failures instead of one list.
  • Enable the modernize linter in .golangci.yml, delete the make modernize
    target, and drop the separate go run …/modernize step from CI. One lint
    invocation now covers both locally and in CI.
  • make lint runs golangci-lint cache clean first, for the same reason
    skip-cache is set.

Two loop sites in internal/git/history_test.go and cmd/nickpit/main_test.go
keep the t.Errorf + continue they picked up along the way: reporting every
bad element instead of stopping at the first is worth having regardless.

@dgrieser dgrieser self-assigned this Aug 14, 2026
@dgrieser dgrieser changed the title Remove confidence scores from visible output Remove confidence scores from visible output and unify lint tooling Aug 14, 2026
@dgrieser dgrieser changed the title Remove confidence scores from visible output and unify lint tooling Remove confidence scores from visible output, fix CI lint cache Aug 14, 2026
dgrieser and others added 5 commits August 14, 2026 13:40
Confidence scores are no longer rendered in terminal output or published
comments. They remain available in the JSON output and the hidden review
envelope only. This simplifies the visible feedback by focusing on the
verdict badge and explanation.

- Remove confidence lines from terminal formatter output
- Remove confidence lines from SCM markdown rendering (GitHub/GitLab)
- Update documentation to reflect new behavior
- Update tests and golden files to match changes
- Add assertion to TestTerminalFormatter to ensure confidence scores are
not rendered in terminal output.
- Update TestCarrierNotesReassemble to include confidence scores in test
findings and verify they are preserved in the carrier envelope after
reassembly.
Move from t.Fatalf to t.Errorf for better error reporting in concurrent
and loop-based tests. Add explicit returns after fatal errors to ensure
subsequent assertions are skipped.

Consolidate modernize analyzers into golangci-lint configuration to unify
local and CI checks. Remove the separate Makefile target and CI step.
A restored golangci-lint analysis cache loses staticcheck's fact that
(*testing.common).Fatal never returns. Every `if x == nil { t.Fatal(...) }`
followed by a dereference is then reported as SA5011, which is why CI kept
failing on code that lints clean locally. A cold run recomputes the fact and
reports nothing, including for a deliberately constructed probe.

- Pass skip-cache to golangci-lint-action so CI matches a clean local run.
- Lift max-same-issues and max-issues-per-linter, which defaulted to 3 and
  turned one pattern into several rounds of CI failures.
- Correct the make lint comment, which had the cause backwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Those `return` statements after t.Fatal only existed to appease SA5011
findings that a cold golangci-lint run does not produce. With the cache
issue fixed at its source, they are noise.

The loop sites keep t.Errorf + continue: reporting every bad element
instead of stopping at the first is worth having on its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dgrieser
dgrieser force-pushed the docs/remove-confidence-from-comments branch from ecb96e3 to 53786db Compare August 14, 2026 11:40
@dgrieser
dgrieser merged commit 7f558ab into main Aug 14, 2026
8 checks passed
@dgrieser
dgrieser deleted the docs/remove-confidence-from-comments branch August 14, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant