Skip to content

codecov/patch posts a red verdict from a partial report while validate-tests is still running #9801

Description

@JSONbored

Problem

codecov/patch posts a red check-run verdict while validate-tests is still running, computed
from a partial coverage report. It later self-corrects, but the red status is real and visible in the
meantime.

codecov.yml sets:

codecov:
  require_ci_to_pass: true
  notify:
    after_n_builds: 1

after_n_builds: 1 fires the verdict on the first upload to land. The uploads are not ordered by
importance:

upload flag job
review-enrichment rees validate-code
control-plane control-plane validate-code
engine engine validate-code
root vitest backend validate-tests

validate-code finishes in ~1-2 minutes; validate-tests takes far longer. So the verdict is
routinely computed from rees + engine alone — without the backend lcov that covers most of
src/**.

Observed on #9799: codecov/patch went pass (after the rees upload), then fail (after the
engine upload), while validate-tests was still pending the whole time.

Why it matters

This is not cosmetic. Per the gate's disposition matrix, any failed CI check is a CLOSE for a
contributor PR — codecov/patch explicitly included. A PR can therefore be auto-closed on a red
status computed from an incomplete report, one that would have resolved green minutes later. The
contributor gets closed for a defect that does not exist, which is the same class of failure as
#9798: a CI signal that corresponds to no real problem.

require_ci_to_pass: true gates the final verdict on the CI run's conclusion but does not stop the
interim status from being posted.

Why bumping after_n_builds is not the fix

The existing comment in codecov.yml already rules it out, and it is right:

It must stay 1, not 2 [...] review-enrichment / control-plane add their own flag uploads on PRs
that touch them -- those uploads only exist on such PRs, and a floor above the guaranteed minimum
would leave codecov/patch permanently un-posted on every other PR.

A count cannot express "wait for the backend upload specifically." Any floor high enough to
guarantee backend has landed would hang the check on PRs that produce fewer uploads.

Expected

Use Codecov's explicit completion signal instead of a heuristic count — codecov.notify.manual_trigger
(or the equivalent upload-then-finalize flow), with the finalize step running after every upload for
the commit has been sent. That makes the verdict fire exactly once, on a complete report, regardless
of how many flags a given PR happens to produce.

Whatever the mechanism, the acceptance criterion is: codecov/patch must not post a conclusion
until every coverage upload for that commit has landed.

Notes

  • Found while validating fix(ci): normalize committed CRLF to LF and name the failure for what it is #9799 (line-ending normalization). That PR is unaffected in the end — its
    patch coverage is genuinely green — but the premature red sent the investigation down a false path
    for a while, which is exactly the cost this issue describes.
  • The same premature-window comment in codecov.yml acknowledges the behavior and calls it
    self-correcting. It does self-correct, but "transiently red" and "auto-closed" are not compatible
    states for a repo whose gate closes on red CI.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions