CNTRLPLANE-3535: Add codecov carryforward flags to stabilize project coverage checks

[bryan-cox]

Summary

• Adds carryforward: true flag definitions for all 5 unit test shards in codecov.yml
• Fixes nondeterministic codecov/project check failures across PRs

Problem

Unit tests run in 5 parallel shards (cpo-hostedcontrolplane, cpo-other, hypershift-operator, cmd-support, other), each uploading a separate coverage file. Without carryforward, Codecov recomputes the project total after each shard arrives and posts intermediate codecov/project statuses. The last shard to arrive determines pass/fail.

Evidence from main branch: coverage on the same commit swings between ~40% and ~47% depending on shard upload timing. For example, commit adfbcddc2e went 41.80% → 47.18% → 45.84% within 10 minutes as shards arrived.

Fix

With carryforward: true, when a shard hasn't uploaded yet for a given commit, Codecov uses its last-known coverage data instead of treating it as empty. This stabilizes the project total from the first status update through the last.

Test plan

• Verify codecov/project status stabilizes on this PR (should not show ~5% swings between shard uploads)
• Confirm the check remains blocking (no informational: true)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Improved code coverage configuration to enable persistent tracking of coverage data across multiple components and continuous integration runs.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@bryan-cox: This pull request references CNTRLPLANE-3535 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

• Adds carryforward: true flag definitions for all 5 unit test shards in codecov.yml
• Fixes nondeterministic codecov/project check failures across PRs

Problem

Unit tests run in 5 parallel shards (cpo-hostedcontrolplane, cpo-other, hypershift-operator, cmd-support, other), each uploading a separate coverage file. Without carryforward, Codecov recomputes the project total after each shard arrives and posts intermediate codecov/project statuses. The last shard to arrive determines pass/fail.

Evidence from main branch: coverage on the same commit swings between ~40% and ~47% depending on shard upload timing. For example, commit adfbcddc2e went 41.80% → 47.18% → 45.84% within 10 minutes as shards arrived.

Fix

With carryforward: true, when a shard hasn't uploaded yet for a given commit, Codecov uses its last-known coverage data instead of treating it as empty. This stabilizes the project total from the first status update through the last.

Test plan

• Verify codecov/project status stabilizes on this PR (should not show ~5% swings between shard uploads)
• Confirm the check remains blocking (no informational: true)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 0684d853-c5df-46e4-accc-bc26e8f88059

📥 Commits

Reviewing files that changed from the base of the PR and between 988f2be and 478e010.

📒 Files selected for processing (1)

• codecov.yml

---

📝 Walkthrough

Walkthrough

This PR updates codecov.yml to add a flags configuration block that defines five coverage flag groups: cpo-hostedcontrolplane, cpo-other, hypershift-operator, cmd-support, and other. Each flag is configured with carryforward: true, which enables Codecov to carry forward coverage information from these flag groups across test runs, ensuring coverage metrics persist when reports are uploaded.

Suggested reviewers

• csrwng
• jparrill

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: adding codecov carryforward flags to stabilize coverage checks, which matches the changeset (adding flags configuration to codecov.yml).
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR modifies codecov.yml configuration only; no Ginkgo test files were modified, so the check for stable test names is not applicable.
Test Structure And Quality	✅ Passed	PR only modifies codecov.yml configuration file; no Ginkgo test code was changed, so the test code quality check is not applicable to this PR.
Topology-Aware Scheduling Compatibility	✅ Passed	PR only modifies codecov.yml, a code coverage tool configuration file unrelated to Kubernetes scheduling, topology, affinity, or deployment manifests. Check is not applicable.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR only modifies codecov.yml configuration; no new Ginkgo e2e tests are added, so IPv6/disconnected network check is not applicable.
No-Weak-Crypto	✅ Passed	PR only modifies codecov.yml with YAML configuration; no cryptographic code added, modified, or implemented.
Container-Privileges	✅ Passed	PR only modifies codecov.yml for coverage configuration; container-privileges check is inapplicable as no K8s/container manifests or container definitions were changed.
No-Sensitive-Data-In-Logs	✅ Passed	PR adds codecov.yml configuration with flag names and carryforward settings. No logging statements or sensitive data are introduced.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cblecker]

/lgtm
/area ci-tooling
/verified bypass

[openshift-ci-robot]

@cblecker: The verified label has been added.

Details

In response to this:

/lgtm
/area ci-tooling
/verified bypass

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

Pipeline controller notification

No second-stage tests were triggered for this PR.

This can happen when:

• The changed files don't match any pipeline_run_if_changed patterns
• All files match pipeline_skip_if_only_changed patterns
• No pipeline-controlled jobs are defined for the main branch

Use /test ? to see all available tests.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, cblecker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bryan-cox,cblecker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bryan-cox]

/retest

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 40.68%. Comparing base (9b67f7b) to head (478e010).
⚠️ Report is 34 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8642      +/-   ##
==========================================
- Coverage   45.84%   40.68%   -5.17%     
==========================================
  Files         440      755     +315     
  Lines       52824    93368   +40544     
==========================================
+ Hits        24218    37985   +13767     
- Misses      26816    52649   +25833     
- Partials     1790     2734     +944     

see 317 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	34.70% <ø> (?)
cpo-hostedcontrolplane	41.80% <ø> (ø)
cpo-other	41.39% <ø> (ø)
hypershift-operator	50.81% <ø> (-0.02%)	⬇️
other	31.61% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bryan-cox]

/override codecov/project

[openshift-ci]

@bryan-cox: Overrode contexts on behalf of bryan-cox: codecov/project

Details

In response to this:

/override codecov/project

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[bryan-cox]

/override "Red Hat Konflux / hypershift-operator-enterprise-contract / hypershift-operator-main"

[openshift-ci]

@bryan-cox: Overrode contexts on behalf of bryan-cox: Red Hat Konflux / hypershift-operator-enterprise-contract / hypershift-operator-main

Details

In response to this:

/override "Red Hat Konflux / hypershift-operator-enterprise-contract / hypershift-operator-main"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

@bryan-cox: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[hypershift-jira-solve-ci]

Test Failure Analysis Complete

Job Information

• Prow Job: codecov/project (Codecov GitHub Check Run — not a Prow CI job)
• Build ID: Check Run 79059830673
• PR: #8642 — CNTRLPLANE-3535: Add codecov carryforward flags to stabilize project coverage checks
• Commit: 478e010 (head) vs 9b67f7b (base)
• Result: Failure — project coverage dropped from 45.84% → 40.68% (−5.17%)

Test Failure Analysis

Error

codecov/project: 40.68% (-5.17%) compared to 9b67f7b
Coverage   45.84%   40.68%   -5.17%
Files         440      755     +315
Lines       52824    93368   +40544
Hits        24218    37985   +13767
Misses      26816    52649   +25833
Partials     1790     2734     +944

Summary

The codecov/project check failed because Codecov reported a −5.17% project coverage decrease (45.84% → 40.68%). This is not caused by the PR's code change — the PR only modifies codecov.yml to add carryforward: true flags and touches zero lines of Go code. The root cause is that Codecov's stored baseline for main at commit 9b67f7b was 34 commits stale, tracking only 440 files / 52,824 lines, while the PR's fresh unit-test uploads cover the current codebase of 755 files / 93,368 lines. The 315 newly-added files (from those 34 intervening commits) are largely untested, diluting the overall percentage below Codecov's zero-tolerance default threshold. The PR was correctly merged via /override codecov/project.

Root Cause

The failure is caused by a stale Codecov baseline combined with uncovered codebase growth, not by any change in this PR.

Mechanism:

1. Stale baseline (34 commits behind): Codecov's stored coverage for main was computed at commit 9b67f7b (May 29 07:11 UTC). The Codecov report itself warns: "Report is 34 commits behind head on main." That baseline tracked 440 files and 52,824 lines, yielding 45.84% coverage.

2. Fresh PR uploads cover a larger codebase: The PR's 5 unit-test shards (cpo-hostedcontrolplane, cpo-other, hypershift-operator, cmd-support, other) all ran successfully and uploaded coverage for the current codebase, which now contains 755 files and 93,368 lines — 315 more files and 40,544 more lines than the stale baseline.

3. Coverage dilution from untested new code: The 34 intervening commits added ~315 files and ~40K lines that are largely untested. While absolute covered lines ("Hits") increased by +13,767, uncovered lines ("Misses") grew nearly twice as fast at +25,833. This dilutes the percentage from 45.84% → 40.68%.

4. Default zero-tolerance threshold violated: The project's codecov.yml has no coverage.status.project section, so Codecov applies its default: target: auto (match the base commit's coverage) with threshold: 0%. Any decrease — even from baseline drift — causes a failure. The −5.17% drop exceeds this zero-tolerance default.

5. The PR's change is not the cause: Adding carryforward: true to flag definitions only affects how Codecov handles missing flag uploads in future PRs. In this PR, all 5 shards uploaded fresh data, so the carryforward setting had no effect on the computed coverage. Codecov itself confirmed: "All modified and coverable lines are covered by tests."

Recommendations

1. Add an explicit project coverage threshold to codecov.yml to absorb baseline drift:

   coverage:
     status:
       project:
         default:
           target: auto
           threshold: 5%   # allow up to 5% decrease to handle baseline staleness

2. Investigate why the main branch baseline is 34 commits stale: Verify that push-triggered CI on main is successfully uploading coverage to Codecov. If main-branch coverage uploads are failing or not running, the baseline will always be stale, and this failure pattern will recur on every PR.

3. No action needed for this specific PR: The override was correct — the PR modifies only codecov.yml, all unit tests passed, and all modified lines have coverage. The failure is purely an artifact of baseline staleness.

4. The carryforward fix is still valuable: While it didn't prevent this failure (which is a baseline staleness issue), it will stabilize the intermediate codecov/project statuses that fluctuate as shards arrive — which was the original problem the PR set out to fix.

Evidence

Evidence	Detail
Check conclusion	failure — project coverage 40.68% vs base 45.84% (−5.17%)
Baseline staleness	Codecov warns: "Report is 34 commits behind head on main"
File count mismatch	Base: 440 files → Head: 755 files (+315 files from stale baseline)
Lines count mismatch	Base: 52,824 lines → Head: 93,368 lines (+40,544 lines)
Coverage of PR's changes	"All modified and coverable lines are covered by tests" ✅
PR diff	Only codecov.yml modified — zero Go source code changed
Missing threshold config	No coverage.status.project section in codecov.yml → Codecov default threshold: 0%
All test shards passed	cmd-support (34.70%), cpo-hostedcontrolplane (41.80%), cpo-other (41.39%), hypershift-operator (50.81%), other (31.61%)
Resolution	PR merged via /override codecov/project by bryan-cox

---