Skip to content

OCPBUGS-85065: Fix repeated CPO Deployment churn due to OPENSHIFT_IMG_OVERRIDES reordering#8656

Draft
dhgautam99 wants to merge 1 commit into
openshift:mainfrom
dhgautam99:fix-image-overrides-cpo-churn
Draft

OCPBUGS-85065: Fix repeated CPO Deployment churn due to OPENSHIFT_IMG_OVERRIDES reordering#8656
dhgautam99 wants to merge 1 commit into
openshift:mainfrom
dhgautam99:fix-image-overrides-cpo-churn

Conversation

@dhgautam99
Copy link
Copy Markdown

@dhgautam99 dhgautam99 commented Jun 3, 2026

What this PR does / why we need it:

Sorts mirrors within each registry source alphabetically when building
the OPENSHIFT_IMG_OVERRIDES environment variable string. Without this,
overlapping mirror definitions across multiple IDMS/ICSP objects for the
same source produce non-deterministic mirror ordering, causing the CPO
Deployment to roll out repeatedly even though the content is identical.

The prior fix (OCPBUGS-57626) sorted ICSP items by name but did not sort
the mirrors within each source. This is a variant where the same source
appears in multiple IDMS/ICSP objects, and the mirror values for that
source get appended in non-deterministic order.

Which issue(s) this PR fixes:

Fixes OCPBUGS-85065

Special notes for your reviewer:

The fix is a single sort.Strings() call in
ConvertOpenShiftImageRegistryOverridesToCommandLineFlag(). This makes
the output deterministic regardless of how mirrors are collected upstream
from IDMS/ICSP objects.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes

    • Image registry mirror replacements are now sorted deterministically when converting to command-line flags, ensuring consistent output.
  • Tests

    • Added test case validating that registry mirrors provided in non-alphabetical order are sorted consistently in the output.

The prior fix for OCPBUGS-57626 sorted ICSP items by name to ensure
deterministic ordering of OPENSHIFT_IMG_OVERRIDES. However, when
overlapping mirror definitions exist across multiple IDMS and/or ICSP
objects for the same source registry, the mirrors within each source
were not sorted. This caused the final OPENSHIFT_IMG_OVERRIDES string
to vary between reconciliation cycles, triggering repeated CPO
Deployment rollouts.

Sort mirrors alphabetically per source in
ConvertOpenShiftImageRegistryOverridesToCommandLineFlag() to guarantee
a deterministic output regardless of the order mirrors are collected
from IDMS/ICSP objects.
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 3, 2026
@openshift-ci-robot openshift-ci-robot added the jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. label Jun 3, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jun 3, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 3, 2026
@openshift-ci-robot
Copy link
Copy Markdown

@dhgautam99: This pull request references Jira Issue OCPBUGS-85065, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

Sorts mirrors within each registry source alphabetically when building
the OPENSHIFT_IMG_OVERRIDES environment variable string. Without this,
overlapping mirror definitions across multiple IDMS/ICSP objects for the
same source produce non-deterministic mirror ordering, causing the CPO
Deployment to roll out repeatedly even though the content is identical.

The prior fix (OCPBUGS-57626) sorted ICSP items by name but did not sort
the mirrors within each source. This is a variant where the same source
appears in multiple IDMS/ICSP objects, and the mirror values for that
source get appended in non-deterministic order.

Which issue(s) this PR fixes:

Fixes OCPBUGS-85065

Special notes for your reviewer:

The fix is a single sort.Strings() call in
ConvertOpenShiftImageRegistryOverridesToCommandLineFlag(). This makes
the output deterministic regardless of how mirrors are collected upstream
from IDMS/ICSP objects.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jun 3, 2026

📝 Walkthrough

Walkthrough

This pull request adds deterministic sorting of registry mirror replacements in the ConvertOpenShiftImageRegistryOverridesToCommandLineFlag function. The change sorts each per-registry list of mirror replacements before they are appended to the command-line flag output. A new test case verifies that mirrors provided in non-alphabetical order are normalized to a consistent sorted order in the resulting flag string.

Suggested reviewers

  • csrwng
  • muraee
  • bryan-cox
🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Test Structure And Quality ❓ Inconclusive Custom check specifies Ginkgo patterns, but test uses standard Go testing with table-driven patterns instead of Describe/It blocks. Clarify if check applies to all tests or only Ginkgo suites; revise criteria to match actual test framework used.
✅ Passed checks (10 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly relates to the main change: sorting mirrors deterministically to fix CPO Deployment churn caused by OPENSHIFT_IMG_OVERRIDES reordering.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR uses standard Go testing, not Ginkgo. The new test case name is stable, deterministic, and contains no dynamic content (no generated IDs, timestamps, IPs, or dynamic substitution).
Topology-Aware Scheduling Compatibility ✅ Passed PR modifies only a utility function for image registry mirror sorting; no deployment manifests, controllers, or scheduling constraints are introduced.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests were added. PR only modifies standard Go unit tests (support/util/util_test.go) with no IPv4 assumptions or external connectivity requirements.
No-Weak-Crypto ✅ Passed No weak crypto (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto, or non-constant-time comparisons detected. Changes only add string sorting for registry mirroring.
Container-Privileges ✅ Passed PR modifies only Go utility code files, not Kubernetes manifests or container configs. No privileged container settings present; check not applicable.
No-Sensitive-Data-In-Logs ✅ Passed No logging statements were added in the PR. The changes only add a sort.Strings() call to deterministically order registry mirrors; no sensitive data logging is present.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jun 3, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dhgautam99
Once this PR has been reviewed and has the lgtm label, please assign devguyio for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Jun 3, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
support/util/util.go (1)

253-255: ⚡ Quick win

Avoid mutating caller-owned mirror slices in place.

sort.Strings(registryReplacements) mutates the slice stored in registryOverrides. Copy before sorting to keep this converter side-effect free.

Proposed fix
 	for _, registrySource := range sortedRegistrySources {
-		registryReplacements := registryOverrides[registrySource]
+		registryReplacements := append([]string(nil), registryOverrides[registrySource]...)
 		sort.Strings(registryReplacements)
 		for _, registryReplacement := range registryReplacements {
 			commandLineFlagArray = append(commandLineFlagArray, fmt.Sprintf("%s=%s", registrySource, registryReplacement))
 		}
 	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@support/util/util.go` around lines 253 - 255, The code is mutating
caller-owned slices by calling sort.Strings on registryReplacements (which is
taken from registryOverrides[registrySource]); to fix, make a fresh copy of the
slice before sorting and iterating so you don't modify registryOverrides.
Specifically, read registryReplacements := registryOverrides[registrySource]
into a new slice (e.g., create a copy via append to an empty slice), call
sort.Strings on that copied slice, and use the copied slice in the for _,
registryReplacement range loop so registryOverrides remains unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@support/util/util.go`:
- Around line 253-255: The code is mutating caller-owned slices by calling
sort.Strings on registryReplacements (which is taken from
registryOverrides[registrySource]); to fix, make a fresh copy of the slice
before sorting and iterating so you don't modify registryOverrides.
Specifically, read registryReplacements := registryOverrides[registrySource]
into a new slice (e.g., create a copy via append to an empty slice), call
sort.Strings on that copied slice, and use the copied slice in the for _,
registryReplacement range loop so registryOverrides remains unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ce0fcb00-9b93-447e-a802-85c65faa35ff

📥 Commits

Reviewing files that changed from the base of the PR and between 435c545 and 7cd8751.

📒 Files selected for processing (2)
  • support/util/util.go
  • support/util/util_test.go

@dhgautam99
Copy link
Copy Markdown
Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 3, 2026
@openshift-ci-robot
Copy link
Copy Markdown

@dhgautam99: This pull request references Jira Issue OCPBUGS-85065, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.27%. Comparing base (fce95fc) to head (7cd8751).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8656      +/-   ##
==========================================
+ Coverage   40.69%   41.27%   +0.58%     
==========================================
  Files         755      755              
  Lines       93373    93447      +74     
==========================================
+ Hits        37994    38567     +573     
+ Misses      52646    52148     -498     
+ Partials     2733     2732       -1     
Files with missing lines Coverage Δ
support/util/util.go 39.71% <100.00%> (+0.17%) ⬆️

... and 35 files with indirect coverage changes

Flag Coverage Δ
cmd-support 34.87% <100.00%> (+0.16%) ⬆️
cpo-hostedcontrolplane 43.50% <ø> (+1.69%) ⬆️
cpo-other 42.79% <ø> (+1.40%) ⬆️
hypershift-operator 51.00% <ø> (+0.16%) ⬆️
other 31.64% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants