Skip to content

[release-4.20] OCPBUGS-92038, OCPBUGS-92039, OCPBUGS-82147, OCPBUGS-92041, OCPBUGS-92042: Replace OLM-based Istio install with Sail Library#1459

Open
gcs278 wants to merge 12 commits into
openshift:release-4.20from
gcs278:backport-noOLM-4.20
Open

[release-4.20] OCPBUGS-92038, OCPBUGS-92039, OCPBUGS-82147, OCPBUGS-92041, OCPBUGS-92042: Replace OLM-based Istio install with Sail Library#1459
gcs278 wants to merge 12 commits into
openshift:release-4.20from
gcs278:backport-noOLM-4.20

Conversation

@gcs278

@gcs278 gcs278 commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.20. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This is part of an approved SBAR to backport the Sail Library (noOLM) from 4.22 to 4.19–4.21. This is an identical backport to the 4.21 PR: #1442 (origin test coverage: openshift/origin#31232).

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Background

Gateway API on OCP 4.19–4.21 uses the Cluster Ingress Operator (CIO) to install Istio via OLM (OSSM operator). This path has several critical bugs:

  • OCPBUGS-88295: OSSM z-stream upgrades are blocked, preventing CVE fixes from being delivered
  • OCPBUGS-82146: OLM-related install failures
  • OCPBUGS-78330: Hardcoded catalog source breaks disconnected environments
  • OCPBUGS-85550: Gateway API fails on clusters without Marketplace capability

In OCP 4.22, NE-2286 replaced OLM with the Sail Library — CIO now installs Istio directly via embedded Helm charts. This feature shipped as GA behind the GatewayAPIWithoutOLM feature gate.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.20.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). On 4.21, the release.openshift.io/feature-set annotation was removed in a separate PR (#1462) before GA promotion. For 4.20, the annotation removal is included as the final commit in this PR to avoid an additional backport PR.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.26.2) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.26.2 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.26.2.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.20, the OLM path is on 3.1.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.20 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate, IngressControllerLBSubnetsAWS, SetEIPForNLBIngressController)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.20 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.20 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.20.
  • go.mod / vendor/: Added replace directives for openshift/api (fork with gate), sail-operator (official OSSM 3.3.1), and dependency pins (see Dependency Pinning Approach below). Re-vendored from scratch.

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Dependency Pinning Approach

Unlike the 4.21 backport which bumped k8s and controller-runtime, this backport keeps all dependencies at their original 4.20 versions. The sail-operator (OSSM 3.3.1) requires k8s 0.34 and controller-runtime 0.22, but its pkg/install package only uses basic CRUD operations (client.New, client.Get, client.Create, client.Update) and stable types (metav1, corev1, runtime, rest.Config) that exist unchanged in the 4.20 versions.

To prevent go mod tidy from bumping dependencies transitively, the following replace directives pin modules to their 4.20 versions:

Module Pinned Version 4.20 Original
k8s.io/api v0.33.2 v0.33.2
k8s.io/apimachinery v0.33.2 v0.33.2
k8s.io/client-go v0.32.1 v0.32.1
k8s.io/apiextensions-apiserver v0.33.0 v0.33.0
k8s.io/apiserver v0.33.0 v0.33.0
k8s.io/component-base v0.33.0 v0.33.0
k8s.io/kube-openapi v0.0.0-20250318... v0.0.0-20250318...
sigs.k8s.io/controller-runtime v0.20.4 v0.20.4
sigs.k8s.io/gateway-api v1.2.1 v1.2.1
github.com/google/gnostic-models v0.6.9 v0.6.9

Risk assessment: The sail-operator install package uses only stable controller-runtime interfaces (client.Client CRUD operations, pkg/log, pkg/scheme). No APIs introduced in controller-runtime 0.21+ or k8s 0.34+ are used. The structured-merge-diff/v4 vs v6 incompatibility that would arise from bumping k8s is avoided entirely. This approach was validated by building successfully and by auditing every import in the sail-operator's pkg/install, api/v1, and resources packages.

Verification

  • make builds successfully
  • No unresolved merge conflict markers in any commit
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 2, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references NE-2286 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.20." or "openshift-4.20.", but it targets "openshift-4.22" instead.

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.20. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1393 OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC Conditionally deploys Sail Library RBAC based on GatewayAPIWithoutOLM feature gate — required for the gate to control RBAC deployment when enabled
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.20.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.26.2) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.26.2 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.26.2.

Dependency Pinning Approach

Unlike the 4.21 backport which bumped k8s and controller-runtime, this backport keeps all dependencies at their original 4.20 versions. The sail-operator (OSSM 3.3.1) requires k8s 0.34 and controller-runtime 0.22, but its pkg/install package only uses basic CRUD operations (client.New, client.Get, client.Create, client.Update) and stable types (metav1, corev1, runtime, rest.Config) that exist unchanged in the 4.20 versions.

To prevent go mod tidy from bumping dependencies transitively, the following replace directives pin modules to their 4.20 versions:

Module Pinned Version 4.20 Original
k8s.io/api v0.33.2 v0.33.2
k8s.io/apimachinery v0.33.2 v0.33.2
k8s.io/client-go v0.32.1 v0.32.1
k8s.io/apiextensions-apiserver v0.33.0 v0.33.0
k8s.io/apiserver v0.33.0 v0.33.0
k8s.io/component-base v0.33.0 v0.33.0
k8s.io/kube-openapi v0.0.0-20250318... v0.0.0-20250318...
sigs.k8s.io/controller-runtime v0.20.4 v0.20.4
sigs.k8s.io/gateway-api v1.2.1 v1.2.1
github.com/google/gnostic-models v0.6.9 v0.6.9

Risk assessment: The sail-operator install package uses only stable controller-runtime interfaces (client.Client CRUD operations, pkg/log, pkg/scheme). No APIs introduced in controller-runtime 0.21+ or k8s 0.34+ are used. The structured-merge-diff/v4 vs v6 incompatibility that would arise from bumping k8s is avoided entirely. This approach was validated by building successfully and by auditing every import in the sail-operator's pkg/install, api/v1, and resources packages.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.20 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate, IngressControllerLBSubnetsAWS, SetEIPForNLBIngressController)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.20 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.20 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.20.
  • go.mod / vendor/: Added replace directives for openshift/api (fork with gate), sail-operator (official OSSM 3.3.1), and dependency pins (see Dependency Pinning Approach above). Re-vendored from scratch.

Merge Order

  1. Merge openshift/api PR — FG as disabled, allows CI to start
  2. TODO: Backport noOLM E2E tests to origin release-4.20
  3. Merge this PR — Sail Library code lands, gate still OFF
  4. Merge openshift/api PR — FG promotion to Default GA, activates noOLM
  5. Verify CI is green

Verification

  • make builds successfully
  • No unresolved merge conflict markers in any commit
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 34538161-fbc8-4217-b4e7-6ac45da38727

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 2, 2026
@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@gcs278 gcs278 changed the title NE-2286: Backport noOLM / Sail Library to release-4.20 [release-4.20] NE-2286: Backport noOLM / Sail Library to release-4.20 Jun 2, 2026
@gcs278

gcs278 commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

/testwith openshift/cluster-ingress-operator/release-4.20/e2e-aws-operator openshift/api#2869

@gcs278

gcs278 commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

/testwith openshift/cluster-ingress-operator/release-4.20/e2e-aws-operator openshift/api#2869

@gcs278 gcs278 marked this pull request as ready for review June 18, 2026 18:19
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 18, 2026
@gcs278

gcs278 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

Ready for early review, but blocked on getting some Jira Tickets set up and the 4.21 NO-OLM backport to merge to GA (openshift/api#2865)
/hold

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 18, 2026
@openshift-ci openshift-ci Bot requested review from grzpiotrowski and rikatz June 18, 2026 18:20
@gcs278 gcs278 changed the title [release-4.20] NE-2286: Backport noOLM / Sail Library to release-4.20 [release-4.20] Replace OLM-based Istio install with Sail Library Jun 22, 2026
@openshift-ci-robot openshift-ci-robot removed the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 22, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.20. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.20.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.26.2) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.26.2 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.26.2.

Dependency Pinning Approach

Unlike the 4.21 backport which bumped k8s and controller-runtime, this backport keeps all dependencies at their original 4.20 versions. The sail-operator (OSSM 3.3.1) requires k8s 0.34 and controller-runtime 0.22, but its pkg/install package only uses basic CRUD operations (client.New, client.Get, client.Create, client.Update) and stable types (metav1, corev1, runtime, rest.Config) that exist unchanged in the 4.20 versions.

To prevent go mod tidy from bumping dependencies transitively, the following replace directives pin modules to their 4.20 versions:

Module Pinned Version 4.20 Original
k8s.io/api v0.33.2 v0.33.2
k8s.io/apimachinery v0.33.2 v0.33.2
k8s.io/client-go v0.32.1 v0.32.1
k8s.io/apiextensions-apiserver v0.33.0 v0.33.0
k8s.io/apiserver v0.33.0 v0.33.0
k8s.io/component-base v0.33.0 v0.33.0
k8s.io/kube-openapi v0.0.0-20250318... v0.0.0-20250318...
sigs.k8s.io/controller-runtime v0.20.4 v0.20.4
sigs.k8s.io/gateway-api v1.2.1 v1.2.1
github.com/google/gnostic-models v0.6.9 v0.6.9

Risk assessment: The sail-operator install package uses only stable controller-runtime interfaces (client.Client CRUD operations, pkg/log, pkg/scheme). No APIs introduced in controller-runtime 0.21+ or k8s 0.34+ are used. The structured-merge-diff/v4 vs v6 incompatibility that would arise from bumping k8s is avoided entirely. This approach was validated by building successfully and by auditing every import in the sail-operator's pkg/install, api/v1, and resources packages.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.20 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate, IngressControllerLBSubnetsAWS, SetEIPForNLBIngressController)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.20 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.20 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.20.
  • go.mod / vendor/: Added replace directives for openshift/api (fork with gate), sail-operator (official OSSM 3.3.1), and dependency pins (see Dependency Pinning Approach above). Re-vendored from scratch.

Merge Order

  1. Merge openshift/api PR — FG as disabled, allows CI to start
  2. TODO: Backport noOLM E2E tests to origin release-4.20
  3. Merge this PR — Sail Library code lands, gate still OFF
  4. Merge [release-4.20] OCPBUGS-92038, OCPBUGS-92039, OCPBUGS-82147, OCPBUGS-92041, OCPBUGS-92042: Promote GatewayAPIWithoutOLM feature gate to TechPreview api#2874 — FG promotion to TechPreview, allows CI soak
  5. Verify CI is green
  6. TODO: Merge CIO PR to remove release.openshift.io/feature-set annotation from Sail Library RBAC manifests
  7. Merge openshift/api PR — FG promotion to Default GA, activates noOLM
  8. Verify CI is green

Verification

  • make builds successfully
  • No unresolved merge conflict markers in any commit
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

/assign @aswinsuryan
/assign @rikatz

@gcs278 gcs278 changed the title [release-4.20] Replace OLM-based Istio install with Sail Library [release-4.20] OCPBUGS-92038, OCPBUGS-92039, OCPBUGS-92040, OCPBUGS-92041, OCPBUGS-92042: Replace OLM-based Istio install with Sail Library Jun 24, 2026
@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 24, 2026
@gcs278 gcs278 force-pushed the backport-noOLM-4.20 branch from a287c29 to 0aea5da Compare June 25, 2026 01:25
gcs278 and others added 2 commits June 24, 2026 21:50
Cherry-picked from: 43c978a
openshift#1404

Conflicts resolved:
- go.mod: Switched sail-operator replace from aslakknutsen's development
  fork to the official openshift-service-mesh/sail-operator v0.0.0-20260327145107
  (OSSM 3.3.1). Added replace directives to pin k8s.io/api, apimachinery,
  apiextensions-apiserver, apiserver, client-go, component-base,
  kube-openapi, controller-runtime, gateway-api, and gnostic-models to
  their original 4.20 versions, preventing the sail-operator's transitive
  dependencies from bumping them. This avoids the structured-merge-diff
  v4/v6 incompatibility and preserves compatibility with the 4.20
  openshift/client-go and openshift/library-go.
- vendor/: Re-vendored from scratch with pinned dependencies.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@gcs278 gcs278 force-pushed the backport-noOLM-4.20 branch from 0aea5da to 5fcd499 Compare June 25, 2026 01:50
Remove the release.openshift.io/feature-set annotation from the Sail
Library ClusterRole and ClusterRoleBinding manifests. This annotation
restricts CVO from deploying these RBAC resources on clusters running
the Default feature set. Removing it is required before promoting the
GatewayAPIWithoutOLM feature gate to GA.

On 4.22, PR openshift#1393 switched to the release.openshift.io/feature-gate
annotation, but CVO on 4.20 does not support that annotation
(openshift/cluster-version-operator#1273 was not backported). On 4.21,
this was done in a separate PR (openshift#1462), but for 4.20 we include it
here to avoid an additional backport PR.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This PR was included in a payload test run from openshift/origin#31322
trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fdb1ad50-703a-11f1-8ca6-25963c2f707b-0

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This PR was included in a payload test run from openshift/origin#31322
trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/02cda910-703b-11f1-8202-b9ab34f52f2b-0

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This PR was included in a payload test run from openshift/origin#31322
trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.20-upgrade-from-stable-4.19-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/09d8a390-703b-11f1-9aa0-93a802cd5146-0

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This PR was included in a payload test run from openshift/origin#31322
trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fcafd840-703b-11f1-962a-ea9f67a1d4d1-0

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This PR was included in a payload test run from openshift/origin#31322
trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/06111ed0-703c-11f1-940d-9b8efc8783d7-0

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This PR was included in a payload test run from openshift/origin#31322
trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.20-upgrade-from-stable-4.19-e2e-metal-ipi-upgrade-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/10ca2d30-703c-11f1-9e9a-43d6051ce295-0

@aswinsuryan

Copy link
Copy Markdown
Contributor

/lgtm

The changes looks good , is the CI failure related to flaky tests?

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 25, 2026
@gcs278

gcs278 commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

@aswinsuryan yea - looks like an unrelated flake. Overall CI is looking good for this PR.

/test e2e-azure-operator

@gcs278

gcs278 commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 25, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-92038, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88295 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88295 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

This pull request references Jira Issue OCPBUGS-92039, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88297 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88297 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-92040, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-82146 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-82146 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-92041, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-78330 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-78330 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-92042, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-85550 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-85550 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot requested a review from melvinjoseph86 June 25, 2026 18:56
@gcs278

gcs278 commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

BTW the code has been spun through CI in the test PR: openshift/origin#31322 with various payload jobs with the FG promoted (testing noOLM here in this PR). Everything is looking good, no failures for 4.20 No-OLM.

@openshift-ci-robot openshift-ci-robot added jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. and removed jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jun 25, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-92038, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88295 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88295 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

This pull request references Jira Issue OCPBUGS-92039, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88297 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88297 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-92040, which is invalid:

  • expected the bug to be open, but it isn't
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Closed (Duplicate) instead
  • expected Jira Issue OCPBUGS-92040 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-92041, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-78330 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-78330 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-92042, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-85550 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-85550 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.20. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This is part of an approved SBAR to backport the Sail Library (noOLM) from 4.22 to 4.19–4.21. This is an identical backport to the 4.21 PR: #1442 (origin test coverage: openshift/origin#31232).

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Background

Gateway API on OCP 4.19–4.21 uses the Cluster Ingress Operator (CIO) to install Istio via OLM (OSSM operator). This path has several critical bugs:

  • OCPBUGS-88295: OSSM z-stream upgrades are blocked, preventing CVE fixes from being delivered
  • OCPBUGS-82146: OLM-related install failures
  • OCPBUGS-78330: Hardcoded catalog source breaks disconnected environments
  • OCPBUGS-85550: Gateway API fails on clusters without Marketplace capability

In OCP 4.22, NE-2286 replaced OLM with the Sail Library — CIO now installs Istio directly via embedded Helm charts. This feature shipped as GA behind the GatewayAPIWithoutOLM feature gate.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.20.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). On 4.21, the release.openshift.io/feature-set annotation was removed in a separate PR (#1462) before GA promotion. For 4.20, the annotation removal is included as the final commit in this PR to avoid an additional backport PR.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.26.2) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.26.2 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.26.2.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.20, the OLM path is on 3.1.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.20 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate, IngressControllerLBSubnetsAWS, SetEIPForNLBIngressController)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.20 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.20 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.20.
  • go.mod / vendor/: Added replace directives for openshift/api (fork with gate), sail-operator (official OSSM 3.3.1), and dependency pins (see Dependency Pinning Approach below). Re-vendored from scratch.

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Dependency Pinning Approach

Unlike the 4.21 backport which bumped k8s and controller-runtime, this backport keeps all dependencies at their original 4.20 versions. The sail-operator (OSSM 3.3.1) requires k8s 0.34 and controller-runtime 0.22, but its pkg/install package only uses basic CRUD operations (client.New, client.Get, client.Create, client.Update) and stable types (metav1, corev1, runtime, rest.Config) that exist unchanged in the 4.20 versions.

To prevent go mod tidy from bumping dependencies transitively, the following replace directives pin modules to their 4.20 versions:

Module Pinned Version 4.20 Original
k8s.io/api v0.33.2 v0.33.2
k8s.io/apimachinery v0.33.2 v0.33.2
k8s.io/client-go v0.32.1 v0.32.1
k8s.io/apiextensions-apiserver v0.33.0 v0.33.0
k8s.io/apiserver v0.33.0 v0.33.0
k8s.io/component-base v0.33.0 v0.33.0
k8s.io/kube-openapi v0.0.0-20250318... v0.0.0-20250318...
sigs.k8s.io/controller-runtime v0.20.4 v0.20.4
sigs.k8s.io/gateway-api v1.2.1 v1.2.1
github.com/google/gnostic-models v0.6.9 v0.6.9

Risk assessment: The sail-operator install package uses only stable controller-runtime interfaces (client.Client CRUD operations, pkg/log, pkg/scheme). No APIs introduced in controller-runtime 0.21+ or k8s 0.34+ are used. The structured-merge-diff/v4 vs v6 incompatibility that would arise from bumping k8s is avoided entirely. This approach was validated by building successfully and by auditing every import in the sail-operator's pkg/install, api/v1, and resources packages.

Verification

  • make builds successfully
  • No unresolved merge conflict markers in any commit
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278 gcs278 changed the title [release-4.20] OCPBUGS-92038, OCPBUGS-92039, OCPBUGS-92040, OCPBUGS-92041, OCPBUGS-92042: Replace OLM-based Istio install with Sail Library [release-4.20] OCPBUGS-92038, OCPBUGS-92039, OCPBUGS-82147, OCPBUGS-92041, OCPBUGS-92042: Replace OLM-based Istio install with Sail Library Jun 25, 2026
@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 25, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-92038, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88295 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88295 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-92039, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88297 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88297 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-82147, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-82146 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-82146 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-92041, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-78330 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-78330 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-92042, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.z) matches configured target version for branch (4.20.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-85550 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-85550 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.20. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This is part of an approved SBAR to backport the Sail Library (noOLM) from 4.22 to 4.19–4.21. This is an identical backport to the 4.21 PR: #1442 (origin test coverage: openshift/origin#31232).

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Background

Gateway API on OCP 4.19–4.21 uses the Cluster Ingress Operator (CIO) to install Istio via OLM (OSSM operator). This path has several critical bugs:

  • OCPBUGS-88295: OSSM z-stream upgrades are blocked, preventing CVE fixes from being delivered
  • OCPBUGS-82146: OLM-related install failures
  • OCPBUGS-78330: Hardcoded catalog source breaks disconnected environments
  • OCPBUGS-85550: Gateway API fails on clusters without Marketplace capability

In OCP 4.22, NE-2286 replaced OLM with the Sail Library — CIO now installs Istio directly via embedded Helm charts. This feature shipped as GA behind the GatewayAPIWithoutOLM feature gate.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.20.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). On 4.21, the release.openshift.io/feature-set annotation was removed in a separate PR (#1462) before GA promotion. For 4.20, the annotation removal is included as the final commit in this PR to avoid an additional backport PR.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.26.2) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.26.2 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.26.2.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.20, the OLM path is on 3.1.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.20 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate, IngressControllerLBSubnetsAWS, SetEIPForNLBIngressController)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.20 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.20 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.20.
  • go.mod / vendor/: Added replace directives for openshift/api (fork with gate), sail-operator (official OSSM 3.3.1), and dependency pins (see Dependency Pinning Approach below). Re-vendored from scratch.

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Dependency Pinning Approach

Unlike the 4.21 backport which bumped k8s and controller-runtime, this backport keeps all dependencies at their original 4.20 versions. The sail-operator (OSSM 3.3.1) requires k8s 0.34 and controller-runtime 0.22, but its pkg/install package only uses basic CRUD operations (client.New, client.Get, client.Create, client.Update) and stable types (metav1, corev1, runtime, rest.Config) that exist unchanged in the 4.20 versions.

To prevent go mod tidy from bumping dependencies transitively, the following replace directives pin modules to their 4.20 versions:

Module Pinned Version 4.20 Original
k8s.io/api v0.33.2 v0.33.2
k8s.io/apimachinery v0.33.2 v0.33.2
k8s.io/client-go v0.32.1 v0.32.1
k8s.io/apiextensions-apiserver v0.33.0 v0.33.0
k8s.io/apiserver v0.33.0 v0.33.0
k8s.io/component-base v0.33.0 v0.33.0
k8s.io/kube-openapi v0.0.0-20250318... v0.0.0-20250318...
sigs.k8s.io/controller-runtime v0.20.4 v0.20.4
sigs.k8s.io/gateway-api v1.2.1 v1.2.1
github.com/google/gnostic-models v0.6.9 v0.6.9

Risk assessment: The sail-operator install package uses only stable controller-runtime interfaces (client.Client CRUD operations, pkg/log, pkg/scheme). No APIs introduced in controller-runtime 0.21+ or k8s 0.34+ are used. The structured-merge-diff/v4 vs v6 incompatibility that would arise from bumping k8s is avoided entirely. This approach was validated by building successfully and by auditing every import in the sail-operator's pkg/install, api/v1, and resources packages.

Verification

  • make builds successfully
  • No unresolved merge conflict markers in any commit
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@gcs278: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants