diff --git a/AGENTS.md b/AGENTS.md index db109ecb7d9..2c929d29402 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -151,6 +151,11 @@ See [docs/content/reference/versioning-support.md](docs/content/reference/versio See [docs/content/how-to/upgrades.md](docs/content/how-to/upgrades.md) for Control Plane and Data Plane upgrades +### Pull secret cycling + +When changing how workers and the guest cluster authenticate to registries, treat **HostedCluster** `spec.pullSecret`, **management-cluster** Secret data, **HCCO** reconciliation into the data plane, and optional **Global Pull Secret** (`kube-system/additional-pull-secret`) as one system. Changing only the Secret bytes in place does not change `spec.pullSecret` and therefore does not drive a **NodePool** rollout, but controllers can still propagate credentials into the control plane namespace, guest `openshift-config`, `kube-system/original-pull-secret`, and (where the Global Pull Secret DaemonSet is scheduled) kubelet configuration. + +See [docs/content/how-to/common/global-pull-secret.md](docs/content/how-to/common/global-pull-secret.md) for behavior, platform and NodePool eligibility (AWS/Azure Replace vs InPlace and other platforms), merge semantics, and operational guidance. ## Dependencies and Modules diff --git a/api/hypershift/v1beta1/hostedcluster_types.go b/api/hypershift/v1beta1/hostedcluster_types.go index c2d63498870..77c272d86ae 100644 --- a/api/hypershift/v1beta1/hostedcluster_types.go +++ b/api/hypershift/v1beta1/hostedcluster_types.go @@ -673,10 +673,11 @@ type HostedClusterSpec struct { // pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. // If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. // TODO(alberto): Signal this in a condition. - // This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - // and it will be injected into the container runtime of all NodePools. - // Changing this value will trigger a rollout for all existing NodePools in the cluster. - // Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + // This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + // Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + // Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + // In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + // will still propagate the updated credentials down to the guest cluster and kubelet config. // +required // +rollout // TODO(alberto): have our own local reference type to include our opinions and avoid transparent changes. diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/AAA_ungated.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/AAA_ungated.yaml index 7838ef9d8f6..cd585082c85 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/AAA_ungated.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/AAA_ungated.yaml @@ -5831,10 +5831,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterUpdateAcceptRisks.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterUpdateAcceptRisks.yaml index 29186fae176..6c5b824f0b6 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterUpdateAcceptRisks.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterUpdateAcceptRisks.yaml @@ -5814,10 +5814,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yaml index 75a647659fa..d88d1cd27a9 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yaml @@ -5834,10 +5834,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDC.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDC.yaml index 9de23f8d67e..3a53c3c05c1 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDC.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDC.yaml @@ -6146,10 +6146,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yaml index c62d6ad473f..850e21ea2a3 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yaml @@ -6286,10 +6286,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yaml index 3e972abdcf7..e6b9e20b742 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yaml @@ -6277,10 +6277,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/GCPPlatform.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/GCPPlatform.yaml index dae5c617ab6..0e0d3a7117f 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/GCPPlatform.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/GCPPlatform.yaml @@ -6260,10 +6260,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HCPEtcdBackup.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HCPEtcdBackup.yaml index d0327faa3de..1dcd56e78f2 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HCPEtcdBackup.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HCPEtcdBackup.yaml @@ -5879,10 +5879,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HyperShiftOnlyDynamicResourceAllocation.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HyperShiftOnlyDynamicResourceAllocation.yaml index 14c27c8efc1..5d96096820b 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HyperShiftOnlyDynamicResourceAllocation.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HyperShiftOnlyDynamicResourceAllocation.yaml @@ -5836,10 +5836,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ImageStreamImportMode.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ImageStreamImportMode.yaml index 5dccbaf9408..01c3fac43d4 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ImageStreamImportMode.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ImageStreamImportMode.yaml @@ -5832,10 +5832,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/KMSEncryptionProvider.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/KMSEncryptionProvider.yaml index 27d0b7170ef..44e48b304db 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/KMSEncryptionProvider.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/KMSEncryptionProvider.yaml @@ -5890,10 +5890,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/OpenStack.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/OpenStack.yaml index 99173632616..7b0e49f3f99 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/OpenStack.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/OpenStack.yaml @@ -6365,10 +6365,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/TLSAdherence.yaml b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/TLSAdherence.yaml index cb7aaf7b0b2..d18481f157f 100644 --- a/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/TLSAdherence.yaml +++ b/api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/TLSAdherence.yaml @@ -5854,10 +5854,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-CustomNoUpgrade.crd.yaml b/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-CustomNoUpgrade.crd.yaml index 6de80f8f0a7..48c159bb721 100644 --- a/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-CustomNoUpgrade.crd.yaml +++ b/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-CustomNoUpgrade.crd.yaml @@ -7652,10 +7652,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-Default.crd.yaml b/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-Default.crd.yaml index 8226337d326..80d34870899 100644 --- a/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-Default.crd.yaml +++ b/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-Default.crd.yaml @@ -6323,10 +6323,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-TechPreviewNoUpgrade.crd.yaml b/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-TechPreviewNoUpgrade.crd.yaml index 919e3b55bbf..985515f0f51 100644 --- a/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-TechPreviewNoUpgrade.crd.yaml +++ b/cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-TechPreviewNoUpgrade.crd.yaml @@ -7523,10 +7523,11 @@ spec: description: |- pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. - This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - and it will be injected into the container runtime of all NodePools. - Changing this value will trigger a rollout for all existing NodePools in the cluster. - Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + will still propagate the updated credentials down to the guest cluster and kubelet config. properties: name: default: "" diff --git a/control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup.go b/control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup.go index 6ec2a951f20..278d71184ab 100644 --- a/control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup.go +++ b/control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup.go @@ -5,6 +5,7 @@ import ( "fmt" "os" + "github.com/openshift/hypershift/control-plane-operator/hostedclusterconfigoperator/controllers/resources/manifests" "github.com/openshift/hypershift/control-plane-operator/hostedclusterconfigoperator/operator" corev1 "k8s.io/api/core/v1" @@ -15,14 +16,12 @@ import ( "sigs.k8s.io/controller-runtime/pkg/controller" "sigs.k8s.io/controller-runtime/pkg/handler" "sigs.k8s.io/controller-runtime/pkg/predicate" + "sigs.k8s.io/controller-runtime/pkg/reconcile" "sigs.k8s.io/controller-runtime/pkg/source" ) func Setup(ctx context.Context, opts *operator.HostedClusterConfigOperatorConfig) error { - // Create a predicate for the pull-secret - secretPredicate := predicate.NewPredicateFuncs(func(o crclient.Object) bool { - return o.GetNamespace() == "kube-system" - }) + secretPredicate := predicate.NewPredicateFuncs(kubeSystemSecretPredicateFunc) // Create a cache for the kube-system namespace kubeSystemCache, err := cache.New(opts.Manager.GetConfig(), cache.Options{ @@ -96,5 +95,28 @@ func Setup(ctx context.Context, opts *operator.HostedClusterConfigOperatorConfig return fmt.Errorf("failed to watch kube-system secrets: %w", err) } + // Watch the CP namespace pull-secret so in-place updates to HostedCluster.spec.pullSecret + // promptly reconcile kube-system/original-pull-secret (and global-pull-secret) in the guest. + cpPullSecret := manifests.PullSecret(opts.Namespace) + cpPullSecretPredicate := predicate.NewPredicateFuncs(namespacedNamePredicateFunc(cpPullSecret.Namespace, cpPullSecret.Name)) + cpEventHandler := handler.EnqueueRequestsFromMapFunc(staticReconcileMapper) + if err := c.Watch(source.Kind[crclient.Object](opts.CPCluster.GetCache(), &corev1.Secret{}, cpEventHandler, cpPullSecretPredicate)); err != nil { + return fmt.Errorf("failed to watch control plane pull secret: %w", err) + } + return nil } + +func kubeSystemSecretPredicateFunc(o crclient.Object) bool { + return o.GetNamespace() == "kube-system" +} + +func namespacedNamePredicateFunc(namespace, name string) func(crclient.Object) bool { + return func(o crclient.Object) bool { + return o.GetNamespace() == namespace && o.GetName() == name + } +} + +func staticReconcileMapper(_ context.Context, _ crclient.Object) []reconcile.Request { + return []reconcile.Request{{}} +} diff --git a/control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup_test.go b/control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup_test.go new file mode 100644 index 00000000000..b471cce2632 --- /dev/null +++ b/control-plane-operator/hostedclusterconfigoperator/controllers/globalps/setup_test.go @@ -0,0 +1,89 @@ +package globalps + +import ( + "context" + "testing" + + . "github.com/onsi/gomega" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +func Test_kubeSystemSecretPredicateFunc(t *testing.T) { + tests := []struct { + name string + object *corev1.Secret + want bool + }{ + { + name: "When secret is in kube-system it should return true", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "kube-system", Name: "any-secret"}, + }, + want: true, + }, + { + name: "When secret is in a different namespace it should return false", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "openshift-config", Name: "pull-secret"}, + }, + want: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + g := NewWithT(t) + g.Expect(kubeSystemSecretPredicateFunc(tt.object)).To(Equal(tt.want)) + }) + } +} + +func Test_namespacedNamePredicateFunc(t *testing.T) { + predicate := namespacedNamePredicateFunc("my-hcp-namespace", "pull-secret") + + tests := []struct { + name string + object *corev1.Secret + want bool + }{ + { + name: "When namespace and name match it should return true", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "my-hcp-namespace", Name: "pull-secret"}, + }, + want: true, + }, + { + name: "When namespace differs it should return false", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "other-namespace", Name: "pull-secret"}, + }, + want: false, + }, + { + name: "When name differs it should return false", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "my-hcp-namespace", Name: "other-secret"}, + }, + want: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + g := NewWithT(t) + g.Expect(predicate(tt.object)).To(Equal(tt.want)) + }) + } +} + +func Test_staticReconcileMapper(t *testing.T) { + t.Run("When called it should return a single empty reconcile request", func(t *testing.T) { + g := NewWithT(t) + requests := staticReconcileMapper(context.Background(), &corev1.Secret{}) + g.Expect(requests).To(HaveLen(1)) + g.Expect(requests[0].NamespacedName.String()).To(Equal("/")) + }) +} diff --git a/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources.go b/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources.go index 9b91e284ccb..f4e9ade5f28 100644 --- a/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources.go +++ b/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources.go @@ -313,9 +313,25 @@ func Setup(ctx context.Context, opts *operator.HostedClusterConfigOperatorConfig return fmt.Errorf("failed to watch Route: %w", err) } + // Watch HostedControlPlane namespace pull-secret on the control plane cluster so guest pull secrets + // (openshift-config, openshift) reconcile promptly when the hypershift-operator + // syncs in-place updates from HostedCluster.spec.pullSecret. + // The globalps controller has its own CP pull-secret watch for kube-system/original-pull-secret. + cpPullSecret := manifests.PullSecret(opts.Namespace) + cpPullSecretPredicate := predicate.NewPredicateFuncs(namespacedNamePredicateFunc(cpPullSecret.Namespace, cpPullSecret.Name)) + if err := c.Watch(source.Kind[client.Object](opts.CPCluster.GetCache(), &corev1.Secret{}, eventHandler(), cpPullSecretPredicate)); err != nil { + return fmt.Errorf("failed to watch control plane pull secret: %w", err) + } + return nil } +func namespacedNamePredicateFunc(namespace, name string) func(client.Object) bool { + return func(o client.Object) bool { + return o.GetNamespace() == namespace && o.GetName() == name + } +} + func (r *reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result, error) { log := ctrl.LoggerFrom(ctx) log.Info("Reconciling") diff --git a/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources_test.go b/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources_test.go index 0ab2c9075a9..def47e1bd6e 100644 --- a/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources_test.go +++ b/control-plane-operator/hostedclusterconfigoperator/controllers/resources/resources_test.go @@ -3061,3 +3061,42 @@ func TestReconcileMetricsForwarder(t *testing.T) { }) } } + +func Test_namespacedNamePredicateFunc(t *testing.T) { + predicate := namespacedNamePredicateFunc("my-hcp-namespace", "pull-secret") + + tests := []struct { + name string + object client.Object + want bool + }{ + { + name: "When namespace and name match it should return true", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "my-hcp-namespace", Name: "pull-secret"}, + }, + want: true, + }, + { + name: "When namespace differs it should return false", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "other-namespace", Name: "pull-secret"}, + }, + want: false, + }, + { + name: "When name differs it should return false", + object: &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "my-hcp-namespace", Name: "other-secret"}, + }, + want: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + g := NewWithT(t) + g.Expect(predicate(tt.object)).To(Equal(tt.want)) + }) + } +} diff --git a/docs/content/how-to/common/global-pull-secret.md b/docs/content/how-to/common/global-pull-secret.md index a7eb1c599a6..3fd0ca98664 100644 --- a/docs/content/how-to/common/global-pull-secret.md +++ b/docs/content/how-to/common/global-pull-secret.md @@ -2,13 +2,28 @@ ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see [Platform and NodePool eligibility](#platform-and-nodepool-eligibility)), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on [eligible nodes](#platform-and-nodepool-eligibility). Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -66,8 +81,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see [Platform and NodePool eligibility](#platform-and-nodepool-eligibility)) You can verify the deployment by checking: @@ -86,42 +101,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node + See [Platform and NodePool eligibility](#platform-and-nodepool-eligibility) for the full picture. + +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -193,20 +211,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -255,9 +271,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -311,42 +327,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see [Platform and NodePool eligibility](#platform-and-nodepool-eligibility)) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling diff --git a/docs/content/reference/aggregated-docs.md b/docs/content/reference/aggregated-docs.md index 9ef5da72720..84cf32ada60 100644 --- a/docs/content/reference/aggregated-docs.md +++ b/docs/content/reference/aggregated-docs.md @@ -2382,13 +2382,28 @@ If you wanna know more about how to expose the ingress service in the Data Plane ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -2446,8 +2461,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -2466,42 +2481,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -2573,20 +2591,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -2635,9 +2651,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -2691,42 +2707,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -6808,13 +6793,28 @@ Let's remark some things from this command:
## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -6872,8 +6872,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -6892,42 +6892,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -6999,20 +7002,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -7061,9 +7062,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -7117,42 +7118,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -10097,13 +10067,28 @@ oc get azureprivatelinkservices -n clusters-${CLUSTER_NAME} -o jsonpath='{.items ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -10161,8 +10146,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -10181,42 +10166,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -10288,20 +10276,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -10350,9 +10336,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -10406,42 +10392,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -12059,13 +12014,28 @@ If you wanna know more about how to expose the ingress service in the Data Plane ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -12123,8 +12093,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -12143,42 +12113,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -12250,20 +12223,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -12312,9 +12283,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -12368,42 +12339,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -20250,13 +20190,28 @@ subjects: ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -20314,8 +20269,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -20334,42 +20289,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -20441,20 +20399,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -20503,9 +20459,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -20559,42 +20515,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -21682,13 +21607,28 @@ If you wanna know more about how to expose the ingress service in the Data Plane ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -21746,8 +21686,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -21766,42 +21706,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -21873,20 +21816,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -21935,9 +21876,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -21991,42 +21932,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -22304,13 +22214,28 @@ We can see the 8GB device for etcd. ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -22368,8 +22293,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -22388,42 +22313,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -22495,20 +22423,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -22557,9 +22483,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -22613,42 +22539,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -23896,13 +23791,28 @@ E.g.: ## Overview -The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to all nodes in the cluster. +The Global Pull Secret functionality enables Hosted Cluster administrators to include additional pull secrets for accessing container images from private registries without requiring assistance from the Management Cluster administrator. This feature allows you to merge your custom pull secret with the original HostedCluster pull secret, making it available to nodes that run the sync DaemonSet. -The implementation uses a DaemonSet approach that automatically detects when you create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster). The system then merges this secret with the original pull secret and deploys the merged result to all nodes via a DaemonSet that updates the kubelet configuration. +The implementation uses a DaemonSet that updates kubelet pull credentials on the node. The pull secret referenced by **`HostedCluster.spec.pullSecret`** is always copied from the HostedControlPlane into the guest cluster as the `original-pull-secret` Secret in `kube-system`. The `sync-global-pullsecret` process writes that content to `/var/lib/kubelet/config.json` on **eligible** worker nodes (see Platform and NodePool eligibility), even if you **never** create `additional-pull-secret`. In that baseline case there is no merge step: the kubelet file is kept aligned with the HostedCluster pull secret that HCCO reconciles into the data plane. + +When you **do** create an `additional-pull-secret` in the `kube-system` namespace of your DataPlane (Hosted Cluster), the system merges it with the original HostedCluster pull secret and deploys the merged result via the same DaemonSet path (still preferring the original secret where registry entries conflict). !!! note - This feature is designed to work autonomously - once you create the additional pull secret, the system automatically handles the rest without requiring Management Cluster administrator intervention. + This feature is designed to work autonomously. With only `HostedCluster.spec.pullSecret`, the Hosted Cluster Config Operator (HCCO) still reconciles `original-pull-secret` and the DaemonSet object in the guest; sync pods run only on eligible nodes. Creating `additional-pull-secret` is optional and only needed to add or layer registry credentials beyond the HostedCluster pull secret. + +## Platform and NodePool eligibility + +HCCO reconciles Global Pull Secret resources for **every** hosted cluster platform: it always maintains `kube-system/original-pull-secret` (and optional `global-pull-secret`), RBAC, and the `global-pull-secret-syncer` DaemonSet **object** in the data plane. + +The DaemonSet pod template requires nodes to have the label **`hypershift.openshift.io/nodepool-globalps-enabled=true`**. Today the HyperShift operator sets that label on **Machines** (and HCCO propagates it to **Nodes**) only for: + +- **AWS** and **Azure** NodePools, and +- the **Replace** upgrade strategy (`MachineDeployment` path). + +It does **not** set the label for **InPlace** NodePools (to avoid conflicting with Machine Config Daemon on kubelet config), or for **Replace** on other platforms such as **KubeVirt** (and other providers) in the current implementation—those workers therefore typically have **no** Global Pull Secret sync pods unless something else applies the label. + +For platforms without sync pods, pull credentials still come from **ignition/bootstrap** and from in-cluster Secrets (for example `openshift-config/pull-secret`); kubelet on-disk config is not updated by this DaemonSet on those nodes. ## Adding your Pull Secret @@ -23960,8 +23870,8 @@ After creating the secret, the system will automatically: 1. Validate the secret format 2. Merge it with the original pull secret -3. Deploy a DaemonSet to all nodes -4. Update the kubelet configuration on each node +3. Ensure the DaemonSet is present in the guest cluster +4. Update kubelet configuration on **eligible** worker nodes (see Platform and NodePool eligibility) You can verify the deployment by checking: @@ -23980,42 +23890,45 @@ kubectl get pods -n kube-system -l name=global-pull-secret-syncer The Global Pull Secret functionality operates through a multi-component system: -### Automatic Detection -- The Hosted Cluster Config Operator (HCCO) continuously monitors the `kube-system` namespace -- When it detects the creation of `additional-pull-secret`, it triggers the reconciliation process - -### Validation and Merging -- The system validates that your secret contains a proper DockerConfigJSON format -- It retrieves the original pull secret from the HostedControlPlane -- Your additional pull secret is merged with the original one -- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries) -- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity - -### Deployment Process -- A `global-pull-secret` is created in the `kube-system` namespace containing the merged result +### Automatic detection and baseline sync +- The Hosted Cluster Config Operator (HCCO) continuously reconciles Global Pull Secret resources and watches Secrets in the `kube-system` namespace of the data plane. +- On every reconcile, HCCO copies the HostedControlPlane pull secret (sourced from **`HostedCluster.spec.pullSecret`**) into `kube-system/original-pull-secret` so the DaemonSet can mount it on the node. +- If `additional-pull-secret` is **not** present, HCCO removes the `global-pull-secret` Secret (if it existed) and the DaemonSet syncs **only** the HostedCluster pull secret copy into `/var/lib/kubelet/config.json` on eligible nodes. +- When `additional-pull-secret` **is** present, reconciliation additionally validates and merges it with the HostedCluster pull secret. + +### Validation and merging (optional additional secret) +- When `additional-pull-secret` exists, the system validates that it contains a proper DockerConfigJSON format. +- It retrieves the original pull secret from the HostedControlPlane (same content as `HostedCluster.spec.pullSecret`). +- Your additional pull secret is merged with the original one. +- **If there are conflicting registry entries, the original pull secret takes precedence** (the additional pull secret entry is ignored for conflicting registries). +- The system supports namespace-specific registry entries (e.g., `quay.io/namespace`) for better credential specificity. + +### Deployment process +- When merging is active, a `global-pull-secret` is created in the `kube-system` namespace containing the merged result. If there is no additional secret, this Secret is absent and the syncer uses `original-pull-secret` only. - RBAC resources (ServiceAccount, Role, RoleBinding) are created for the DaemonSet in both `kube-system` and `openshift-config` namespaces - We use Role and RoleBinding in both namespaces to access secrets in `kube-system` and `openshift-config` namespaces - A DaemonSet named `global-pull-secret-syncer` is deployed to eligible nodes -!!! warning "NodePool InPlace Strategy Restriction" +!!! warning "InPlace and unsupported platforms" + + **InPlace NodePools:** workers are intentionally **not** labeled `hypershift.openshift.io/nodepool-globalps-enabled`, so the Global Pull Secret sync **pods do not schedule** there. That avoids conflicts between edits to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD). - The Global Pull Secret DaemonSet is **not deployed** to nodes that belong to NodePools using the **InPlace upgrade strategy**. This restriction prevents conflicts between the DaemonSet's modifications to `/var/lib/kubelet/config.json` and the Machine Config Daemon (MCD) during InPlace upgrades. + **AWS and Azure, Replace:** workers **are** labeled (via Machine → Node propagation), so sync pods **can** run and reconcile kubelet pull configuration from `original-pull-secret` / `global-pull-secret`. - - **Nodes with Replace strategy**: ✅ Receive Global Pull Secret DaemonSet - - **Nodes with InPlace strategy**: ❌ Do not receive Global Pull Secret DaemonSet + **Other platforms (for example KubeVirt, GCP, Agent, …):** the DaemonSet object still exists in `kube-system`, but nodes usually **lack** the selector label, so you will typically see **no** (or very few) sync pods unless you set that label yourself. - This ensures that MCD operations during InPlace upgrades do not fail due to unexpected changes in kubelet configuration files. + See Platform and NodePool eligibility for the full picture. -### Node-Level Synchronization -- Each DaemonSet pod runs a controller that watches the secrets under kube-system namespace -- When changes are detected, it updates `/var/lib/kubelet/config.json` on the node +### Node-level synchronization +- Each DaemonSet pod runs `sync-global-pullsecret`, which periodically reads the mounted pull secret files (`global-pull-secret` when present, otherwise `original-pull-secret`, which holds the **`HostedCluster.spec.pullSecret`** payload reconciled by HCCO). +- When the desired content differs from `/var/lib/kubelet/config.json`, it updates the file on the node - The kubelet service is restarted via DBus to apply the new configuration - If the restart fails after 3 attempts, the system rolls back the file changes -### Automatic Cleanup -- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret -- The system reverts to using only the original pull secret from the HostedControlPlane -- The DaemonSet continues running but now syncs only the original pull secret to nodes +### Automatic cleanup +- If you delete the `additional-pull-secret`, the HCCO automatically removes the `global-pull-secret` secret. +- The system reverts to syncing **only** the HostedCluster pull secret (via `original-pull-secret`, still sourced from the HostedControlPlane). +- The DaemonSet continues to run on eligible nodes and keeps `/var/lib/kubelet/config.json` aligned with that HostedCluster pull secret. ## Registry Precedence and Conflict Resolution @@ -24087,20 +24000,18 @@ The implementation consists of several key components working together: - Handles validation of user-provided pull secrets - Manages the merging logic between original and additional pull secrets - Creates and manages RBAC resources - - Deploys and manages the DaemonSet - - **Node eligibility assessment**: Labels nodes from InPlace NodePools and configures DaemonSet scheduling restrictions + - Deploys and manages the DaemonSet in Nodes labeled with `hypershift.openshift.io/nodepool-globalps-enabled=true` 2. **Sync Global Pull Secret Command** (`sync-global-pullsecret` package) - - Runs as a DaemonSet on each node - - Watches for changes to the `global-pull-secret` in `kube-system` namespace - - Accesses the original `pull-secret` in `openshift-config` namespace - - Updates the kubelet configuration file + - Runs in the DaemonSet pod on eligible nodes + - Reads mounted Docker config JSON from `global-pull-secret` when that volume exists; otherwise uses `original-pull-secret` (the copy of **`HostedCluster.spec.pullSecret`** reconciled into `kube-system`) + - Updates `/var/lib/kubelet/config.json` on the host - Manages kubelet service restarts via DBus -3. **Hosted Cluster Config Operator Integration** - - Monitors for the presence of `additional-pull-secret` - - Orchestrates the entire process - - Handles cleanup when the secret is removed +3. **Hosted Cluster Config Operator integration** + - Reconciles `original-pull-secret` on every pass from the HostedControlPlane pull secret (`HostedCluster.spec.pullSecret`) + - When `additional-pull-secret` exists, validates, merges, and reconciles `global-pull-secret`; when it does not, removes `global-pull-secret` and relies on `original-pull-secret` only for kubelet sync + - Orchestrates RBAC and the DaemonSet for both paths ### Architecture Diagram @@ -24149,9 +24060,9 @@ graph TB Container --> |Executes| SyncCommand[sync-global-pullsecret command] %% Sync Process - SyncCommand --> |Watches global-pull-secret| SyncController[Global Pull Secret Reconciler] - SyncController --> |Reads secret| ReadGlobalPS[Read global-pull-secret] - SyncController --> |Reads original| ReadOriginalPS[Read original pull-secret] + SyncCommand --> |Reads mounted files| SyncController[sync-global-pullsecret loop] + SyncController --> |Reads if present| ReadGlobalPS[Read global-pull-secret mount] + SyncController --> |Reads HostedCluster PS copy| ReadOriginalPS[Read original-pull-secret mount] %% File Update Process ReadGlobalPS --> |Gets data| GlobalPSBytes[Global Pull Secret Bytes] @@ -24205,42 +24116,11 @@ graph TB - Write to `/var/lib/kubelet/config.json` (kubelet configuration file) - Connect to systemd via DBus for service management - Restart kubelet.service, which requires root privileges -- **Smart node targeting**: Automatically excludes nodes from InPlace NodePools to prevent MCD conflicts - -### InPlace NodePool Handling - -To prevent conflicts with Machine Config Daemon operations, the implementation includes intelligent node targeting: - -#### Node Labeling Process -1. **MachineSets Discovery**: The controller queries the management cluster for MachineSets with InPlace-specific annotations (`hypershift.openshift.io/nodePoolTargetConfigVersion`) -2. **Machine Enumeration**: For each InPlace MachineSets, it lists all associated Machines -3. **Node Identification**: Maps Machine objects to their corresponding nodes via `machine.Status.NodeRef.Name` -4. **Labeling**: Applies `hypershift.openshift.io/nodepool-inplace-strategy=true` label to identified nodes - -#### DaemonSet Scheduling Configuration -The DaemonSet uses NodeAffinity to exclude InPlace nodes: - -```yaml -spec: - template: - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: hypershift.openshift.io/nodepool-inplace-strategy - operator: DoesNotExist -``` +- **Smart node targeting**: The DaemonSet uses a `nodeSelector` for `hypershift.openshift.io/nodepool-globalps-enabled=true`; the HyperShift operator only applies that label on **AWS** and **Azure** **Replace** NodePools, so InPlace and other platforms do not get sync pods by default (see Platform and NodePool eligibility) -This ensures that: -- **Nodes without the label**: ✅ Are eligible for DaemonSet scheduling -- **Nodes with the label** (any value): ❌ Are excluded from DaemonSet scheduling +### How scheduling avoids InPlace conflicts -#### Conflict Prevention Benefits -- **Prevents MCD failures**: Avoids conflicts when MCD expects specific kubelet configuration during InPlace upgrades -- **Maintains upgrade reliability**: InPlace upgrade processes are not interrupted by Global Pull Secret modifications -- **Automatic detection**: No manual intervention required - the system automatically identifies and handles InPlace nodes +Eligibility is **positive selection**, not NodeAffinity on an InPlace label: InPlace workers simply **never** receive `hypershift.openshift.io/nodepool-globalps-enabled=true`, so the sync DaemonSet does not place pods on them. Replace workers on AWS/Azure **do** receive the label so the DaemonSet can run there without colliding with MCD on InPlace upgrade paths. ### Error Handling @@ -32554,10 +32434,11 @@ Kubernetes core/v1.LocalObjectReference

pullSecret is a local reference to a Secret that must have a “.dockerconfigjson” key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition. -This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster -and it will be injected into the container runtime of all NodePools. -Changing this value will trigger a rollout for all existing NodePools in the cluster. -Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. +This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. +Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). +Updating the referenced Secret’s data in place (without changing this reference) does not trigger that rollout. +In AWS and Azure NodePools using the Replace upgrade strategy, the Secret’s data in place changes +will still propagate the updated credentials down to the guest cluster and kubelet config. TODO(alberto): have our own local reference type to include our opinions and avoid transparent changes.

@@ -40842,10 +40723,11 @@ Kubernetes core/v1.LocalObjectReference

pullSecret is a local reference to a Secret that must have a “.dockerconfigjson” key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition. -This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster -and it will be injected into the container runtime of all NodePools. -Changing this value will trigger a rollout for all existing NodePools in the cluster. -Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. +This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. +Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). +Updating the referenced Secret’s data in place (without changing this reference) does not trigger that rollout. +In AWS and Azure NodePools using the Replace upgrade strategy, the Secret’s data in place changes +will still propagate the updated credentials down to the guest cluster and kubelet config. TODO(alberto): have our own local reference type to include our opinions and avoid transparent changes.

diff --git a/docs/content/reference/api.md b/docs/content/reference/api.md index a9ce510fc01..5de01ff9124 100644 --- a/docs/content/reference/api.md +++ b/docs/content/reference/api.md @@ -726,10 +726,11 @@ Kubernetes core/v1.LocalObjectReference

pullSecret is a local reference to a Secret that must have a “.dockerconfigjson” key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition. -This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster -and it will be injected into the container runtime of all NodePools. -Changing this value will trigger a rollout for all existing NodePools in the cluster. -Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. +This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. +Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). +Updating the referenced Secret’s data in place (without changing this reference) does not trigger that rollout. +In AWS and Azure NodePools using the Replace upgrade strategy, the Secret’s data in place changes +will still propagate the updated credentials down to the guest cluster and kubelet config. TODO(alberto): have our own local reference type to include our opinions and avoid transparent changes.

@@ -9014,10 +9015,11 @@ Kubernetes core/v1.LocalObjectReference

pullSecret is a local reference to a Secret that must have a “.dockerconfigjson” key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition. -This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster -and it will be injected into the container runtime of all NodePools. -Changing this value will trigger a rollout for all existing NodePools in the cluster. -Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. +This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. +Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). +Updating the referenced Secret’s data in place (without changing this reference) does not trigger that rollout. +In AWS and Azure NodePools using the Replace upgrade strategy, the Secret’s data in place changes +will still propagate the updated credentials down to the guest cluster and kubelet config. TODO(alberto): have our own local reference type to include our opinions and avoid transparent changes.

diff --git a/test/e2e/util/util.go b/test/e2e/util/util.go index 8fbc6f2c169..d38d775a32f 100644 --- a/test/e2e/util/util.go +++ b/test/e2e/util/util.go @@ -6,6 +6,7 @@ import ( "context" "crypto/x509" "crypto/x509/pkix" + "encoding/json" "fmt" "io" "net" @@ -1986,6 +1987,108 @@ func EnsureGlobalPullSecret(t *testing.T, ctx context.Context, mgmtClient crclie nodeCount := *np.Spec.Replicas t.Logf("NodePool replicas: %d", nodeCount) + // Extract the DaemonSet image early so it is available for all subtests. + g.Eventually(func() error { + daemonSet := hccomanifests.GlobalPullSecretDaemonSet() + if err := guestClient.Get(ctx, crclient.ObjectKey{Name: daemonSet.Name, Namespace: daemonSet.Namespace}, daemonSet); err != nil { + return err + } + dsImage = daemonSet.Spec.Template.Spec.Containers[0].Image + return nil + }, 30*time.Second, 5*time.Second).Should(Succeed(), "DaemonSet is not present") + + // Verify that in-place management-cluster pull secret updates propagate to the guest cluster + // without triggering a NodePool rollout. + t.Run("When management-cluster hostedCluster.Spec.PullSecret is updated in-place it should propagate to guest without rollout", func(t *testing.T) { + CPOAtLeast(t, Version423, entryHostedCluster) + g := NewWithT(t) + t.Logf("Reading management-cluster pull secret %s/%s", entryHostedCluster.Namespace, entryHostedCluster.Spec.PullSecret.Name) + mgmtSecret := &corev1.Secret{} + g.Expect(mgmtClient.Get(ctx, crclient.ObjectKey{ + Namespace: entryHostedCluster.Namespace, + Name: entryHostedCluster.Spec.PullSecret.Name, + }, mgmtSecret)).To(Succeed(), "failed to get management-cluster pull secret") + + originalData := make([]byte, len(mgmtSecret.Data[corev1.DockerConfigJsonKey])) + copy(originalData, mgmtSecret.Data[corev1.DockerConfigJsonKey]) + + t.Cleanup(func() { + t.Log("Restoring original management-cluster pull secret data") + fresh := &corev1.Secret{} + if err := mgmtClient.Get(ctx, crclient.ObjectKey{ + Namespace: entryHostedCluster.Namespace, + Name: entryHostedCluster.Spec.PullSecret.Name, + }, fresh); err != nil { + t.Logf("Warning: failed to re-read pull secret for cleanup: %v", err) + return + } + fresh.Data[corev1.DockerConfigJsonKey] = originalData + if err := mgmtClient.Update(ctx, fresh); err != nil { + t.Logf("Warning: failed to restore pull secret: %v", err) + } + }) + + // Merge a dummy auth entry into the pull secret without removing existing auths + type dockerConfigJSON struct { + Auths map[string]json.RawMessage `json:"auths"` + } + var cfg dockerConfigJSON + g.Expect(json.Unmarshal(originalData, &cfg)).To(Succeed(), "failed to parse pull secret") + cfg.Auths["e2e-dummy.example.com"] = json.RawMessage(`{"auth":"e2e-dummy-token"}`) + modifiedData, err := json.Marshal(cfg) + g.Expect(err).NotTo(HaveOccurred(), "failed to marshal modified pull secret") + + t.Log("Patching management-cluster pull secret with dummy auth entry") + mgmtSecret.Data[corev1.DockerConfigJsonKey] = modifiedData + g.Expect(mgmtClient.Update(ctx, mgmtSecret)).To(Succeed(), "failed to update management-cluster pull secret") + + // Wait for openshift-config/pull-secret in the guest cluster to pick up the change + t.Log("Waiting for openshift-config/pull-secret to update in guest cluster") + g.Eventually(func() bool { + secret := &corev1.Secret{} + if err := guestClient.Get(ctx, crclient.ObjectKey{Name: "pull-secret", Namespace: "openshift-config"}, secret); err != nil { + return false + } + return bytes.Contains(secret.Data[corev1.DockerConfigJsonKey], []byte("e2e-dummy.example.com")) + }, 150*time.Second, 5*time.Second).Should(BeTrue(), "openshift-config/pull-secret did not propagate dummy entry") + + // Wait for kube-system/original-pull-secret to pick up the change (globalps controller path) + t.Log("Waiting for kube-system/original-pull-secret to update in guest cluster") + g.Eventually(func() bool { + secret := hccomanifests.OriginalPullSecret() + if err := guestClient.Get(ctx, crclient.ObjectKey{Name: secret.Name, Namespace: secret.Namespace}, secret); err != nil { + return false + } + return bytes.Contains(secret.Data[corev1.DockerConfigJsonKey], []byte("e2e-dummy.example.com")) + }, 150*time.Second, 5*time.Second).Should(BeTrue(), "kube-system/original-pull-secret did not propagate dummy entry") + + // Verify no NodePool rollout was triggered + t.Log("Verifying no NodePool rollout was triggered") + nodePool := &hyperv1.NodePool{} + g.Expect(mgmtClient.Get(ctx, crclient.ObjectKeyFromObject(np), nodePool)).To(Succeed(), "failed to get NodePool") + for _, cond := range nodePool.Status.Conditions { + if cond.Type == hyperv1.NodePoolUpdatingConfigConditionType { + g.Expect(string(cond.Status)).To(Equal(string(metav1.ConditionFalse)), + "UpdatingConfig should be False — in-place pull secret update must not trigger a rollout") + break + } + } + + nodeList := &corev1.NodeList{} + g.Expect(guestClient.List(ctx, nodeList, crclient.MatchingLabels{ + hyperv1.NodePoolLabel: np.Name, + })).To(Succeed(), "failed to list nodes") + g.Expect(len(nodeList.Items)).To(Equal(int(nodeCount)), "node count changed — unexpected rollout") + + // Verify the on-disk kubelet config.json reflects the updated pull secret. + // The global-pull-secret-syncer DaemonSet polls every 30s, so wait for it + // to sync original-pull-secret → /var/lib/kubelet/config.json, then deploy + // a verifier DaemonSet that compares the on-disk file against the cluster pull secret. + VerifyKubeletConfigWithDaemonSet(t, ctx, guestClient, dsImage, nodeCount) + + t.Log("Pull secret propagated to guest cluster and on-disk kubelet config.json without triggering rollout") + }) + // Verify that nodes from Replace NodePools have the globalPS label applied via CAPI propagation. // This label is set on the MachineDeployment template so it flows to Nodes at creation time. t.Run("Check Replace nodes have globalPS label from CAPI propagation", func(t *testing.T) { @@ -2029,18 +2132,6 @@ func EnsureGlobalPullSecret(t *testing.T, ctx context.Context, mgmtClient crclie }, 30*time.Second, 5*time.Second).Should(Succeed(), "global-pull-secret secret is not present") }) - // Check if the DaemonSet is present in the DataPlane - t.Run("Check if the DaemonSet is present in the DataPlane", func(t *testing.T) { - g.Eventually(func() error { - daemonSet := hccomanifests.GlobalPullSecretDaemonSet() - if err := guestClient.Get(ctx, crclient.ObjectKey{Name: daemonSet.Name, Namespace: daemonSet.Namespace}, daemonSet); err != nil { - return err - } - dsImage = daemonSet.Spec.Template.Spec.Containers[0].Image - return nil - }, 30*time.Second, 5*time.Second).Should(Succeed(), "DaemonSet is not present") - }) - t.Run("Wait for critical DaemonSets to be ready - first check", func(t *testing.T) { g.Expect(waitForDaemonSetReady(t, ctx, guestClient, "ovnkube-node", "openshift-ovn-kubernetes", nodeCount)).To(Succeed()) g.Expect(waitForDaemonSetReady(t, ctx, guestClient, hccomanifests.GlobalPullSecretDSName, hccomanifests.GlobalPullSecretNamespace, nodeCount)).To(Succeed()) diff --git a/vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go b/vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go index c2d63498870..77c272d86ae 100644 --- a/vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go +++ b/vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go @@ -673,10 +673,11 @@ type HostedClusterSpec struct { // pullSecret is a local reference to a Secret that must have a ".dockerconfigjson" key whose content must be a valid Openshift pull secret JSON. // If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. // TODO(alberto): Signal this in a condition. - // This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster - // and it will be injected into the container runtime of all NodePools. - // Changing this value will trigger a rollout for all existing NodePools in the cluster. - // Changing the content of the secret inplace will not trigger a rollout and might result in unpredictable behaviour. + // This pull secret is included in NodePool ignition/bootstrap payloads and applied to the container runtime when nodes provision. + // Changing this value will trigger a rollout for all existing NodePools in the cluster (for both replace and inplace upgrade types). + // Updating the referenced Secret's data in place (without changing this reference) does not trigger that rollout. + // In AWS and Azure NodePools using the Replace upgrade strategy, the Secret's data in place changes + // will still propagate the updated credentials down to the guest cluster and kubelet config. // +required // +rollout // TODO(alberto): have our own local reference type to include our opinions and avoid transparent changes.