AzurePrivateSpec configures private connectivity to an Azure hosted cluster’s API server. It is a discriminated union keyed on the type field, which selects the private connectivity -mechanism. Currently only PrivateLink is supported; additional mechanisms (e.g., Swift) may -be added in the future.
+mechanism.|
type specifies the private connectivity mechanism used for the hosted cluster’s API server. “PrivateLink” selects Azure Private Link Service for private API server access. +“Swift” selects Azure Swift pod networking for private API server access, used by ARO HCP. This field is immutable once set. |
@@ -40107,6 +40107,23 @@ AzurePrivateLinkSpec
This field is required when type is “PrivateLink” and must not be set otherwise.
+|
+swift,omitzero
+
+
+AzureSwiftSpec
+
+
+ |
+
+(Optional)
+ swift configures Azure Swift pod networking for private API server access. +Swift networking requires the management cluster to be pre-configured with +Azure Swift support; this is not provisioned by HyperShift automatically. +This field is required when type is “Swift” and must not be set otherwise. + |
+
"Swift"
AzurePrivateTypeSwift specifies private connectivity using Azure Swift pod networking. +In this mode, Azure Swift assigns a private IP from the customer VNet directly +to the hosted cluster’s router pods, providing private API server access without a +separate Private Link Service. This is used by ARO HCP managed clusters.
++(Appears on: +AzurePrivateSpec) +
++
AzureSwiftSpec configures Azure Swift pod networking for private API server access. +Swift assigns a private IP from the customer VNet directly to the hosted cluster’s +router pods, providing private connectivity without a separate Private Link Service.
+ +| Field | +Description | +
|---|---|
+podNetworkInstance
+
+string
+
+ |
+
+ podNetworkInstance is the name of a PodNetworkInstance custom resource in the +hosted control plane namespace. This resource configures Azure Swift pod networking +for private connectivity to the hosted cluster’s router pods. +The value must be a valid Kubernetes object name (RFC 1123 DNS label): lowercase +alphanumeric characters or hyphens, must start and end with an alphanumeric character. +This field is immutable once set. + |
+
(Appears on: @@ -41640,6 +41699,12 @@ hosted cluster can be live migrated without experiencing a node restart
desired platform are valid. A failure here is unlikely to resolve without the changing user input. +"PublicEndpointExposed"
PublicEndpointExposed indicates whether public API server endpoints are +currently configured and exposed for this cluster via the management +cluster’s shared ingress. Status reflects observed state: True means +public endpoints are reachable, False means they are not.
+"ReconciliationActive"
ReconciliationActive indicates if reconciliation of the HostedCluster is active or paused hostedCluster.spec.pausedUntil.
diff --git a/docs/content/reference/api.md b/docs/content/reference/api.md index 828e8531ff1..7992d2a1d54 100644 --- a/docs/content/reference/api.md +++ b/docs/content/reference/api.md @@ -4381,8 +4381,7 @@ in the format xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
AzurePrivateSpec configures private connectivity to an Azure hosted cluster’s API server. It is a discriminated union keyed on the type field, which selects the private connectivity -mechanism. Currently only PrivateLink is supported; additional mechanisms (e.g., Swift) may -be added in the future.
+mechanism.|
type specifies the private connectivity mechanism used for the hosted cluster’s API server. “PrivateLink” selects Azure Private Link Service for private API server access. +“Swift” selects Azure Swift pod networking for private API server access, used by ARO HCP. This field is immutable once set. |
@@ -4422,6 +4422,23 @@ AzurePrivateLinkSpec
This field is required when type is “PrivateLink” and must not be set otherwise.
+|
+swift,omitzero
+
+
+AzureSwiftSpec
+
+
+ |
+
+(Optional)
+ swift configures Azure Swift pod networking for private API server access. +Swift networking requires the management cluster to be pre-configured with +Azure Swift support; this is not provisioned by HyperShift automatically. +This field is required when type is “Swift” and must not be set otherwise. + |
+
"Swift"
AzurePrivateTypeSwift specifies private connectivity using Azure Swift pod networking. +In this mode, Azure Swift assigns a private IP from the customer VNet directly +to the hosted cluster’s router pods, providing private API server access without a +separate Private Link Service. This is used by ARO HCP managed clusters.
++(Appears on: +AzurePrivateSpec) +
++
AzureSwiftSpec configures Azure Swift pod networking for private API server access. +Swift assigns a private IP from the customer VNet directly to the hosted cluster’s +router pods, providing private connectivity without a separate Private Link Service.
+ +| Field | +Description | +
|---|---|
+podNetworkInstance
+
+string
+
+ |
+
+ podNetworkInstance is the name of a PodNetworkInstance custom resource in the +hosted control plane namespace. This resource configures Azure Swift pod networking +for private connectivity to the hosted cluster’s router pods. +The value must be a valid Kubernetes object name (RFC 1123 DNS label): lowercase +alphanumeric characters or hyphens, must start and end with an alphanumeric character. +This field is immutable once set. + |
+
(Appears on: @@ -5955,6 +6014,12 @@ hosted cluster can be live migrated without experiencing a node restart
desired platform are valid. A failure here is unlikely to resolve without the changing user input. +"PublicEndpointExposed"
PublicEndpointExposed indicates whether public API server endpoints are +currently configured and exposed for this cluster via the management +cluster’s shared ingress. Status reflects observed state: True means +public endpoints are reachable, False means they are not.
+"ReconciliationActive"
ReconciliationActive indicates if reconciliation of the HostedCluster is active or paused hostedCluster.spec.pausedUntil.
diff --git a/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go b/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go index 095522e7bd2..f7c2deac532 100644 --- a/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go +++ b/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go @@ -213,6 +213,11 @@ type HostedClusterReconciler struct { FeatureSet configv1.FeatureSet OpenShiftTrustedCAFilePath string + + // ProbeSharedIngressEndpoint tests whether a public endpoint is reachable + // via the shared ingress. Defaults to probeSharedIngressEndpoint. Override + // in tests to avoid real network calls. + ProbeSharedIngressEndpoint func(context context.Context, serviceIP string, servicePort int, kasHostname string) bool } // +kubebuilder:rbac:groups=hypershift.openshift.io,resources=hostedclusters,verbs=get;list;watch;create;update;patch;delete @@ -991,7 +996,7 @@ func (r *HostedClusterReconciler) reconcile(ctx context.Context, req ctrl.Reques // Copy Azure Private Link conditions from the AzurePrivateLinkService resources. // ARO HCP uses Swift networking, not Private Link Services. - if hcluster.Spec.Platform.Type == hyperv1.AzurePlatform && !azureutil.IsAroHCP() { + if hcluster.Spec.Platform.Type == hyperv1.AzurePlatform && !netutil.UseSwiftNetworkingHC(hcluster) { hcpNamespace := manifests.HostedControlPlaneNamespace(hcluster.Namespace, hcluster.Name) var azPLSList hyperv1.AzurePrivateLinkServiceList if err := r.List(ctx, &azPLSList, &client.ListOptions{Namespace: hcpNamespace}); err != nil { @@ -2069,7 +2074,7 @@ func (r *HostedClusterReconciler) reconcile(ctx context.Context, req ctrl.Reques return ctrl.Result{}, err } case hyperv1.AzurePlatform: - if azureutil.IsAroHCP() { + if azureutil.IsAroHCPByHC(hcluster) { // Reconcile CPO SecretProviderClass CR cpoSecretProviderClass := cpomanifests.ManagedAzureSecretProviderClass(config.ManagedAzureCPOSecretProviderClassName, hcp.Namespace) if _, err = createOrUpdate(ctx, r, cpoSecretProviderClass, func() error { @@ -2090,7 +2095,7 @@ func (r *HostedClusterReconciler) reconcile(ctx context.Context, req ctrl.Reques } if hcluster.Spec.SecretEncryption != nil && hcluster.Spec.SecretEncryption.KMS != nil { - if azureutil.IsAroHCP() { + if azureutil.IsAroHCPByHC(hcluster) { // Reconcile KMS SecretProviderClass CR kmsSecretProviderClass := cpomanifests.ManagedAzureSecretProviderClass(config.ManagedAzureKMSSecretProviderClassName, hcp.Namespace) if _, err := createOrUpdate(ctx, r, kmsSecretProviderClass, func() error { @@ -2113,6 +2118,12 @@ func (r *HostedClusterReconciler) reconcile(ctx context.Context, req ctrl.Reques return ctrl.Result{}, fmt.Errorf("failed to reconcile karpenter operator: %w", err) } + if pubEndpointRequeue, err := r.reconcilePublicEndpointExposedCondition(ctx, hcluster); err != nil { + return ctrl.Result{}, fmt.Errorf("failed to reconcile PublicEndpointExposed condition: %w", err) + } else if pubEndpointRequeue != nil && (requeueAfter == nil || *pubEndpointRequeue < *requeueAfter) { + requeueAfter = pubEndpointRequeue + } + log.Info("successfully reconciled") result := ctrl.Result{} if requeueAfter != nil { @@ -3996,7 +4007,7 @@ func (r *HostedClusterReconciler) validateAzureConfig(hc *hyperv1.HostedCluster) // When topology is Private or PublicAndPrivate, the private field must be configured // to provide Azure Private Link Service settings for private API server access. // ARO HCP uses Swift networking, not Private Link Services. - if !azureutil.IsAroHCP() && + if !netutil.UseSwiftNetworkingHC(hc) && hc.Spec.Platform.Azure.Topology != "" && hc.Spec.Platform.Azure.Topology != hyperv1.AzureTopologyPublic && hc.Spec.Platform.Azure.Private.Type == "" { @@ -5217,7 +5228,7 @@ func ensureReferencedResourceAnnotation(ctx context.Context, client client.Clien } func ensureHostedResourcesAreEmpty(ctx context.Context, crclient client.Client, hcluster *hyperv1.HostedCluster, obj client.Object) error { - if !azureutil.IsAroHCP() || !k8sutil.HasAnnotationWithValue(obj, hyperv1.HostedClusterSourcedAnnotation, "true") { + if !azureutil.IsAroHCPByHC(hcluster) || !k8sutil.HasAnnotationWithValue(obj, hyperv1.HostedClusterSourcedAnnotation, "true") { return nil } var cm corev1.Secret @@ -5408,3 +5419,110 @@ func computeAzurePLSCondition(azPLSList hyperv1.AzurePrivateLinkServiceList, con } return computeEndpointServiceCondition(resourceConditions, conditionType, hyperv1.AzurePLSErrorReason, hyperv1.AzurePLSSuccessReason, "AzurePrivateLinkService conditions not found") } + +const publicEndpointProbeTimeout = 3 * time.Second + +func (r *HostedClusterReconciler) reconcilePublicEndpointExposedCondition(ctx context.Context, hc *hyperv1.HostedCluster) (*time.Duration, error) { + if !netutil.UseSwiftNetworkingHC(hc) { + return nil, nil + } + + kasStrategy := netutil.ServicePublishingStrategyByTypeByHC(hc, hyperv1.APIServer) + if kasStrategy == nil || kasStrategy.Route == nil || kasStrategy.Route.Hostname == "" { + return nil, nil + } + kasHostname := kasStrategy.Route.Hostname + + svc := &corev1.Service{} + svcKey := client.ObjectKey{Namespace: "hypershift-sharedingress", Name: "router"} + if err := r.Client.Get(ctx, svcKey, svc); err != nil { + if apierrors.IsNotFound(err) { + return nil, nil + } + return nil, fmt.Errorf("failed to get shared ingress service: %w", err) + } + if svc.Spec.ClusterIP == "" || svc.Spec.ClusterIP == "None" { + return nil, nil + } + + lbReady := len(svc.Status.LoadBalancer.Ingress) > 0 && + (svc.Status.LoadBalancer.Ingress[0].Hostname != "" || svc.Status.LoadBalancer.Ingress[0].IP != "") + usesSharedIngress := netutil.UseSharedIngressHC(hc) + + if !lbReady && usesSharedIngress { + condition := metav1.Condition{ + Type: string(hyperv1.PublicEndpointExposed), + Status: metav1.ConditionFalse, + Reason: hyperv1.PublicEndpointConvergenceInProgressReason, + Message: "Shared ingress LoadBalancer is not ready", + ObservedGeneration: hc.Generation, + } + if meta.SetStatusCondition(&hc.Status.Conditions, condition) { + if err := r.Client.Status().Update(ctx, hc); err != nil { + return nil, fmt.Errorf("failed to update PublicEndpointExposed condition: %w", err) + } + } + d := 10 * time.Second + return &d, nil + } + + probeFn := r.ProbeSharedIngressEndpoint + if probeFn == nil { + probeFn = probeSharedIngressEndpoint + } + exposed := probeFn(ctx, svc.Spec.ClusterIP, 443, kasHostname) + + var condition metav1.Condition + condition.Type = string(hyperv1.PublicEndpointExposed) + condition.ObservedGeneration = hc.Generation + var requeue *time.Duration + + switch { + case exposed && usesSharedIngress: + condition.Status = metav1.ConditionTrue + condition.Reason = hyperv1.PublicEndpointSharedIngressConfiguredReason + condition.Message = "Public API server endpoint is reachable via shared ingress" + + case !exposed && usesSharedIngress: + condition.Status = metav1.ConditionFalse + condition.Reason = hyperv1.PublicEndpointConvergenceInProgressReason + condition.Message = "Public endpoint configuration in progress" + d := 10 * time.Second + requeue = &d + + case exposed && !usesSharedIngress: + condition.Status = metav1.ConditionTrue + condition.Reason = hyperv1.PublicEndpointConvergenceInProgressReason + condition.Message = "Public endpoint removal in progress" + d := 10 * time.Second + requeue = &d + + case !exposed && !usesSharedIngress: + condition.Status = metav1.ConditionFalse + condition.Reason = hyperv1.PublicEndpointTopologyPrivateReason + condition.Message = "Public API server endpoint is not exposed via shared ingress" + } + + if meta.SetStatusCondition(&hc.Status.Conditions, condition) { + if err := r.Client.Status().Update(ctx, hc); err != nil { + return nil, fmt.Errorf("failed to update PublicEndpointExposed condition: %w", err) + } + } + return requeue, nil +} + +func probeSharedIngressEndpoint(context context.Context, serviceIP string, servicePort int, kasHostname string) bool { + dialer := &tls.Dialer{ + NetDialer: &net.Dialer{Timeout: publicEndpointProbeTimeout}, + Config: &tls.Config{ + ServerName: kasHostname, + InsecureSkipVerify: true, //nolint:gosec + }, + } + conn, err := dialer.DialContext(context, "tcp", net.JoinHostPort(serviceIP, strconv.Itoa(servicePort))) + if err != nil { + return false + } + conn.Close() + return true +} diff --git a/hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go b/hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go index 668ffd25206..a2bb4c59082 100644 --- a/hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go +++ b/hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go @@ -1804,7 +1804,7 @@ func TestHostedClusterWatchesEverythingItCreates(t *testing.T) { SubscriptionID: "12345678-1234-1234-1234-123456789abc", SecurityGroupID: "/subscriptions/12345678-1234-1234-1234-123456789abc/resourceGroups/test-resource-group/providers/Microsoft.Network/networkSecurityGroups/test-nsg", AzureAuthenticationConfig: hyperv1.AzureAuthenticationConfiguration{ - AzureAuthenticationConfigType: "WorkloadIdentities", + AzureAuthenticationConfigType: hyperv1.AzureAuthenticationTypeManagedIdentities, WorkloadIdentities: &hyperv1.AzureWorkloadIdentities{ ImageRegistry: hyperv1.WorkloadIdentity{ClientID: "12345678-1234-1234-1234-123456789abc"}, Ingress: hyperv1.WorkloadIdentity{ClientID: "12345678-1234-1234-1234-123456789abc"}, @@ -1868,6 +1868,12 @@ func TestHostedClusterWatchesEverythingItCreates(t *testing.T) { }, }, TenantID: "12345678-1234-1234-1234-123456789abc", + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, }, }, Release: hyperv1.Release{ @@ -5395,6 +5401,22 @@ func TestEnsureHostedResourcesAreEmpty(t *testing.T) { Namespace: namespace, }, } + if tc.setAROHCP { + hcluster.Spec.Platform = hyperv1.PlatformSpec{ + Type: hyperv1.AzurePlatform, + Azure: &hyperv1.AzurePlatformSpec{ + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, + AzureAuthenticationConfig: hyperv1.AzureAuthenticationConfiguration{ + AzureAuthenticationConfigType: hyperv1.AzureAuthenticationTypeManagedIdentities, + }, + }, + } + } // Create a Secret that would exist in the cluster secret := &corev1.Secret{ @@ -6939,6 +6961,12 @@ func TestValidateAzureConfig(t *testing.T) { Type: hyperv1.AzurePlatform, Azure: &hyperv1.AzurePlatformSpec{ Topology: hyperv1.AzureTopologyPrivate, + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, }, }, }, diff --git a/hypershift-operator/controllers/hostedcluster/network_policies.go b/hypershift-operator/controllers/hostedcluster/network_policies.go index 056d3ce3611..684fbbd710a 100644 --- a/hypershift-operator/controllers/hostedcluster/network_policies.go +++ b/hypershift-operator/controllers/hostedcluster/network_policies.go @@ -15,6 +15,7 @@ import ( "github.com/openshift/hypershift/support/awsutil" "github.com/openshift/hypershift/support/capabilities" "github.com/openshift/hypershift/support/config" + "github.com/openshift/hypershift/support/netutil" "github.com/openshift/hypershift/support/rhobsmonitoring" "github.com/openshift/hypershift/support/upsert" @@ -79,7 +80,7 @@ func (r *HostedClusterReconciler) reconcileNetworkPolicies(ctx context.Context, return err } - if sharedingress.UseSharedIngress() { + if netutil.UseSharedIngressHC(hcluster) { policy := networkpolicy.SharedIngressNetworkPolicy(controlPlaneNamespaceName) if _, err := createOrUpdate(ctx, r.Client, policy, func() error { return reconcileSharedIngressNetworkPolicy(policy, hcluster) diff --git a/hypershift-operator/controllers/hostedcluster/public_endpoint_condition_test.go b/hypershift-operator/controllers/hostedcluster/public_endpoint_condition_test.go new file mode 100644 index 00000000000..0e7f88805d1 --- /dev/null +++ b/hypershift-operator/controllers/hostedcluster/public_endpoint_condition_test.go @@ -0,0 +1,209 @@ +package hostedcluster + +import ( + "context" + "testing" + "time" + + . "github.com/onsi/gomega" + + hyperv1 "github.com/openshift/hypershift/api/hypershift/v1beta1" + api "github.com/openshift/hypershift/support/api" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + crclient "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" +) + +func sharedIngressService(clusterIP string, lbHostname string) *corev1.Service { + svc := &corev1.Service{ + ObjectMeta: metav1.ObjectMeta{ + Name: "router", + Namespace: "hypershift-sharedingress", + }, + Spec: corev1.ServiceSpec{ + ClusterIP: clusterIP, + }, + } + if lbHostname != "" { + svc.Status.LoadBalancer.Ingress = []corev1.LoadBalancerIngress{ + {Hostname: lbHostname}, + } + } + return svc +} + +func swiftHostedCluster(name string, topology hyperv1.AzureTopologyType, kasHostname string) *hyperv1.HostedCluster { + return &hyperv1.HostedCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: "clusters", + Generation: 1, + }, + Spec: hyperv1.HostedClusterSpec{ + Platform: hyperv1.PlatformSpec{ + Type: hyperv1.AzurePlatform, + Azure: &hyperv1.AzurePlatformSpec{ + Topology: topology, + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, + AzureAuthenticationConfig: hyperv1.AzureAuthenticationConfiguration{ + AzureAuthenticationConfigType: hyperv1.AzureAuthenticationTypeManagedIdentities, + }, + }, + }, + Services: []hyperv1.ServicePublishingStrategyMapping{ + { + Service: hyperv1.APIServer, + ServicePublishingStrategy: hyperv1.ServicePublishingStrategy{ + Type: hyperv1.Route, + Route: &hyperv1.RoutePublishingStrategy{ + Hostname: kasHostname, + }, + }, + }, + }, + }, + } +} + +func TestReconcilePublicEndpointExposedCondition(t *testing.T) { + tests := []struct { + name string + hc *hyperv1.HostedCluster + svc *corev1.Service + probeResult bool + expectCondition bool + expectStatus metav1.ConditionStatus + expectReason string + expectRequeue bool + }{ + { + name: "When cluster is not Swift it should not set condition", + hc: &hyperv1.HostedCluster{ + ObjectMeta: metav1.ObjectMeta{Name: "test", Namespace: "clusters", Generation: 1}, + Spec: hyperv1.HostedClusterSpec{ + Platform: hyperv1.PlatformSpec{ + Type: hyperv1.AWSPlatform, + }, + }, + }, + expectCondition: false, + }, + { + name: "When shared ingress service is not found it should not set condition", + hc: swiftHostedCluster("test", hyperv1.AzureTopologyPublicAndPrivate, "api.test.example.com"), + svc: nil, + expectCondition: false, + }, + { + name: "When KAS hostname is not configured it should not set condition", + hc: swiftHostedCluster("test", hyperv1.AzureTopologyPublicAndPrivate, ""), + svc: sharedIngressService("10.0.0.1", "lb.example.com"), + expectCondition: false, + }, + { + name: "When LB is not ready and cluster uses shared ingress it should set False with ConvergenceInProgress", + hc: swiftHostedCluster("test", hyperv1.AzureTopologyPublicAndPrivate, "api.test.example.com"), + svc: sharedIngressService("10.0.0.1", ""), + expectCondition: true, + expectStatus: metav1.ConditionFalse, + expectReason: hyperv1.PublicEndpointConvergenceInProgressReason, + expectRequeue: true, + }, + { + name: "When probe succeeds and cluster uses shared ingress it should set True with SharedIngressConfigured", + hc: swiftHostedCluster("test", hyperv1.AzureTopologyPublicAndPrivate, "api.test.example.com"), + svc: sharedIngressService("10.0.0.1", "lb.example.com"), + probeResult: true, + expectCondition: true, + expectStatus: metav1.ConditionTrue, + expectReason: hyperv1.PublicEndpointSharedIngressConfiguredReason, + expectRequeue: false, + }, + { + name: "When probe fails and cluster uses shared ingress it should set False with ConvergenceInProgress", + hc: swiftHostedCluster("test", hyperv1.AzureTopologyPublicAndPrivate, "api.test.example.com"), + svc: sharedIngressService("10.0.0.1", "lb.example.com"), + probeResult: false, + expectCondition: true, + expectStatus: metav1.ConditionFalse, + expectReason: hyperv1.PublicEndpointConvergenceInProgressReason, + expectRequeue: true, + }, + { + name: "When probe succeeds but cluster is Private it should set True with ConvergenceInProgress", + hc: swiftHostedCluster("test", hyperv1.AzureTopologyPrivate, "api.test.example.com"), + svc: sharedIngressService("10.0.0.1", "lb.example.com"), + probeResult: true, + expectCondition: true, + expectStatus: metav1.ConditionTrue, + expectReason: hyperv1.PublicEndpointConvergenceInProgressReason, + expectRequeue: true, + }, + { + name: "When probe fails and cluster is Private it should set False with TopologyPrivate", + hc: swiftHostedCluster("test", hyperv1.AzureTopologyPrivate, "api.test.example.com"), + svc: sharedIngressService("10.0.0.1", "lb.example.com"), + probeResult: false, + expectCondition: true, + expectStatus: metav1.ConditionFalse, + expectReason: hyperv1.PublicEndpointTopologyPrivateReason, + expectRequeue: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + g := NewWithT(t) + + objects := []crclient.Object{tt.hc} + if tt.svc != nil { + objects = append(objects, tt.svc) + } + cl := fake.NewClientBuilder(). + WithScheme(api.Scheme). + WithObjects(objects...). + WithStatusSubresource(tt.hc). + Build() + + r := &HostedClusterReconciler{ + Client: cl, + ProbeSharedIngressEndpoint: func(context context.Context, serviceIP string, servicePort int, kasHostname string) bool { + return tt.probeResult + }, + } + + requeue, err := r.reconcilePublicEndpointExposedCondition(t.Context(), tt.hc) + g.Expect(err).ToNot(HaveOccurred()) + + if tt.expectRequeue { + g.Expect(requeue).ToNot(BeNil(), "expected requeue") + g.Expect(*requeue).To(Equal(10 * time.Second)) + } else { + g.Expect(requeue).To(BeNil(), "expected no requeue") + } + + updatedHC := &hyperv1.HostedCluster{} + g.Expect(cl.Get(t.Context(), crclient.ObjectKeyFromObject(tt.hc), updatedHC)).To(Succeed()) + + condition := meta.FindStatusCondition(updatedHC.Status.Conditions, string(hyperv1.PublicEndpointExposed)) + if !tt.expectCondition { + g.Expect(condition).To(BeNil(), "expected no PublicEndpointExposed condition") + return + } + + g.Expect(condition).ToNot(BeNil(), "expected PublicEndpointExposed condition to be set") + g.Expect(condition.Status).To(Equal(tt.expectStatus), "unexpected condition status") + g.Expect(condition.Reason).To(Equal(tt.expectReason), "unexpected condition reason") + g.Expect(condition.ObservedGeneration).To(Equal(tt.hc.Generation)) + }) + } +} diff --git a/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy.go b/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy.go index df8ba33615e..1118e8f1f2a 100644 --- a/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy.go +++ b/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy.go @@ -157,7 +157,7 @@ func (r *HAProxy) reconcileHAProxyIgnitionConfig(ctx context.Context, hcluster * } // This is true for ARO in CI while swift is not available. - if sharedingress.UseSharedIngress() && !netutil.IsPrivateHC(hcluster) { + if netutil.UseSharedIngressHC(hcluster) && !netutil.IsPrivateHC(hcluster) { sharedIngressRouteSVC := &corev1.Service{ TypeMeta: metav1.TypeMeta{}, ObjectMeta: metav1.ObjectMeta{ @@ -175,7 +175,7 @@ func (r *HAProxy) reconcileHAProxyIgnitionConfig(ctx context.Context, hcluster * apiServerExternalPort = sharedingress.KASSVCLBPort } - useProxyProtocol := sharedingress.UseSharedIngress() && !netutil.IsPrivateHC(hcluster) + useProxyProtocol := netutil.UseSharedIngressHC(hcluster) && !netutil.IsPrivateHC(hcluster) serializedConfig, err := apiServerProxyConfig(r.HAProxyImage, controlPlaneOperatorImage, hcluster.Spec.ClusterID, apiServerExternalAddress, apiServerInternalAddress, apiServerExternalPort, apiServerInternalPort, diff --git a/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy_test.go b/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy_test.go index 44b32b29ba3..c938f1a9962 100644 --- a/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy_test.go +++ b/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy_test.go @@ -396,6 +396,14 @@ kind: Config` hc: hc(func(hc *hyperv1.HostedCluster) { hc.Spec.Platform.Type = hyperv1.AzurePlatform hc.Spec.Platform.AWS = nil + hc.Spec.Platform.Azure = &hyperv1.AzurePlatformSpec{ + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-swift-instance", + }, + }, + } hc.ObjectMeta.Annotations = map[string]string{ hyperv1.SwiftPodNetworkInstanceAnnotation: "test-swift-instance", } @@ -422,13 +430,25 @@ kind: Config` expectedHAProxyConfigContent: []string{"api.hc.hypershift.local:443"}, }, { - name: "When ARO and no Swift is used (CI) it should use the shared ingress LB service IP, port 6443 and proxy protocol", + name: "When ARO shared ingress is used it should use the shared ingress LB service IP, port 6443 and proxy protocol", setupEnv: func(t *testing.T) { azureutil.SetAsAroHCPTest(t) }, hc: hc(func(hc *hyperv1.HostedCluster) { hc.Spec.Platform.Type = hyperv1.AzurePlatform hc.Spec.Platform.AWS = nil + hc.Spec.Platform.Azure = &hyperv1.AzurePlatformSpec{ + Topology: hyperv1.AzureTopologyPublic, + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, + AzureAuthenticationConfig: hyperv1.AzureAuthenticationConfiguration{ + AzureAuthenticationConfigType: hyperv1.AzureAuthenticationTypeManagedIdentities, + }, + } hc.Status.KubeConfig = &corev1.LocalObjectReference{Name: "kk"} hc.Spec.Networking.ServiceNetwork = []hyperv1.ServiceNetworkEntry{{CIDR: *ipnet.MustParseCIDR("192.168.1.0/24")}} hc.Spec.Services = []hyperv1.ServicePublishingStrategyMapping{ diff --git a/hypershift-operator/controllers/nodepool/apiserver-haproxy/testdata/zz_fixture_TestReconcileHAProxyIgnitionConfig_When_ARO__and_no_Swift_is_used_CI__it_should_use_the_shared_ingress_LB_service_IP__port_6443_and_proxy_protocol.yaml b/hypershift-operator/controllers/nodepool/apiserver-haproxy/testdata/zz_fixture_TestReconcileHAProxyIgnitionConfig_When_ARO_shared_ingress_is_used_it_should_use_the_shared_ingress_LB_service_IP__port_6443_and_proxy_protocol.yaml similarity index 100% rename from hypershift-operator/controllers/nodepool/apiserver-haproxy/testdata/zz_fixture_TestReconcileHAProxyIgnitionConfig_When_ARO__and_no_Swift_is_used_CI__it_should_use_the_shared_ingress_LB_service_IP__port_6443_and_proxy_protocol.yaml rename to hypershift-operator/controllers/nodepool/apiserver-haproxy/testdata/zz_fixture_TestReconcileHAProxyIgnitionConfig_When_ARO_shared_ingress_is_used_it_should_use_the_shared_ingress_LB_service_IP__port_6443_and_proxy_protocol.yaml diff --git a/hypershift-operator/controllers/nodepool/nodepool_controller.go b/hypershift-operator/controllers/nodepool/nodepool_controller.go index 8a550972e4a..b2b89f4f769 100644 --- a/hypershift-operator/controllers/nodepool/nodepool_controller.go +++ b/hypershift-operator/controllers/nodepool/nodepool_controller.go @@ -16,12 +16,12 @@ import ( haproxy "github.com/openshift/hypershift/hypershift-operator/controllers/nodepool/apiserver-haproxy" "github.com/openshift/hypershift/hypershift-operator/controllers/nodepool/instancetype" "github.com/openshift/hypershift/hypershift-operator/controllers/nodepool/kubevirt" - "github.com/openshift/hypershift/hypershift-operator/controllers/sharedingress" kvinfra "github.com/openshift/hypershift/kubevirtexternalinfra" "github.com/openshift/hypershift/support/awsapi" "github.com/openshift/hypershift/support/capabilities" "github.com/openshift/hypershift/support/images" "github.com/openshift/hypershift/support/k8sutil" + "github.com/openshift/hypershift/support/netutil" "github.com/openshift/hypershift/support/releaseinfo" "github.com/openshift/hypershift/support/supportedversion" "github.com/openshift/hypershift/support/upsert" @@ -1084,21 +1084,17 @@ func (r *NodePoolReconciler) getAdditionalTrustBundle(ctx context.Context, hoste // resolveHAProxyImage determines which HAProxy image to use based on priority: // 1. NodePool annotation (highest priority) -// 2. Environment variable override (when shared ingress enabled) -// 3. Hardcoded default (when shared ingress enabled) -// 4. Release payload (default) -func resolveHAProxyImage(nodePool *hyperv1.NodePool, releaseImage *releaseinfo.ReleaseImage) (string, error) { - // Check NodePool annotation first (highest priority) +// 2. Shared ingress image (when cluster uses shared ingress for public endpoints) +// 3. Release payload (default) +func resolveHAProxyImage(nodePool *hyperv1.NodePool, hcluster *hyperv1.HostedCluster, releaseImage *releaseinfo.ReleaseImage) (string, error) { if annotationImage := strings.TrimSpace(nodePool.Annotations[hyperv1.NodePoolHAProxyImageAnnotation]); annotationImage != "" { return annotationImage, nil } - // Check if shared ingress is enabled - if sharedingress.UseSharedIngress() { + if netutil.UseSharedIngressHC(hcluster) { return images.GetSharedIngressHAProxyImage(), nil } - // Fall back to release payload image haProxyImage, ok := releaseImage.ComponentImages()[haproxy.HAProxyRouterImageName] if !ok { return "", fmt.Errorf("release image doesn't have a %s image", haproxy.HAProxyRouterImageName) @@ -1107,7 +1103,7 @@ func resolveHAProxyImage(nodePool *hyperv1.NodePool, releaseImage *releaseinfo.R } func (r *NodePoolReconciler) generateHAProxyRawConfig(ctx context.Context, nodePool *hyperv1.NodePool, hcluster *hyperv1.HostedCluster, releaseImage *releaseinfo.ReleaseImage) (string, error) { - haProxyImage, err := resolveHAProxyImage(nodePool, releaseImage) + haProxyImage, err := resolveHAProxyImage(nodePool, hcluster, releaseImage) if err != nil { return "", err } diff --git a/hypershift-operator/controllers/nodepool/nodepool_controller_test.go b/hypershift-operator/controllers/nodepool/nodepool_controller_test.go index 210b69afcef..2783e89f120 100644 --- a/hypershift-operator/controllers/nodepool/nodepool_controller_test.go +++ b/hypershift-operator/controllers/nodepool/nodepool_controller_test.go @@ -2652,12 +2652,29 @@ kind: Config`), Namespace: "clusters", }, Spec: hyperv1.HostedClusterSpec{ - Platform: hyperv1.PlatformSpec{ - Type: hyperv1.AWSPlatform, - AWS: &hyperv1.AWSPlatformSpec{ - EndpointAccess: hyperv1.Public, - }, - }, + Platform: func() hyperv1.PlatformSpec { + if tc.useSharedIngress { + return hyperv1.PlatformSpec{ + Type: hyperv1.AzurePlatform, + Azure: &hyperv1.AzurePlatformSpec{ + Topology: hyperv1.AzureTopologyPublicAndPrivate, + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{PodNetworkInstance: "test-pni"}, + }, + AzureAuthenticationConfig: hyperv1.AzureAuthenticationConfiguration{ + AzureAuthenticationConfigType: hyperv1.AzureAuthenticationTypeManagedIdentities, + }, + }, + } + } + return hyperv1.PlatformSpec{ + Type: hyperv1.AWSPlatform, + AWS: &hyperv1.AWSPlatformSpec{ + EndpointAccess: hyperv1.Public, + }, + } + }(), Networking: hyperv1.ClusterNetworking{ ServiceNetwork: []hyperv1.ServiceNetworkEntry{{CIDR: *ipnet.MustParseCIDR("192.168.1.0/24")}}, }, diff --git a/sharedingress-config-generator/config.go b/sharedingress-config-generator/config.go index 6810c004e99..998bdd04e66 100644 --- a/sharedingress-config-generator/config.go +++ b/sharedingress-config-generator/config.go @@ -85,6 +85,10 @@ func generateRouterConfig(ctx context.Context, client crclient.Client, w io.Writ if !hc.DeletionTimestamp.IsZero() { continue } + // Skip clusters that don't use shared ingress (Private topology or non-Swift). + if !netutil.UseSharedIngressHC(&hc) { + continue + } backends, externalDNSBackends, err := getBackendsForHostedCluster(ctx, hc, client) if err != nil { diff --git a/sharedingress-config-generator/config_test.go b/sharedingress-config-generator/config_test.go index 53bd460f05c..962d6698e78 100644 --- a/sharedingress-config-generator/config_test.go +++ b/sharedingress-config-generator/config_test.go @@ -108,6 +108,21 @@ func TestGenerateConfig(t *testing.T) { Spec: hyperv1.HostedClusterSpec{ ClusterID: "hc1-UUID", KubeAPIServerDNSName: "kube-apiserver-public-custom.example.com", + Platform: hyperv1.PlatformSpec{ + Type: hyperv1.AzurePlatform, + Azure: &hyperv1.AzurePlatformSpec{ + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, + Topology: hyperv1.AzureTopologyPublicAndPrivate, + AzureAuthenticationConfig: hyperv1.AzureAuthenticationConfiguration{ + AzureAuthenticationConfigType: hyperv1.AzureAuthenticationTypeManagedIdentities, + }, + }, + }, }, }, routes: []client.Object{ @@ -139,6 +154,21 @@ func TestGenerateConfig(t *testing.T) { }, }, }, + Platform: hyperv1.PlatformSpec{ + Type: hyperv1.AzurePlatform, + Azure: &hyperv1.AzurePlatformSpec{ + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, + Topology: hyperv1.AzureTopologyPublicAndPrivate, + AzureAuthenticationConfig: hyperv1.AzureAuthenticationConfiguration{ + AzureAuthenticationConfigType: hyperv1.AzureAuthenticationTypeManagedIdentities, + }, + }, + }, }, }, routes: []client.Object{ @@ -174,14 +204,20 @@ func TestGenerateConfig(t *testing.T) { ObjectMeta: metav1.ObjectMeta{ Name: "hc1", Namespace: "test", - Annotations: map[string]string{ - hyperv1.SwiftPodNetworkInstanceAnnotation: "test-swift-instance", - }, }, Spec: hyperv1.HostedClusterSpec{ ClusterID: "hc1-UUID", Platform: hyperv1.PlatformSpec{ Type: hyperv1.AzurePlatform, + Azure: &hyperv1.AzurePlatformSpec{ + Private: hyperv1.AzurePrivateSpec{ + Type: hyperv1.AzurePrivateTypeSwift, + Swift: hyperv1.AzureSwiftSpec{ + PodNetworkInstance: "test-pni", + }, + }, + Topology: hyperv1.AzureTopologyPrivate, + }, }, }, }, diff --git a/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_ARO_Swift_is_enabled_it_should_skip_the_dataplane_kas_svc.cfg b/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_ARO_Swift_is_enabled_it_should_skip_the_dataplane_kas_svc.cfg index f658e9893cf..0c125167bdc 100644 --- a/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_ARO_Swift_is_enabled_it_should_skip_the_dataplane_kas_svc.cfg +++ b/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_ARO_Swift_is_enabled_it_should_skip_the_dataplane_kas_svc.cfg @@ -39,10 +39,6 @@ frontend external-dns tcp-request content set-var(sess.src_ip) src log-format "%{+Q}o\ src_ip = %[var(sess.src_ip)], %ci:%cp / %si:%sp -> %fi:%fp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq" - acl is_test-hc1-ignition req_ssl_sni -i ignition-server.example.com - acl is_test-hc1-apiserver req_ssl_sni -i kube-apiserver-public.example.com - use_backend test-hc1-ignition if is_test-hc1-ignition - use_backend test-hc1-apiserver if is_test-hc1-apiserver default_backend no-match @@ -54,10 +50,6 @@ listen health_check_http_url # Backends for the KAS SVCs to support traffic coming form the data plane via kubernetes.svc. # Backends support any traffic coming from external DNS. -backend test-hc1-ignition - server test-hc1-ignition 1.1.1.1:443 -backend test-hc1-apiserver - server test-hc1-apiserver 4.4.4.4:6443 backend no-match tcp-request content reject diff --git a/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_there_s_two_HostedClusters_with_all_their_Routes_and_SVCs_it_should_generate_the_config_with_frontends_and_backends_for_both.cfg b/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_there_s_two_HostedClusters_with_all_their_Routes_and_SVCs_it_should_generate_the_config_with_frontends_and_backends_for_both.cfg index dac4686b904..d8ea59b32dc 100644 --- a/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_there_s_two_HostedClusters_with_all_their_Routes_and_SVCs_it_should_generate_the_config_with_frontends_and_backends_for_both.cfg +++ b/sharedingress-config-generator/testdata/zz_fixture_TestGenerateConfig_When_there_s_two_HostedClusters_with_all_their_Routes_and_SVCs_it_should_generate_the_config_with_frontends_and_backends_for_both.cfg @@ -27,12 +27,6 @@ frontend dataplane-kas-svc # example output: cluster_id = "4954e6da-fc87-4e74-9861-05895fe9549d", "169.254.169.1":38984 -> "172.20.0.1":6443 [10/Jul/2024:14:02:09.943] "dataplane-kas-svc" "