GCP CAPI: bootstrap in master instance group causes worker ignition failure via ILB pinning

[kaovilai]

Bug

Worker nodes never join GCP CAPI-based OCP 4.22 clusters. Workers are provisioned (GCP VMs running) but stay stuck in an ignition fetch loop indefinitely, receiving HTTP 500 from the Machine Config Server (MCS).

Root Cause

In GCP CAPI installs, the installer places the bootstrap node in the same unmanaged instance group as master-0 (zone-a/b). Both are backends for the GCP Internal Load Balancer (ILB) on ports 6443 and 22623.

The GCP ILB uses connection-based session affinity (CONNECTION mode) — once a worker's TCP connection is established to a backend, it stays pinned for the connection lifetime.

During the critical window when workers first boot and fetch ignition config from api-int:22623, the bootstrap node is the healthiest/first-responding backend. All worker connections get pinned to bootstrap.

Bootstrap MCS is designed to refuse worker ignition requests — it only serves master configs:

refusing to serve bootstrap configuration to pool "worker"

Workers receive HTTP 500 forever. The in-cluster MCS running on master-0 receives zero worker requests despite being healthy and ready.

Evidence

• Worker serial console: 900+ ignition GET attempts, all returning Internal Server Error
• Bootstrap MCS logs: explicit refusing to serve bootstrap configuration to pool "worker"
• In-cluster MCS logs: zero worker requests despite running 40+ minutes
• Zero worker CSRs in the cluster
• gcloud compute instance-groups unmanaged list-instances confirms bootstrap and master-0 in same group

Workaround

Remove bootstrap from the master instance group after masters are ready:

gcloud compute instance-groups unmanaged remove-instances \
  <cluster-id>-master-<zone> \
  --zone=<zone> \
  --instances=<cluster-id>-bootstrap \
  --project=<project>

All 3 workers completed ignition within minutes and joined the cluster after applying this.

Suggested Fix

The installer's GCP CAPI code should either:

1. Give bootstrap its own separate instance group (as Terraform/UPI installs did), or
2. Remove bootstrap from the master instance group once masters are healthy and in-cluster MCS is serving

Environment

• OCP: 4.22.0-rc.5 (also reproduced on 4.22.0-rc.4)
• Platform: GCP (IPI with CAPI, Workload Identity Federation)
• Region: us-east1
• Confirmed across 5+ cluster creation attempts

Impact

This affects all GCP CAPI installs in OCP 4.22, not just WIF/STS. CAPI became the default for GCP IPI in 4.22.

Note

Responses generated with Claude

[kaovilai]

cc: @weshayutin @tsanders-rh

[kaovilai]

Root Cause Analysis & Suggested Fix

Why bootstrap ends up in the master instance group

The issue is in pkg/asset/machines/gcp/gcpmachines.go. The createGCPMachine() function (line ~130) unconditionally adds the cluster.x-k8s.io/control-plane label to every GCPMachine, including bootstrap:

// createGCPMachine() - line ~130
gcpMachine := &capg.GCPMachine{
    ObjectMeta: metav1.ObjectMeta{
        Name: name,
        Labels: map[string]string{
            "cluster.x-k8s.io/control-plane": "",  // ← always added, even for bootstrap
        },
    },

The same issue exists in createCAPIMachine() (line ~194):

machine := &capi.Machine{
    ObjectMeta: metav1.ObjectMeta{
        Name: name,
        Labels: map[string]string{
            "cluster.x-k8s.io/control-plane": "",  // ← always added
        },
    },

Meanwhile, pkg/asset/manifests/gcp/cluster.go (line ~163) sets:

LoadBalancer: capg.LoadBalancerSpec{
    APIServerInstanceGroupTagOverride: ptr.To(InstanceGroupRoleTag), // "master"
},

CAPG uses the cluster.x-k8s.io/control-plane label to determine which machines go into the API server instance groups. Since bootstrap has this label, it gets placed in the same instance group as master-0 (same zone).

The chain

1. GenerateBootstrapMachines() calls createGCPMachine() → bootstrap gets control-plane label
2. Line 96 adds install.openshift.io/bootstrap label but doesn't remove control-plane label
3. CAPG controller sees control-plane label → adds bootstrap to master instance group
4. GCP ILB pins worker connections to bootstrap (CONNECTION affinity) → bootstrap refuses worker ignition → 0 workers join

Suggested fix

Option A (cleanest): Add a isControlPlane bool parameter to createGCPMachine() and createCAPIMachine(), only add the label when true:

func createGCPMachine(..., isControlPlane bool) (*capg.GCPMachine, error) {
    labels := map[string]string{}
    if isControlPlane {
        labels["cluster.x-k8s.io/control-plane"] = ""
    }
    gcpMachine := &capg.GCPMachine{
        ObjectMeta: metav1.ObjectMeta{
            Name:   name,
            Labels: labels,
        },
    // ...

Update callers:

• GenerateMachines() (line ~53): createGCPMachine(..., true)
• GenerateBootstrapMachines() (line ~90): createGCPMachine(..., false)

Same pattern for createCAPIMachine().

Option B (simpler, minimal diff): Remove the label from bootstrap after creation in GenerateBootstrapMachines():

// After createGCPMachine() call (~line 90):
delete(bootstrapGCPMachine.Labels, "cluster.x-k8s.io/control-plane")

// After createCAPIMachine() call (~line 107):
delete(bootstrapCapiMachine.Labels, "cluster.x-k8s.io/control-plane")

Key files

File	Lines	What to change
pkg/asset/machines/gcp/gcpmachines.go	~130-136	createGCPMachine() unconditionally adds control-plane label
pkg/asset/machines/gcp/gcpmachines.go	~194-200	createCAPIMachine() same issue
pkg/asset/machines/gcp/gcpmachines.go	~82-114	GenerateBootstrapMachines() — caller that should NOT get control-plane label

Note

Responses generated with Claude