GCP destroy: instance group deletion fails due to dependency ordering with backend services

[kaovilai]

Bug

openshift-install destroy cluster on GCP fails to delete instance groups because the backend service that references them hasn't been deleted yet. The destroy loop retries indefinitely but the dependency is never resolved cleanly, leaving orphaned resources.

Error

WARNING failed to delete instance group tkaovila-260601-wif-fwj68-master-us-east1-c: googleapi: Error 400:
  The instance_group resource 'projects/.../instanceGroups/...-master-us-east1-c' is already being used by
  'projects/.../backendServices/...-apiserver', resourceInUseByAnotherResource

This repeats for all 3 instance groups (one per zone), cycling many times before the backend service is eventually deleted.

Root Cause

In pkg/destroy/gcp/gcp.go lines 204-223, all resource types in stage 3 are destroyed in parallel:

{
    {name: "Instances", execute: o.destroyInstances},
    // ... other resources ...
    {name: "Instance groups", execute: o.destroyInstanceGroups},   // line 215
    {name: "Target TCP Proxies", execute: o.destroyTargetTCPProxies}, // line 216
    {name: "Backend services", execute: o.destroyBackendServices}, // line 217
    // ...
}

GCP enforces a dependency chain: forwarding rules → target TCP proxies → backend services → instance groups. A backend service cannot be deleted while a forwarding rule references it, and an instance group cannot be deleted while a backend service references it.

Since all deletions run concurrently, instance group deletion is attempted before (or simultaneously with) backend service deletion. The GCP API returns 400 resourceInUseByAnotherResource. The retry loop does eventually resolve this (backend service succeeds on attempt N, instance groups succeed on attempt N+1), but it produces many warning logs and can leave orphaned resources if the destroy times out.

Suggested Fix

Split the load balancer resources into dependency-ordered stages:

// Stage 3a: Delete LB frontend (forwarding rules, target proxies)
{
    {name: "Forwarding rules", execute: o.destroyForwardingRules},
    {name: "Target Pools", execute: o.destroyTargetPools},
    {name: "Target TCP Proxies", execute: o.destroyTargetTCPProxies},
},
// Stage 3b: Delete LB backend
{
    {name: "Backend services", execute: o.destroyBackendServices},
    {name: "Health checks", execute: o.destroyHealthChecks},
    {name: "HTTP Health checks", execute: o.destroyHTTPHealthChecks},
},
// Stage 3c: Delete compute resources (instance groups now safe to delete)
{
    {name: "Instances", execute: o.destroyInstances},
    {name: "Disks", execute: o.destroyDisks},
    {name: "Instance groups", execute: o.destroyInstanceGroups},
    // ... remaining resources ...
},

This ensures backend services are fully deleted before instance group deletion is attempted.

Environment

• openshift-install version: 4.22.0-ec.5 (commit f8a1613fc01e0b5de8ee9906e8aa067ba2fdb98e)
• Platform: GCP (Workload Identity Federation)
• Region: us-east1

Orphaned Resources After Destroy

After destroy timed out, these resources remained:

$ gcloud compute backend-services list --filter='name~cluster-name'
tkaovila-260601-wif-fwj68-apiserver

$ gcloud compute instance-groups list --filter='name~cluster-name'
tkaovila-260601-wif-fwj68-master-us-east1-b
tkaovila-260601-wif-fwj68-master-us-east1-c
tkaovila-260601-wif-fwj68-master-us-east1-d

Note

Responses generated with Claude

[kaovilai]

Additional findings

GCP operation stuck in RUNNING state

During destroy, one of the delete operations for the backend service got stuck with a GCP internal error:

$ gcloud compute operations describe <op-id> --global --project=gc-acm-demo
error:
  errors:
  - code: INTERNAL_ERROR
    message: "Internal error. Please try again or contact Google Support. (Code: '7298347784567734096')"
progress: 0
status: RUNNING

This operation has been stuck in RUNNING state for 5+ hours with 0 progress. While in this state, the backend service returns "not ready" and cannot be deleted by any means:

$ gcloud compute backend-services delete ...-apiserver --global --quiet
ERROR: The resource '...backendServices/...-apiserver' is not ready

This means even with correct dependency ordering, a stuck GCP operation could leave resources permanently orphaned until GCP resolves the internal error. The installer's retry loop would spin indefinitely on this.

Suggested additional handling

The destroy code could check for operations stuck in RUNNING with INTERNAL_ERROR and log a clear message directing the user to contact Google Support with the error code, rather than silently retrying forever.

Note

Responses generated with Claude

[weshayutin]

@PrasadJoshi12 @tsanders-rh fyi

[kaovilai]

opened also #10590

[kaovilai]

from support

We have reached out to our specialist team and are actively investigating the issue you experienced when deleting the backend service. We will let you know once we have new information, please expect our next update by June 04 at 4:00 PM PDT.

In the meantime, I have tried to reproduce your configuration in a test environment. I created a classic proxy load balancer with an instance group backend and then attempted to delete these resources.I first confirmed that as our documentation [1] states, an instance group cannot be deleted when it is being used by a backend service. While for the backend service itself, I confirmed that the deletion on it is successful when the components are removed in the following order:

Delete the forwarding rule [2].

Delete the target proxy [3].

Delete the backend service [4].

hopefully PR fixes it.