ARO-24544: migrate admin API endpoint for control plane VM resize

[tuxerrante]

Which issue this PR addresses:

Fixes ARO-24544

What this PR does / why we need it:

Migrates the control plane VM resize orchestration from the external C# Geneva Action (ResizeControlPlaneVMsOperation.cs) into the ARO-RP as a native admin API endpoint. This eliminates split-brain logic between the Geneva Action and the RP, gives the operation access to the RP's existing validation infrastructure, and makes the orchestration testable with standard Go tooling.

The new POST /admin/.../resizecontrolplane?vmSize=<sku>&deallocateVM=<bool> endpoint performs pre-flight health checks and then sequentially resizes each master node through the full lifecycle: cordon, drain (with retry), stop VM, resize VM, start VM, wait for Node Ready, uncordon, update Machine object metadata, and update Node instance-type labels.

Design choices and deviations from the original C# implementation

• Synchronous admin operation. Matches the pattern used by stopvm, startvm, redeployvm. The UnplannedMaintenanceSignal middleware signals maintenance state.
• deallocateVM defaults to true because Azure requires deallocation for most cross-family resizes. Callers can pass deallocateVM=false explicitly.
• CPMS-active blocks the operation with 409 Conflict. CPMS-aware resize (patching the CPMS spec) is planned for a follow-up.
• Reverse name-order processing (master-2 → master-1 → master-0) minimises etcd leader elections, matching the C# behaviour.
• Pre-flight health validation via the shared _getPreResizeControlPlaneVMsValidation endpoint, which checks API server health (synchronous gate), etcd health, service principal validity, VM SKU/quota, and CPMS state.
• API server health as a synchronous gate: if the kube-apiserver is unreachable, we fail immediately instead of spawning parallel kube-based checks that would all fail with connection errors.
• Reuse of existing helpers: validateAdminMasterVMSize, getClusterMachines (via getControlPlaneMachines wrapper), and health checks from the pre-resize validation endpoint.
• Machine/Node metadata updates with retry (up to 3 attempts, ctx-aware delays). The Machine update uses typed machinev1beta1.Machine objects for safety.
• Explicit cordon/uncordon wrappers: cordonNode() and uncordonNode() wrap CordonNode(bool) to make intent visible at every call site.

Test plan for issue:

• Unit tests (pkg/frontend/admin_openshiftcluster_resize_controlplane_test.go):

  • TestCheckCPMSNotActive — CPMS not found, Inactive, Active (blocked), empty state, non-NotFound error (fails closed), invalid JSON (fails closed)
  • TestIsNodeReady — node ready, not ready, not found
  • TestResizeControlPlane — all-nodes-already-at-size no-op, single-node full sequence (verifies exact call ordering via gomock.InOrder), no machines found, drain/stop/resize failure cases
  • TestUpdateMachineVMSize — success, retry on conflict
  • TestUpdateNodeInstanceTypeLabels — success, retry on conflict
  • TestAdminResizeControlPlane — full HTTP integration: invalid VM size (400), cluster not found (404), subscription not found (400)

• E2E tests (test/e2e/adminapi_resize_controlplane.go):

  • Rejects unsupported VM size (400)
  • Rejects missing vmSize parameter (400)
  • (Full resize E2E requires a live cluster and is validated manually in dev environments)

• Local integration testing against containerized RP:

  Prerequisites: Docker (or Podman on Linux), env file at repo root (from env.example), secrets/ directory with valid credentials.

  # Build the dev container image (first time or after Dockerfile changes)
  make dev-env-build
  
  # Start the containerized RP (first run compiles from source, ~2-5 min)
  make dev-env-start
  
  # Wait for the RP to become healthy
  until curl -ksSf https://localhost:8443/healthz/ready 2>/dev/null; do sleep 5; done
  
  # Test: invalid vmSize → 400
  curl -ksS -X POST \
    "https://localhost:8443/admin/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/test-rg/providers/microsoft.redhatopenshift/openshiftclusters/test-cluster/resizecontrolplane?vmSize=Standard_Invalid_Fake&deallocateVM=true"
  
  # Test: missing vmSize → 400
  curl -ksS -X POST \
    "https://localhost:8443/admin/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/test-rg/providers/microsoft.redhatopenshift/openshiftclusters/test-cluster/resizecontrolplane?deallocateVM=true"
  
  # Test: valid vmSize, nonexistent cluster → 404
  curl -ksS -X POST \
    "https://localhost:8443/admin/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/test-rg/providers/microsoft.redhatopenshift/openshiftclusters/nonexistent/resizecontrolplane?vmSize=Standard_D8s_v3&deallocateVM=true"
  
  # Stop the containerized RP
  make dev-env-stop

  Test case	Expected
  Invalid vmSize	400 InvalidParameter
  Missing vmSize	400 InvalidParameter
  Valid vmSize, nonexistent cluster	404 ResourceNotFound

Is there any documentation that needs to be updated for this PR?

The admin API endpoint documentation in docs/deploy-development-rp.md should be updated to include an example curl command for the new /resizecontrolplane endpoint. This can be done in a follow-up.

How do you know this will function as expected in production?

• The operation is 1:1 with the proven C# Geneva Action sequence — same steps, same order.
• Structured logging at every step provides full observability.
• Failure modes are handled explicitly: drain retries (3×2s), kube update retries (3×1s), node-ready polling (30min×5s), context cancellation respected throughout.
• The UnplannedMaintenanceSignal middleware, CPMS-active check, and pre-flight health checks (apiserver gate + etcd + SP + CPMS) prevent conflicting or unsafe operations.
• Best-effort node recovery is planned as a separate follow-up PR for easier review and revertability.

[Copilot]

Pull request overview

Adds a new admin API endpoint to orchestrate control plane VM resizes directly within ARO-RP, migrating the resize workflow from the external Geneva Action into the RP so it can reuse existing validation and be unit-tested.

Changes:

• Adds POST /admin/.../resizecontrolplane route guarded by UnplannedMaintenanceSignal.
• Implements synchronous, sequential master resize orchestration with best-effort recovery and kube metadata/label updates.
• Adds unit tests for orchestration helpers and minimal E2E validations for request rejection cases.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
test/e2e/adminapi_resize_controlplane.go	Adds E2E coverage for basic 400 validation cases (unsupported size, missing vmSize).
pkg/frontend/frontend.go	Wires the new /resizecontrolplane admin route with maintenance signaling middleware.
pkg/frontend/admin_openshiftcluster_resize_controlplane.go	New resize orchestration implementation (cordon/drain/stop/resize/start/wait/uncordon + metadata updates + recovery).
pkg/frontend/admin_openshiftcluster_resize_controlplane_test.go	Adds unit tests for CPMS gating, readiness checks, drain retries, full sequence ordering, and recovery paths.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mociarain]

LGTM but there a bunch of copilot suggestions. I'm gonna treat these like the PR is still a draft i.e. when they're resolved I'll give it a more thorough review... Is this a good or bad heuristic?

[Copilot]

Pull request overview

Copilot reviewed 40 out of 44 changed files in this pull request and generated 11 comments.

Comments suppressed due to low confidence (1)

pkg/util/azureclient/azuresdk/common/options.go:51

• shouldRetry() no longer reads the response body correctly: var b []byte; resp.Body.Read(b) reads 0 bytes, so the retry checks for AADSTS/AuthorizationFailed will never match. It also doesn’t restore/close the body, which can break downstream SDK error handling and leak connections. Please revert to fully reading the body (e.g., io.ReadAll), close the original body, and restore resp.Body so it can be read again; keep/restore the unit tests that validated this behavior.

	// Check if the body contains the certain strings that can be retried.
	var b []byte
	_, err = resp.Body.Read(b)
	if err != nil {
		return true
	}
	body := string(b)
	return strings.Contains(body, ErrCodeInvalidClientSecretProvided) ||
		strings.Contains(body, ErrCodeMissingRequiredParameters) ||
		strings.Contains(body, AuthorizationFailed)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bizz001]

@tuxerrante Since we are moving from various smaller individual API calls to a single long running synchronous call, should we be concerned by the possibility of connection drops or load balancer timeouts? If I am not mistaken If the connection is closed the context is instantly killed.

[tuxerrante]

@tuxerrante Since we are moving from various smaller individual API calls to a single long running synchronous call, should we be concerned by the possibility of connection drops or load balancer timeouts? If I am not mistaken If the connection is closed the context is instantly killed.

@bizz001 Yes, this is a valid concern.

This endpoint currently follows the same request-scoped synchronous model used by existing admin imperative APIs, rather than the ARM async-operation path:

• Admin route wiring for these endpoints (redeployvm, stopvm, startvm, resize, resizecontrolplane, cordonnode, drainnode):
  pkg/frontend/frontend.go#L374-L393
• Existing handlers all start from request context (ctx := r.Context()), e.g.
  postAdminOpenShiftClusterStopVM,
  postAdminOpenShiftClusterStartVM,
  postAdminOpenShiftClusterRedeployVM,
  postAdminOpenShiftClusterVMResize,
  postAdminOpenShiftClusterCordonNode,
  postAdminOpenShiftClusterDrainNode
• New endpoint uses the same model:
  postAdminResizeControlPlane ->
  _postAdminResizeControlPlane ->
  resizeControlPlaneNode

So yes: if client/LB drops the connection, context cancellation can abort an in-flight operation. That behavior is consistent with the current admin API model; the main difference here is that this operation is longer, so the exposure window is larger.

A full async approach would require a broader architectural change (persisted progress + background worker/reconciler), closer to the async ARM pattern (operationsstatus/operationresults) here:
pkg/frontend/frontend.go#L292-L294

Given scope, this PR keeps parity with existing admin behavior; near-term hardening is idempotency/recovery so reruns are safe after interruptions (ARO-24543).

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Please rebase pull request.

[mrWinston]

Thanks for all your effort! Looks good to me now 🚀

[mociarain]

LGTM