feat: sequenced rolling restart, MAIN and Raft leader last

[as51340]

Carries out a changed pod template without a coordinator-driven failover per pod. Both role StatefulSets move to updateStrategy: OnDelete, so Kubernetes replaces nothing on its own and the operator owns every restart from here on. That cost is real and permanent — a template change no reconcile acts on takes effect never, which is what the new Updated condition exists to report — and it buys the one property RollingUpdate cannot express: it sweeps highest ordinal to lowest and partition is a descending cutoff rather than a set, so a MAIN on any ordinal but 0 is restarted mid-sweep, and every such restart buys another failover.

The order is data instances before coordinators, the observed MAIN last of its role, the observed Raft leader last of its. One pod of the cluster is down at a time, across both roles. internal/rollout decides it as a pure function — both roles' pods reduced to {name, revisionHash, ready} plus each StatefulSet's UpdateRevision, the SHOW INSTANCES view and SHOW REPLICATION LAG in, exactly one Done/Wait(reason)/Delete(pod) out. One action and never a list, because every step re-gates on a fresh observation: lag measured one pod ago says nothing about the next. Nothing is persisted — the pods already carrying the new revision are the ones already restarted, so a spec reverted halfway through, or Raft moving MAIN or leadership mid-roll, self-corrects with nothing to unwind.

What gates a step is not pod readiness. Probes are TCP connects to a port, so Ready says an instance answers, not that it rejoined replication. Taking another replica down reduces the number of instances holding recent writes, so every already-restarted pod must be reachable and caught up first — which is what waits out a fresh volume's full snapshot resync. Untouched pods are held to readiness alone, so a replica that was already lagging cannot block the roll. Restarting the MAIN forces a promotion, so it needs one survivor that is reachable and caught up: one, not all, because the coordinators promote the most up-to-date instance they can reach, so whoever wins is at least as current as the one proven — and because a chronically sick replica must not freeze the cluster's pod template. A coordinator restart does neither of those things, so lag does not gate it; leader health does, which with three or more coordinators is the quorum question itself. Without a caught-up survivor the roll parks at the MAIN indefinitely and says so (NoCaughtUpSurvivor); the cluster keeps serving, so waiting costs only the upgrade. A single data instance is the exception: it can never satisfy that precondition, so it is restarted with acknowledged downtime rather than left frozen forever.

The operator promotes nothing here — it deletes the MAIN's pod and the coordinators fail over, which is what keeps two control systems from choosing a MAIN at once. That is only safe on a Memgraph reporting an unreachable MAIN as role=main, health=down. A release that vacates the main row instead leaves planner.Plan believing the cluster has no MAIN, and it will race the failover with a promotion of its own on every single restart. Which operator version is safe against which Memgraph release is documented outside this repository.

The registration planner keeps running throughout a roll and should return empty on every pass; if it ever does not, that is real drift the cluster wants repaired now rather than after the roll. workloadsReady therefore tolerates one existing but unready pod — otherwise the first deleted pod would end the pass before it ever connected to a coordinator, and the roll could never learn whether that pod came back. Both halves of the exception matter: only while the role has outdated pods, and only when all of its pods exist, or a 3-to-4 scale-up's stale readyReplicas would read as "one pod down, mid-roll, tolerated" and let registration run against a pod that does not exist yet. A scale converges before any roll step, so the retirement's own MAIN handover is never moving MAIN at the same time as this is.

observedMain now requires IsMain() && IsUp(). A dead MAIN keeps its role in Raft, so the old check claimed the cluster served writes for the whole failover window — including every window this feature opens deliberately. Ready is consequently False for a few seconds of every upgrade, which is honest.

Also: terminationGracePeriodSeconds: 300 on both roles, because Kubernetes' 30 seconds was harmless while nothing routinely deleted these pods and is wrong now that every one of them is deleted on every template change — an instance killed mid-shutdown recovers from its WAL and lengthens exactly the catch-up the roll waits on. It is a ceiling, not a delay. The two scale-down pod-shed assertions and the scaling namespace teardown were widened above it, since a five-minute timeout against a five-minute grace period is a coin flip rather than an assertion.

Pods are read through a label-scoped informer (app.kubernetes.io/managed-by), so the cache holds this operator's pods and not the cluster's, and the manager's ClusterRole gains pods: get;list;watch;delete — a user-visible widening worth naming in the chart's release notes. Deletes carry the observed pod's UID as a precondition, so a pod already replaced between observation and delete is left alone instead of restarted twice.

Instance naming is collapsed to one spelling while it is being relied on from a third place: resources.CoordinatorInstanceName/DataInstanceName and their inverses, with the coordinator name's format and parser adjacent in internal/memgraph. test/utils/names.go is deleted — its "spelled out independently so tests catch a rename" rationale is already covered by topology_test.go, which pins the literals in its golden expectations, so the parallel implementation bought nothing and cost a third copy.

Not attempted here: proving no acknowledged write is lost across a roll. That needs a write workload running through the whole sequence and belongs to the chaos-testing project, not to a spec gating every pull request. A PodDisruptionBudget is also left out — a node drain can still take the MAIN and a replica together, but that gap predates this and is unrelated to operator-driven restarts.

See specs/operator-mvp/issues/17-sequenced-rolling-restart.md.