See template drift, and optionally roll cells to fix it

CHANGELOG.md

@@ -3,6 +3,37 @@
 All notable changes to this project are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [Unreleased]
+
+### Added
+
+- **Template drift is visible.** Every cell records the template it was created
+  from, surfaced as `status.cells[].templateHash`, and a new `Updated` condition
+  compares it with the pool's current template. The hash was being written to each
+  cell and never read back, so a `spec.cell` change was invisible to the pool.
+- **`spec.updatePolicy.type: RollingUpdate`** replaces stale cells, one at a time,
+  through the same drain gate automatic scale-down uses: only cells the capacity
+  provider reports idle, never during a resize, never while another cell is coming
+  or going, never while the workload cluster is unreachable, lowest index first.
+  Opt-in, because a template edit is not consent to destroy running work — and it
+  stalls visibly (`UpdateBlocked`) rather than evicting anything. See
+  `docs/updates.md`.
+
+### Fixed
+
+- **A replacement cell inherited its predecessor's teardown.** Cell names are
+  reused (index 0 is always `<pool>-0`) and status rows are keyed by name, so a
+  freshly created cell adopted the previous incarnation's `Draining` phase and was
+  deleted on the pass that created it — create, destroy, create, destroy, with no
+  timeout that could break the loop. Rows for a different guest UID no longer lend
+  their phase; the failure counter still carries, because the replacement backoff
+  is counted per index. This affected any replacement path, not just the new
+  rolling update, and was only masked because nothing had previously refilled an
+  index whose row still said `Draining`.
+- Cell idleness was read from the capacity provider only when *automatic
+  scale-down* was enabled, so a rolling update always saw zero idle cells and
+  silently never acted.
+
 ## [v0.1.0] — 2026-08-08
 
 First release. Every capability below has been run on real hardware (one

api/v1alpha1/conditions.go

@@ -31,6 +31,16 @@ const (
 	// being drained or cordoned. Cells are never offline-migrated, so this is a
 	// signal for the operator (and, later, for automated replacement).
 	ConditionCellDrainRequested = "CellDrainRequested"
+
+	// ConditionUpdated reports whether every cell was created from the CURRENT
+	// cell template. False means at least one cell is running an older shape — a
+	// previous image, guest class or interface set.
+	//
+	// It is only ever informational until spec.updatePolicy.type is RollingUpdate,
+	// because replacing a cell destroys whatever the old one was still running
+	// unless it is drained first. Knowing you have drifted is useful on its own;
+	// acting on it is a decision the operator opts into.
+	ConditionUpdated = "Updated"
 )
 
 // Condition reasons. Every reason names a distinct operator action: "your token
@@ -44,6 +54,9 @@ const (
 	ReasonCellCreating    = "CellCreating"
 	ReasonCellDraining    = "CellDraining"
 	ReasonTemplateChanged = "TemplateChanged"
+	ReasonAllCellsCurrent = "AllCellsCurrent"
+	ReasonRollingUpdate   = "RollingUpdate"
+	ReasonUpdateBlocked   = "UpdateBlocked"
 
 	ReasonConnected         = "Connected"
 	ReasonUnreachable       = "Unreachable"

api/v1alpha1/gpucellpool_types.go

@@ -51,6 +51,38 @@ type GPUCellPoolSpec struct {
 	// Deletion controls teardown behaviour for the pool.
 	// +optional
 	Deletion *DeletionSpec `json:"deletion,omitempty"`
+
+	// UpdatePolicy decides what happens to existing cells when spec.cell changes.
+	// +optional
+	UpdatePolicy *UpdatePolicySpec `json:"updatePolicy,omitempty"`
+}
+
+// Cell update modes.
+const (
+	// UpdateManual leaves existing cells alone when the template changes. The
+	// pool reports Updated=False/TemplateChanged and which cells are stale; you
+	// replace them when it suits you.
+	UpdateManual = "Manual"
+	// UpdateRolling replaces stale cells one at a time, using the same drain gate
+	// as automatic scale-down.
+	UpdateRolling = "RollingUpdate"
+)
+
+// UpdatePolicySpec decides what happens to cells already running an older
+// spec.cell than the pool now declares.
+//
+// Manual is the default because replacing a cell destroys whatever the old one was
+// still running unless it is drained first, and a template edit is not consent to
+// that. RollingUpdate is opt-in, and it is deliberately slow: one cell at a time,
+// only cells the capacity provider reports as IDLE, and never while another cell is
+// already being created or drained. On a pool whose cells are all busy it will
+// therefore make no progress — and says so, rather than forcing its way through.
+type UpdatePolicySpec struct {
+	// Type is Manual (default) or RollingUpdate.
+	// +kubebuilder:validation:Enum=Manual;RollingUpdate
+	// +kubebuilder:default=Manual
+	// +optional
+	Type string `json:"type,omitempty"`
 }
 
 // Scale-down modes.
@@ -556,6 +588,13 @@ type CellStatus struct {
 	// +optional
 	ReadyOnce bool `json:"readyOnce,omitempty"`
 
+	// TemplateHash is the cell template this cell was CREATED from. When it differs
+	// from the pool's current template the cell is running an older shape, which is
+	// what the Updated condition reports — and, under
+	// updatePolicy.type: RollingUpdate, what gets it replaced.
+	// +optional
+	TemplateHash string `json:"templateHash,omitempty"`
+
 	// LastTransitionTime is when Phase last changed.
 	// +optional
 	LastTransitionTime *metav1.Time `json:"lastTransitionTime,omitempty"`

charts/gpucellpool/crds/cells.kubeswift.io_gpucellpools.yaml

@@ -442,6 +442,18 @@ spec:
                 format: int32
                 minimum: 0
                 type: integer
+              updatePolicy:
+                description: UpdatePolicy decides what happens to existing cells when
+                  spec.cell changes.
+                properties:
+                  type:
+                    default: Manual
+                    description: Type is Manual (default) or RollingUpdate.
+                    enum:
+                    - Manual
+                    - RollingUpdate
+                    type: string
+                type: object
               workloadCluster:
                 description: WorkloadCluster is the cluster the cells join and where
                   HAMi runs.
@@ -640,6 +652,13 @@ spec:
                         ReadyOnce records that this cell reached Ready at least once, so a later
                         Ready after a regression is not mistaken for a startup.
                       type: boolean
+                    templateHash:
+                      description: |-
+                        TemplateHash is the cell template this cell was CREATED from. When it differs
+                        from the pool's current template the cell is running an older shape, which is
+                        what the Updated condition reports — and, under
+                        updatePolicy.type: RollingUpdate, what gets it replaced.
+                      type: string
                   required:
                   - index
                   - name

config/crd/bases/cells.kubeswift.io_gpucellpools.yaml

@@ -442,6 +442,18 @@ spec:
                 format: int32
                 minimum: 0
                 type: integer
+              updatePolicy:
+                description: UpdatePolicy decides what happens to existing cells when
+                  spec.cell changes.
+                properties:
+                  type:
+                    default: Manual
+                    description: Type is Manual (default) or RollingUpdate.
+                    enum:
+                    - Manual
+                    - RollingUpdate
+                    type: string
+                type: object
               workloadCluster:
                 description: WorkloadCluster is the cluster the cells join and where
                   HAMi runs.
@@ -640,6 +652,13 @@ spec:
                         ReadyOnce records that this cell reached Ready at least once, so a later
                         Ready after a regression is not mistaken for a startup.
                       type: boolean
+                    templateHash:
+                      description: |-
+                        TemplateHash is the cell template this cell was CREATED from. When it differs
+                        from the pool's current template the cell is running an older shape, which is
+                        what the Updated condition reports — and, under
+                        updatePolicy.type: RollingUpdate, what gets it replaced.
+                      type: string
                   required:
                   - index
                   - name

docs/README.md

@@ -10,6 +10,7 @@ Start here if you are installing or operating a pool.
 | [concepts](concepts.md) | the two-layer model — cells, the two identities, the two capacities, layered isolation |
 | [api-reference](api-reference.md) | the full `GPUCellPool` v1alpha1 spec/status, generated from the Go types and the validating webhook |
 | [autoscaling](autoscaling.md) | `spec.autoscaling` — both directions, the safety gates, the remembered cell shape |
+| [updates](updates.md) | changing `spec.cell`: seeing drift, replacing cells by hand, `updatePolicy.type: RollingUpdate` |
 | [networking](networking.md) | the routable-interface requirement, `nodeIPFrom`, NADs, DNS, `port-forward` |
 | [security](security.md) | why creating a pool is node-root-equivalent authority, the webhook, the two RBAC scopes |
 | [cell-image](cell-image.md) | building and publishing a cell image with `hack/build-cell-image.sh` |

docs/limitations.md

@@ -15,22 +15,6 @@ demand signal (`docs/autoscaling.md`), and readiness all fail loudly rather
 than silently reading zero. **`DevicePlugin` mode is the only implementation.**
 Use it (the default) even if your HAMi install also has DRA mode available.
 
-## No rolling update on `guestTemplate` change
-
-Tracked as [#2](https://github.com/kubeswift-io/gpucellpool/issues/2).
-
-Changing `spec.cell.guestTemplate` (a new `imageRef`, a driver bump, a
-different `guestClassRef`) bumps the per-cell template-hash annotation but
-**does not roll existing cells**. `status.conditions` will not tell you a
-template drifted either — there is no `Updated` condition in v1alpha1.
-
-This is the most likely real operational task you will hit: **updating the
-NVIDIA driver (or anything else) in the cell image means manually recreating
-every cell**, one at a time, and doing the *inner* drain yourself first —
-cordon and drain the cell's workload Node before deleting the cell's
-`SwiftGuest`/`Machine`, or you destroy running HAMi workloads. See
-`docs/runbook.md` for the sequence.
-
 ## No automated outer-drain sequencing
 
 Tracked as [#3](https://github.com/kubeswift-io/gpucellpool/issues/3).
@@ -91,6 +75,11 @@ is 1 — use `resourceClaimTemplateName` for anything larger.
 
 ## What is *not* a limitation, stated for clarity
 
+Replacing cells after a `spec.cell` change is **implemented**: the `Updated`
+condition reports drift, and `updatePolicy.type: RollingUpdate` replaces stale
+cells one at a time behind the drain gate. See `docs/updates.md`. It is opt-in
+because a template edit is not consent to destroy running work.
+
 - Scale-up and scale-down are both implemented (`docs/autoscaling.md`) — the
   earlier design draft that called scale-down "postponed" is stale; ignore
   any doc under `docs/design/` that still says so (they carry a banner).

docs/runbook.md

@@ -21,6 +21,7 @@ kubectl get cellpool <name> -n <ns> -o jsonpath='{range .status.conditions[*]}{.
 | `PhysicalGPUsAvailable` | the infrastructure cluster's GPU inventory |
 | `CapacityAvailable` | the workload cluster's GPU is full, or unreadable |
 | `ScalingActive` | demand could not be read (only present when autoscaling is on) |
+| `Updated` | cells are running an older `spec.cell` than the pool declares — see below |
 | `Ready` alone | individual cells — read `status.cells[]` |
 
 Per-cell detail, which names the layer and the reason:
@@ -193,6 +194,24 @@ An empty providerID with the Machine in `Provisioning` means the provider is sti
 working. A `Provisioned` Machine with no providerID is a capi-kubeswift problem, not
 a pool problem.
 
+### `Updated=False` — cells are running an older template
+
+Expected after any `spec.cell` edit. The reason says what will happen next:
+
+| Reason | Meaning |
+|---|---|
+| `TemplateChanged` | drift detected, and `updatePolicy.type` is `Manual` — nothing will be replaced. Replace cells yourself, or switch to `RollingUpdate`. See `docs/updates.md` |
+| `RollingUpdate` | a cell is being replaced right now |
+| `UpdateBlocked` | a rolling update is wanted but cannot proceed. The message names the cause: every stale cell still holds workloads, the pool is mid-resize, another cell is already being replaced, or the workload cluster is unreachable |
+
+`status.cells[].templateHash` tells you which cells are stale. An empty hash
+counts as current — it predates the field, and treating unknown as out-of-date
+would replace a healthy pool.
+
+A rolling update that is blocked on busy cells stays blocked indefinitely, by
+design; it will not evict anything. `kubectl drain <cell>` in the workload
+cluster releases the GPU and the rollout proceeds on its next pass.
+
 ### Capacity numbers look stale
 
 They are, and deliberately: a failed read retains the previous values rather than

docs/updates.md

@@ -0,0 +1,95 @@
+# Changing the cell template
+
+Editing `spec.cell` — a new `imageRef` after a driver bump, a different
+`guestClassRef`, another interface — changes the shape of a cell. Existing cells
+are already running the old shape, and there is no in-place update of a VM's
+image: adopting a new template means replacing cells.
+
+The pool always tells you the cells have drifted. Whether it replaces them is
+yours to decide.
+
+## Seeing drift
+
+Every cell records the template it was created from, and the `Updated` condition
+compares that with the pool's current one:
+
+```bash
+kubectl get cellpool <name> -n <ns> \
+  -o jsonpath='{range .status.conditions[?(@.type=="Updated")]}{.status} {.reason} {.message}{"\n"}{end}'
+
+# which cells specifically
+kubectl get cellpool <name> -n <ns> \
+  -o jsonpath='{range .status.cells[*]}{.name} {.templateHash}{"\n"}{end}'
+```
+
+| `Updated` | Reason | Meaning |
+|---|---|---|
+| True | `AllCellsCurrent` | every cell matches the current template |
+| False | `TemplateChanged` | cells are out of date and `updatePolicy.type` is `Manual`, so nothing will happen to them |
+| False | `RollingUpdate` | a cell is being replaced right now |
+| False | `UpdateBlocked` | replacement is wanted but cannot proceed — the message says why |
+
+A cell with an empty `templateHash` is treated as current, not stale. It predates
+the hash being read back, and treating unknown as out-of-date would replace a
+whole healthy pool the first time you switched the policy on.
+
+## Manual (the default)
+
+Nothing is replaced. You do it when it suits you, one cell at a time:
+
+```bash
+# in the WORKLOAD cluster: stop new work landing, move what is there
+kubectl cordon <cell>
+kubectl drain <cell> --ignore-daemonsets --delete-emptydir-data
+
+# in the INFRASTRUCTURE cluster: remove the cell; the pool refills the index
+kubectl delete swiftguest <cell> -n <ns>        # or: kubectl delete machine <cell> -n <ns>
+```
+
+Wait for the replacement to reach `Ready` before doing the next one. The pool
+refills the freed index from the current template, so the cell comes back with
+the same name and the new shape.
+
+## RollingUpdate
+
+```yaml
+spec:
+  updatePolicy:
+    type: RollingUpdate
+```
+
+The pool replaces stale cells itself — deliberately slowly, and through the same
+gate automatic scale-down uses:
+
+- **one cell at a time**, never while another cell is being created or drained;
+- **only cells the capacity provider reports as idle.** A cell whose allocations
+  cannot be read is never replaced either — "empty" and "unknown" are different
+  answers;
+- **not while the pool is resizing**, so a rollout cannot race a scaling decision
+  for the index it is about to free;
+- **not while the workload cluster is unreachable**, because then "is this cell
+  busy?" has no answer;
+- lowest index first, so the order is predictable.
+
+The drain gate re-checks allocations again immediately before the cell is
+removed, so a workload that lands between the decision and the deletion is still
+safe.
+
+### It can stall, and that is the point
+
+On a pool whose stale cells all hold workloads, a rolling update makes no
+progress — indefinitely. `Updated` stays False with reason `UpdateBlocked` and a
+message naming the cause. It will not evict anything to make room: this operator
+waits for a GPU to be released rather than taking it away, which is why it holds
+no `pods/eviction` right in the workload cluster at all.
+
+If you need the cell back sooner, drain it yourself with `kubectl drain` and the
+rollout proceeds on its next pass.
+
+### Capacity during a rollout
+
+Replacing a cell costs its capacity for the duration of a full cell startup
+(measured 4m45s). There is no surge: a cell holds a *physical* GPU, so bringing up
+a replacement alongside the old one would need a spare device. On a single-cell
+pool a rolling update therefore means a gap in service — which is a reason to run
+`replicas: 2` if the workload cannot tolerate one.