See template drift, and optionally roll cells to fix it

[wrkode]

Closes #2.

Two layers, because the first turned out to be missing entirely.

Drift was invisible

Each cell was stamped with the hash of the template it was created from — and nothing ever read it back. Changing spec.cell left the pool silently unaware; the design doc's claim that a template change is "observable (Updated=False, TemplateChanged)" was not true, because no such condition was ever computed.

The hash is now observed, surfaced per cell as status.cells[].templateHash, and compared in a new Updated condition. That is worth having on its own: knowing you have drifted is useful even if you replace cells by hand.

spec.updatePolicy.type: RollingUpdate

Opt-in, because a template edit is not consent to destroy running work. Replacement is deletion plus recreation — there is no in-place update of a VM's image — so every gate is protecting something:

• one cell at a time, never while another is coming or going
• only cells the capacity provider reports idle; a cell whose allocations cannot be read is never replaced either
• not while the pool is resizing, so a rollout cannot race a scaling decision for the index it is about to free
• not while the workload cluster is unreachable, because then "is this cell busy?" has no answer
• lowest index first, so the order is predictable

The existing drain gate re-checks allocations again immediately before the object goes.

It can stall indefinitely, and that is the point. On a pool whose stale cells are all busy, Updated stays False with UpdateBlocked and a message naming the cause. It will not evict anything to make room — this operator waits for a GPU to be released rather than taking it away, which is why it holds no pods/eviction right at all.

A cell with no recorded hash counts as current, not stale: it predates the field, and treating unknown as out-of-date would replace an entire healthy pool the first time someone enabled the policy.

A latent bug this exposed, which was never about rolling updates

Cell names are reused — index 0 is always <pool>-0 — and status rows are keyed by name. So a replacement inherited its predecessor's Draining phase and was deleted on the pass that created it: create, destroy, create, destroy, with no timeout that could ever break the loop.

The harness caught it as "the stale cell was never replaced"; tracing showed a new guest UID every single pass. Rows describing a different guest UID no longer lend their phase. The failure counter still carries, because the replacement backoff is counted per index rather than per incarnation.

Any replacement path could hit this — a failed cell, or a scale-down followed by a scale-up on the same index — and it was masked only because nothing had previously refilled an index whose row still said Draining. It has its own regression test that does not involve templates at all.

Also fixed: cell idleness was read from the capacity provider only when automatic scale-down was enabled, so a rolling update always saw zero idle cells and silently never acted.

Docs

New docs/updates.md covers seeing drift, the manual cordon/drain/delete sequence, and the rolling policy including its stall behaviour and the capacity gap during a replacement (no surge: a cell holds a physical GPU). docs/limitations.md loses the entry, the runbook gains the new reasons, and the index links it.

make test, make lint and make verify pass.

🤖 Generated with Claude Code