fix: stop reaping a Node the live replacement kubelet already adopted (#14)

[wrkode]

Closes #14.

The bug

A Node carrying a previous incarnation's cells.kubeswift.io/instance label was reaped as stale on that label alone. But a replacement's kubelet registers under the reused node name and adopts the existing Node object, keeping labels it did not set — so a live cell is indistinguishable from a leftover by label. Deleting it is unrecoverable: a kubelet whose Node is removed under it never re-registers, it logs Error updating node status, will retry forever and the cell waits in Joining. Only restarting the guest's kubelet recovered it.

The fix

Liveness is the discriminator. NodeState.KubeletLive comes from the kubelet's heartbeat Lease, falling back to the Ready condition's heartbeat, and finally to live when neither can be read — an unknown must never authorise a delete. Foreign label + live kubelet → adopt and re-label. Foreign label + nothing heartbeating → phantom, reap.

Orphan cleanup is a sweep, not a one-shot. My first attempt cleaned up when a retired cell's status row was dropped. That is the exact moment its kubelet has only just died, so the Node still looks live and is correctly left alone — and nothing remembers the cell afterwards. Measured on hardware: the Node sat NotReady for ten minutes with no further attempt, waiting for the next cell at that index to adopt it. So orphan cleanup is now keyed on the pool label and runs every reconcile, converging instead of getting one chance.

Hardware validation (dev cluster, GTX 1080 cell, real k0s workload cluster)

Adoption path — relabelled the live cell's Node to a foreign instance value:

node: cells-0 Ready   instance: uid-from-a-previous-life   ->   ae415ab9-…   (re-stamped, node survived)
cell=AwaitingGPUCapacity throughout

The real race, caught in the wild — killed the guest, let the pool build a replacement:

11:18:21 node=[NotReady/ae415ab9-…]  cell=GuestProvisioning
11:19:03 node=[NotReady/592cbf5b-…]  cell=Joining          <- adopted + re-labelled, NOT deleted
11:19:44 node=[Ready/592cbf5b-…]     cell=AwaitingGPUCapacity

The replacement's kubelet had renewed the lease while the Node was still NotReady — precisely the instant that used to be the fatal delete.

Sweep — A/B on the same abandoned Node:

one-shot build:  cells-0 NotReady 10m   (never cleaned up)
sweep build:     removed within 25s of rollout; inner cluster back to just the control plane

Also

• coordination.k8s.io/leases: [get] added to the workload-cluster observer role — absent, the operator degrades to the Ready-condition heartbeat rather than failing.
• TestReconcileLeavesForeignNodesAlone keeps the sweep from becoming a licence to delete: an unlabelled, cold Node is untouched.
• Fixed two stale doc claims found while writing this up: the runbook and API reference still said there is no rolling update (See template drift, and optionally roll cells to fix it #10 shipped it).

make lint 0 issues; full suite green including the four node-lifecycle harness tests.

🤖 Generated with Claude Code