Skip to content

fix: stop rebuilding cells against a fault that is not in the cell - #16

Merged
wrkode merged 1 commit into
mainfrom
fix/no-rebuild-against-poolwide-fault
Aug 9, 2026
Merged

fix: stop rebuilding cells against a fault that is not in the cell#16
wrkode merged 1 commit into
mainfrom
fix/no-rebuild-against-poolwide-fault

Conversation

@wrkode

@wrkode wrkode commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Found by watching the pool's actual response to a broken workload cluster, before cutting v0.1.1.

The behaviour

A cell whose Node joins and never advertises a GPU fails at spec.capacity.readyTimeout, and the pool replaced it: delete the guest, allocate a GPU, clone a 30 GiB root disk, boot, join, wait out the timeout again — five times per index, destroying the evidence each round.

Right when the cell is broken. Useless when the workload cluster is — HAMi missing, its DaemonSet not tolerating the pool's taints, a broken inner CNI. A fresh VM reaches exactly the same place.

The discriminator was already there

HAMiProvider.Health reports usable as soon as any cell node advertises a device (detected == 0 is the not-ready case). So:

Situation Before After
one cell advertises nothing, others fine replaced replaced (unchanged)
no cell node advertises anything replaced ×5 stalls, Progressing=False/FaultNotInTheCell, VM left up
cell never joined replaced replaced (unchanged — its fault may be its own)
workload cluster unreachable replaced not replaced, consistent with "doubt freezes"

No deadlock is possible: with no cells at all the provider is usable by definition, so a pool can never wedge itself out of ever creating one, and the veto is re-evaluated every pass.

Hardware validation (dev cluster, GTX 1080, inner k0s cluster where HAMi genuinely cannot register)

Failed on the pool-wide fault, then held — same guest UID, one failure, one clone job, zero rebuilds. The old code would have rebuilt at +30s, +60s, +120s, +240s:

12:00:32Z cell=Failed fails=1 guest=Running
12:01:28Z cell=Failed fails=1 sameGuest=yes clones=1
…
12:05:16Z cell=Failed fails=1 sameGuest=yes clones=1
Progressing=False (FaultNotInTheCell) not replacing cell cells-0: its Node joined and
advertises no GPU, and no cell node advertises one — the fault is in the workload
cluster, so a rebuilt VM would fail the same way. Fix the capacity provider (see
CapacityProviderReady) and the cell is replaced then

Then the recovery direction — wrote a valid registration onto the node, as fixing HAMi would:

12:06:19Z provider=True  cell=Failed
12:08:21Z provider=True  cell=AllocatingGPU fails=1   <- index refilled, failure count carried

Also

  • GPUCellPoolCellFailed's description and the runbook now explain the "Failed with no replacement attempts at all" case, which would otherwise look like the operator having stopped working.

make lint 0 issues, full suite green, helm lint clean.

🤖 Generated with Claude Code

A cell whose Node joins and never advertises a GPU fails at
spec.capacity.readyTimeout, and the pool replaced it: delete the guest, allocate a
GPU, clone a 30 GiB root disk, boot, join, wait out the timeout again — five times
per index, destroying the evidence each round.

That is right when the cell is broken and useless when the workload cluster is.
HAMi missing, its DaemonSet not tolerating the pool's taints, a broken inner CNI:
a fresh VM reaches exactly the same place. Observed on hardware — HAMi could not
register and the pool's answer was to rebuild the VM.

The pool's capacity verdict already distinguishes the two, because HAMiProvider
reports usable as soon as ANY cell node advertises a device. So a failed cell whose
Node is Ready, advertising nothing, in a pool where nothing advertises anything, is
no longer replaced: the pool stalls with Progressing=False/FaultNotInTheCell and a
message naming CapacityProviderReady as the thing to fix, and leaves the VM up to
be inspected. A single broken cell among healthy ones is still replaced, and a cell
that never joined is still replaced — its failure really may be its own.

No deadlock is possible: with no cells the provider is usable by definition, so a
pool can never wedge itself out of ever creating one, and the veto is re-evaluated
every pass. Validated on hardware in both directions — held Failed with zero
rebuilds for four minutes, then refilled the index within two minutes of the
provider becoming usable, carrying the failure count forward.

Signed-off-by: William Rizzo <william.rizzo@gmail.com>
@wrkode
wrkode merged commit 6c928d1 into main Aug 9, 2026
4 checks passed
@wrkode wrkode mentioned this pull request Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant