fix: stop rebuilding cells against a fault that is not in the cell - #16
Merged
Conversation
A cell whose Node joins and never advertises a GPU fails at spec.capacity.readyTimeout, and the pool replaced it: delete the guest, allocate a GPU, clone a 30 GiB root disk, boot, join, wait out the timeout again — five times per index, destroying the evidence each round. That is right when the cell is broken and useless when the workload cluster is. HAMi missing, its DaemonSet not tolerating the pool's taints, a broken inner CNI: a fresh VM reaches exactly the same place. Observed on hardware — HAMi could not register and the pool's answer was to rebuild the VM. The pool's capacity verdict already distinguishes the two, because HAMiProvider reports usable as soon as ANY cell node advertises a device. So a failed cell whose Node is Ready, advertising nothing, in a pool where nothing advertises anything, is no longer replaced: the pool stalls with Progressing=False/FaultNotInTheCell and a message naming CapacityProviderReady as the thing to fix, and leaves the VM up to be inspected. A single broken cell among healthy ones is still replaced, and a cell that never joined is still replaced — its failure really may be its own. No deadlock is possible: with no cells the provider is usable by definition, so a pool can never wedge itself out of ever creating one, and the veto is re-evaluated every pass. Validated on hardware in both directions — held Failed with zero rebuilds for four minutes, then refilled the index within two minutes of the provider becoming usable, carrying the failure count forward. Signed-off-by: William Rizzo <william.rizzo@gmail.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by watching the pool's actual response to a broken workload cluster, before cutting v0.1.1.
The behaviour
A cell whose Node joins and never advertises a GPU fails at
spec.capacity.readyTimeout, and the pool replaced it: delete the guest, allocate a GPU, clone a 30 GiB root disk, boot, join, wait out the timeout again — five times per index, destroying the evidence each round.Right when the cell is broken. Useless when the workload cluster is — HAMi missing, its DaemonSet not tolerating the pool's taints, a broken inner CNI. A fresh VM reaches exactly the same place.
The discriminator was already there
HAMiProvider.Healthreports usable as soon as any cell node advertises a device (detected == 0is the not-ready case). So:Progressing=False/FaultNotInTheCell, VM left upNo deadlock is possible: with no cells at all the provider is usable by definition, so a pool can never wedge itself out of ever creating one, and the veto is re-evaluated every pass.
Hardware validation (dev cluster, GTX 1080, inner k0s cluster where HAMi genuinely cannot register)
Failed on the pool-wide fault, then held — same guest UID, one failure, one clone job, zero rebuilds. The old code would have rebuilt at +30s, +60s, +120s, +240s:
Then the recovery direction — wrote a valid registration onto the node, as fixing HAMi would:
Also
GPUCellPoolCellFailed's description and the runbook now explain the "Failed with no replacement attempts at all" case, which would otherwise look like the operator having stopped working.make lint0 issues, full suite green,helm lintclean.🤖 Generated with Claude Code