Spotted while fixing the same class of thrash on the capacity path (#16).
The gap
Two guards bound futile cell rebuilds today:
Neither covers a pool that was working and whose bootstrap credential then expires. EverReady is true, so the hopeless guard never fires; the Node never registers, so #16's veto (which requires NodeReady) never fires either. Every new cell fails at spec.bootstrap.readyTimeout and is rebuilt up to MaxFailuresPerIndex (5) times per index — each attempt a GPU allocation, a 30 GiB root-disk clone, a boot, and the full join timeout.
A k0s worker token defaults to a finite expiry, and a kubeadm bootstrap token to 24h, so this is the ordinary end state of a long-lived pool, not an exotic one.
Why it was not fixed alongside #16
The capacity case has an authoritative signal — HAMiProvider.Health reports whether anything advertises a device, so "the fault is not in this cell" is an observation. There is no equivalent for a join credential: the operator cannot validate a k0s or kubeadm token without trying it. Any fix here is a heuristic (N consecutive cells failing at JoinTimeout with no Node ever appearing ⇒ suspect the credential), and inventing the thresholds under release pressure is how a guard ends up firing on a slow image pull instead.
ReasonBootstrapCredentialSuspect already exists in api/v1alpha1/conditions.go and is never used — it is the placeholder for exactly this.
Suggested shape
- Count consecutive cells that reached
Joining and timed out without any Node object ever appearing, across indexes rather than per index.
- At a threshold (2 seems enough — one cell may be unlucky, two in a row is the credential or the network), stall with
Progressing=False/BootstrapCredentialSuspect naming spec.bootstrap.joinSecretRef and pointing at the runbook's join section.
- Do not gate on the credential's own expiry field: the join blob is opaque to this operator by design, and parsing distribution-specific tokens is exactly what the Opaque provider avoids.
- Clear the suspicion the moment any cell joins.
Test shape: PlanMembership with two Joining-timeout failures and no node ⇒ stalled; one such failure ⇒ still replaces; a failure with a Node present ⇒ not this guard's business (that is #16's).
Spotted while fixing the same class of thrash on the capacity path (#16).
The gap
Two guards bound futile cell rebuilds today:
HopelessPoolFailures = 3stops creation when no cell has ever been Ready — covers a pool that was misconfigured from the start.Neither covers a pool that was working and whose bootstrap credential then expires.
EverReadyis true, so the hopeless guard never fires; the Node never registers, so #16's veto (which requiresNodeReady) never fires either. Every new cell fails atspec.bootstrap.readyTimeoutand is rebuilt up toMaxFailuresPerIndex(5) times per index — each attempt a GPU allocation, a 30 GiB root-disk clone, a boot, and the full join timeout.A k0s worker token defaults to a finite expiry, and a kubeadm bootstrap token to 24h, so this is the ordinary end state of a long-lived pool, not an exotic one.
Why it was not fixed alongside #16
The capacity case has an authoritative signal —
HAMiProvider.Healthreports whether anything advertises a device, so "the fault is not in this cell" is an observation. There is no equivalent for a join credential: the operator cannot validate a k0s or kubeadm token without trying it. Any fix here is a heuristic (N consecutive cells failing atJoinTimeoutwith no Node ever appearing ⇒ suspect the credential), and inventing the thresholds under release pressure is how a guard ends up firing on a slow image pull instead.ReasonBootstrapCredentialSuspectalready exists inapi/v1alpha1/conditions.goand is never used — it is the placeholder for exactly this.Suggested shape
Joiningand timed out without any Node object ever appearing, across indexes rather than per index.Progressing=False/BootstrapCredentialSuspectnamingspec.bootstrap.joinSecretRefand pointing at the runbook's join section.Test shape:
PlanMembershipwith twoJoining-timeout failures and no node ⇒ stalled; one such failure ⇒ still replaces; a failure with a Node present ⇒ not this guard's business (that is #16's).