Public-release readiness: five defects and a documentation set - #1
Merged
Conversation
An architecture review of the docs turned up five things that are code or config, not prose. In severity order. **A session-local path in a script we were about to publish.** hack/build-cell-image.sh defaulted OUT to a scratchpad directory on the authoring machine — it would not work for anyone else and should not be in a public repo. Now $PWD/build. The driver comment also now says plainly that the metapackage tracks a BRANCH (570-server resolved to 580.173.02), so "pinned" meant less than it claimed. **The observer RBAC we tell users to apply cannot tear a cell down.** nodes: delete was commented out and labelled a Phase 4 drain feature, but DeleteNode is called from two ALWAYS-ON paths: reaping a Node left by a previous incarnation of a cell, and removing a cell's Node when the cell goes. Anyone following our documented least-privilege path got Forbidden on every teardown, and stale Nodes then accumulate — which is exactly the condition that produces a phantom Ready cell or HAMi advertising a GPU that is gone. Granted, with the real reason. The commented-out pods/eviction block is deleted: this operator waits for a GPU to be released rather than evicting, so it never needed it, and shipping it as a suggestion was misleading. **Stale text was being served to users by the apiserver.** Doc comments in api/ are compiled into the CRD's OpenAPI, so `kubectl explain gpucellpool.spec.autoscaling` told people the feature is not implemented, and the provisioner field documented the MachineDeployment design we explicitly REJECTED. Six such claims, all now corrected and regenerated: autoscaling "a later phase", "Scale-UP only", ScaleDownAuto "Not implemented", Manual "the only supported value", ClusterAPI "the only mode today / sizes a MachineDeployment", and the obsolete ~14-minute startup figure. **bootstrap.provider: KubeadmToken was accepted and unimplemented.** The enum admitted it; nothing implemented it; and the failure was not even clean — with no joinSecretRef it passed admission and then errored about the Opaque provider the user had not selected, and with one it silently behaved as Opaque and minted no token. Removed from the enum rather than left accepted, since a value the apiserver admits and the controller ignores is the silent failure this project refuses. **A shared DRA claim was not held to one cell.** V3's justification says a named ResourceClaim is one claim and a VFIO device backs one VM, so N cells double-book it — but the rule only enforced the XOR between the two references. resourceClaimName with replicas: 2 was admitted. Now rejected, including via autoscaling.maxReplicas, because a pool that can grow past one cell is the same bug deferred until demand arrives. Signed-off-by: William Rizzo <william.rizzo@gmail.com>
Written for the repo going public. The problem was not missing prose — it was that the
first documentation click landed on a pre-implementation design doc whose header said
"No hardware proof yet", and that two things a user needs to reach a running cell were
never shipped at all.
New, user-facing:
- quickstart.md — the whole journey, ending on the assertion that is the product: two
pods, each limited to a fraction, both naming the SAME GPU UUID.
- concepts.md, api-reference.md (written from the types and the validator, not from the
design doc), autoscaling.md (the headline feature had no user-facing page at all),
networking.md, security.md, limitations.md, cell-image.md, and docs/README.md as an
index that separates "using this" from "why it is built this way".
- config/samples/cell-join-secret.yaml — k0s and kubeadm cloud-init. There was no join
example anywhere, and nobody was going to reconstruct one from prose: it needs the
closed substitution set, a quoted token (a bare {{ token }} in a value position is not
valid YAML), the node IP derived by subnet, containerd 2.x's v3 config schema, a pinned
resolver, and ssh keys or a broken cell is undiagnosable. Plus a NAD sample.
Two API fixes fell out of verifying the samples actually apply:
- spec.bootstrap is now optional. It is genuinely unused when Cluster API's own bootstrap
provider supplies the join data, and requiring it forced an empty `bootstrap: {}` into
the manifest to satisfy the schema — a field you must write and nothing reads. The
ClusterAPI sample could not be applied at all before this.
- A test now runs every shipped GPUCellPool sample through the real webhook rules. A
sample the webhook rejects is worse than no sample: it is the first thing a new user
applies and it fails looking like their mistake. The CRD schema half is covered by a
server-side dry-run.
The seven design docs are kept — the D1-D10 decision record is worth publishing — but
demoted behind a banner saying they document rationale, not behaviour, and every "Phase
N", "later", "not implemented" and "to be measured" that had become false is corrected.
gpucellpool-poc.md is renamed to gpucellpool-validation-record.md, because it is a
results record and had been read as a plan.
Signed-off-by: William Rizzo <william.rizzo@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything here came out of an architecture review of the repo done specifically for going public. Most of it was not prose.
Five defects (commit 1)
hack/build-cell-image.shdefaultedOUTto a scratchpad directory on the authoring machine. It would not work for anyone else.nodes: deletewas commented out as a "Phase 4 drain feature", butDeleteNoderuns on two always-on paths — reaping a Node left by a previous cell incarnation, and removing a cell's Node. Following our documented least-privilege path meantForbiddenon every teardown, and accumulating stale Nodes are precisely what produce a phantom Ready cell or HAMi advertising a GPU that is gone. The commented-outpods/evictionblock is deleted: this operator waits for a GPU to be released rather than evicting, so it never needed it.api/compile into the CRD's OpenAPI, sokubectl explain gpucellpool.spec.autoscalingsaid the feature is not implemented, and the provisioner field documented the MachineDeployment design we explicitly rejected. Six such claims, corrected and regenerated.bootstrap.provider: KubeadmTokenaccepted but unimplementedjoinSecretRefit passed admission then errored about the Opaque provider the user had not chosen; with one it silently behaved as Opaque and minted no token. Removed from the enum.resourceClaimNamewithreplicas: 2was admitted and double-booked the device. Now rejected, including viaautoscaling.maxReplicas.Documentation (commit 2)
The problem was not missing prose: the first documentation click landed on a pre-implementation design doc headed "No hardware proof yet", and two things a user needs to reach a running cell were never shipped.
New:
quickstart.md(ending on the assertion that is the product — two pods, each limited to a fraction, both naming the same GPU UUID),concepts.md,api-reference.md,autoscaling.md(the headline feature had no user-facing page),networking.md,security.md,limitations.md,cell-image.md, and adocs/README.mdindex separating "using this" from "why it is built this way".Also
config/samples/cell-join-secret.yaml— there was no join example anywhere, and nobody was going to reconstruct one from prose (closed substitution set, a quoted token, the node IP derived by subnet, containerd 2.x's v3 schema, a pinned resolver, ssh keys) — plus a NAD sample.The seven design docs are kept, because the D1–D10 decision record is worth publishing, but demoted behind a banner saying they document rationale rather than behaviour, with every stale "Phase N" / "not implemented" / "to be measured" corrected.
gpucellpool-poc.mdbecomesgpucellpool-validation-record.md: it is a results record and was being read as a plan.Two API fixes that fell out of verifying the samples actually apply
spec.bootstrapis now optional. It is genuinely unused when Cluster API's own bootstrap provider supplies the join data, and requiring it forced an emptybootstrap: {}into the manifest to satisfy the schema. The ClusterAPI sample could not be applied at all before this.Verification
GPUCellPoolsample passesValidate().ScalingActivereason and every autoscaling default named in the docs cross-checked against the constants and kubebuilder markers.make test,make lint,make verifypass.🤖 Generated with Claude Code