Skip to content

Public-release readiness: five defects and a documentation set - #1

Merged
wrkode merged 2 commits into
mainfrom
fix/public-release-defects
Aug 8, 2026
Merged

Public-release readiness: five defects and a documentation set#1
wrkode merged 2 commits into
mainfrom
fix/public-release-defects

Conversation

@wrkode

@wrkode wrkode commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Everything here came out of an architecture review of the repo done specifically for going public. Most of it was not prose.

Five defects (commit 1)

A session-local path in a script we were about to publish hack/build-cell-image.sh defaulted OUT to a scratchpad directory on the authoring machine. It would not work for anyone else.
The observer RBAC we tell users to apply cannot tear a cell down nodes: delete was commented out as a "Phase 4 drain feature", but DeleteNode runs on two always-on paths — reaping a Node left by a previous cell incarnation, and removing a cell's Node. Following our documented least-privilege path meant Forbidden on every teardown, and accumulating stale Nodes are precisely what produce a phantom Ready cell or HAMi advertising a GPU that is gone. The commented-out pods/eviction block is deleted: this operator waits for a GPU to be released rather than evicting, so it never needed it.
Stale text served to users by the apiserver Doc comments in api/ compile into the CRD's OpenAPI, so kubectl explain gpucellpool.spec.autoscaling said the feature is not implemented, and the provisioner field documented the MachineDeployment design we explicitly rejected. Six such claims, corrected and regenerated.
bootstrap.provider: KubeadmToken accepted but unimplemented And it failed dirtily: with no joinSecretRef it passed admission then errored about the Opaque provider the user had not chosen; with one it silently behaved as Opaque and minted no token. Removed from the enum.
A shared DRA claim was not held to one cell V3's justification says a named claim is one claim and a VFIO device backs one VM, but only the XOR between the two references was enforced — so resourceClaimName with replicas: 2 was admitted and double-booked the device. Now rejected, including via autoscaling.maxReplicas.

Documentation (commit 2)

The problem was not missing prose: the first documentation click landed on a pre-implementation design doc headed "No hardware proof yet", and two things a user needs to reach a running cell were never shipped.

New: quickstart.md (ending on the assertion that is the product — two pods, each limited to a fraction, both naming the same GPU UUID), concepts.md, api-reference.md, autoscaling.md (the headline feature had no user-facing page), networking.md, security.md, limitations.md, cell-image.md, and a docs/README.md index separating "using this" from "why it is built this way".

Also config/samples/cell-join-secret.yaml — there was no join example anywhere, and nobody was going to reconstruct one from prose (closed substitution set, a quoted token, the node IP derived by subnet, containerd 2.x's v3 schema, a pinned resolver, ssh keys) — plus a NAD sample.

The seven design docs are kept, because the D1–D10 decision record is worth publishing, but demoted behind a banner saying they document rationale rather than behaviour, with every stale "Phase N" / "not implemented" / "to be measured" corrected. gpucellpool-poc.md becomes gpucellpool-validation-record.md: it is a results record and was being read as a plan.

Two API fixes that fell out of verifying the samples actually apply

  • spec.bootstrap is now optional. It is genuinely unused when Cluster API's own bootstrap provider supplies the join data, and requiring it forced an empty bootstrap: {} into the manifest to satisfy the schema. The ClusterAPI sample could not be applied at all before this.
  • A test runs every shipped sample through the real webhook rules. A sample the webhook rejects is worse than no sample: it is the first thing a new user applies and it fails looking like their mistake.

Verification

  • All four samples pass a server-side dry-run against the real CRD, and every GPUCellPool sample passes Validate().
  • No dead internal links; no leaked private paths or AI-tooling references.
  • Every ScalingActive reason and every autoscaling default named in the docs cross-checked against the constants and kubebuilder markers.
  • Confirmed zero stale phrases remain in the generated CRD.
  • make test, make lint, make verify pass.

🤖 Generated with Claude Code

wrkode added 2 commits August 8, 2026 16:07
An architecture review of the docs turned up five things that are code or config, not
prose. In severity order.

**A session-local path in a script we were about to publish.** hack/build-cell-image.sh
defaulted OUT to a scratchpad directory on the authoring machine — it would not work
for anyone else and should not be in a public repo. Now $PWD/build. The driver comment
also now says plainly that the metapackage tracks a BRANCH (570-server resolved to
580.173.02), so "pinned" meant less than it claimed.

**The observer RBAC we tell users to apply cannot tear a cell down.** nodes: delete was
commented out and labelled a Phase 4 drain feature, but DeleteNode is called from two
ALWAYS-ON paths: reaping a Node left by a previous incarnation of a cell, and removing a
cell's Node when the cell goes. Anyone following our documented least-privilege path got
Forbidden on every teardown, and stale Nodes then accumulate — which is exactly the
condition that produces a phantom Ready cell or HAMi advertising a GPU that is gone.
Granted, with the real reason. The commented-out pods/eviction block is deleted: this
operator waits for a GPU to be released rather than evicting, so it never needed it, and
shipping it as a suggestion was misleading.

**Stale text was being served to users by the apiserver.** Doc comments in api/ are
compiled into the CRD's OpenAPI, so `kubectl explain gpucellpool.spec.autoscaling` told
people the feature is not implemented, and the provisioner field documented the
MachineDeployment design we explicitly REJECTED. Six such claims, all now corrected and
regenerated: autoscaling "a later phase", "Scale-UP only", ScaleDownAuto "Not
implemented", Manual "the only supported value", ClusterAPI "the only mode today /
sizes a MachineDeployment", and the obsolete ~14-minute startup figure.

**bootstrap.provider: KubeadmToken was accepted and unimplemented.** The enum admitted
it; nothing implemented it; and the failure was not even clean — with no joinSecretRef it
passed admission and then errored about the Opaque provider the user had not selected,
and with one it silently behaved as Opaque and minted no token. Removed from the enum
rather than left accepted, since a value the apiserver admits and the controller ignores
is the silent failure this project refuses.

**A shared DRA claim was not held to one cell.** V3's justification says a named
ResourceClaim is one claim and a VFIO device backs one VM, so N cells double-book it —
but the rule only enforced the XOR between the two references. resourceClaimName with
replicas: 2 was admitted. Now rejected, including via autoscaling.maxReplicas, because a
pool that can grow past one cell is the same bug deferred until demand arrives.

Signed-off-by: William Rizzo <william.rizzo@gmail.com>
Written for the repo going public. The problem was not missing prose — it was that the
first documentation click landed on a pre-implementation design doc whose header said
"No hardware proof yet", and that two things a user needs to reach a running cell were
never shipped at all.

New, user-facing:
- quickstart.md — the whole journey, ending on the assertion that is the product: two
  pods, each limited to a fraction, both naming the SAME GPU UUID.
- concepts.md, api-reference.md (written from the types and the validator, not from the
  design doc), autoscaling.md (the headline feature had no user-facing page at all),
  networking.md, security.md, limitations.md, cell-image.md, and docs/README.md as an
  index that separates "using this" from "why it is built this way".
- config/samples/cell-join-secret.yaml — k0s and kubeadm cloud-init. There was no join
  example anywhere, and nobody was going to reconstruct one from prose: it needs the
  closed substitution set, a quoted token (a bare {{ token }} in a value position is not
  valid YAML), the node IP derived by subnet, containerd 2.x's v3 config schema, a pinned
  resolver, and ssh keys or a broken cell is undiagnosable. Plus a NAD sample.

Two API fixes fell out of verifying the samples actually apply:
- spec.bootstrap is now optional. It is genuinely unused when Cluster API's own bootstrap
  provider supplies the join data, and requiring it forced an empty `bootstrap: {}` into
  the manifest to satisfy the schema — a field you must write and nothing reads. The
  ClusterAPI sample could not be applied at all before this.
- A test now runs every shipped GPUCellPool sample through the real webhook rules. A
  sample the webhook rejects is worse than no sample: it is the first thing a new user
  applies and it fails looking like their mistake. The CRD schema half is covered by a
  server-side dry-run.

The seven design docs are kept — the D1-D10 decision record is worth publishing — but
demoted behind a banner saying they document rationale, not behaviour, and every "Phase
N", "later", "not implemented" and "to be measured" that had become false is corrected.
gpucellpool-poc.md is renamed to gpucellpool-validation-record.md, because it is a
results record and had been read as a plan.

Signed-off-by: William Rizzo <william.rizzo@gmail.com>
@wrkode
wrkode merged commit 03c66b0 into main Aug 8, 2026
4 checks passed
@wrkode
wrkode deleted the fix/public-release-defects branch August 8, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant