Public-release readiness: five defects and a documentation set

CHANGELOG.md

@@ -5,128 +5,136 @@ All notable changes to this project are documented here. The format follows
 
 ## [v0.1.0] — 2026-08-08
 
-### Added
+First release. Every capability below has been run on real hardware (one
+physical GPU, one KubeSwift VM, two independent workloads sharing it through
+HAMi) except where noted in **Known gaps**. See `docs/quickstart.md` to try it
+and `docs/design/gpucellpool-validation-record.md` /
+`docs/design/clusterapi-cells-validation.md` for the full hardware record,
+including the bugs a live cluster found that the test harness could not.
+
+### Added — core API and controller
 
 - `GPUCellPool` v1alpha1 API (`cells.kubeswift.io`): one CRD, namespaced, with a
   scale subresource on `spec.replicas`. A cell is an owned `SwiftGuest` named
   `<pool>-<index>`, not a second kind. `spec.cell.guestTemplate` is an opaque
-  `SwiftGuestSpec` passthrough so this API never mirrors KubeSwift's.
-- Controller: cell state machine, membership planning with churn control,
-  physical-inventory pre-flight, per-cell bootstrap rendering, drain-gated
-  teardown, and status that separates the two layers (`physicalCapacity` =
-  whole devices, `workloadCapacity` = HAMi fractions).
-- HAMi capacity provider (DevicePlugin mode), written against annotations
-  captured from real hardware rather than upstream documentation.
-- Validating webhook enforcing the API's rules, fail-closed.
-- Helm chart, container image (distroless, non-root, read-only rootfs), CI.
-- Design set under `docs/design/`, including the Phase-1 hardware proof: one
-  physical GPU, one KubeSwift VM, two workloads sharing it through HAMi, with
-  neither upstream project modified.
-
-### Added — demand-driven scale-up (Phase 3)
-
-- `spec.autoscaling`: `enabled`, `minReplicas`, `maxReplicas`, `stabilizationWindow`,
-  `scaleDown` (`Manual` only; `Auto` is rejected). `maxReplicas` is required when
-  enabled, because an unbounded pool that misreads demand can consume every GPU in
-  the cluster.
-- `PendingDemand` for HAMi DevicePlugin mode. Two filters are the whole safety of
-  the feature: only pods the scheduler could not place count (`PodScheduled=False`
-  / `Unschedulable`, which naturally excludes a pod stuck on a missing ConfigMap —
-  that pod was scheduled), and only requests one fresh cell could actually satisfy
-  count toward a decision.
-- Scale-up is one cell at a time, behind a stabilization window, and never on
-  unread demand, an unknown cell shape, a saturated cluster, or a ceiling already
-  reached. `status.demand`, `status.desiredReplicas`, `status.lastScaleUpTime` and
-  the `ScalingActive` condition report every decision and its reason.
-
-### Added — automatic scale-down (Phase 4)
-
-- `spec.autoscaling.scaleDown: Auto` now works, gated on `minReplicas` being set:
-  without a floor the pool could shrink to zero and every later request would pay a
-  full cell boot.
-- Only cells the capacity provider reports as **idle** are removable, and a cell
-  whose allocations cannot be read is never treated as idle — "empty" and "unknown"
-  are different answers, and only the first may lead to a deletion.
-- Demand must have been absent for the whole `scaleDownStabilizationWindow`
-  (default 30m, deliberately longer than scale-up), tracked by
-  `status.demandFreeSince`. Absent *right now* is not the same thing: a pool that
-  shrinks between two bursts is worse than one that waits.
-- Membership removes the autoscaler's named idle cells rather than the highest
-  index; highest-index-first remains the rule for an operator-driven shrink, where
-  the intent is "make it smaller" rather than "remove that one".
-
-### Added — metrics
-
-- `gpucell_*` Prometheus metrics, with the two layers deliberately kept apart:
-  `physical_gpus{held,free}` counts whole devices from the infrastructure cluster,
-  `capacity_gpu_*` reports fractional capacity from the workload cluster. An alert
-  can therefore tell "no GPU left in the cluster" from "the shared GPU is full".
-- `cell_startup_seconds` measures creation to first Ready — the number that decides
-  whether autoscaling can be reactive at all.
-- `cell_transitions_total` makes an oscillating cell visible even though its
-  instantaneous phase looks healthy; `capacity_scrape_errors_total` says the gauges
-  went stale, since a failed read retains the previous values rather than zeroing;
-  `scale_decisions_total` records refusals as well as scale-ups.
-
-### Fixed (from the first live run)
+  `SwiftGuestSpec` passthrough so this API never mirrors KubeSwift's — see
+  `docs/api-reference.md`.
+- Cell state machine (`Pending` → `AllocatingGPU` → `GuestProvisioning` →
+  `Booting` → `Joining` → `AwaitingGPUCapacity` → `Ready`), membership planning
+  with churn control, physical-inventory pre-flight, per-cell bootstrap
+  rendering, drain-gated teardown, and status that separates the two layers
+  (`physicalCapacity` = whole devices, `workloadCapacity` = HAMi fractions).
+- Validating webhook enforcing the API's rules, fail-closed — see
+  `docs/security.md`.
+
+### Added — GPU allocation
+
+- Two backends behind `spec.cell.gpu.backend`: `DRA`
+  (`gpuResourceClaim`/`ResourceClaimTemplate`, scheduler-time) and `Native`
+  (`gpuProfileRef`, controller-time). One whole `pcie`-tier GPU per cell.
+
+### Added — bootstrap
+
+- `spec.bootstrap.provider: Opaque` — user-supplied cloud-init with a closed
+  substitution set (`{{ cellName }}`, `{{ poolName }}`, `{{ nodeLabels }}`,
+  `{{ nodeIPInterface }}`, `{{ expectedGPUs }}`), rendered per cell into an
+  operator-owned Secret. The join credential is only ever referenced, never
+  written into a CR.
+- `hack/build-cell-image.sh`: a reference cell image build (Ubuntu Noble +
+  NVIDIA driver + `nvidia-container-toolkit` + k0s worker), documented in
+  `docs/cell-image.md`.
+
+### Added — capacity and autoscaling
+
+- HAMi capacity provider (`DevicePlugin` mode), written against annotations
+  captured from real hardware rather than upstream documentation — see
+  `docs/design/gpucellpool-capacity.md`.
+- `spec.autoscaling`, both directions — see `docs/autoscaling.md`:
+  - **Scale-up**: `enabled`, `minReplicas`, `maxReplicas` (required when
+    enabled), `stabilizationWindow` (default 10m). Demand is read from pods
+    the scheduler could not place (`PodScheduled=False`/`Unschedulable`) and
+    filtered against whether a fresh cell of this pool's shape would actually
+    satisfy it — the only two gates that make scale-up safe.
+  - **Scale-down**: `scaleDown: Auto` (requires `minReplicas`), removing only
+    cells the capacity provider reports **idle**, after demand has been
+    absent for the whole `scaleDownStabilizationWindow` (default 30m, longer
+    than scale-up on purpose).
+  - `status.cellDeviceShape`: the remembered GPU shape, kept after the last
+    cell is removed, so `minReplicas: 0` is recoverable instead of a one-way
+    door.
+
+### Added — Cluster API cells
+
+- `spec.cell.provisioner: ClusterAPI` + `spec.cell.clusterAPI`
+  (`clusterName`, `version`, `bootstrapConfigTemplateRef`): each cell becomes
+  a `Machine` + `KubeSwiftMachine`, one per cell (never a `MachineDeployment`
+  — see `docs/clusterapi-cells.md` for why that would have broken cell
+  identity). When `bootstrapConfigTemplateRef` is set, the workload cluster's
+  own bootstrap provider supplies join data instead of `spec.bootstrap`.
+- `KubeSwiftMachine` can express only `imageRef`, `guestClassRef`,
+  `interfaces` — any other `guestTemplate` field is rejected at admission
+  under this provisioner rather than silently dropped.
+
+### Added — observability
+
+- `gpucell_*` Prometheus metrics (twelve series, `{pool, namespace}` labels
+  throughout): cell counts by phase, `cell_startup_seconds`,
+  `cell_transitions_total` (catches an oscillating cell even when its current
+  phase looks healthy), physical vs. workload capacity kept as separate
+  metric families, `capacity_scrape_errors_total` (a failed read retains the
+  previous value rather than reporting zero — the errors counter is how you'd
+  know), `scale_decisions_total` (including refusals), `reconcile_errors_total`.
+
+### Added — packaging and security
+
+- Helm chart, distroless/non-root/read-only-rootfs container image, CI.
+- `config/rbac/workload-cluster-observer.yaml`: the minimal
+  `ServiceAccount`+`ClusterRole` a workload-cluster credential needs — no
+  `cluster-admin`, no `pods/eviction`, `nodes: delete` granted as an
+  always-on right (stale-Node reaping and cell teardown both need it
+  unconditionally, not only under automatic scale-down).
+
+### Fixed (found during hardware validation)
 
 - `cell.nodeIPFrom` is an observation field, not a readiness gate. KubeSwift
-  v0.13.4 reports a secondary NAD interface's MAC but not its IP, and the operator
-  does not control what the kubelet registers anyway (cloud-init derives the node
-  IP in-guest), so gating on it parked every bridge-NAD cell in `Booting`.
-- The sample and design doc showed a `networkRef.kind` field that does not exist;
-  KubeSwift's strict decoding rejects it, so the sample could not have been applied.
-
-### Added — Cluster API cells (Phase 5)
-
-- `cell.provisioner: ClusterAPI` plus `cell.clusterAPI` (`clusterName`, `version`,
-  `bootstrapConfigTemplateRef`). Each cell becomes a `Machine` and a
-  `KubeSwiftMachine`, so a GPU cell added to a CAPI-managed cluster is a member of
-  it — with a `providerID`, visible to the cluster's own controllers — rather than a
-  node attached out of band.
-- One Machine per cell, named after the cell, not a MachineDeployment sized to the
-  replica count. A MachineDeployment generates Machine names, and capi-kubeswift
-  derives the guest hostname (and so the Node name) from the Machine name, which
-  would break the cell-name==Node-name identity and leave no way to drain one cell.
-- Bootstrap comes from the workload cluster's own provider when
-  `bootstrapConfigTemplateRef` is set: the operator instantiates the template once
-  per cell, as a MachineSet does, so tokens and CA hashes are the cluster's rather
-  than a secret somebody maintains. `spec.bootstrap.joinSecretRef` is then rejected
-  instead of ignored. Without a template ref, the pool's rendered Secret is handed
-  over as `dataSecretName`.
-- A `KubeSwiftMachine` can express only image, class, two networks and GPU, so any
-  other `guestTemplate` field is rejected at admission rather than silently dropped.
-- Validated end to end on hardware: pool to Ready in 6m29s, two workloads sharing the
-  cell's GTX 1080, teardown returning the GPU claim in under a minute.
-  See `docs/clusterapi-cells.md`, which also records what the workload cluster needs.
-
-### Fixed (from the Cluster API validation)
-
-- The pool claimed the **controller** owner reference on its Machines. Kubernetes
-  allows one per object and Cluster API needs it, so CAPI failed every reconcile with
-  "already owned by another GPUCellPool controller" — no bootstrap data, no VM, ever.
-  Cells are co-owned now, which is all garbage collection requires.
-- Pool teardown listed SwiftGuests, so a ClusterAPI pool saw no cells, drained
-  nothing and dropped its own finalizer — orphaning each Machine with the drain
-  finalizer still on it, unclearable, GPU claim leaked. Teardown goes through the
-  provisioner now.
-- Scale-to-zero was a one-way door. The satisfiability reference device came from
-  live capacity only, so an emptied pool had nothing to judge a request against,
-  counted nothing satisfiable, and never grew back — while blaming the request.
-  `status.cellDeviceShape` outlives the cells; a pool that never advertised a device
-  reports `CellShapeUnknown` and says what to do about it.
-- `Ready` was measured against `spec.replicas`, which stops being the target once
-  autoscaling is on: a healthy pool holding two autoscaled cells reported "2 of 1
-  cells are Ready" and False. A pool that deliberately holds none also reported
-  itself broken; both now say `ScaledToZero`.
+  v0.13.4 reports a secondary NAD interface's MAC but not its IP, and the
+  operator does not control what the kubelet registers anyway (cloud-init
+  derives the node IP in-guest), so gating readiness on it parked every
+  bridge-NAD cell in `Booting` forever.
+- `networkRef` takes `{name, namespace}` only; an earlier sample and design
+  doc showed a `kind` field that does not exist and that KubeSwift's strict
+  decoding rejects — the sample as first written could not have been applied.
+- Scale-to-zero was a one-way door: with the satisfiability reference device
+  coming from live capacity only, an emptied pool had nothing to judge new
+  demand against, counted nothing satisfiable, and never grew back — while
+  blaming the request. Fixed by `status.cellDeviceShape` (above).
+- `Ready` was measured against `spec.replicas`, which stops being the target
+  once autoscaling is on: a healthy pool holding two autoscaled cells reported
+  "2 of 1 cells are Ready" and `False`. A pool deliberately holding zero cells
+  also reported itself broken. Both now report `ScaledToZero` correctly
+  against `status.desiredReplicas`.
+- The pool claimed the **controller** owner reference on ClusterAPI Machines.
+  Kubernetes allows one per object and Cluster API needs it, so every
+  reconcile failed with "already owned by another GPUCellPool controller" —
+  no bootstrap data, no VM, ever. Cells are co-owned now.
+- ClusterAPI pool teardown listed SwiftGuests, so it saw no cells, drained
+  nothing, and dropped its own finalizer — orphaning each Machine with the
+  drain finalizer still on it, GPU claim leaked. Teardown goes through the
+  provisioner's own lister now.
 
 ### Known gaps
 
-- `hami.mode: DRA` reports `ErrUnsupported`; only DevicePlugin mode is implemented.
-- Pools of two or more cells are covered by the two-apiserver harness, not by
-  hardware — the lab has one GPU. So are `deletion.policy: Force` and cell
-  replacement backoff.
-- Cell startup measures 4m45s, about three minutes of which is cloning a 30 GiB root
-  disk. A smaller disk or a copy-on-write clone is where the next minute is.
-- A join template must derive the cell's routable address by subnet: `nodeIPFrom`
-  names a KubeSwift interface, and cloud-init cannot map that to a guest device.
+- `hami.mode: DRA` reports `ErrUnsupported`; only `DevicePlugin` mode is
+  implemented.
+- Pools of two or more cells are covered by the two-apiserver test harness,
+  not by hardware — the reference lab has one GPU. So are
+  `deletion.policy: Force` and cell replacement backoff.
+- Cell startup measures 4m45s to Ready, about three minutes of which is
+  cloning a 30 GiB root disk. A smaller disk or a copy-on-write clone
+  strategy is where the next minute would come from.
+- A join template must derive the cell's routable address by subnet:
+  `nodeIPFrom` names a KubeSwift interface, and cloud-init cannot map that to
+  a guest device.
+- No rolling update on `guestTemplate` change, no automated outer-drain
+  sequencing, no bootstrap token minting. See `docs/limitations.md` for the
+  full list and the operational workarounds.

README.md

@@ -24,6 +24,10 @@ boundary; HAMi owns GPU → workload allocation; this operator owns the lifecycl
 between them. A HAMi fraction is never handed to VFIO — the two layers nest, they
 do not translate.
 
+This is **layered isolation**, not tenant isolation: a cell is a VM boundary
+around the GPU, not a hard security perimeter, and every KubeSwift launcher pod
+is privileged in the infrastructure cluster — see `docs/security.md`.
+
 ## Status
 
 **v0.1.0 — alpha.** Every capability has been run on real hardware: one physical GPU
@@ -65,6 +69,7 @@ spec:
 
 ```bash
 helm install gpucellpool oci://ghcr.io/kubeswift-io/charts/gpucellpool \
+  --version 0.1.0 \
   --namespace gpucellpool-system --create-namespace
 ```
 
@@ -85,22 +90,19 @@ privileged in the infrastructure cluster.
 
 | | |
 |---|---|
-| infrastructure cluster | KubeSwift ≥ v0.13.4, a GPU node (`kubeswift.io/gpu-node=true`), a `DeviceClass` for VFIO GPUs |
+| infrastructure cluster | KubeSwift ≥ v0.13.4, a GPU node (`kubeswift.io/gpu-node=true`), a `DeviceClass` + `ResourceClaimTemplate` for VFIO GPUs, Multus + a NAD carrying a routable address, a `SwiftGuestClass` for the cell VM |
 | workload cluster | HAMi installed, reachable from the operator, and reachable **both ways** for kubelet (cells need a routable interface, not just egress) |
 | cell image | a `SwiftImage` with the NVIDIA driver, containerd + CDI, and your distribution's node binaries |
 
+Budget **~5 minutes** for a first cell to go `Pending` → `Ready` — most of it is
+cloning the root disk, not booting. See `docs/quickstart.md`.
+
 ## Documentation
 
-| Doc | Contents |
-|---|---|
-| [overview](docs/design/gpucellpool-overview.md) | architecture, decisions, scope, phases |
-| [api](docs/design/gpucellpool-api.md) | the v1alpha1 CRD and its validation rules |
-| [reconciliation](docs/design/gpucellpool-reconciliation.md) | cell state machine, identity, RBAC, deletion |
-| [bootstrap](docs/design/gpucellpool-bootstrap.md) | cell image strategy, join credentials |
-| [capacity](docs/design/gpucellpool-capacity.md) | how HAMi capacity is read |
-| [failure-model](docs/design/gpucellpool-failure-model.md) | what breaks and what the operator does about it |
-| [poc](docs/design/gpucellpool-poc.md) | hardware proof and test strategy |
-| [runbook](docs/runbook.md) | what to check when a pool misbehaves |
+Start at [`docs/README.md`](docs/README.md) — it separates operator-facing
+docs (quickstart, concepts, API reference, networking, security, autoscaling,
+runbook) from the design record (`docs/design/`, decisions and rationale, kept
+for history rather than as the current spec).
 
 ## Licence