diff --git a/README.md b/README.md index 8a54501..9cc7036 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,7 @@ privileged in the infrastructure cluster. |---|---| | infrastructure cluster | KubeSwift ≥ v0.13.4, a GPU node (`kubeswift.io/gpu-node=true`), a `DeviceClass` + `ResourceClaimTemplate` for VFIO GPUs, Multus + a NAD carrying a routable address, a `SwiftGuestClass` for the cell VM | | workload cluster | HAMi installed, reachable from the operator, and reachable **both ways** for kubelet (cells need a routable interface, not just egress) | -| cell image | a `SwiftImage` with the NVIDIA driver, containerd + CDI, and your distribution's node binaries | +| cell image | a `SwiftImage` with the NVIDIA driver, containerd + CDI, and your distribution's node binaries. A public reference image is published — see `docs/cell-image.md` | Budget **~5 minutes** for a first cell to go `Pending` → `Ready` — most of it is cloning the root disk, not booting. See `docs/quickstart.md`. diff --git a/config/samples/swiftimage-cell.yaml b/config/samples/swiftimage-cell.yaml new file mode 100644 index 0000000..bce47cf --- /dev/null +++ b/config/samples/swiftimage-cell.yaml @@ -0,0 +1,41 @@ +# A ready-made cell image, so you do not have to bake one to try this out. +# +# This is the exact image the project's own hardware validation ran on, published as a +# public OCI artifact. No pull secret is needed. Apply it, wait for Ready (it pulls +# ~11 GiB and expands to a 30 GiB raw disk, so allow a few minutes), then reference it +# from your pool's guestTemplate as `imageRef: {name: gpu-worker-noble}`. +# +# It is a REFERENCE image, not a universal one. Two things pin it, and either can make +# it the wrong choice for you: +# +# * Driver 580.173.02, the PROPRIETARY branch, chosen for a Pascal card (GTX 1080). +# The open kernel modules are Turing and newer only. On newer hardware you may +# prefer -open, which means baking your own. +# * The node binaries are k0s. If your workload cluster is kubeadm, RKE2 or k3s, this +# image has the driver and container toolkit you need but not your node binaries — +# either install them from the join cloud-init (slower first boot, no rebake) or +# bake an image for your distribution. +# +# Build your own with hack/build-cell-image.sh; docs/cell-image.md covers both paths and +# what any cell image must satisfy. +apiVersion: image.kubeswift.io/v1alpha1 +kind: SwiftImage +metadata: + name: gpu-worker-noble + namespace: gpu-cells +spec: + # The published artifact is already raw. format is the INPUT format: declaring raw for + # a qcow2 source skips the conversion and hands Cloud Hypervisor a qcow2 it reads as + # raw, which fails at boot with a refcount error rather than at import. + format: raw + osType: linux + cloneStrategy: copy + rootDisk: + # Must be at least the baked disk size. The clone is most of a cell's startup time + # (measured: ~3 minutes of 4m45s), so a smaller image is the cheapest latency win + # available if you bake your own. + size: 30Gi + source: + oci: + repository: ghcr.io/kubeswift-io/gpucellpool/images + tag: gpu-worker-noble-580-v2 diff --git a/docs/cell-image.md b/docs/cell-image.md index 01615bd..8c4e89c 100644 --- a/docs/cell-image.md +++ b/docs/cell-image.md @@ -2,8 +2,31 @@ A cell boots from a prebaked disk image — a GPU-capable Kubernetes worker with the driver, container runtime and node binaries already installed, so joining -is thin cloud-init rather than a 5-15 minute install-at-boot. This page covers -building one and what it must contain. +is thin cloud-init rather than a 5-15 minute install-at-boot. + +## Use the published reference image + +You do not have to bake one to try this out. The image the project's own +hardware validation ran on is published as a **public** OCI artifact, so this +needs no pull secret: + +```bash +kubectl apply -f config/samples/swiftimage-cell.yaml # namespace gpu-cells +kubectl wait swiftimage/gpu-worker-noble -n gpu-cells --for=jsonpath='{.status.phase}'=Ready --timeout=15m +``` + +Then reference it from your pool: `imageRef: {name: gpu-worker-noble}`. + +It is a reference image, not a universal one, and two things pin it: + +| | | +|---|---| +| driver **580.173.02**, the *proprietary* branch | chosen for a Pascal card (GTX 1080); the open kernel modules are Turing and newer only. On newer hardware you may want `-open` instead, which means baking your own | +| node binaries are **k0s** | if your workload cluster is kubeadm, RKE2 or k3s, this image still has the driver and container toolkit you need but not your node binaries. Either install them from the join cloud-init (slower first boot, no rebake — see `config/samples/cell-join-secret.yaml`) or bake for your distribution | + +It pulls ~11 GiB and expands to a 30 GiB raw disk. Cloning that disk is most of +a cell's startup time (~3 minutes of the measured 4m45s), so if you bake your +own, a smaller root disk is the cheapest latency win available. ## What the image must satisfy diff --git a/docs/limitations.md b/docs/limitations.md index d142497..a773050 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -2,10 +2,13 @@ Everything here is a real, current gap — not a hedge. Where there is a practical workaround, it is stated; where the answer is "a human has to act", -that is stated too. +that is stated too. Each gap that is tracked links its issue, so you can see +whether it is being worked on rather than guessing. ## HAMi DRA mode is unimplemented +Tracked as [#4](https://github.com/kubeswift-io/gpucellpool/issues/4). + `spec.capacity.hami.mode: DRA` is accepted by the API but `internal/capacity`'s DRA path returns `ErrUnsupported` — capacity reads, demand signal (`docs/autoscaling.md`), and readiness all fail loudly rather @@ -14,6 +17,8 @@ Use it (the default) even if your HAMi install also has DRA mode available. ## No rolling update on `guestTemplate` change +Tracked as [#2](https://github.com/kubeswift-io/gpucellpool/issues/2). + Changing `spec.cell.guestTemplate` (a new `imageRef`, a driver bump, a different `guestClassRef`) bumps the per-cell template-hash annotation but **does not roll existing cells**. `status.conditions` will not tell you a @@ -28,6 +33,8 @@ cordon and drain the cell's workload Node before deleting the cell's ## No automated outer-drain sequencing +Tracked as [#3](https://github.com/kubeswift-io/gpucellpool/issues/3). + A cell is a VFIO guest, so KubeSwift can only move it with an *offline* migration — a VM restart (`docs/concepts.md` — "Cells are cattle"). Every cell is therefore pinned with `migration.enabled: false`. If the @@ -52,6 +59,8 @@ token expires, new cells fail at `Joining` with `JoinTimeout` ## Pools of two or more cells: harness-only +Tracked as [#5](https://github.com/kubeswift-io/gpucellpool/issues/5). + The reference lab has exactly one GPU. Everything about a *single* cell — provisioning, boot, join, HAMi accounting, autoscale-to-one, teardown — is hardware-validated. Behaviour specific to **multiple concurrent cells** — diff --git a/docs/quickstart.md b/docs/quickstart.md index 1f1d3d5..09192a9 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -25,7 +25,7 @@ by GPUCellPool. | a `DeviceClass` + `ResourceClaimTemplate` for the VFIO GPU | do not re-derive these — apply KubeSwift's own samples at `config/samples/dra-gpu/` in the KubeSwift repo (`resourceclaimtemplate-single-gpu.yaml` creates the `single-vfio-gpu` template this quickstart references) | | Multus + a `NetworkAttachmentDefinition` carrying a **routable** address | not optional — see `docs/networking.md`. A minimal sample is at `config/samples/network-attachment-definition.yaml` | | a `SwiftGuestClass` | CPU/memory/disk shape for the cell VM | -| a cell `SwiftImage` | a disk image with the NVIDIA driver, containerd + CDI, and your distribution's node binaries baked in — see `docs/cell-image.md` to build one | +| a cell `SwiftImage` | **we publish one**: `kubectl apply -f config/samples/swiftimage-cell.yaml`. It is a reference image (proprietary driver 580.173.02 for Pascal, k0s node binaries) — `docs/cell-image.md` says when to bake your own instead | **In the workload cluster:**