Found while testing #117 live on an AWS Nebari cluster.
When GPU nodes carry a taint (e.g. nvidia.com/gpu) and Longhorn's components do not tolerate it, longhorn-manager is not scheduled on the GPU nodes, so they never register as Longhorn nodes and Longhorn cannot attach PVCs there:
FailedAttachVolume: ... unable to attach volume <pvc> to <gpu-node>:
node.longhorn.io "<gpu-node>" not found
The pack defaults model storage to the longhorn StorageClass (defaults.storage.storageClassName), so a served LLMModel scheduled onto a tainted GPU node gets stuck in Init with its model-storage PVC unattached, and never serves.
Workaround (per model): set spec.model.storage.storageClassName to a class that attaches on the GPU node, e.g. gp2/EBS (WaitForFirstConsumer, provisions in the node's AZ).
Suggested fix: document this for GPU node groups, and/or default served-model storage to a node-local class when the model requests a GPU. Not addressed by code yet.
Found while testing #117 live on an AWS Nebari cluster.
When GPU nodes carry a taint (e.g.
nvidia.com/gpu) and Longhorn's components do not tolerate it,longhorn-manageris not scheduled on the GPU nodes, so they never register as Longhorn nodes and Longhorn cannot attach PVCs there:The pack defaults model storage to the
longhornStorageClass (defaults.storage.storageClassName), so a servedLLMModelscheduled onto a tainted GPU node gets stuck inInitwith its model-storage PVC unattached, and never serves.Workaround (per model): set
spec.model.storage.storageClassNameto a class that attaches on the GPU node, e.g.gp2/EBS (WaitForFirstConsumer, provisions in the node's AZ).Suggested fix: document this for GPU node groups, and/or default served-model storage to a node-local class when the model requests a GPU. Not addressed by code yet.