Auto-add nvidia.com/gpu toleration to GPU JupyterHub profiles

[dcmcand]

Background

nebari-dev/nebari-infrastructure-core#370 makes NIC automatically taint AWS GPU node groups with nvidia.com/gpu=true:NoSchedule, so ordinary pods stay off GPU nodes.

Nothing injects the matching toleration for us on EKS:

• The NVIDIA GPU Operator only tolerates this taint on its own operands (driver, device plugin), not on user pods.
• The Kubernetes ExtendedResourceToleration admission controller would auto-tolerate pods that request nvidia.com/gpu, but it is not enabled on EKS and cannot be enabled on the managed control plane. AWS's own GPU guide hand-writes the toleration on every example pod for this reason.

So once #370 lands, a JupyterHub GPU profile will stop scheduling new servers onto GPU nodes unless the profile carries the toleration.

Request

Auto-add a toleration for nvidia.com/gpu to any JupyterHub/KubeSpawner profile that targets GPUs, so users do not have to hand-write it. A GPU profile is one that requests nvidia.com/gpu in its resources, or is otherwise marked as a GPU profile (for example a GPU node selector). The injected toleration should be:

tolerations:
- key: "nvidia.com/gpu"
  operator: "Exists"
  effect: "NoSchedule"

operator: Exists keeps it working regardless of the taint value, so it matches both NIC's value: "true" and other values.

Acceptance criteria

• A GPU profile launched against a cluster whose GPU nodes carry nvidia.com/gpu=true:NoSchedule schedules onto the GPU node and starts.
• Non-GPU profiles are unchanged and gain no toleration.
• Inside a running GPU server, the GPU is visible (nvidia-smi, CUDA available to the framework).

Blocks

This blocks nebari-dev/nebari-infrastructure-core#370. The taint change should not roll out until GPU profiles tolerate the taint, otherwise GPU notebooks break.

[kenafoster]

May already be supported - need documented workaround

[kcpevey]

Documentation of workaround is in https://github.com/nebari-dev/data-science-pack/pull/139/changes

[oren-openteams]

Validated #370 + a candidate fix for this on an EKS 1.34 cluster — surfacing two findings that change the criticality calculus here.

ExtendedResourceToleration IS enabled on EKS

The issue body's load-bearing claim — "the ExtendedResourceToleration admission controller … is not enabled on EKS and cannot be enabled on the managed control plane" — doesn't match what I observed. Reproduction on stock EKS 1.34:

# Submitted with kubectl apply -- no kubespawner, no manual toleration
apiVersion: v1
kind: Pod
metadata: {name: gpu-no-toleration, namespace: default}
spec:
  containers:
  - name: c
    image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources: {limits: {nvidia.com/gpu: 1}}

Read back from the apiserver, the persisted pod has this toleration auto-injected:

{"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}

Ruled out all mutating webhooks on the cluster (aws-load-balancer-webhook, pod-identity-webhook, vpc-resource-mutating-webhook, envoy-gateway-topology-injector, cert-manager-webhook, longhorn-webhook-mutator — none would touch GPU tolerations). By elimination the injection comes from the apiserver's built-in admission chain, and the behavior matches ERT's documented semantics exactly.

AWS appears to enable ERT silently — it isn't in their published admission-plugin list, which is probably what led to the original assessment. AWS's GPU example docs hand-write the toleration defensively even though it's not strictly needed; that doesn't prove ERT is off.

End state on EKS: GPU profiles that set extra_resource_limits.nvidia.com/gpu: 1 schedule cleanly onto NIC-tainted GPU nodes with no chart change and no per-profile toleration. The "Blocks #370" claim in this issue doesn't hold on EKS — #370 ships standalone.

The proposed implementation has a chart-level gotcha worth heading off

If this is implemented by following PR #139's pattern (adding tolerations: inside each GPU profile's kubespawner_override), it silently replaces c.KubeSpawner.tolerations rather than appending to it. Any global tolerations the chart populated at hub startup from scheduling.userPods.tolerations get wiped for the affected profile. I caught this in testing:

Toleration source	Without workaround	With kubespawner_override.tolerations: [...]
hub.jupyter.org/dedicated=user (chart, global)	present	gone
hub.jupyter.org_dedicated=user (chart, global)	present	gone
nvidia.com/gpu	ERT-injected	explicit

Concretely: on a cluster that uses z2jh's dedicated-user-nodes pattern (taint hub.jupyter.org/dedicated=user:NoSchedule on the user nodegroup, common in production Nebari deployments), implementing #117 the obvious way would break GPU profiles from scheduling onto those nodes — exactly the opposite of the goal.

A correct implementation needs to merge the GPU toleration into the existing kubespawner toleration list at spawn time, not replace it. The natural place is the chart's bundled 01-spawner.py (which already does this kind of merge for c.KubeSpawner.volumes), gated on the profile actually requesting nvidia.com/gpu in extra_resource_limits.

Criticality suggestion

• Managed EKS / GKE / AKS users (where ERT runs by default): not critical. PR #370 + the existing chart is sufficient. Recommend dropping from "Critical for MVP" for these targets.
• kubeadm / bare-metal / on-prem clusters (where ERT typically isn't enabled): still genuinely useful — saves operators from hand-writing the toleration on every GPU profile. Issue stays relevant for portability.

Happy to follow up with the merge-style implementation on 01-spawner.py if helpful.