Add nvidia.com/gpu toleration to Ray Serve GPU workloads

[dcmcand]

Background

nebari-dev/nebari-infrastructure-core#370 makes NIC automatically taint AWS GPU node groups with nvidia.com/gpu=true:NoSchedule, so only pods that tolerate it land on GPU nodes.

Nothing injects the toleration for us on EKS:

• The NVIDIA GPU Operator only tolerates this taint on its own operands, not on Ray workloads.
• The ExtendedResourceToleration admission controller is not available on EKS, so GPU-requesting pods do not get an automatic toleration. AWS's GPU guide hand-writes the toleration on every example.

Once #370 lands, Ray Serve head/worker pods that request GPUs will stop scheduling onto GPU nodes unless they tolerate the taint.

Request

Add a toleration for nvidia.com/gpu to the GPU-bearing Ray workloads this pack deploys. For the RayService CRD this means setting the toleration on the worker group pod template (and the head if it requests a GPU), ideally only for groups that request nvidia.com/gpu. The toleration:

tolerations:
- key: "nvidia.com/gpu"
  operator: "Exists"
  effect: "NoSchedule"

operator: Exists matches any taint value, so it works with NIC's value: "true" and any other.

Acceptance criteria

• A Ray Serve worker group that requests a GPU schedules onto a node tainted nvidia.com/gpu=true:NoSchedule and serves.
• Ray pods that do not request a GPU are unchanged.

Blocks

This blocks nebari-dev/nebari-infrastructure-core#370. The taint should not roll out until GPU Ray workloads tolerate it.

[oren-openteams]

Verified fixed on an EKS 1.34 cluster running the chart at v0.3.1. Three-layer check:

1. Chart contents

v0.3.1 is ahead_by: 2, behind_by: 0 relative to PR #18's merge commit b6b54fc, and was published 2026-06-19 — one day after PR #18 merged on 2026-06-18. The nebari-rayserve.tolerations helper in chart/templates/_helpers.tpl at ref v0.3.1 is byte-identical to PR #18's diff, and both headGroupSpec and workerGroupSpecs pod templates in chart/templates/rayservice.yaml correctly wire it in with {{- with (include "nebari-rayserve.tolerations" .Values.head/worker) }}.

2. helm template across three scenarios

Ran helm template against the chart at v0.3.1 with three values files, extracted the rendered RayService and inspected head + worker tolerations:

Case	Worker GPU request	Explicit user tolerations	Rendered worker tolerations
1 — chart defaults	none	none	none
2 — GPU worker, implicit	nvidia.com/gpu: 1	none	[{key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}] (auto-injected)
3 — GPU worker + explicit	nvidia.com/gpu: 1	nvidia.com/gpu Equal custom + dedicated Equal ml	both user tolerations preserved, no double-injection

All three match the documented behavior and the issue's acceptance criteria.

3. Live cluster

The deployed RayService on the cluster (no GPU worker configured currently) renders no tolerations block on either head or worker pod template — matches case 1 ✅.

Closing out the verification side. Reopen if behavior regresses on a chart upgrade.

---

Side note for future reference: the ExtendedResourceToleration admission controller IS active on EKS managed control planes (contrary to the original premise in this issue and PR #18's body) — verified by submitting a raw pod with resources.limits.nvidia.com/gpu: 1 and no manual toleration, and observing the toleration appear in the post-admission pod spec. So on EKS, the auto-injection PR #18 performs is technically redundant; ERT would inject the same toleration during admission.

This does NOT affect the correctness of PR #18 — the implementation is conditional on a GPU request AND uses append-with-dedupe semantics, so on EKS it cleanly composes with ERT (the chart-injected toleration matches ERT's, so ERT's idempotency skips the duplicate) and on non-managed clusters it's the only thing keeping GPU Ray workloads scheduling. The fix is correct in both environments. Just noting the EKS observation for anyone trying to reason about admission-chain behavior here.