Summary
chart/values.yaml exposes image.repository and image.tag for the Ray head + worker pods, but doesn't expose image.pullSecrets. As a result, charts deployed against a private container registry can render successfully but the resulting pods sit in ImagePullBackOff because there's no way to wire up registry credentials through helm values.
The chart's templates also don't override serviceAccountName for the pods, so they run as the default ServiceAccount in the release namespace.
Reproduction
-
Deploy the chart with a private-registry image, e.g.:
image:
repository: registry.example.internal/team/ray-worker
tag: 1.0.0
-
Without an out-of-band step, head + worker pods stay in ImagePullBackOff:
$ kubectl -n ray describe pod rayservice-pack-rayservice-... | grep -A2 Failed
Warning Failed … kubelet Failed to pull image "registry.example.internal/…":
failed to resolve reference "...": pulling from host
registry.example.internal failed with status code 401 Unauthorized
-
Currently the only way to fix this is to attach an imagePullSecret to the default ServiceAccount in the release namespace, which means the credential lives outside the chart's helm values and the gitops repo:
kubectl -n ray create secret docker-registry registry-creds \
--docker-server=registry.example.internal \
--docker-username=<user> --docker-password=<token>
kubectl -n ray patch serviceaccount default --type=merge \
-p '{"imagePullSecrets":[{"name":"registry-creds"}]}'
This is awkward for a few reasons:
- The chart consumer has to know the chart uses the
default SA — that's an implementation detail leaking through.
- The patch is imperative and lives on the cluster, not in git. If the SA is ever recreated (e.g. namespace prune-and-recreate), the binding silently disappears and the next pod restart fails.
- It's noisy in multi-tenant clusters where the
default SA is shared across unrelated workloads.
Proposed change
Add image.pullSecrets: [] to chart/values.yaml and render it into both pod templates:
# chart/values.yaml
image:
repository: rayproject/ray
tag: 2.43.0
# List of image pull secret names already present in the release
# namespace. Names must reference Secrets of type
# kubernetes.io/dockerconfigjson. Defaulted empty so existing
# deployments using public images are unaffected.
pullSecrets: []
# Example:
# pullSecrets:
# - name: my-registry-creds
Then in chart/templates/rayservice.yaml, add the field to both the head pod template (around line 42) and the worker pod template (around line 96), gated so it only renders when set:
template:
spec:
{{- with .Values.image.pullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 6 }}
{{- end }}
containers:
- name: ray-head # or ray-worker
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
# …
{{- with }} ensures the field is only rendered when pullSecrets is non-empty, so the existing rendered output is byte-identical when nobody sets it — no diff for users who don't need this.
Acceptance criteria
Out of scope
- Auto-creating the pull Secret from a helm value. The Secret should be pre-created via whatever mechanism the operator uses (sealed-secrets, external-secrets, gitops-managed Secret, manual
kubectl create). The chart only consumes a reference, not the credential.
- A separate
serviceAccountName override. That's a different feature (custom SA with extra IAM, IRSA bindings, etc.). The pullSecrets fix doesn't require it — imagePullSecrets on the pod spec works regardless of which SA the pod runs under.
Alternatives considered
- Patch the
default ServiceAccount. What we do today. Documented above — leaky and not gitops-friendly.
- Add a
serviceAccount.create: true with imagePullSecrets. Works but introduces a new SA per release, plus requires serviceAccountName plumbing through the pod templates. Larger surface area than necessary when the only thing that's missing is the per-pod imagePullSecrets field.
- Use a registry mirror. Workable for organizations that have a public mirror of their private images, but doesn't help the common case where the images themselves are private.
Summary
chart/values.yamlexposesimage.repositoryandimage.tagfor the Ray head + worker pods, but doesn't exposeimage.pullSecrets. As a result, charts deployed against a private container registry can render successfully but the resulting pods sit inImagePullBackOffbecause there's no way to wire up registry credentials through helm values.The chart's templates also don't override
serviceAccountNamefor the pods, so they run as thedefaultServiceAccount in the release namespace.Reproduction
Deploy the chart with a private-registry image, e.g.:
Without an out-of-band step, head + worker pods stay in
ImagePullBackOff:Currently the only way to fix this is to attach an
imagePullSecretto thedefaultServiceAccount in the release namespace, which means the credential lives outside the chart's helm values and the gitops repo:kubectl -n ray create secret docker-registry registry-creds \ --docker-server=registry.example.internal \ --docker-username=<user> --docker-password=<token> kubectl -n ray patch serviceaccount default --type=merge \ -p '{"imagePullSecrets":[{"name":"registry-creds"}]}'This is awkward for a few reasons:
defaultSA — that's an implementation detail leaking through.defaultSA is shared across unrelated workloads.Proposed change
Add
image.pullSecrets: []tochart/values.yamland render it into both pod templates:Then in
chart/templates/rayservice.yaml, add the field to both the head pod template (around line 42) and the worker pod template (around line 96), gated so it only renders when set:{{- with }}ensures the field is only rendered whenpullSecretsis non-empty, so the existing rendered output is byte-identical when nobody sets it — no diff for users who don't need this.Acceptance criteria
chart/values.yamlexposesimage.pullSecrets(list of{name: ...}entries), defaulted empty, with a comment explaining the format and that the referenced Secrets must already exist in the release namespace.chart/templates/rayservice.yamlhead pod spec rendersimagePullSecretswhenimage.pullSecretsis non-empty.image.pullSecretsis empty (the default), the rendered manifest is byte-identical to v0.3.1 — confirmed byhelm templatediff.Out of scope
kubectl create). The chart only consumes a reference, not the credential.serviceAccountNameoverride. That's a different feature (custom SA with extra IAM, IRSA bindings, etc.). The pullSecrets fix doesn't require it —imagePullSecretson the pod spec works regardless of which SA the pod runs under.Alternatives considered
defaultServiceAccount. What we do today. Documented above — leaky and not gitops-friendly.serviceAccount.create: truewithimagePullSecrets. Works but introduces a new SA per release, plus requiresserviceAccountNameplumbing through the pod templates. Larger surface area than necessary when the only thing that's missing is the per-podimagePullSecretsfield.