feat(ca-bundle): inject org CA into Ray head + worker pods via ConfigMap (#15)

[oren-openteams]

Update (post-initial-push): This PR was originally drafted to use a Secret for the org CA bundle. Switched to ConfigMap mid-review — CA certs are public material by design, and Kubernetes itself uses a ConfigMap for kube-root-ca.crt (the cluster's own CA distribution). Latest commit on the branch reflects the change. Old orgCABundle.secretName is no longer recognized; use orgCABundle.configMapName instead.

What

Adds opt-in injection of an organization-supplied CA bundle into the Ray head and worker pods, so outbound HTTPS calls succeed in environments running a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.). Resolves the symptom that motivated #15:

httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)

Implemented as a new orgCABundle values block driving an initContainer that builds a combined CA bundle (system trust + org CA) and a set of env vars / volume mounts on the head and worker pods.

Behavior

Opt-in via Secret. Deployer creates a Secret with key ca.crt:

apiVersion: v1
kind: Secret
metadata:
  name: org-ca-bundle
type: Opaque
stringData:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

…then points the chart at it:

orgCABundle:
  secretName: org-ca-bundle

When enabled, both head and worker pods get:

• initContainer (default alpine:3.20) that concatenates /etc/ssl/certs/ca-certificates.crt + /var/local/org-ca/ca.crt → /shared/combined-ca.crt
• Volumes: org-ca (the Secret, read-only) + combined-ca (emptyDir)
• Env: SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE → /shared/combined-ca.crt
• volumeMount of the combined bundle (read-only)

When configMapName is empty (default), zero CA-related output is rendered — no initContainer, no volumes, no env vars. Backward-compatible for existing deploys.

Implementation

Five named helpers in _helpers.tpl (single source of truth for head + worker):

• nebari-rayserve.orgCABundle.enabled — predicate
• nebari-rayserve.orgCABundle.initContainers — the build step
• nebari-rayserve.orgCABundle.volumes — Secret + emptyDir
• nebari-rayserve.orgCABundle.volumeMounts — read-only mount of combined bundle
• nebari-rayserve.orgCABundle.env — SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE

Each helper renders empty when the predicate is false, so the rayservice.yaml template can wrap them in {{- with (include ...) }} and the no-op case produces byte-identical output to today.

head.containerEnv and worker.containerEnv are preserved — the CA env vars are concat'd, not overwritten:

{{- $headEnv := concat (.Values.head.containerEnv | default list) (fromYamlArray (include "nebari-rayserve.orgCABundle.env" .)) }}

Coverage

This covers anything that honors SSL_CERT_FILE: requests, urllib3, curl, git, pip, torch.hub, stdlib urllib, and properly-constructed httpx clients (i.e. ones using verify=ssl.create_default_context()).

Known coverage gap — httpx default verify=True: httpx's default verify path hardcodes cafile=certifi.where(), which ignores env vars. Application code making httpx calls must construct an SSL context explicitly:

import ssl, httpx
client = httpx.Client(verify=ssl.create_default_context())

README documents this caveat. (One specific upstream case — checkmaite's httpx.stream in VisdroneODModel — is being fixed in a separate app-side change by the same team, since this chart can mount the bundle but can't fix the consumer's SSL context construction.)

Verification

helm template rendering, both modes (the chart's kuberay-operator dep was pulled with helm dependency build chart/ first):

Enabled (--set orgCABundle.configMapName=org-ca-bundle) — both head and worker get:

initContainers:
  - name: build-ca-bundle
    image: "alpine:3.20"
    command:
      - sh
      - -c
      - |
        cat /etc/ssl/certs/ca-certificates.crt \
            /var/local/org-ca/ca.crt > /shared/combined-ca.crt
    volumeMounts:
      - {name: org-ca, mountPath: /var/local/org-ca, readOnly: true}
      - {name: combined-ca, mountPath: /shared}
volumes:
  - {name: org-ca, configMap: {name: "org-ca-bundle"}}
  - {name: combined-ca, emptyDir: {}}
containers:
  - name: ray-head    # also ray-worker
    env:
      - {name: SSL_CERT_FILE,       value: /shared/combined-ca.crt}
      - {name: REQUESTS_CA_BUNDLE,  value: /shared/combined-ca.crt}
      - {name: CURL_CA_BUNDLE,      value: /shared/combined-ca.crt}
    volumeMounts:
      - {name: combined-ca, mountPath: /shared, readOnly: true}

Disabled (default): grepping helm template foo ./chart for initContainers|build-ca|SSL_CERT_FILE|combined-ca|org-ca returns zero hits.

Files

• chart/values.yaml — new orgCABundle block (42 lines, all defaults that keep behaviour unchanged)
• chart/templates/_helpers.tpl — five new helpers (78 lines, all gated by the same predicate so no-op when disabled)
• chart/templates/rayservice.yaml — wraps the five helpers into head + worker specs (30 line diff, additive, preserves head.runtimeClassName/readinessProbe/livenessProbe and worker.containerEnv unchanged)
• README.md — new ### Organization CA Bundle Injection subsection under ## Chart Configuration

Test plan

• helm template with configMapName=org-ca-bundle renders all five components on head + worker
• helm template without configMapName produces byte-identical RayService to today (verified by absence of CA-related strings)
• helm template with both head.containerEnv and orgCABundle.configMapName set produces the union of both env lists (deployer env vars are preserved, not overwritten by CA env vars)
• Deploy to a real cluster behind a TLS-inspecting proxy, verify a Python workload calling out via requests succeeds (covered by SSL_CERT_FILE)
• Same as above with an httpx client using verify=ssl.create_default_context() — confirms the coverage caveat path

Closes #15.

[oren-openteams]

Heads-up on what we learned integrating this PR into a Netskope-fronted production cluster deployed via ArgoCD gitops. Net: the chart works as designed end-to-end. A few findings might be worth folding back before merge.

End-to-end verification ✓

Deployed PR HEAD (fix/org-ca-bundle-injection) via ArgoCD with an org-ca-bundle ConfigMap (data.ca.crt: = corp CA) in the ray namespace and orgCABundle.configMapName: org-ca-bundle in values. Confirmed:

• initContainer produces /shared/combined-ca.crt (~219 KB = system bundle + 1 corp cert)
• SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE set on head + worker pods
• Inside a worker pod: ssl.create_default_context() loads 146 CAs incl. corp root; httpx.head against an intercepted endpoint (issuer confirmed as the corp Netskope CA) returns 200

Re: "Known coverage gap — httpx default verify=True"

This caveat is no longer accurate for current httpx. Empirically with httpx 0.28.1 and trust_env=True (the default):

httpx._config.create_ssl_context(verify=True, trust_env=True)  → 146 CAs (system + corp)
httpx._config.create_ssl_context(verify=True, trust_env=False) → 137 CAs (certifi only)

Reason: httpx 0.28.0's release silently added env-var honoring in commit 80960fa. No changelog entry; docs caught up 10 months later in encode/httpx#3579. So the "application must construct an explicit SSL context" guidance only applies to:

• httpx < 0.28
• Callers passing trust_env=False
• Callers passing verify=ssl.create_default_context(cafile=…) explicitly (the antipattern)

For the chart's typical audience (modern Python apps with httpx defaults), SSL_CERT_FILE already Just Works. Worth softening the gap callout. The downstream issue referenced in the PR can be updated similarly — we closed it without merging an app-layer wrapper, since httpx 0.28+ handles it.

Possibly missing env: GIT_SSL_CAINFO

The chart sets SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE, but not GIT_SSL_CAINFO. git's libcurl ignores those three and reads GIT_SSL_CAINFO (or git config http.sslCAInfo). We hit this trying to pip install git+https://... from a worker pod:

fatal: unable to access 'https://...': server certificate verification failed.
CAfile: none CRLfile: none

Workaround: prefix with GIT_SSL_CAINFO=/shared/combined-ca.crt. Git is commonly invoked from worker pods (pip install git+..., Ray runtime envs with pip deps, custom user code), so adding it to the chart's env list would close that gap.

Related: ArgoCD footgun (#17)

Filed #17 separately, but cross-referencing here since it affects deploys of this PR specifically. The chart's recommended ArgoCD configuration (README.md "On a Nebari cluster (via ArgoCD)") combines RespectIgnoreDifferences=true in syncOptions with a broad /spec/rayClusterConfig entry in ignoreDifferences. With server-side apply, that combination makes Argo skip every field under rayClusterConfig on the patch path — including this PR's orgCABundle additions. Argo reports "Synced" while the live RayService head spec has no initContainers / volumes. Suggested fix is documented in #17.

Worth at least a doc note in this PR too — anyone using the README's ArgoCD example will hit it the moment they enable orgCABundle.

---

Happy to send PRs for any of these (env-var addition, httpx caveat softening, README footgun warning) if it'd help land the parent PR faster.

[oren-openteams]

I took the initiative to address a couple of the items from the comment above: added GIT_SSL_CAINFO to the env vars that are mounted into the ray serve head/worker pods, and added some documentation to cover the ArgoCD configuration footgun. These two items improve the robustness of this feature as well as the ease of configuration.