From 594b5188c77a37bcf0a2baba7ac45ad846ad2fd3 Mon Sep 17 00:00:00 2001 From: Oren Fromberg Date: Thu, 11 Jun 2026 09:14:39 -0400 Subject: [PATCH 1/4] feat(ca-bundle): inject org CA into Ray head + worker pods (#15) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a `singleuser.gallery`-style opt-in for organisation-supplied CA bundles. When `orgCABundle.secretName` is set, the chart renders: - An initContainer (default image `alpine:3.20`) that concatenates the base image's system trust store with the deployer-supplied CA into a shared emptyDir at /shared/combined-ca.crt - Two volumes on each pod spec: the deployer Secret (read-only) and the shared emptyDir - SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE env vars on both head and worker pointing at the combined bundle, alongside any existing head.containerEnv / worker.containerEnv entries - A read-only volumeMount of the combined bundle on each main container When `secretName` is empty (default), zero CA-related output is rendered — no initContainer, no volumes, no env vars. Backward- compatible. Implemented as five named helpers in _helpers.tpl (.enabled, .initContainers, .volumes, .volumeMounts, .env) so head and worker share one source of truth, and the rayservice template gates each block on the same `if .secretName` predicate. Covers everything that honours SSL_CERT_FILE: requests, urllib3, curl, git, pip, torch.hub, stdlib urllib, properly-constructed httpx clients. Does NOT cover httpx.Client(verify=True) defaults, which hardcode cafile=certifi.where() and ignore env vars — caller code needs to pass verify=ssl.create_default_context() explicitly. README calls this out as a known coverage caveat. Closes #15 --- README.md | 37 ++++++++++++++++ chart/templates/_helpers.tpl | 78 +++++++++++++++++++++++++++++++++ chart/templates/rayservice.yaml | 30 ++++++++++++- chart/values.yaml | 42 ++++++++++++++++++ 4 files changed, 185 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3a515dc..0aa5098 100644 --- a/README.md +++ b/README.md @@ -225,6 +225,43 @@ Key values in `chart/values.yaml`: |-------|---------|-------------| | `serveApplications` | `[]` | Declarative Serve applications (see [Ray Serve config](https://docs.ray.io/en/latest/serve/production-guide/config.html)) | +### Organisation CA Bundle Injection + +For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.secretName` at a Secret containing your organisation's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` so any TLS client honouring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. + +| Value | Default | Description | +|-------|---------|-------------| +| `orgCABundle.secretName` | `""` | Name of a Secret with key `ca.crt` containing the org CA (PEM). Empty disables injection — no behavior change. | +| `orgCABundle.initImage` | `alpine:3.20` | Image used by the bundle-building initContainer. Only needs `sh` + `cat`. | + +```yaml +# Create the Secret out-of-band (gitops, kubectl, sealed-secrets, etc.): +apiVersion: v1 +kind: Secret +metadata: + name: org-ca-bundle +type: Opaque +stringData: + ca.crt: | + -----BEGIN CERTIFICATE----- + ...your org CA... + -----END CERTIFICATE----- +--- +# Then point the chart at it: +orgCABundle: + secretName: org-ca-bundle +``` + +**Coverage caveat — httpx default `verify=True`:** httpx hardcodes its SSL context to `cafile=certifi.where()`, which means it **ignores** `SSL_CERT_FILE`. Application code making httpx calls that need to traverse a TLS-inspecting proxy must construct an explicit context: + +```python +import ssl, httpx +client = httpx.Client(verify=ssl.create_default_context()) +# or per-call: httpx.get(url, verify=ssl.create_default_context()) +``` + +`ssl.create_default_context()` with no `cafile=` honours `SSL_CERT_FILE` / `SSL_CERT_DIR` per the standard OpenSSL convention, so it picks up the bundle this chart injected. Other Python HTTP clients (`requests`, `urllib3`, stdlib `urllib`) and most non-Python TLS tooling honour the env vars automatically. + ## Architecture ```mermaid diff --git a/chart/templates/_helpers.tpl b/chart/templates/_helpers.tpl index d8f8d69..778c7eb 100644 --- a/chart/templates/_helpers.tpl +++ b/chart/templates/_helpers.tpl @@ -61,3 +61,81 @@ Ray serve service name - RayService creates a service named -se {{- define "nebari-rayserve.serve-service-name" -}} {{- printf "%s-serve-svc" (include "nebari-rayserve.fullname" .) }} {{- end }} + +{{/* +Whether organisation CA bundle injection is enabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.enabled" -}} +{{- if and .Values.orgCABundle .Values.orgCABundle.secretName -}} +true +{{- end }} +{{- end }} + +{{/* +initContainers block for the combined-CA bundle build step. Renders empty +when orgCABundle injection is disabled. Used by both head and worker pod +specs so the SSL_CERT_FILE bundle exists before the main container starts. +*/}} +{{- define "nebari-rayserve.orgCABundle.initContainers" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: build-ca-bundle + image: {{ .Values.orgCABundle.initImage | quote }} + command: + - sh + - -c + - | + cat /etc/ssl/certs/ca-certificates.crt \ + /var/local/org-ca/ca.crt > /shared/combined-ca.crt + volumeMounts: + - name: org-ca + mountPath: /var/local/org-ca + readOnly: true + - name: combined-ca + mountPath: /shared +{{- end }} +{{- end }} + +{{/* +Volumes block for the org-ca Secret (deployer-supplied) + the shared +emptyDir that the initContainer writes the combined bundle into. Renders +empty when orgCABundle injection is disabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.volumes" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: org-ca + secret: + secretName: {{ .Values.orgCABundle.secretName | quote }} +- name: combined-ca + emptyDir: {} +{{- end }} +{{- end }} + +{{/* +Container volumeMounts for the combined-CA bundle (read-only). The main +Ray container mounts only the combined-ca volume — it never sees the raw +org-ca Secret. Renders empty when orgCABundle injection is disabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.volumeMounts" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: combined-ca + mountPath: /shared + readOnly: true +{{- end }} +{{- end }} + +{{/* +Container env entries pointing the standard OpenSSL trust-store env vars +at the combined-CA bundle. Anything that honours SSL_CERT_FILE / +REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE picks it up automatically. +Renders empty when orgCABundle injection is disabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.env" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: SSL_CERT_FILE + value: /shared/combined-ca.crt +- name: REQUESTS_CA_BUNDLE + value: /shared/combined-ca.crt +- name: CURL_CA_BUNDLE + value: /shared/combined-ca.crt +{{- end }} +{{- end }} diff --git a/chart/templates/rayservice.yaml b/chart/templates/rayservice.yaml index c05ce9f..e9ff085 100644 --- a/chart/templates/rayservice.yaml +++ b/chart/templates/rayservice.yaml @@ -27,13 +27,26 @@ spec: {{- with .Values.head.runtimeClassName }} runtimeClassName: {{ . }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.initContainers" .) }} + initContainers: + {{- . | nindent 12 }} + {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumes" .) }} + volumes: + {{- . | nindent 12 }} + {{- end }} containers: - name: ray-head image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - {{- with .Values.head.containerEnv }} + {{- $headEnv := concat (.Values.head.containerEnv | default list) (fromYamlArray (include "nebari-rayserve.orgCABundle.env" .)) }} + {{- with $headEnv }} env: {{- toYaml . | nindent 16 }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumeMounts" .) }} + volumeMounts: + {{- . | nindent 16 }} + {{- end }} resources: {{- toYaml .Values.head.resources | nindent 16 }} ports: @@ -64,13 +77,26 @@ spec: {{- with .Values.worker.runtimeClassName }} runtimeClassName: {{ . }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.initContainers" .) }} + initContainers: + {{- . | nindent 14 }} + {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumes" .) }} + volumes: + {{- . | nindent 14 }} + {{- end }} containers: - name: ray-worker image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - {{- with .Values.worker.containerEnv }} + {{- $workerEnv := concat (.Values.worker.containerEnv | default list) (fromYamlArray (include "nebari-rayserve.orgCABundle.env" .)) }} + {{- with $workerEnv }} env: {{- toYaml . | nindent 18 }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumeMounts" .) }} + volumeMounts: + {{- . | nindent 18 }} + {{- end }} resources: {{- toYaml .Values.worker.resources | nindent 18 }} {{- with .Values.worker.readinessProbe }} diff --git a/chart/values.yaml b/chart/values.yaml index 2a4c92f..545b2bd 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -72,6 +72,48 @@ serve: # num_replicas: 2 serveApplications: [] +# ============================================================================= +# Organisation CA Bundle Injection +# ============================================================================= +# Mount an organisation-managed CA bundle into the Ray head and worker pods +# so outbound HTTPS calls (model registries, dataset URLs, Hugging Face hub, +# S3, etc.) succeed in environments running a TLS-inspecting proxy +# (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.). +# +# When `secretName` is set, the chart adds: +# - An initContainer that concatenates the base image's system trust store +# (/etc/ssl/certs/ca-certificates.crt) with the deployer-supplied CA +# (mounted from the Secret at the key `ca.crt`) into +# /shared/combined-ca.crt +# - A shared emptyDir volume that the main container mounts read-only +# - SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE env vars on both +# head and worker pointing at /shared/combined-ca.crt +# +# When `secretName` is empty (default), nothing is rendered — the resulting +# RayService is byte-identical to the no-CA-injection case. +# +# Known coverage gap: httpx clients constructed with the default +# `verify=True` hardcode `cafile=certifi.where()` and ignore SSL_CERT_FILE. +# Application code that needs its httpx calls covered must pass +# `verify=ssl.create_default_context()` (or equivalent) explicitly. +# +# Secret shape (created out-of-band, e.g. by your gitops layer): +# apiVersion: v1 +# kind: Secret +# metadata: +# name: org-ca-bundle +# type: Opaque +# stringData: +# ca.crt: | +# -----BEGIN CERTIFICATE----- +# ...your org CA... +# -----END CERTIFICATE----- +orgCABundle: + secretName: "" + # Image used by the initContainer that builds the combined CA bundle. + # Only needs `sh` and `cat` — defaults to a tiny Alpine image. + initImage: "alpine:3.20" + # ============================================================================= # Ray Image # ============================================================================= From e4087a504cca3a8d3289d99240a7fb0024f1278d Mon Sep 17 00:00:00 2001 From: Oren Fromberg Date: Thu, 11 Jun 2026 09:19:39 -0400 Subject: [PATCH 2/4] Normalize British spellings to American MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit s/organisation/organization/, s/honour/honor/ across values.yaml comment block, _helpers.tpl docstrings, and README section. No functional change — rendered RayService is byte-identical to the prior commit on this branch. Keeps the chart's user-facing copy consistent with the rest of the project. --- README.md | 6 +++--- chart/templates/_helpers.tpl | 4 ++-- chart/values.yaml | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 0aa5098..fe70441 100644 --- a/README.md +++ b/README.md @@ -225,9 +225,9 @@ Key values in `chart/values.yaml`: |-------|---------|-------------| | `serveApplications` | `[]` | Declarative Serve applications (see [Ray Serve config](https://docs.ray.io/en/latest/serve/production-guide/config.html)) | -### Organisation CA Bundle Injection +### Organization CA Bundle Injection -For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.secretName` at a Secret containing your organisation's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` so any TLS client honouring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. +For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.secretName` at a Secret containing your organization's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` so any TLS client honoring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. | Value | Default | Description | |-------|---------|-------------| @@ -260,7 +260,7 @@ client = httpx.Client(verify=ssl.create_default_context()) # or per-call: httpx.get(url, verify=ssl.create_default_context()) ``` -`ssl.create_default_context()` with no `cafile=` honours `SSL_CERT_FILE` / `SSL_CERT_DIR` per the standard OpenSSL convention, so it picks up the bundle this chart injected. Other Python HTTP clients (`requests`, `urllib3`, stdlib `urllib`) and most non-Python TLS tooling honour the env vars automatically. +`ssl.create_default_context()` with no `cafile=` honors `SSL_CERT_FILE` / `SSL_CERT_DIR` per the standard OpenSSL convention, so it picks up the bundle this chart injected. Other Python HTTP clients (`requests`, `urllib3`, stdlib `urllib`) and most non-Python TLS tooling honor the env vars automatically. ## Architecture diff --git a/chart/templates/_helpers.tpl b/chart/templates/_helpers.tpl index 778c7eb..64f873a 100644 --- a/chart/templates/_helpers.tpl +++ b/chart/templates/_helpers.tpl @@ -63,7 +63,7 @@ Ray serve service name - RayService creates a service named -se {{- end }} {{/* -Whether organisation CA bundle injection is enabled. +Whether organization CA bundle injection is enabled. */}} {{- define "nebari-rayserve.orgCABundle.enabled" -}} {{- if and .Values.orgCABundle .Values.orgCABundle.secretName -}} @@ -125,7 +125,7 @@ org-ca Secret. Renders empty when orgCABundle injection is disabled. {{/* Container env entries pointing the standard OpenSSL trust-store env vars -at the combined-CA bundle. Anything that honours SSL_CERT_FILE / +at the combined-CA bundle. Anything that honors SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE picks it up automatically. Renders empty when orgCABundle injection is disabled. */}} diff --git a/chart/values.yaml b/chart/values.yaml index 545b2bd..a57d7ba 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -73,9 +73,9 @@ serve: serveApplications: [] # ============================================================================= -# Organisation CA Bundle Injection +# Organization CA Bundle Injection # ============================================================================= -# Mount an organisation-managed CA bundle into the Ray head and worker pods +# Mount an organization-managed CA bundle into the Ray head and worker pods # so outbound HTTPS calls (model registries, dataset URLs, Hugging Face hub, # S3, etc.) succeed in environments running a TLS-inspecting proxy # (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.). From a862de7ee7a19fb3bf0837a0b0e9826d69c99d74 Mon Sep 17 00:00:00 2001 From: Oren Fromberg Date: Thu, 11 Jun 2026 14:57:24 -0400 Subject: [PATCH 3/4] Switch org CA bundle from Secret to ConfigMap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CA certificates are public material by design — the PKI trust model relies on root CAs being widely distributed. Kubernetes itself uses a ConfigMap for the cluster's own CA distribution (kube-root-ca.crt, auto-projected into every namespace), and cert-manager's trust-manager distributes Bundle CRs as ConfigMaps. Following that precedent. Breaking change against the prior commit on this branch (PR not yet merged, so no consumer impact): - values.yaml: orgCABundle.secretName -> orgCABundle.configMapName - _helpers.tpl: enabled predicate gates on configMapName; volumes helper renders as configMap.name instead of secret.secretName - README + values.yaml comment block: rewrite example to ConfigMap; add 'Why ConfigMap rather than Secret' explainer citing kube-root-ca.crt and trust-manager precedent helm template verified in three modes: configMapName set -> initContainer + ConfigMap volume + env vars no value (default) -> zero CA-related output old secretName set -> no injection (regression guard) --- README.md | 15 ++++++++------- chart/templates/_helpers.tpl | 10 +++++----- chart/values.yaml | 26 +++++++++++++++++--------- 3 files changed, 30 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index fe70441..861aabc 100644 --- a/README.md +++ b/README.md @@ -227,21 +227,20 @@ Key values in `chart/values.yaml`: ### Organization CA Bundle Injection -For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.secretName` at a Secret containing your organization's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` so any TLS client honoring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. +For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.configMapName` at a ConfigMap containing your organization's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` so any TLS client honoring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. | Value | Default | Description | |-------|---------|-------------| -| `orgCABundle.secretName` | `""` | Name of a Secret with key `ca.crt` containing the org CA (PEM). Empty disables injection — no behavior change. | +| `orgCABundle.configMapName` | `""` | Name of a ConfigMap with key `ca.crt` containing the org CA (PEM). Empty disables injection — no behavior change. | | `orgCABundle.initImage` | `alpine:3.20` | Image used by the bundle-building initContainer. Only needs `sh` + `cat`. | ```yaml -# Create the Secret out-of-band (gitops, kubectl, sealed-secrets, etc.): +# Create the ConfigMap out-of-band (gitops, kubectl, etc.): apiVersion: v1 -kind: Secret +kind: ConfigMap metadata: name: org-ca-bundle -type: Opaque -stringData: +data: ca.crt: | -----BEGIN CERTIFICATE----- ...your org CA... @@ -249,9 +248,11 @@ stringData: --- # Then point the chart at it: orgCABundle: - secretName: org-ca-bundle + configMapName: org-ca-bundle ``` +> **Why ConfigMap rather than Secret?** A CA certificate is public material by design — the PKI trust model relies on root CAs being widely distributed. Kubernetes itself uses a ConfigMap for the cluster's own CA distribution (`kube-root-ca.crt`, auto-projected into every namespace), and cert-manager's trust-manager subproject does the same. Use a ConfigMap here; reserve Secret for things that actually need confidentiality (private keys, OAuth client secrets, etc.). + **Coverage caveat — httpx default `verify=True`:** httpx hardcodes its SSL context to `cafile=certifi.where()`, which means it **ignores** `SSL_CERT_FILE`. Application code making httpx calls that need to traverse a TLS-inspecting proxy must construct an explicit context: ```python diff --git a/chart/templates/_helpers.tpl b/chart/templates/_helpers.tpl index 64f873a..71a097b 100644 --- a/chart/templates/_helpers.tpl +++ b/chart/templates/_helpers.tpl @@ -66,7 +66,7 @@ Ray serve service name - RayService creates a service named -se Whether organization CA bundle injection is enabled. */}} {{- define "nebari-rayserve.orgCABundle.enabled" -}} -{{- if and .Values.orgCABundle .Values.orgCABundle.secretName -}} +{{- if and .Values.orgCABundle .Values.orgCABundle.configMapName -}} true {{- end }} {{- end }} @@ -96,15 +96,15 @@ specs so the SSL_CERT_FILE bundle exists before the main container starts. {{- end }} {{/* -Volumes block for the org-ca Secret (deployer-supplied) + the shared +Volumes block for the org-ca ConfigMap (deployer-supplied) + the shared emptyDir that the initContainer writes the combined bundle into. Renders empty when orgCABundle injection is disabled. */}} {{- define "nebari-rayserve.orgCABundle.volumes" -}} {{- if include "nebari-rayserve.orgCABundle.enabled" . -}} - name: org-ca - secret: - secretName: {{ .Values.orgCABundle.secretName | quote }} + configMap: + name: {{ .Values.orgCABundle.configMapName | quote }} - name: combined-ca emptyDir: {} {{- end }} @@ -113,7 +113,7 @@ empty when orgCABundle injection is disabled. {{/* Container volumeMounts for the combined-CA bundle (read-only). The main Ray container mounts only the combined-ca volume — it never sees the raw -org-ca Secret. Renders empty when orgCABundle injection is disabled. +org-ca ConfigMap. Renders empty when orgCABundle injection is disabled. */}} {{- define "nebari-rayserve.orgCABundle.volumeMounts" -}} {{- if include "nebari-rayserve.orgCABundle.enabled" . -}} diff --git a/chart/values.yaml b/chart/values.yaml index a57d7ba..f1d3bb5 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -80,36 +80,44 @@ serveApplications: [] # S3, etc.) succeed in environments running a TLS-inspecting proxy # (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.). # -# When `secretName` is set, the chart adds: +# When `configMapName` is set, the chart adds: # - An initContainer that concatenates the base image's system trust store # (/etc/ssl/certs/ca-certificates.crt) with the deployer-supplied CA -# (mounted from the Secret at the key `ca.crt`) into +# (mounted from the ConfigMap at the key `ca.crt`) into # /shared/combined-ca.crt # - A shared emptyDir volume that the main container mounts read-only # - SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE env vars on both # head and worker pointing at /shared/combined-ca.crt # -# When `secretName` is empty (default), nothing is rendered — the resulting -# RayService is byte-identical to the no-CA-injection case. +# When `configMapName` is empty (default), nothing is rendered — the +# resulting RayService is byte-identical to the no-CA-injection case. +# +# Why ConfigMap rather than Secret: a CA certificate is public material +# by design (the entire PKI trust model relies on root CAs being widely +# distributed — Mozilla's bundle ships in every browser/OS, corporate +# inspecting proxy roots are pushed to every device that traverses them). +# Kubernetes itself uses a ConfigMap for the cluster's own CA distribution +# (`kube-root-ca.crt`, auto-projected into every namespace). cert-manager's +# trust-manager subproject distributes CA bundles as ConfigMaps via its +# Bundle CR. We follow that precedent here. # # Known coverage gap: httpx clients constructed with the default # `verify=True` hardcode `cafile=certifi.where()` and ignore SSL_CERT_FILE. # Application code that needs its httpx calls covered must pass # `verify=ssl.create_default_context()` (or equivalent) explicitly. # -# Secret shape (created out-of-band, e.g. by your gitops layer): +# ConfigMap shape (created out-of-band, e.g. by your gitops layer): # apiVersion: v1 -# kind: Secret +# kind: ConfigMap # metadata: # name: org-ca-bundle -# type: Opaque -# stringData: +# data: # ca.crt: | # -----BEGIN CERTIFICATE----- # ...your org CA... # -----END CERTIFICATE----- orgCABundle: - secretName: "" + configMapName: "" # Image used by the initContainer that builds the combined CA bundle. # Only needs `sh` and `cat` — defaults to a tiny Alpine image. initImage: "alpine:3.20" From 97b1e0e655c1dc88bcd55bc867e1914874e6fb23 Mon Sep 17 00:00:00 2001 From: Oren Fromberg Date: Thu, 18 Jun 2026 08:04:17 -0400 Subject: [PATCH 4/4] Add GIT_SSL_CAINFO env var and document ArgoCD ignoreDifferences footgun git's libcurl ignores SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE and reads only GIT_SSL_CAINFO, so `pip install git+https://...` and other git-over-HTTPS calls failed certificate verification in worker pods even with the CA bundle mounted. Set GIT_SSL_CAINFO to the combined bundle too. Also document the ArgoCD footgun: the example sync policy's broad /spec/rayClusterConfig ignore rule combined with RespectIgnoreDifferences makes ArgoCD silently skip the CA injection (which lives under that path) while still reporting a healthy sync. See #17. --- README.md | 9 ++++++++- chart/templates/_helpers.tpl | 5 +++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 861aabc..a74e09c 100644 --- a/README.md +++ b/README.md @@ -123,6 +123,7 @@ spec: - `managedNamespaceMetadata` with `nebari.dev/managed: "true"` is required for the nebari-operator to manage NebariApp resources - `redirectURI` must be `/oauth2/callback` (Envoy Gateway rejects `/`) - Set `serve.enabled: false` to keep the serve endpoint internal-only (recommended — notebooks access it via cluster DNS) +- The `/spec/rayClusterConfig` ignore rule combined with `RespectIgnoreDifferences=true` makes ArgoCD stop managing everything under `rayClusterConfig`. If you enable [`orgCABundle`](#organization-ca-bundle-injection), the CA injection lives under that path and will **silently not apply** — see the ArgoCD footgun warning in that section before turning it on. ## Connecting from Jupyter @@ -227,7 +228,9 @@ Key values in `chart/values.yaml`: ### Organization CA Bundle Injection -For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.configMapName` at a ConfigMap containing your organization's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` so any TLS client honoring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. +For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.configMapName` at a ConfigMap containing your organization's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` / `GIT_SSL_CAINFO` so any TLS client honoring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. + +`GIT_SSL_CAINFO` is set in addition to the three OpenSSL env vars because git's libcurl ignores `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` and reads only `GIT_SSL_CAINFO`. Without it, `pip install git+https://...` and other git-over-HTTPS operations in worker contexts fail certificate verification even though plain `requests`/`pip` calls succeed. | Value | Default | Description | |-------|---------|-------------| @@ -253,6 +256,10 @@ orgCABundle: > **Why ConfigMap rather than Secret?** A CA certificate is public material by design — the PKI trust model relies on root CAs being widely distributed. Kubernetes itself uses a ConfigMap for the cluster's own CA distribution (`kube-root-ca.crt`, auto-projected into every namespace), and cert-manager's trust-manager subproject does the same. Use a ConfigMap here; reserve Secret for things that actually need confidentiality (private keys, OAuth client secrets, etc.). +> **⚠️ ArgoCD footgun — the CA bundle silently won't apply.** The ArgoCD `Application` shown under [On a Nebari cluster (via ArgoCD)](#on-a-nebari-cluster-via-argocd) sets `RespectIgnoreDifferences=true` together with an `ignoreDifferences` rule on `/spec/rayClusterConfig`. With server-side apply, that combination tells ArgoCD to **stop managing every field under `rayClusterConfig`** — which is exactly where this chart injects the initContainer, volumes, volumeMounts, and CA env vars for the head and worker pods. The result is a silent failure: ArgoCD reports a healthy, fully-synced `Application`, but the running RayService never gets the CA bundle, and TLS calls keep failing with `CERTIFICATE_VERIFY_FAILED`. +> +> If you enable `orgCABundle` on a cluster managed by ArgoCD with the example sync policy, you must **narrow the ignore rule** so the CA fields are still reconciled. The broad `/spec/rayClusterConfig` ignore exists only to suppress the autoscaler/runtime mutations KubeRay makes; replace it with targeted pointers (or drop it and ignore only the specific subpaths KubeRay rewrites). After changing it, confirm the head and worker pods actually carry `SSL_CERT_FILE` (`kubectl exec ... -- printenv SSL_CERT_FILE`) rather than trusting the ArgoCD sync status. See [#17](https://github.com/nebari-dev/nebari-rayserve-pack/issues/17) for details. + **Coverage caveat — httpx default `verify=True`:** httpx hardcodes its SSL context to `cafile=certifi.where()`, which means it **ignores** `SSL_CERT_FILE`. Application code making httpx calls that need to traverse a TLS-inspecting proxy must construct an explicit context: ```python diff --git a/chart/templates/_helpers.tpl b/chart/templates/_helpers.tpl index 71a097b..c703200 100644 --- a/chart/templates/_helpers.tpl +++ b/chart/templates/_helpers.tpl @@ -127,6 +127,9 @@ org-ca ConfigMap. Renders empty when orgCABundle injection is disabled. Container env entries pointing the standard OpenSSL trust-store env vars at the combined-CA bundle. Anything that honors SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE picks it up automatically. +GIT_SSL_CAINFO is set separately because git's libcurl ignores the three +above and reads only GIT_SSL_CAINFO — without it `pip install git+https://...` +and other git-over-HTTPS calls fail certificate verification. Renders empty when orgCABundle injection is disabled. */}} {{- define "nebari-rayserve.orgCABundle.env" -}} @@ -137,5 +140,7 @@ Renders empty when orgCABundle injection is disabled. value: /shared/combined-ca.crt - name: CURL_CA_BUNDLE value: /shared/combined-ca.crt +- name: GIT_SSL_CAINFO + value: /shared/combined-ca.crt {{- end }} {{- end }}