diff --git a/README.md b/README.md index 3a515dc..a74e09c 100644 --- a/README.md +++ b/README.md @@ -123,6 +123,7 @@ spec: - `managedNamespaceMetadata` with `nebari.dev/managed: "true"` is required for the nebari-operator to manage NebariApp resources - `redirectURI` must be `/oauth2/callback` (Envoy Gateway rejects `/`) - Set `serve.enabled: false` to keep the serve endpoint internal-only (recommended — notebooks access it via cluster DNS) +- The `/spec/rayClusterConfig` ignore rule combined with `RespectIgnoreDifferences=true` makes ArgoCD stop managing everything under `rayClusterConfig`. If you enable [`orgCABundle`](#organization-ca-bundle-injection), the CA injection lives under that path and will **silently not apply** — see the ArgoCD footgun warning in that section before turning it on. ## Connecting from Jupyter @@ -225,6 +226,50 @@ Key values in `chart/values.yaml`: |-------|---------|-------------| | `serveApplications` | `[]` | Declarative Serve applications (see [Ray Serve config](https://docs.ray.io/en/latest/serve/production-guide/config.html)) | +### Organization CA Bundle Injection + +For deployments behind a TLS-inspecting proxy (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.), point `orgCABundle.configMapName` at a ConfigMap containing your organization's root CA. The chart adds an initContainer that builds a combined CA bundle (system trust + org CA), mounts it into the head and worker pods, and sets `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` / `GIT_SSL_CAINFO` so any TLS client honoring those env vars (requests, urllib3, curl, git, pip, `torch.hub`, etc.) trusts the proxy's re-signed certs. + +`GIT_SSL_CAINFO` is set in addition to the three OpenSSL env vars because git's libcurl ignores `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE` and reads only `GIT_SSL_CAINFO`. Without it, `pip install git+https://...` and other git-over-HTTPS operations in worker contexts fail certificate verification even though plain `requests`/`pip` calls succeed. + +| Value | Default | Description | +|-------|---------|-------------| +| `orgCABundle.configMapName` | `""` | Name of a ConfigMap with key `ca.crt` containing the org CA (PEM). Empty disables injection — no behavior change. | +| `orgCABundle.initImage` | `alpine:3.20` | Image used by the bundle-building initContainer. Only needs `sh` + `cat`. | + +```yaml +# Create the ConfigMap out-of-band (gitops, kubectl, etc.): +apiVersion: v1 +kind: ConfigMap +metadata: + name: org-ca-bundle +data: + ca.crt: | + -----BEGIN CERTIFICATE----- + ...your org CA... + -----END CERTIFICATE----- +--- +# Then point the chart at it: +orgCABundle: + configMapName: org-ca-bundle +``` + +> **Why ConfigMap rather than Secret?** A CA certificate is public material by design — the PKI trust model relies on root CAs being widely distributed. Kubernetes itself uses a ConfigMap for the cluster's own CA distribution (`kube-root-ca.crt`, auto-projected into every namespace), and cert-manager's trust-manager subproject does the same. Use a ConfigMap here; reserve Secret for things that actually need confidentiality (private keys, OAuth client secrets, etc.). + +> **⚠️ ArgoCD footgun — the CA bundle silently won't apply.** The ArgoCD `Application` shown under [On a Nebari cluster (via ArgoCD)](#on-a-nebari-cluster-via-argocd) sets `RespectIgnoreDifferences=true` together with an `ignoreDifferences` rule on `/spec/rayClusterConfig`. With server-side apply, that combination tells ArgoCD to **stop managing every field under `rayClusterConfig`** — which is exactly where this chart injects the initContainer, volumes, volumeMounts, and CA env vars for the head and worker pods. The result is a silent failure: ArgoCD reports a healthy, fully-synced `Application`, but the running RayService never gets the CA bundle, and TLS calls keep failing with `CERTIFICATE_VERIFY_FAILED`. +> +> If you enable `orgCABundle` on a cluster managed by ArgoCD with the example sync policy, you must **narrow the ignore rule** so the CA fields are still reconciled. The broad `/spec/rayClusterConfig` ignore exists only to suppress the autoscaler/runtime mutations KubeRay makes; replace it with targeted pointers (or drop it and ignore only the specific subpaths KubeRay rewrites). After changing it, confirm the head and worker pods actually carry `SSL_CERT_FILE` (`kubectl exec ... -- printenv SSL_CERT_FILE`) rather than trusting the ArgoCD sync status. See [#17](https://github.com/nebari-dev/nebari-rayserve-pack/issues/17) for details. + +**Coverage caveat — httpx default `verify=True`:** httpx hardcodes its SSL context to `cafile=certifi.where()`, which means it **ignores** `SSL_CERT_FILE`. Application code making httpx calls that need to traverse a TLS-inspecting proxy must construct an explicit context: + +```python +import ssl, httpx +client = httpx.Client(verify=ssl.create_default_context()) +# or per-call: httpx.get(url, verify=ssl.create_default_context()) +``` + +`ssl.create_default_context()` with no `cafile=` honors `SSL_CERT_FILE` / `SSL_CERT_DIR` per the standard OpenSSL convention, so it picks up the bundle this chart injected. Other Python HTTP clients (`requests`, `urllib3`, stdlib `urllib`) and most non-Python TLS tooling honor the env vars automatically. + ## Architecture ```mermaid diff --git a/chart/templates/_helpers.tpl b/chart/templates/_helpers.tpl index 6d5c4b2..a918c40 100644 --- a/chart/templates/_helpers.tpl +++ b/chart/templates/_helpers.tpl @@ -62,6 +62,89 @@ Ray serve service name - RayService creates a service named -se {{- printf "%s-serve-svc" (include "nebari-rayserve.fullname" .) }} {{- end }} +{{/* +Whether organization CA bundle injection is enabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.enabled" -}} +{{- if and .Values.orgCABundle .Values.orgCABundle.configMapName -}} +true +{{- end }} +{{- end }} + +{{/* +initContainers block for the combined-CA bundle build step. Renders empty +when orgCABundle injection is disabled. Used by both head and worker pod +specs so the SSL_CERT_FILE bundle exists before the main container starts. +*/}} +{{- define "nebari-rayserve.orgCABundle.initContainers" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: build-ca-bundle + image: {{ .Values.orgCABundle.initImage | quote }} + command: + - sh + - -c + - | + cat /etc/ssl/certs/ca-certificates.crt \ + /var/local/org-ca/ca.crt > /shared/combined-ca.crt + volumeMounts: + - name: org-ca + mountPath: /var/local/org-ca + readOnly: true + - name: combined-ca + mountPath: /shared +{{- end }} +{{- end }} + +{{/* +Volumes block for the org-ca ConfigMap (deployer-supplied) + the shared +emptyDir that the initContainer writes the combined bundle into. Renders +empty when orgCABundle injection is disabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.volumes" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: org-ca + configMap: + name: {{ .Values.orgCABundle.configMapName | quote }} +- name: combined-ca + emptyDir: {} +{{- end }} +{{- end }} + +{{/* +Container volumeMounts for the combined-CA bundle (read-only). The main +Ray container mounts only the combined-ca volume — it never sees the raw +org-ca ConfigMap. Renders empty when orgCABundle injection is disabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.volumeMounts" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: combined-ca + mountPath: /shared + readOnly: true +{{- end }} +{{- end }} + +{{/* +Container env entries pointing the standard OpenSSL trust-store env vars +at the combined-CA bundle. Anything that honors SSL_CERT_FILE / +REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE picks it up automatically. +GIT_SSL_CAINFO is set separately because git's libcurl ignores the three +above and reads only GIT_SSL_CAINFO — without it `pip install git+https://...` +and other git-over-HTTPS calls fail certificate verification. +Renders empty when orgCABundle injection is disabled. +*/}} +{{- define "nebari-rayserve.orgCABundle.env" -}} +{{- if include "nebari-rayserve.orgCABundle.enabled" . -}} +- name: SSL_CERT_FILE + value: /shared/combined-ca.crt +- name: REQUESTS_CA_BUNDLE + value: /shared/combined-ca.crt +- name: CURL_CA_BUNDLE + value: /shared/combined-ca.crt +- name: GIT_SSL_CAINFO + value: /shared/combined-ca.crt +{{- end }} +{{- end }} + {{/* Tolerations for a Ray group (head or worker). Pass the group config (.Values.head or .Values.worker). diff --git a/chart/templates/rayservice.yaml b/chart/templates/rayservice.yaml index ebfec82..b2475ee 100644 --- a/chart/templates/rayservice.yaml +++ b/chart/templates/rayservice.yaml @@ -27,6 +27,14 @@ spec: {{- with .Values.head.runtimeClassName }} runtimeClassName: {{ . }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.initContainers" .) }} + initContainers: + {{- . | nindent 12 }} + {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumes" .) }} + volumes: + {{- . | nindent 12 }} + {{- end }} {{- with (include "nebari-rayserve.tolerations" .Values.head) }} tolerations: {{- . | nindent 12 }} @@ -34,10 +42,15 @@ spec: containers: - name: ray-head image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - {{- with .Values.head.containerEnv }} + {{- $headEnv := concat (.Values.head.containerEnv | default list) (fromYamlArray (include "nebari-rayserve.orgCABundle.env" .)) }} + {{- with $headEnv }} env: {{- toYaml . | nindent 16 }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumeMounts" .) }} + volumeMounts: + {{- . | nindent 16 }} + {{- end }} resources: {{- toYaml .Values.head.resources | nindent 16 }} ports: @@ -68,6 +81,14 @@ spec: {{- with .Values.worker.runtimeClassName }} runtimeClassName: {{ . }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.initContainers" .) }} + initContainers: + {{- . | nindent 14 }} + {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumes" .) }} + volumes: + {{- . | nindent 14 }} + {{- end }} {{- with (include "nebari-rayserve.tolerations" .Values.worker) }} tolerations: {{- . | nindent 14 }} @@ -75,10 +96,15 @@ spec: containers: - name: ray-worker image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - {{- with .Values.worker.containerEnv }} + {{- $workerEnv := concat (.Values.worker.containerEnv | default list) (fromYamlArray (include "nebari-rayserve.orgCABundle.env" .)) }} + {{- with $workerEnv }} env: {{- toYaml . | nindent 18 }} {{- end }} + {{- with (include "nebari-rayserve.orgCABundle.volumeMounts" .) }} + volumeMounts: + {{- . | nindent 18 }} + {{- end }} resources: {{- toYaml .Values.worker.resources | nindent 18 }} {{- with .Values.worker.readinessProbe }} diff --git a/chart/values.yaml b/chart/values.yaml index a51ca38..b25c4d7 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -72,6 +72,56 @@ serve: # num_replicas: 2 serveApplications: [] +# ============================================================================= +# Organization CA Bundle Injection +# ============================================================================= +# Mount an organization-managed CA bundle into the Ray head and worker pods +# so outbound HTTPS calls (model registries, dataset URLs, Hugging Face hub, +# S3, etc.) succeed in environments running a TLS-inspecting proxy +# (Netskope, Zscaler, BlueCoat, internal corporate CAs, etc.). +# +# When `configMapName` is set, the chart adds: +# - An initContainer that concatenates the base image's system trust store +# (/etc/ssl/certs/ca-certificates.crt) with the deployer-supplied CA +# (mounted from the ConfigMap at the key `ca.crt`) into +# /shared/combined-ca.crt +# - A shared emptyDir volume that the main container mounts read-only +# - SSL_CERT_FILE / REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE env vars on both +# head and worker pointing at /shared/combined-ca.crt +# +# When `configMapName` is empty (default), nothing is rendered — the +# resulting RayService is byte-identical to the no-CA-injection case. +# +# Why ConfigMap rather than Secret: a CA certificate is public material +# by design (the entire PKI trust model relies on root CAs being widely +# distributed — Mozilla's bundle ships in every browser/OS, corporate +# inspecting proxy roots are pushed to every device that traverses them). +# Kubernetes itself uses a ConfigMap for the cluster's own CA distribution +# (`kube-root-ca.crt`, auto-projected into every namespace). cert-manager's +# trust-manager subproject distributes CA bundles as ConfigMaps via its +# Bundle CR. We follow that precedent here. +# +# Known coverage gap: httpx clients constructed with the default +# `verify=True` hardcode `cafile=certifi.where()` and ignore SSL_CERT_FILE. +# Application code that needs its httpx calls covered must pass +# `verify=ssl.create_default_context()` (or equivalent) explicitly. +# +# ConfigMap shape (created out-of-band, e.g. by your gitops layer): +# apiVersion: v1 +# kind: ConfigMap +# metadata: +# name: org-ca-bundle +# data: +# ca.crt: | +# -----BEGIN CERTIFICATE----- +# ...your org CA... +# -----END CERTIFICATE----- +orgCABundle: + configMapName: "" + # Image used by the initContainer that builds the combined CA bundle. + # Only needs `sh` and `cat` — defaults to a tiny Alpine image. + initImage: "alpine:3.20" + # ============================================================================= # Ray Image # =============================================================================