Summary
On clusters with a TLS-inspecting egress proxy (corporate SSL inspection / mitmproxy), the proxy's CA has to be in the trust store of every container that makes outbound HTTPS. Currently this is not exposed in nebari-data-science-pack values — deployers have to override c.KubeSpawner.volumes and c.KubeSpawner.volume_mounts via hub.extraConfig Python to append the mount to spawned singleuser pods. ~30 lines of load-bearing Python that mirror the pattern already merged in nebari-nebi-pack#38 and nebari-rayserve-pack#16.
Why the z2jh-native paths don't work today
config/jupyterhub/01-spawner.py sets c.KubeSpawner.volumes = [...] and c.KubeSpawner.volume_mounts = [...] (assignment, not append). That overwrites z2jh's singleuser.storage.extraVolumes and singleuser.extraFiles — the standard z2jh idioms for injecting per-pod files. So deployers can't use the documented z2jh path and have to work around by loading a 99-*.py snippet from hub.extraConfig (which fires AFTER the chart configs) that APPENDS to the already-set lists.
Proposal
Two options, pick one (both is best):
Option A — new orgCABundle.configMapName value (mirrors nebi-pack)
When set, the chart:
- Mounts the ConfigMap into the hub pod at
/etc/ssl/certs/org-ca.crt (via subPath) and sets SSL_CERT_FILE — so the hub can reach any HTTPS endpoint behind the intercepting proxy.
- Appends the same volume + mount to
c.KubeSpawner.volumes / .volume_mounts in 01-spawner.py (or in a new bundled snippet loaded before extraConfig) — so singleuser pods trust the CA too.
- Sets
SSL_CERT_FILE in the singleuser env via c.KubeSpawner.environment.
Values shape:
orgCABundle:
configMapName: org-ca-bundle # empty = feature off (default)
configMapKey: ca-bundle.crt # optional; defaults to ca-bundle.crt
Option B — fix 01-spawner.py to append instead of overwrite
Change:
c.KubeSpawner.volumes = [ ...chart stuff... ]
to:
c.KubeSpawner.volumes = list(c.KubeSpawner.volumes or []) + [ ...chart stuff... ]
(and same for volume_mounts). Then z2jh's native singleuser.storage.extraVolumes and singleuser.extraFiles work as documented, and deployers can inject the CA (and anything else) via those without chart-side changes.
Option B is smaller but doesn't help the hub pod (a separate Deployment, not a spawned singleuser). Recommend both — A for the hub pod's own trust, B for a general fix to the chart's z2jh-compat break.
Rust tooling nuance
Worth surfacing in the values docstring: Rust's rustls-native-certs (used by pixi, rattler, uv) honors SSL_CERT_FILE but does NOT iterate /etc/ssl/certs/. So a bare subPath mount at /etc/ssl/certs/org-ca.crt is invisible to those tools without the env var. This is a subtle difference from Go and Python tooling. Same finding as nebari-nebi-pack#38; the chart should set the env var whenever the feature is enabled.
Precedent
Both landed clean; retires the deployer-side Python workaround.
Impact
Retires the ~30-line hub.extraConfig.99-org-ca-bundle Python block that private-VPC deployments currently maintain out-of-band. Also opens the door for z2jh's standard idioms (singleuser.storage.extraVolumes, singleuser.extraFiles) to work — useful for other volume-mount cases beyond just the CA bundle.
Environment context
- EKS on private VPC, mitmproxy TLS interception on outbound HTTPS
- Rust-based tools (pixi/rattler/uv) require
SSL_CERT_FILE in singleuser env
- Verified on
nebari-data-science-pack@nebari-data-science-pack-0.1.0
Summary
On clusters with a TLS-inspecting egress proxy (corporate SSL inspection / mitmproxy), the proxy's CA has to be in the trust store of every container that makes outbound HTTPS. Currently this is not exposed in
nebari-data-science-packvalues — deployers have to overridec.KubeSpawner.volumesandc.KubeSpawner.volume_mountsviahub.extraConfigPython to append the mount to spawned singleuser pods. ~30 lines of load-bearing Python that mirror the pattern already merged innebari-nebi-pack#38andnebari-rayserve-pack#16.Why the z2jh-native paths don't work today
config/jupyterhub/01-spawner.pysetsc.KubeSpawner.volumes = [...]andc.KubeSpawner.volume_mounts = [...](assignment, not append). That overwrites z2jh'ssingleuser.storage.extraVolumesandsingleuser.extraFiles— the standard z2jh idioms for injecting per-pod files. So deployers can't use the documented z2jh path and have to work around by loading a99-*.pysnippet fromhub.extraConfig(which fires AFTER the chart configs) that APPENDS to the already-set lists.Proposal
Two options, pick one (both is best):
Option A — new
orgCABundle.configMapNamevalue (mirrors nebi-pack)When set, the chart:
/etc/ssl/certs/org-ca.crt(via subPath) and setsSSL_CERT_FILE— so the hub can reach any HTTPS endpoint behind the intercepting proxy.c.KubeSpawner.volumes/.volume_mountsin01-spawner.py(or in a new bundled snippet loaded before extraConfig) — so singleuser pods trust the CA too.SSL_CERT_FILEin the singleuser env viac.KubeSpawner.environment.Values shape:
Option B — fix
01-spawner.pyto append instead of overwriteChange:
to:
(and same for
volume_mounts). Then z2jh's nativesingleuser.storage.extraVolumesandsingleuser.extraFileswork as documented, and deployers can inject the CA (and anything else) via those without chart-side changes.Option B is smaller but doesn't help the hub pod (a separate Deployment, not a spawned singleuser). Recommend both — A for the hub pod's own trust, B for a general fix to the chart's z2jh-compat break.
Rust tooling nuance
Worth surfacing in the values docstring: Rust's
rustls-native-certs(used bypixi,rattler,uv) honorsSSL_CERT_FILEbut does NOT iterate/etc/ssl/certs/. So a bare subPath mount at/etc/ssl/certs/org-ca.crtis invisible to those tools without the env var. This is a subtle difference from Go and Python tooling. Same finding asnebari-nebi-pack#38; the chart should set the env var whenever the feature is enabled.Precedent
nebari-nebi-packPR #38 — feat: exposeorgCABundle.configMapNamefor TLS trust injectionnebari-rayserve-packPR #16 — same shape for Ray head/worker podsBoth landed clean; retires the deployer-side Python workaround.
Impact
Retires the ~30-line
hub.extraConfig.99-org-ca-bundlePython block that private-VPC deployments currently maintain out-of-band. Also opens the door for z2jh's standard idioms (singleuser.storage.extraVolumes,singleuser.extraFiles) to work — useful for other volume-mount cases beyond just the CA bundle.Environment context
SSL_CERT_FILEin singleuser envnebari-data-science-pack@nebari-data-science-pack-0.1.0