Skip to content

feat(chart): auto-wire hub OAuth from nebariapp.hostname#74

Merged
aktech merged 9 commits into
mainfrom
feat/auto-wire-hub-oauth
May 16, 2026
Merged

feat(chart): auto-wire hub OAuth from nebariapp.hostname#74
aktech merged 9 commits into
mainfrom
feat/auto-wire-hub-oauth

Conversation

@aktech

@aktech aktech commented May 16, 2026

Copy link
Copy Markdown
Member
  • Auto-wire OAUTH_CALLBACK_URL + OAUTH_EXTERNAL_URL into 00-gateway-auth.py from nebariapp.hostname
  • Mount operator-provisioned OIDC client Secret at /etc/oauth by default (hub.extraVolumes / extraVolumeMounts)
  • Auto-wire hub.extraEnv.JUPYTERHUB_OIDC_CLIENT_SECRET via secretKeyRef
  • Derive KC_REALM_API_URL from /etc/oauth/issuer-url (env-var override still wins)
  • Default profile_list with explicit slugs small-instance + medium-instance
  • Add profile_options.image.choices so each profile exposes an image dropdown
  • Workload-oriented profile descriptions
  • Enable rbac.bootstrap by default + auto-derive hubClientId from release/chart name
  • Gate rbac.bootstrap Job on nebariapp.enabled so kind / local-dev installs don't fail post-install
  • Bootstrap reconciles existing group-membership mapper config (forces full.path: true) — fixes silently empty /shared/<group>
  • Bootstrap sets rootUrl / baseUrl / initiate.login.uri on the hub OIDC client — fixes 400 OAuth state mismatch on KC-initiated SSO
  • Unit + integration tests for every new path

aktech added 7 commits May 16, 2026 06:13
Drops the deployer-side boilerplate of setting OAUTH_CALLBACK_URL +
OAUTH_EXTERNAL_URL via hub.extraEnv. The chart now renders these
values directly into 00-gateway-auth.py via Helm replace at template
time, using nebariapp.hostname, and adds the operator-provisioned
oidc-client Secret mount as a default in hub.extraVolumes /
extraVolumeMounts.

Changes:
- 00-gateway-auth.py: read URLs from chart-rendered constants first,
  fall back to OAUTH_CALLBACK_URL env (legacy escape hatch). Add
  _resolve_oauth_urls() helper to encapsulate the precedence.
- templates/hub-config.yaml: substitute __CHART_OAUTH_*__ placeholders
  with values computed from nebariapp.hostname. Skip substitution when
  nebariapp is disabled or hostname is unset, leaving the env-var path
  intact for kind / local-dev deploys.
- values.yaml: add oauth-client volume + mount defaults referencing
  the operator's standard secret name pattern. Document the z2jh
  list-replace gotcha so deployers re-include both entries if they
  override extraVolumes.
- tests/unit: new cases for _resolve_oauth_urls() — chart wins over
  env, env serves as fallback, neither returns None.

Fixes the regression seen on nebari.openteams.ai after switching from
git-branch chart source to published chart 0.1.0-alpha.12: deployers
without the manual OAUTH_* extraEnv wiring fell back to dummy auth.
Every Nebari deployment uses bitnami/keycloakx + nebari-operator
NebariApps, so the chart can self-bootstrap the KC role + group
mapper the spawner needs. Without this, every deployer had to copy
the same rbac.bootstrap block into their values, and forgetting it
left the hub OIDC client with no roles — spawner could not gate
shared mounts and the Roles tab in KC was empty.

- values.yaml: flip enabled to true, default namespace=keycloak +
  kcAdminCredentialSecret=keycloak-admin-credentials so a vanilla
  Nebari install needs no extra config.
- keycloak-rbac-bootstrap-job.yaml: derive hubClientId from
  `jupyterhub-<release>-<chart>` when unset (the operator's
  NebariApp client-id pattern). Removes the deployer-must-set
  failure when the rest of the layout is standard.

Override any field for non-Nebari (kind, BYO-Keycloak) deployments.
…rofiles

Restores the resource-sizing profile selector that was split out in 67a452e.
Without a profile_list, users with a saved profile_slug from earlier chart
versions hit:

  ValueError: No such profile: small-instance. Options include:

Adds explicit slug fields ('small-instance', 'medium-instance') so the
identifiers are stable across display_name changes and match the slugs
users already have saved. Wires custom.profiles → c.KubeSpawner.profile_list
in 01-spawner.py; an empty list disables the selector (single-instance mode).
The post-install Job runs in the 'keycloak' namespace by default,
which only exists when nebari-operator + keycloakx are deployed
alongside (the same precondition as nebariapp). On bare-chart
installs (kind, local-dev) the namespace is absent and the Job
fails post-install with:

  failed post-install: ... namespaces "keycloak" not found

Gate the template on nebariapp.enabled too, so deployers who turn
nebariapp off implicitly opt out of RBAC bootstrap.
Replace bare spec strings ("1 CPU / 2 GB RAM") with workload-oriented
hints so deployers see a use-case alongside the resource numbers.
Matches the classic Nebari profile-selector layout: each instance
profile exposes an Image dropdown so users see (and can later choose
between) the container image they're spawning into.

Single default choice per profile for now, pointing at the chart's
bundled singleuser image. Comment notes that bumping
singleuser.image.tag also requires bumping the profile choices since
z2jh values.yaml cannot reference other values.
…in URL

Two more deployer-side env vars that previously had to be added by
hand are now bundled with the chart:

- hub.extraEnv.JUPYTERHUB_OIDC_CLIENT_SECRET is now a default
  secretKeyRef against the operator-provisioned oidc-client Secret
  (same secret already mounted at /etc/oauth). Token exchange in
  02-jhub-apps.py / 03-nebi-envs.py works out of the box.

- KC_REALM_API_URL is derived in 00-gateway-auth.py from the
  issuer-url file by inserting /admin before /realms/. The
  KC realm admin API lives on the same host as the OIDC issuer for a
  standard KC layout, so deployers no longer need to compute and pin
  the admin URL separately. The explicit KC_REALM_API_URL env var
  still wins for non-standard deployments (KC admin behind a different
  gateway).

Both are gated on the operator Secret being present (optional: true
on the secretKeyRef, file-read inside the OAuth wiring block), so
plain kind / local-dev installs still come up on dummy auth.
@viniciusdc

viniciusdc commented May 16, 2026

Copy link
Copy Markdown
Contributor

this PR is fixing a live issue. On my Hetzner nic-nebari cluster, JupyterHub OAuth login started failing with unauthorized_client / Invalid client credentials from Keycloak at 2026-05-16 12:18 UTC — about 12h after PR #53 (feat(auth): forward access token via Bearer header + stale-token re-fetch) landed on main and ArgoCD synced it in.

Diagnosis chain ruled out the obvious suspects (operator-managed k8s secret stale, Keycloak credential corruption, Envoy interception, duplicate clients) and landed on the same gap this PR addresses:

  • 00-gateway-auth.py:477 reads client_secret=_read_secret_file(_secret_dir, "client-secret") where _secret_dir defaults to /etc/oauth
  • The chart's default hub.extraVolumes/extraVolumeMounts (pre-feat(chart): auto-wire hub OAuth from nebariapp.hostname #74) only mount custom-config; nothing populates /etc/oauth
  • _read_secret_file returns empty → c.KeyCloakOAuthenticator.client_secret = "" → Keycloak rejects the token exchange
  • Confirmed via Python introspection in the running hub pod: env-var-set secret is 32 chars (correct), but oauthenticator.client_secret resolves to "" (sha256 e3b0c44298fc... = empty string)
  • The same secret, supplied via curl against Keycloak's token endpoint, authenticates cleanly (only the deliberately-fake code was rejected) — proving the credentials in the env are correct and the chart-side wiring is the missing piece.

The REPLACE-on-override note you added is exactly the trap. Our deployment's values overrode hub.extraEnv (legacy JUPYTERHUB_OAUTH_CLIENT_SECRET env var) but never touched extraVolumes, so the chart default applied — which pre-#74 lacks the oauth-client mount. From a deployer's perspective the chart silently broke.

I can easily add a workaround (explicit oauth-client mount in our values) to unblock the cluster while this PR is in review. Happy to test the merged version against the same cluster once it lands.

aktech added 2 commits May 16, 2026 15:15
…lient URLs

Two operator-provisioned KC objects shipped with config that silently
broke chart features; rbac bootstrap now reconciles them on every run.

1. ``groups`` client-scope ``group-membership`` mapper with
   ``full.path: "false"`` — token emits ``["admin"]`` (no leading
   slash) but the KC admin API role-groups endpoint returns
   ``["/admin"]``. The hub's ``filter_user_groups_by_role`` intersects
   the two and silently returned ``[]`` for every user, so
   ``/shared/<group>`` never mounted. Bootstrap now detects the mismatch
   and PUTs the full desired config (was: skip when a mapper of that
   name existed, regardless of config).

2. Hub OIDC client with ``rootUrl``/``baseUrl: null`` and no
   ``initiate.login.uri``. KC-initiated SSO flows (account console
   "Sign in", third-party launchers) skip ``/hub/oauth_login`` and hit
   ``/hub/oauth_callback`` directly. JupyterHub has no
   ``oauthenticator-state`` cookie set for the flow and raises
   ``400 OAuth state mismatch``. Bootstrap now patches the client to
   route KC-initiated launches through ``/hub/oauth_login`` first.

Wiring:
- Adds ``rbac.bootstrap.hubExternalUrl`` value (defaults to
  ``https://{nebariapp.hostname}``).
- ``HUB_EXTERNAL_URL`` env on the Job, ``BootstrapConfig.hub_external_url``,
  new ``ensure_hub_client_urls`` step that no-ops if the URL is empty.
- Unit tests for both reconcile paths + the opt-out empty-URL case.

Both root causes are operator-side defaults that need separate fixes
upstream; until then the chart self-heals each install/upgrade.
Integration tests instantiate BootstrapConfig directly. Pass empty
string for hub_external_url so the existing tests stay focused on the
RBAC flow they cover; the new ensure_hub_client_urls path is unit-
tested separately.
@aktech aktech merged commit 0cdf3f2 into main May 16, 2026
14 checks passed
@aktech aktech deleted the feat/auto-wire-hub-oauth branch May 16, 2026 14:32
@aktech aktech mentioned this pull request May 16, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants