feat(auth): support split-horizon OIDC for private-VPC clusters

[oren-openteams]

Closes #158.

Summary

KeyCloakConfig.build() and the module-level configure() in 00-gateway-auth.py derive all four OIDC URLs (authorize / token / userinfo / end_session) from a single issuer. On private-VPC clusters where the external Keycloak hostname resolves ONLY outside the cluster (browser side, via a private DNS zone or external resolver), the hub cannot resolve issuer from inside the cluster and the token + userinfo backchannel legs fail. Deployers work around this by overriding the entire authenticator via hub.extraConfig — a ~40-line GenericOAuthenticator block with hardcoded split URLs.

This PR adds first-class split-horizon support so that workaround retires.

Values surface — two new opt-in fields, both default empty

No behaviour change on existing deploys.

• keycloak.backchannelURL (top-level) — base URL used for backchannel legs. The realm suffix (/realms/<realm>) is appended by the helper. Typical value: http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080.
• jupyterhub.custom.keycloak-backchannel-issuer-url (explicit override) — full backchannel issuer URL INCLUDING /realms/<realm>. Provided for the rare case where the backchannelURL + realm pattern cannot express the desired shape.

Chart-side wiring

• New helper nebari-data-science-pack.keycloakBackchannelIssuerURL in _helpers.tpl. Order of precedence: explicit override → backchannelURL + realm → empty (= no split-horizon). Emits without the /protocol/openid-connect suffix so the Python side owns the URL shape.
• templates/hub-config.yaml emits the helper's output into _CHART_DERIVED under key keycloak-backchannel-issuer-url, so get_chart_config returns the deployer's value when set and the chart-rendered default otherwise (mirrors the existing keycloak-token-url pattern).

Python-side wiring

config/jupyterhub/00-gateway-auth.py:

• KeyCloakConfig.build() gains an optional backchannel_issuer kwarg. When non-empty, token_url and userdata_url are derived from it; authorize_url and end_session_url keep using issuer. The dataclass's issuer field itself is unchanged — that's what downstream validators check against iss claims. Empty string is treated as "no split-horizon" and is byte-identical to the prior single-issuer signature (backward-compat guard).

  @classmethod
  def build(
      cls,
      *,
      issuer: str,
      post_logout_redirect_uri: str,
      backchannel_issuer: str = "",
  ) -> "KeyCloakConfig":
      base = f"{issuer}/protocol/openid-connect"
      bc_base = f"{(backchannel_issuer or issuer)}/protocol/openid-connect"
      return cls(
          issuer=issuer,
          authorize_url=f"{base}/auth",       # BROWSER
          token_url=f"{bc_base}/token",        # HUB backchannel
          userdata_url=f"{bc_base}/userinfo",  # HUB backchannel
          end_session_url=f"{base}/logout",   # BROWSER
          post_logout_redirect_uri=post_logout_redirect_uri,
      )

• configure() gains a matching optional kwarg and threads it through.

• The module-level call site reads keycloak-backchannel-issuer-url via get_chart_config (which already handles the deployer-override + chart-default precedence) and passes it to configure. Missing / empty is fine — the hub keeps the historical single-issuer behaviour.

Runtime contract with Keycloak

For the iss claim in issued tokens to keep matching the external issuer (so JWKS validators still work), run Keycloak with KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true. That flag makes KC dynamically choose the iss value based on the request's Host header, so a token minted via the backchannel URL still embeds the external hostname in iss. This dependency is documented in the keycloak.backchannelURL docstring.

Tests

6 new assertions in tests/unit/test_keycloak_authenticator.py:

• test_kc_config_build_without_backchannel_derives_all_from_issuer — default (backward-compat) shape
• test_kc_config_build_with_backchannel_splits_urls — the split-horizon contract; verifies which URLs use which base
• test_kc_config_build_with_empty_backchannel_matches_default — empty-string backchannel is treated as no-split-horizon (footgun guard: Helm renders unset values as "" and that must not produce //protocol/openid-connect/... URLs)
• test_configure_wires_backchannel_issuer_onto_authenticator — end-to-end through configure()
• test_configure_without_backchannel_leaves_single_issuer_shape — backward-compat guard
• test_configure_treats_empty_backchannel_string_as_no_split_horizon — same footgun guarded at the configure() layer

Full test_keycloak_authenticator.py suite (23 pre-existing + 6 new):

$ python -m pytest tests/unit/test_keycloak_authenticator.py -v
=============== 29 passed in 1.34s ===============

Related

Companion PR #160 adds the hub-side org-CA-bundle wiring — the two together let the chart-native KeyCloakOAuthenticator fully replace the ~70-line hub.extraConfig override that private-VPC deployments currently maintain out-of-band.

[oren-openteams]

Verified end-to-end on a private-VPC dev cluster (combined with #160 for the hub CA trust).

• keycloak.backchannelURL: http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080 — chart derives keycloak-token-url / keycloak-backchannel-issuer-url from it, 00-gateway-auth.py picks them up
• Browser: authorize + logout hit the external Keycloak URL as expected
• Hub: token + userinfo go to the in-cluster Service — no external-DNS resolution needed, no NLB hop, iss claim on tokens still matches the external URL (with KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true on the Keycloak side, as documented)
• Let me drop ~40 lines of hub.extraConfig that had been hard-coding the split URLs via a GenericOAuthenticator override — chart-native KeyCloakOAuthenticator fully replaces it

The one thing I hit that turned out to be out of scope for this PR is worth flagging: update_auth_model() calls a third Keycloak endpoint — the Admin API at realm_api_url — that auto-derives from the external issuer URL and thus hit the same DNS gap in-cluster. Fail-closed behavior meant /shared/<group> mounts and jupyterlab-profiles gating both stayed empty until I bridged it via the KC_REALM_API_URL env-var escape hatch. Opened #162 to propose extending the same backchannel treatment to that third leg, either by widening backchannelURL to cover the admin URL by convention or adding an optional adminBackchannelURL explicit override.

CI is green except one hub_login_page job that timed out during kind cluster setup with kubectl get pv ... timed out after 15 seconds — an infra flake unrelated to the PR (setup step, before any test code ran). Happy to push an empty commit to re-trigger if reviewers want.