feat(auth): extend split-horizon OIDC to Keycloak Admin API (realm_api_url)

[oren-openteams]

Summary

PR #161 gave the hub split-horizon for two of the three Keycloak backchannel legs — token_url and userdata_url. But KeyCloakOAuthenticator also calls a third Keycloak endpoint that #161 didn't cover: the Admin API at realm_api_url, used by update_auth_model() and refresh_user() to filter group membership by client role. On private-VPC clusters where the external Keycloak hostname doesn't resolve in-cluster (the exact scenario #158/#161 addressed), this fails the same way — the auto-derived URL points at an unreachable hostname.

Failure mode

config/jupyterhub/00-gateway-auth.py derives realm_api_url from the issuer URL read out of the OIDC-client Secret:

_issuer = _read_secret_file(_secret_dir, "issuer-url")   # external URL
_realm_api_url = (
    os.environ.get("KC_REALM_API_URL")
    or _derive_realm_api_url(_issuer)                     # -> /admin/... same external host
)

On a private-VPC deploy this yields https://keycloak.<external>/admin/realms/<realm>, which the hub pod can't resolve. Downstream:

• update_auth_model() catches the exception and stamps auth_state["groups_with_permission_to_mount"] = []
• _resolve_allowed_profiles() catches the exception and stamps auth_state["allowed_jupyterlab_profiles"] = []
• Spawner's _get_user_groups() sees the empty key present in auth_state and skips its fallback to auth_state.get("groups", [])
• Result: no /shared/<group> mounts, no keycloak-gated profiles, silent — the only signal is a WARNING in the hub log

The chart's KC_REALM_API_URL env-var escape hatch works as a bridge (I've verified it end-to-end on a real deploy) but it means every private-VPC deployer has to know about it and configure it out-of-band.

Proposal — universal backchannel base, plus targeted override

Two additive changes to keep the existing backchannelURL mental model:

1. Widen keycloak.backchannelURL to be the base for all three backchannel legs

The docstring already frames backchannelURL as "base URL used for backchannel legs" — extend that to cover the admin URL derivation too. <backchannelURL>/admin/realms/<realm> becomes the default realm_api_url when backchannelURL is set. No new value; existing users of #161 get the Admin-API leg for free.

2. Optional keycloak.adminBackchannelURL explicit override

For the rare case where the Admin API sits somewhere the backchannelURL + /admin/realms/<realm> pattern doesn't reach (different port, different path prefix, separate admin gateway). Mirrors #161's custom.keycloak-backchannel-issuer-url explicit override.

Precedence at the Python side

Same layering shape as #161 already established for keycloak-token-url:

os.environ.get("KC_REALM_API_URL")                               # legacy env override, kept
  or get_chart_config("keycloak-admin-api-url")                  # NEW — chart-derived
  or _derive_realm_api_url(_issuer)                              # existing fallback

Helper (_helpers.tpl) emits keycloak-admin-api-url with the precedence:

explicit `keycloak.adminBackchannelURL`
  else `keycloak.backchannelURL` + "/admin/realms/" + `keycloak.realm`
  else "" (Python falls back to issuer-derived)

Fully backward-compat — deploys not on private-VPC never set backchannelURL so admin URL stays issuer-derived exactly as today.

Related

• Closes the third backchannel leg feat: support split-horizon OIDC URLs (separate browser vs backchannel) so hub can reach in-cluster Keycloak on private-VPC deploys #158/feat(auth): support split-horizon OIDC for private-VPC clusters #161 opened
• Suggest also adding a hub-log INFO line at update_auth_model startup echoing which realm_api_url the authenticator resolved to — the current failure mode is entirely silent to users unless someone knows to kubectl logs | grep rbac:. Even a one-line "using realm_api_url=" at startup would have saved the discovery time on my side.

Runtime dependency (unchanged from #161)

Keycloak must be running with KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true so tokens minted via the backchannel URL still carry the browser-facing iss claim — same requirement as #161 documented for token/userinfo.

Verification path once implemented

• On a cluster with keycloak.backchannelURL set (no explicit adminBackchannelURL): update_auth_model succeeds, groups_with_permission_to_mount populated, /shared/<group> mounts appear
• On a cluster without backchannelURL: no behavior change, admin URL still derives from issuer
• Explicit adminBackchannelURL takes precedence when set
• KC_REALM_API_URL env var still overrides both (legacy escape hatch preserved)

Happy to open a PR against this if the maintainers agree on shape. Would build on top of #161.