feat: support split-horizon OIDC URLs (separate browser vs backchannel) so hub can reach in-cluster Keycloak on private-VPC deploys

[oren-openteams]

Summary

The chart's KeyCloakOAuthenticator (config/jupyterhub/00-gateway-auth.py) derives all four OIDC endpoints from a single issuer via KeyCloakConfig.build():

# config/jupyterhub/00-gateway-auth.py
base = f"{issuer}/protocol/openid-connect"
return cls(
    issuer=issuer,
    authorize_url=f"{base}/auth",       # BROWSER
    token_url=f"{base}/token",           # HUB (backchannel)
    userdata_url=f"{base}/userinfo",     # HUB (backchannel)
    end_session_url=f"{base}/logout",    # BROWSER
    post_logout_redirect_uri=post_logout_redirect_uri,
)

On private-VPC deployments where DNS for the external Keycloak hostname is only served outside the cluster (i.e. browsers can resolve it via a private DNS zone / external resolver, but in-cluster pods cannot), the hub pod cannot reach the external URL for the OAuth backchannel — curl from the hub returns exit code 6 (host not resolved). Meanwhile the iss claim in tokens has to remain the external URL (that's what token validators and browsers see), so we can't just point everything at the in-cluster URL.

Result: deployers hit this every time on private-VPC clusters and have to override the entire authenticator via hub.extraConfig — a ~40-line Python block that instantiates a plain GenericOAuthenticator and hardcodes split URLs (authorize_url external, token_url + userdata_url in-cluster). That's a lot of load-bearing code that only exists because the chart can't express two-URL configuration.

Proposal

Accept an optional backchannel URL that, when set, overrides token_url and userdata_url while authorize_url and end_session_url keep using the primary (external) issuer.

Values shape:

keycloak:
  hostname: keycloak.example.com          # external — used for authorize + iss claim
  # Optional: in-cluster base URL for backchannel calls (token + userinfo).
  # Empty = same as hostname (backward compatible).
  backchannelURL: http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080

Then in KeyCloakConfig.build():

@classmethod
def build(
    cls,
    *,
    issuer: str,
    backchannel_issuer: str = "",
    post_logout_redirect_uri: str = "",
) -> "KeyCloakConfig":
    """Derive KC endpoint URLs from the realm issuer.

    When `backchannel_issuer` is set, `token_url` and `userdata_url` use it
    instead of `issuer`. Enables split-horizon OIDC on private-VPC clusters
    where hub pods can't reach the external Keycloak hostname.
    """
    base = f"{issuer}/protocol/openid-connect"
    bc_base = f"{(backchannel_issuer or issuer)}/protocol/openid-connect"
    return cls(
        issuer=issuer,
        authorize_url=f"{base}/auth",       # browser
        token_url=f"{bc_base}/token",        # hub → in-cluster if set
        userdata_url=f"{bc_base}/userinfo",  # hub → in-cluster if set
        end_session_url=f"{base}/logout",   # browser
        post_logout_redirect_uri=post_logout_redirect_uri,
    )

The chart's templates/hub-config.yaml would compute backchannel_issuer from keycloak.backchannelURL (defaulting to "", so behavior on existing deploys is byte-identical).

Precedent in the ecosystem

• nebari-nebi-pack PR fix: don't invalidate session when IdToken cookie is missing #34 added the analogous pattern for OIDC validation: auth.oidc.issuerURL (external, for iss claim matching) + auth.oidc.discoveryURL (in-cluster, for JWKS fetch). Same two-URL split, same motivation.
• Keycloak itself supports host-dynamic backchannel URLs via KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true — but only for clients that auto-discover from .well-known/openid-configuration. jupyterhub-oauthenticator uses hardcoded URLs, so this KC feature doesn't help.

Impact

Retires the ~40-line hub.extraConfig.99-oauth Python override that private-VPC deployments currently maintain out-of-band. All the other things that block does (auth_state, groups claim, scopes, allow_all, refresh) are already handled by the chart's KeyCloakOAuthenticator — the single-issuer design is the last blocker.

Environment context

• Private-VPC Kubernetes cluster, endpoint_public_access: false
• TLS-inspecting egress proxy on outbound HTTPS
• Keycloak reachable in-cluster at keycloak-keycloakx-http.keycloak.svc.cluster.local:8080 (HTTP, no CA needed)
• External Keycloak URL served only via internal NLB + Envoy Gateway; DNS for that hostname is not served by in-cluster CoreDNS
• Verified on nebari-data-science-pack@nebari-data-science-pack-0.1.0

[oren-openteams]

Update after browsing the chart on main: the raw material is already present — keycloak.serviceHost: keycloak-keycloakx-http.keycloak.svc.cluster.local:8080 and the values comment even mentions "used by the hub <-> nebi token-exchange path when keycloak.hostname is empty" — but 00-gateway-auth.pys KeyCloakConfig.build() still derives all four URLs from a single issuer and does not use serviceHost for the backchannel legs.

Revised scope: wire keycloak.serviceHost (or a new keycloak.backchannelURL) through templates/hub-config.yaml into an env var (e.g. KC_BACKCHANNEL_ISSUER_URL) read by configure() and passed to KeyCloakConfig.build(backchannel_issuer=...). When set, token_url and userdata_url use the backchannel form while authorize_url and end_session_url keep using the primary issuer.

Blocker for retiring the deployer-side 99-oauth override on private-VPC clusters where in-cluster CoreDNS does not resolve the external Keycloak hostname. Related: #159 (hub-side CA mount) — the two together let the chart-native KeyCloakOAuthenticator fully replace the override.