panda-server hard-logs-out hourly vs production proxy: refresh-token TTL shorter than access-token TTL

[lodekeeper]

Summary

Running panda-server (ethpandaops/panda:server-0.35.0) as an OIDC client against the production proxy (https://panda-proxy.ethpandaops.io, authentik issuer .../application/o/panda-proxy/) loses authentication roughly once an hour. Every background token refresh returns 400 invalid_grant, after which all datasource queries 401 until a fresh panda auth login.

Root cause (confirmed empirically): the refresh token's lifetime is much shorter than the access token's, so by the time panda refreshes, the refresh token has already expired.

Environment

• panda CLI + ethpandaops/panda:server-0.35.0, proxy.auth.mode: oidc, client_id: panda-proxy (public client, no secret)
• Provider: authentik, token endpoint .../application/o/token/

Symptom

Once the access token passes its refresh point, every ~60s:

level=warning msg="Background datasource refresh failed" component=proxy-client
error="... token endpoint returned status 400: {\"error\": \"invalid_grant\",
\"error_description\": \"The provided authorization grant or refresh token is invalid, expired, revoked, ...\"}"

All ClickHouse datasources then return 401 missing or invalid Authorization header. Recovers only via panda auth login.

Root cause: refresh-token TTL << access-token TTL

Access token lives 3600s (decoded exp - iat). The refresh token expires in ~15-25 min. Reproduced by replaying panda's exact refresh request by hand (grant_type=refresh_token + client_id=panda-proxy, no secret — identical to pkg/auth/client/client.go's Refresh):

refresh token age	result
~10 min	HTTP 200, rotates fine
~25-27 min	HTTP 400 invalid_grant

panda (correctly) refreshes at DefaultRefreshFraction (50% ≈ 30 min) or near expiry — but the refresh token is already dead by then. The token response carries no refresh_expires_in, so the client cannot discover the short refresh-token TTL and adapt on its own.

This is provider-side: the panda-proxy authentik provider issues refresh tokens with a validity shorter than the access token. Refresh tokens should normally outlive access tokens by a wide margin. Suggested fix: raise the panda-proxy provider's refresh-token validity well above the access-token TTL.

Secondary: proxy.auth.refresh_token_ttl not honored by the server-0.35.0 image

refresh_token_ttl is the intended mitigation (store refreshes at the TTL half-life, before the refresh token dies — pkg/auth/store/store.go needsRefresh). I set proxy.auth.refresh_token_ttl: 10m and restarted, but the running server-0.35.0 image never fires the half-life refresh — the token is never rotated and still dies at ~the same age.

The v0.35.0 source does honor it — verified against the tagged source:

• config.Load() on the real config resolves Proxy.Auth.RefreshTokenTTL = 10m0s
• (&App{}).proxyClientConfig(...) carries RefreshTokenTTL = 10m0s into ClientConfig (pkg/app/app.go)
• a unit test on the real store (RefreshTokenTTL=10m, refresh token issued 6 min ago, access token still fresh) fires exactly one proactive refresh via the half-life branch

So config + code path are correct in the v0.35.0 source, yet the published image doesn't exhibit the behavior. Could you confirm server-0.35.0 is built from the v0.35.0 tag (and includes the refresh_token_ttl → client-store plumbing)? It looks like the image may predate the complete feature.

Impact / interim workaround

Until the provider TTL is raised (or the image honors refresh_token_ttl), the only reliable mitigation is an external refresher that rotates the on-disk refresh token every few minutes — the server re-reads creds per request, so this keeps it alive.