What happened
When the operator-provisioned OIDC client Secret doesn't exist yet, the hub crash-loops at startup:
File "00-gateway-auth.py", line 631, in <module>
File "00-gateway-auth.py", line 565, in _read_secret_file
FileNotFoundError: [Errno 2] No such file or directory: '/etc/oauth/issuer-url'
The chart mounts the secret with optional: true, and the values.yaml comments read as though the chart degrades gracefully when Keycloak isn't wired ("Empty is OK if...", dummy authenticator as the default). In practice 00-gateway-auth.py reads /etc/oauth/issuer-url unconditionally; with the secret missing, kubelet mounts an empty directory and the read throws.
How we hit it
The NebariApp wasn't being reconciled because the namespace was missing the nebari.dev/managed=true label, so the operator never created the Secret. The operator's status condition (NamespaceNotOptedIn) pointed at the real cause, but the hub traceback is what a deployer sees first, and it doesn't hint at the Secret name or the NebariApp at all.
Once the label was fixed and the operator created the Secret, the crash-looping pod recovered on its own (kubelet re-syncs optional secret volumes), which is nice, but the intermediate state is confusing.
Suggestion
Either of these would have saved a debugging round-trip:
- Fall back to dummy auth with a loud warning when
/etc/oauth is empty, matching what the comments imply, or
- Fail with an explicit message: "OIDC client secret not found; the nebari-operator has not provisioned it yet, check
kubectl get nebariapp -n <ns> status".
Given the secret appearing later heals the pod automatically, option 2 with a clear message (still crashing, so the pod retries) may actually be the better behavior; option 1 risks briefly exposing a dummy-auth hub on a real deployment.
Seen on 0.1.0-alpha.16 with nebari-operator 0.1.0-alpha.20.
What happened
When the operator-provisioned OIDC client Secret doesn't exist yet, the hub crash-loops at startup:
The chart mounts the secret with
optional: true, and the values.yaml comments read as though the chart degrades gracefully when Keycloak isn't wired ("Empty is OK if...", dummy authenticator as the default). In practice00-gateway-auth.pyreads/etc/oauth/issuer-urlunconditionally; with the secret missing, kubelet mounts an empty directory and the read throws.How we hit it
The NebariApp wasn't being reconciled because the namespace was missing the
nebari.dev/managed=truelabel, so the operator never created the Secret. The operator's status condition (NamespaceNotOptedIn) pointed at the real cause, but the hub traceback is what a deployer sees first, and it doesn't hint at the Secret name or the NebariApp at all.Once the label was fixed and the operator created the Secret, the crash-looping pod recovered on its own (kubelet re-syncs optional secret volumes), which is nice, but the intermediate state is confusing.
Suggestion
Either of these would have saved a debugging round-trip:
/etc/oauthis empty, matching what the comments imply, orkubectl get nebariapp -n <ns>status".Given the secret appearing later heals the pod automatically, option 2 with a clear message (still crashing, so the pod retries) may actually be the better behavior; option 1 risks briefly exposing a dummy-auth hub on a real deployment.
Seen on 0.1.0-alpha.16 with nebari-operator 0.1.0-alpha.20.