Skip to content

api: fix /oauth2/jwks 500 — add bcpkix runtime dep, cache JWKS#190

Merged
ExtraToast merged 1 commit into
mainfrom
platform/jwks-cache-and-resilience
May 11, 2026
Merged

api: fix /oauth2/jwks 500 — add bcpkix runtime dep, cache JWKS#190
ExtraToast merged 1 commit into
mainfrom
platform/jwks-cache-and-resilience

Conversation

@ExtraToast

@ExtraToast ExtraToast commented May 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Vault OIDC sign-in (and every other OIDC RP) failed at id_token signature verification:

Provider.VerifyIDToken: invalid id_token: failed to verify signature:
fetching keys oidc: get keys failed: 500 Internal Server Error
{"timestamp":"...","status":500,"error":"Internal Server Error","path":"/oauth2/jwks"}

Stacktrace from the running api pod (fresh 6-min-old pod with valid Vault token — rules out token expiry):

java.lang.ClassNotFoundException: org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter
  at com.nimbusds.jose.jwk.PEMEncodedKeyParser.<clinit>(PEMEncodedKeyParser.java:48)
  at com.nimbusds.jose.jwk.JWK.parseFromPEMEncodedObjects(JWK.java:977)
  at net.blueshell.api.platform.oidc.OidcJwtConfig.vaultTransitJwkSource$lambda$0(OidcJwtConfig.kt:50)

Nimbus's PEM parser requires bcpkix; only bcprov was on the runtime classpath. The Vault-Transit JWKS path has been broken since auth.transit.enabled=true first shipped.

Verified locally

Reproduced the production failure end-to-end against a real Vault in docker-compose, then verified the fix flips the same path green:

# RED on bug-state (bcpkix removed from build.gradle.kts):
> Expected /oauth2/jwks to publish ≥1 key; got []

# GREEN with fix (4 consecutive runs):
> Test result: SUCCESS (1 tests, 1 passed, 0 failed, 0 skipped)

What's in the PR

Fix:

  1. Add org.bouncycastle:bcpkix-jdk18on:1.81 as runtimeOnly to :services:api (lockfile refreshed). Root-cause fix.
  2. Replace the inline JWKSource lambda with a cached VaultTransitJwkSource: fetches Vault Transit public keys at startup and on a fixed schedule (auth.transit.jwks-refresh-ms, default 5 min), serves the cached JWKSet on every request, preserves the previous set on refresh failure. A transient Vault hiccup never produces a 500 on JWKS again.
  3. VaultTransitJwkSource catches Throwable, not Exception — the bcpkix-missing failure is a NoClassDefFoundError (Error subclass), so the narrow catch let it abort bean creation instead of degrading.

Verification (regression):

  • Unit tests on VaultTransitJwkSource: initial-success, initial-failure-then-recover (uses NoClassDefFoundError, the actual prod failure mode), scheduled-refresh-failure-preserves-cache, empty-result-preserves-cache.
  • New Playwright system test VaultTransitJwksPlaywrightTest (@Tag("vault-oidc-live")) drives /oauth2/jwks against a real api wired to a real Vault.
  • New vaultOidcLiveTest gradle task with the right tag filter.
  • New services/vault/docker-compose.yml (oidc-e2e profile) — dev-mode Vault + one-shot init enabling transit and creating the RSA api-jwt key.
  • New docker-compose.oidc-e2e.yml overlay — flips api into transit mode and drops the listmonk dep so the api can boot without Listmonk for OIDC e2e.
  • Dockerfile-dev pinned to eclipse-temurin:21-jdk-noble — the unpinned 21-jdk tag is now ubuntu26.04, which Playwright 1.58 can't install browsers for on arm64.

Repro / local run

docker compose -f docker-compose.yml -f docker-compose.oidc-e2e.yml \
  --profile oidc-e2e up -d
curl http://localhost:8080/oauth2/jwks   # 200 with keys[…]
./gradlew :services:system-tests:vaultOidcLiveTest

Test plan

  • ./gradlew :services:api:test --tests VaultTransitJwkSourceTest — 4/4
  • Local repro red (bcpkix removed) → "Expected ≥1 key; got []"
  • Local repro green (with fix) — 4 consecutive vaultOidcLiveTest runs SUCCESS
  • After merge + deploy: curl https://v2.esa-blueshell.nl/api/oauth2/jwks returns 200
  • Vault OIDC sign-in via https://vault.esa-blueshell.nl completes end-to-end
  • Headlamp OIDC sign-in via https://kube.esa-blueshell.nl completes end-to-end

Out of scope

  • Vault token auto-renewal for the api pod (still agent-pre-populate-only). Cache/resilience changes here decouple JWKS availability from token freshness, which is the user-visible failure mode; renewal hygiene is a separate platform PR.
  • Caching public-key reads inside VaultTransitJwtEncoder.encode. Low-rate path; optional follow-up.

@ExtraToast
api: fix /oauth2/jwks 500 — add missing bcpkix runtime dep, cache JWKS
8eb0608 
Vault OIDC login was failing at id_token signature verification because
/oauth2/jwks returned 500 on every call. Production stacktrace:

  ClassNotFoundException: org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter
    at com.nimbusds.jose.jwk.PEMEncodedKeyParser.<clinit>(...)
    at com.nimbusds.jose.jwk.JWK.parseFromPEMEncodedObjects(...)
    at OidcJwtConfig.vaultTransitJwkSource$lambda$0(OidcJwtConfig.kt:50)

Nimbus's PEM parser requires bcpkix, which was not on the runtime
classpath — only bcprov was, pulled transitively. Adding bcpkix-jdk18on
as runtimeOnly resolves the immediate failure.

While here, replace the inline JWKSource lambda — which called Vault on
every JWKS request and let every exception surface as a 500 — with a
cached JWKSource that refreshes on a fixed schedule (5m default, knob
via auth.transit.jwks-refresh-ms). On refresh failure it logs and keeps
the previously cached set, so a transient Vault hiccup no longer breaks
OIDC for every relying party.

Unit tests cover: initial fetch success, initial fetch failure recovers
on next schedule, scheduled refresh failure preserves cache, empty
refresh result preserves cache.
@ExtraToast ExtraToast added the bug Something isn't working label May 11, 2026
@ExtraToast ExtraToast self-assigned this May 11, 2026
@ExtraToast ExtraToast merged commit fda4be9 into main May 11, 2026
16 checks passed
@ExtraToast ExtraToast mentioned this pull request May 11, 2026
Merged
5 tasks
ExtraToast added a commit that referenced this pull request May 11, 2026
@ExtraToast
api: stamp kid/alg/use on Transit JWKS so Vault's go-oidc verifier pi…
013c965 
…cks the key

After #190 fixed the /oauth2/jwks 500, the next Vault sign-in failed with:

  Provider.VerifyIDToken: invalid id_token: failed to verify signature:
  failed to verify id token signature: invalid signature

Diagnosed by signing a known payload via Vault Transit and verifying with
the published JWKS key directly — the crypto is fine, the JWKS public key
matches the Transit-engine public key, RSA-PKCS1v15-SHA256 verifies
cleanly. So the failure was upstream of the actual signature check.

Root cause: VaultTransitJwtEncoder writes kid="api-jwt:v1" into the JWT
header, but VaultTransitJwkSource published JWK objects with no kid/alg/
use (Nimbus's RSAKey.parseFromPEMEncodedObjects leaves those unset). The
verifier (coreos/go-oidc-v3, used by Vault's OIDC auth) filters JWKS keys
by kid:

  for _, key := range keys {
      if keyID == "" || key.KeyID == keyID {
          if payload, err := jws.Verify(&key); err == nil { return payload }
      }
  }
  return errors.New("failed to verify id token signature")

JWT header.kid="api-jwt:v1" != JWK.KeyID="" → every key skipped → error.

Fix is mechanical: rebuild each published JWK with kid="$keyName:v$version"
(matching the encoder), alg=RS256, use=sig.

Unit test now asserts kid/alg/use on the published JWK. System test
(VaultTransitJwksPlaywrightTest) asserts the same on /oauth2/jwks against
the live api. Verified locally: signing-then-verifying round-trips
through the JWKS public key.
@ExtraToast ExtraToast mentioned this pull request May 11, 2026
Merged
4 tasks
ExtraToast added a commit that referenced this pull request May 11, 2026
@ExtraToast
api: stamp kid/alg/use on Transit JWKS so Vault's go-oidc verifier pi…
58ea6dc 
…cks the key (#192)

After #190 fixed the /oauth2/jwks 500, the next Vault sign-in failed with:

  Provider.VerifyIDToken: invalid id_token: failed to verify signature:
  failed to verify id token signature: invalid signature

Diagnosed by signing a known payload via Vault Transit and verifying with
the published JWKS key directly — the crypto is fine, the JWKS public key
matches the Transit-engine public key, RSA-PKCS1v15-SHA256 verifies
cleanly. So the failure was upstream of the actual signature check.

Root cause: VaultTransitJwtEncoder writes kid="api-jwt:v1" into the JWT
header, but VaultTransitJwkSource published JWK objects with no kid/alg/
use (Nimbus's RSAKey.parseFromPEMEncodedObjects leaves those unset). The
verifier (coreos/go-oidc-v3, used by Vault's OIDC auth) filters JWKS keys
by kid:

  for _, key := range keys {
      if keyID == "" || key.KeyID == keyID {
          if payload, err := jws.Verify(&key); err == nil { return payload }
      }
  }
  return errors.New("failed to verify id token signature")

JWT header.kid="api-jwt:v1" != JWK.KeyID="" → every key skipped → error.

Fix is mechanical: rebuild each published JWK with kid="$keyName:v$version"
(matching the encoder), alg=RS256, use=sig.

Unit test now asserts kid/alg/use on the published JWK. System test
(VaultTransitJwksPlaywrightTest) asserts the same on /oauth2/jwks against
the live api. Verified locally: signing-then-verifying round-trips
through the JWKS public key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@ExtraToast ExtraToast

Labels

bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ExtraToast