Skip to content

api: real actuator health probes + raise k8s startup budget#186

Merged
ExtraToast merged 1 commit into
mainfrom
fix-api-health-probes
May 11, 2026
Merged

api: real actuator health probes + raise k8s startup budget#186
ExtraToast merged 1 commit into
mainfrom
fix-api-health-probes

Conversation

@ExtraToast

Copy link
Copy Markdown
Contributor

Summary

  • Replace tcpSocket liveness/readiness with httpGet against Spring Boot's /actuator/health/{liveness,readiness} on the management port (8081). Tomcat binding alone said nothing about Flyway, Hibernate, Vault, or the OAuth2 AS being ready — pods were going Ready while half-initialized.
  • Add a dedicated @Order(0) SecurityFilterChain matching EndpointRequest.toAnyEndpoint() that permits all and skips the HTTPS-redirect filter so the kubelet's plain-HTTP probes aren't 302'd.
  • Expose containerPort: 8081 name: management and bump startupProbe.failureThreshold from 60 to 180 (5s × 180 = 15 min) so slow cold boots aren't crashlooped while startup-optimization work is still pending.

Test plan

  • CI: full :services:api:test passes (architecture tests pass locally; full suite needs the CI MariaDB).
  • Deploy to cluster and confirm: pod stays in Init/Startup until Flyway + Hibernate + Vault + SAS warm up, then flips to Ready.
  • kubectl exec into a pod and curl -fsS http://localhost:8081/actuator/health/liveness and /readiness return 200 with {"status":"UP"}.
  • External Gatus probe at https://v2.esa-blueshell.nl/api/health still returns 200 (unchanged path on port 8080).
  • Prometheus scrape of /actuator/prometheus still works.

The kubelet was marking the api Ready as soon as Tomcat bound port
8080, well before Flyway migrations, Hibernate metadata scan, Vault
Agent handshake and the OAuth2 Authorization Server had finished
initializing — in-flight requests could land on half-initialized
beans. Switch liveness/readiness to httpGet probes against
/actuator/health/{liveness,readiness} on the management port (8081),
which only report UP once AvailabilityState and all HealthIndicators
agree. A dedicated @order(0) SecurityFilterChain matching
EndpointRequest.toAnyEndpoint() permits unauthenticated access and
bypasses the HTTPS redirect that breaks plain-HTTP kubelet probes.
Startup budget bumped to 15 min (5s * 180) so slow cold boots aren't
crashlooped while startup optimisation lands.
@ExtraToast ExtraToast merged commit 4cb684d into main May 11, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant