Skip to content

Decouple K8s pod readiness from Ray Serve application state#8

Merged
oren-openteams merged 1 commit into
nebari-dev:mainfrom
oren-openteams:fix/decouple-readiness-from-serve
May 1, 2026
Merged

Decouple K8s pod readiness from Ray Serve application state#8
oren-openteams merged 1 commit into
nebari-dev:mainfrom
oren-openteams:fix/decouple-readiness-from-serve

Conversation

@oren-openteams

Copy link
Copy Markdown
Collaborator

Summary

Closes #7.

The chart's worker pods stay 0/1 Ready indefinitely on a fresh install, blocking KubeRay from completing its rollout. The cause is two layered defaults:

  1. KubeRay's default worker readiness/liveness probe chains a raylet healthz check with wget http://localhost:8000/-/healthz | grep success. The Serve check requires both a deployed Serve application AND a Serve HTTP proxy on the local pod.
  2. The chart's defaults are serveApplications: [] (no app deployed) and no explicit proxy_location in serveConfigV2 (Ray Serve ≥2.10 defaults to HeadOnly, so workers don't get local proxies even when an app is deployed).

The result is the worker probe fails forever with the confusing message Readiness probe failed: success — the literal word "success" is the grep match from the first half of the chained command; the pipeline exits non-zero because wget against the dead port 8000 fails.

What this PR does

  1. Overrides the head and worker readiness/liveness probes to drop the Serve healthz check. The new probes check only the raylet healthz endpoint (port 52365), which reflects whether the Ray node itself is healthy (raylet up, resources allocated). Serve application health is tracked separately by the Serve controller and shouldn't gate Kubernetes pod readiness. The chart's serve-svc Service already targets only the head pod, so worker readiness state has no effect on user-visible HTTP routing.

  2. Adds proxy_location: EveryNode to serveConfigV2. With KubeRay placing each Ray node in its own pod, having a local HTTP proxy on each node is typically what users want. Users with a different intent can override.

Verification

# Install chart on a fresh cluster with default values:
helm install rayserve ./chart --namespace ray --create-namespace

# Before this PR: worker stays 0/1 forever, kubectl describe shows
#   "Readiness probe failed: success"
#
# After this PR: head and worker reach 1/1 within ~30s, no manual
# `serve.run()` workaround needed.

I ran helm template against the modified chart to confirm the output is well-formed YAML and the probe overrides land in the right places in the rendered RayService.

Reproduce the original failure

The full diagnosis lives in #7 — short version: install the chart on a clean cluster, observe worker stays 0/1 indefinitely, manually exec into the head pod and run serve.shutdown(); serve.start(proxy_location='EveryNode'); serve.run(SomeApp.bind(), ...) to make the worker become Ready. The current PR removes the need for that manual step.

Considered alternatives

  • Auto-deploy a healthz stub as part of the chart. Rejected: requires solving the import_path problem (no stub is universally importable from arbitrary Ray images), and adds opinionated state to the cluster that users will then have to remove.
  • Make the readiness probe configurable via values. Rejected for the initial fix: punts the design choice to users, and most will hit the broken default first. Could be added as a follow-up if maintainers want it.
  • Documentation only. Rejected: leaves the broken default install experience.

Notes

  • The probe values (initialDelaySeconds, failureThreshold, etc.) match KubeRay's defaults for those fields, just with the Serve check removed from the command.
  • This change does not affect users who already deploy serveApplications and configure proxy_location: EveryNode — for them, the probe was passing anyway. It only changes behavior for fresh installs and users without a Serve app deployed.

Override KubeRay's default worker readiness/liveness probes which chain a
raylet healthz check with `wget http://localhost:8000/-/healthz | grep
success`. The Serve check requires a deployed Serve application AND a
Serve HTTP proxy on the local pod — neither holds on a fresh cluster
install (default `serveApplications: []`, default Ray Serve
`proxy_location: HeadOnly` in Ray ≥2.10), so the worker stays 0/1 Ready
indefinitely and KubeRay can't complete its rollout.

Kubernetes pod readiness should reflect whether the Ray node itself is
healthy (raylet up, resources allocated). Serve application health is
tracked separately by the Serve controller and need not gate K8s
readiness. The chart's serve-svc Service already targets only the head
pod, so worker readiness state has no effect on user-visible HTTP
routing.

Changes:

- chart/values.yaml: add `serve.proxyLocation` (default `EveryNode`),
  `worker.readinessProbe`, `worker.livenessProbe`, `head.readinessProbe`,
  `head.livenessProbe`. All probe fields are tunable; setting either
  worker probe to null (~) suppresses it and lets KubeRay's default
  apply. Head probes default to {} (KubeRay default applies); users can
  set explicit probes to override.
- chart/templates/rayservice.yaml: template proxy_location from values;
  conditionally render head/worker probes from values via `with`.
- chart/Chart.yaml: bump version 0.2.0 → 0.3.0.
- .github/workflows/test.yaml: bump worker.replicas from 0 to 1 and add
  a wait for worker Ready condition. The previous test only ran with
  zero workers, so it didn't exercise the failure mode this PR fixes.

Closes #7
@oren-openteams oren-openteams force-pushed the fix/decouple-readiness-from-serve branch from e89e1fa to 5ce6bdb Compare May 1, 2026 12:02
@oren-openteams oren-openteams merged commit 5456c40 into nebari-dev:main May 1, 2026
3 checks passed
@oren-openteams oren-openteams deleted the fix/decouple-readiness-from-serve branch May 1, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker readiness probe fails on fresh cluster: requires Ray Serve EveryNode proxies + at least one deployed app

1 participant