Skip to content

DaemonSet health probes fail: /healthz on proxy port has no route #62

@yairfalse

Description

@yairfalse

Summary

The DaemonSet readiness/liveness/startup probes are configured to hit GET /healthz on port 80 (the proxy port). Since port 80 only serves Gateway API routes, /healthz returns "No route found" and the probes fail, causing pods to restart in a loop.

Steps to reproduce

./deploy/deploy-to-kind.sh
kubectl get pods -n rauta-system  # pods show 0/1, restarts increasing
kubectl logs -n rauta-system -l app=rauta | grep healthz
# "No route found for GET /healthz (host: 172.18.0.2:80)"

Expected

Pods become ready and stay running.

Actual

Pods restart every ~30s because all three probes fail. The proxy accepts the TCP connection on port 80 but has no matching route for /healthz.

Workaround

Patch probes to TCP socket check on port 80:

kubectl patch daemonset rauta -n rauta-system --type='json' -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/startupProbe","value":{"tcpSocket":{"port":80},"initialDelaySeconds":3,"periodSeconds":3,"failureThreshold":30}},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe","value":{"tcpSocket":{"port":80},"initialDelaySeconds":10,"periodSeconds":10,"failureThreshold":3}},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe","value":{"tcpSocket":{"port":80},"initialDelaySeconds":3,"periodSeconds":5,"failureThreshold":3}}
]'

Suggested fix

Either:

  1. Add a built-in /healthz route to the proxy that bypasses Gateway API routing
  2. Point probes at the admin server (port 9091) which is always available
  3. Use TCP probes in the DaemonSet manifest

Option 1 is the most robust — dedicated health routes are standard for proxies (envoy, nginx, haproxy all have them).

Discovered during

Luotain blackbox testing — deploying RAUTA to kind via deploy/rauta-daemonset.yaml.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions