Ops hardening: graceful shutdown, readiness probe, production metrics

[balegas]

Graceful shutdown (engine): axum::serve has no with_graceful_shutdown and no SIGTERM handler (apps/engine/src/main.rs:67) — the process is hard-killed, dropping in-flight appends and long-polls (recovered only via at-least-once re-peek). Add SIGTERM handling: stop accepting, complete in-flight appends, close long-polls cleanly, stop the ingestor after its current batch (post-advance). The TS ds-server/api-server already handle SIGTERM.

Readiness vs liveness: /health returns a static "ok" before Postgres or DS are reachable (http.rs:16). Add /ready gated on {PG connected, slot exists and streaming, DS reachable}; keep /health as pure liveness. Optionally upstream's hold-briefly-then-503 pattern for requests arriving during startup (5s, then retry-after).

Metrics (gaps vs the set upstream exports; ours cover engine internals well but not ops):

• slot_retained_wal_size + confirmed_flush_lsn_lag (bytes) — the two that page you before the primary's disk fills
• replication receive-lag distribution (commit-to-ingest ms)
• request counts by status (/v1/shape + control plane), tagged known-error vs unexpected
• backfill duration/bytes/rows histograms; per-shape append-failure counter
• active vs total shapes (with the Shape retention: three-tier lifecycle (active / dormant / evicted), layered eviction #9 tiers: active/dormant), DS disk usage per class (table/* vs shape/*)
• error-rate counters on ingest decode/append paths

These land on the utility port (see auth issue). Wire the existing /metrics/prometheus exporter; add OTel trace-sampling knobs before enabling tracing at production volume (upstream defaults to 1% span sampling).

🤖 Generated with Claude Code