feat: Docker events watcher for near-instant liveness reconcile (#1018 slice B)

[milkway]

What & why

Slice B of #1018 — the second, independent half of the follow-up to #1015/#1016.

• fix: authoritative + bidirectional liveness reconcile for external docker changes (#1015, #1016) #1017 made the scaler's liveness reconcile authoritative + bidirectional.
• feat: reactive proxy recovery reaps a confirmed-dead upstream (#1018 slice A) #1019 (slice A) added reactive recovery so a request that hits a dead upstream
  recovers immediately instead of waiting for the tick.

But a spec with no incoming traffic — or a min-replicas warm pool that nobody
is actively hitting — still waited up to a full ~10 s scaler tick to notice an
external docker rm -f / docker restart. This subscribes to the Docker event
stream so the runtime reconciles within ~1 s, with no request needed to trigger
it.

How

• core: ContainerEvent / ContainerEventKind + ContainerEventStream, and a
  ContainerBackend::container_events() trait method. Default returns an empty
  stream — events are a latency optimization, never the source of truth, so mocks
  and any backend without support just rely on the periodic reconcile.
• ruscker-docker (LocalDockerBackend): streams bollard events filtered
  server-side to our containers' lifecycle actions
  (type=container, label=ruscker.replica_id, event ∈ {start,die,stop,kill,destroy,oom}),
  so daemon-wide churn and noisy actions (exec/health) never reach us. The stream
  ends on a transport error so the consumer reconnects.
• ruscker-docker (MultiHostDockerBackend): merges per-host streams with
  select_all; a host that can't open one is skipped (its containers still
  recover via the periodic reconcile) and never fails the whole call. A host stream
  that later ends just drops out of the merge; the watcher's reconnect re-establishes
  every host.
• ruscker-admin: scaler::reconcile_liveness_once factors the fix: authoritative + bidirectional liveness reconcile for external docker changes (#1015, #1016) #1017 reconcile
  pass so the events watcher and the periodic tick take the exact same
  authoritative, idempotent path. spawn_event_watcher consumes the stream,
  coalesces a burst (a restart is die+start; a rm is die+destroy) into one
  reconcile once the stream goes quiet for 400 ms, is leader-gated like the tick,
  and reconnects with backoff. Wired into serve next to the scaler.

The event is only a "reconcile now" nudge — the authoritative replica_liveness pass
decides the action — so a missed or duplicated event is always safe, and the
periodic reconcile remains the fallback for anything missed during a reconnect gap.

Acceptance criteria (slice B)

• An external removal/stop of a Ruscker container is reflected in ~1 s (not ~10 s),
  with no request needed to trigger it.
• The watcher reconnects after a stream drop; no replica is pruned on event
  "silence" — only the authoritative reconcile prunes.
• Multi-host: a host whose stream is unavailable degrades to periodic reconcile,
  with no prune or duplicate.

Interaction with #1016 / #1019

The watcher just runs the same reconcile: a destroy → Missing → prune + respawn;
a restart's die+start coalesce into one pass that sees the container already
Running → re-adopt (or Stopped → grace window if the restart is slow). Concurrent
runs with the periodic tick are safe (registry writes serialized, every action
re-checks under lock).

Tests

• Unit: the action → ContainerEventKind map; EventMessage → ContainerEvent
  (a non-container or unlabelled event maps to None); reconcile_liveness_once
  prunes a Missing replica (the watcher's entry point).
• docker-it (real daemon): open the stream, then assert a Start on spawn and a
  die/stop/destroy on external removal (fired from delayed tasks while the stream is
  already being polled, since bollard connects lazily). Run:
  cargo test -p ruscker-docker --features docker-it.

Gate

cargo test · clippy (clean on touched files) · ./scripts/i18n-check.sh ·
docker-it (ruscker-docker 52 passed, ruscker-admin 429 passed) against real
Docker 29.6.1 — all green.

🤖 Generated with Claude Code

[milkway]

Review (adversarial pass) — MERGE-OK

• Events are a nudge, not truth. The watcher only calls reconcile_liveness_once, which runs the same authoritative replica_liveness pass as the tick. A missed, duplicated, or reordered event is always safe — the reconcile re-derives state. This is the property that makes the whole slice low-risk.
• Idempotent + lock-safe concurrency. Watcher and periodic tick can reconcile at once: registry writes are serialized, cleanup_replica_state/mark_restarting/readopt all re-check under lock (a double-reap gets None, a double-readopt is a no-op via the spawn-lock + already_ready check).
• Leader-gated like the tick (state.leader.is_leader() before reconcile) — non-leaders drain the stream but never prune/spawn.
• Reconnect + no false prune on silence. The stream ends on a transport error (take_while on Ok) → reconnect with backoff; the periodic reconcile covers the gap. Nothing is pruned because a stream went quiet — only the authoritative pass prunes.
• Multi-host safety. select_all merges reachable hosts; an unopenable host is skipped (periodic covers it), and the empty-Vec case is guarded (select_all panics on empty).
• Burst coalescing. die+start (restart) / die+destroy (rm) collapse into one reconcile after a 400 ms quiet window, so a restart doesn't trigger prune-then-respawn churn — the single coalesced pass usually sees the container already Running → re-adopt.

Real smoke: docker-it opens the stream and asserts Start on spawn + die/stop/destroy on external removal against a live daemon (fired from delayed tasks because bollard connects lazily — same as the real watcher, which polls continuously). Full docker-it: ruscker-docker 52 passed, ruscker-admin 429 passed.

No blockers.