feat: reactive proxy recovery reaps a confirmed-dead upstream (#1018 slice A)

[milkway]

What & why

Slice A of #1018 (the follow-up to #1015/#1016). The merged #1017 made the scaler's
liveness reconcile authoritative + bidirectional, but recovery still waited for the
periodic ~10 s tick: between an external docker rm -f and the next tick, a
request routed to the vanished replica returned 502 upstream error, and the sticky
cookie kept the visitor pinned to the dead upstream.

This adds reactive recovery at the two transport-failure points in the proxy —
the HTTP forward error and the WS handshake failure (which connects the upstream
before the 101, so it's the same signal).

How

• reap_if_confirmed_dead(state, replica) asks the backend
  (ContainerBackend::replica_liveness, from fix: authoritative + bidirectional liveness reconcile for external docker changes (#1015, #1016) #1017) whether the selected replica is
  authoritatively gone, and reaps it through the same
  scaler::cleanup_replica_state the periodic reconcile uses (registry + sessions +
  metrics-cache + in-flight gauge) — so the reactive and periodic paths can't drift.
• The bar is deliberately conservative:
  • Only Missing reaps. A Stopped container may be mid-docker restart,
    which is owned by the scaler's grace window (External docker restart can orphan a running container because runtime reconciliation is one-way #1016) — the reactive path must not
    prune it and risk a duplicate replacement.
  • An app-level 5xx is an Ok upstream response and never reaches this arm.
  • A transient daemon error or a multi-host Unknown (owning host
    unreachable) returns false — a live replica behind a hiccup is never pruned.
• On a reap: a coalesced replacement spawn is kicked in the background (if the spec
  can still scale), and a GET document navigation recovers on the familiar
  cold-start splash (which polls readiness and opens the app once the replacement is
  up) instead of a raw 502. Subresources/XHR get the 502 and the browser retries
  onto the fresh replica. Non-idempotent methods are never blindly replayed —
  the reap lets the next call recover.

The stale sticky binding is cleared server-side immediately, so recovery no longer
waits out the scaler tick.

Acceptance criteria (slice A)

• A GET to a spec whose replica was docker rm -f'd recovers without waiting for
  the tick and without a persistent 502 (bounced to the Starting… splash).
• The stale sticky is invalidated (server-side reap); a fresh binding is created
  on the next pick.
• Non-idempotent methods are not blindly replayed; they get an explicit error
  after the reap.
• An app 5xx or a transient daemon error does not prune the replica.
• Multi-host Unknown (host unreachable) does not prune.

Out of scope (slice B, tracked in #1018)

The Docker die/destroy/start events listener for near-instant recovery. The
reactive path here already collapses the recovery window; the periodic reconcile
remains the fallback either way.

Tests

• Unit (every classification): Missing reaps + drops the sticky session;
  Stopped, Unknown, Running, and a whole-call backend error all spare the
  replica and its session.
• docker-it (new opt-in docker-it feature on ruscker-admin): spawns a real
  container, seeds the registry + a session, removes it externally
  (ContainerBackend::remove_container), and asserts the production glue
  (reap_if_confirmed_dead → replica_liveness → cleanup_replica_state) reaps it.
  Run: cargo test -p ruscker-admin --features docker-it.

Gate

cargo test · clippy (clean on touched files) · ./scripts/i18n-check.sh ·
cargo test -p ruscker-admin --features docker-it (real Docker 29.6.1) — all green.

🤖 Generated with Claude Code

[milkway]

Review (adversarial pass) — MERGE-OK

Focused pass on the hot-path risk areas (reactive recovery runs on the busiest handler in the product):

• Only runs on the error path. reap_if_confirmed_dead is called solely from the two transport-failure arms (HTTP forward Err, WS handshake Err) — zero cost on the success path.
• No lock held across the backend call. do_forward/ws::connect have already returned before the reap; replica_liveness runs with no registry lock, and cleanup_replica_state then takes replicas.write() cleanly. No lock inversion vs the scaler.
• Conservative prune bar. Only an authoritative Missing reaps. Stopped is spared (owned by the External docker restart can orphan a running container because runtime reconciliation is one-way #1016 restart grace window → no reactive double-prune); an app 5xx is an Ok response and never reaches the arm; a whole-call daemon error and multi-host Unknown return false. Covered by unit tests for every classification.
• Move/borrow correctness. req_is_get_visit is captured before req is consumed by do_forward; replica/spec/state are owned in forward and only borrowed afterward.
• Recovery UX. On reap: a background coalesced respawn is kicked (guarded by at_capacity), a GET navigation returns the existing cold-start splash (polls readiness), non-idempotent methods are never replayed. Stale sticky is cleared server-side, so the next request re-picks without waiting for the tick.

Real smoke: new docker-it test spawns a real container, removes it externally, and asserts the production glue reaps it (replica_liveness → cleanup_replica_state). CI 6/6 green.

No blockers.