Make headless runtime fleet-safe across Maestro web replicas

[haasonsaas]

Current Status

• 2026-04-24: Drain signal PR open via Expose hosted runner drain signal #169 and internal source PR evalops/maestro-internal#1482. Hosted runner readiness/identity now expose latest drain status and manifest metadata for future transfer controllers.
• 2026-04-23: Deployment guardrail merged via Guard multi-replica headless Helm routing #137 (dcd2f0d8fbaeccbb81823cc831e0aa35a459befc).
• 2026-04-23: runtime_owned_elsewhere response contract merged via Add structured hosted runtime ownership errors #136 (0cc289e7ba27becccbf66b5d1937e03d52291e26).
• Remaining Make headless runtime fleet-safe across Maestro web replicas #78 work: durable owner/session routing and shutdown drain/transfer behavior.

From the Maestro remote/runtime review on 2026-04-22.

Reviewed heads:

• evalops/maestro@ae6e88a22d3a36d34d07c5c507375cfd7fdcd372
• evalops/platform@7b003938349939052f90ae41241599993a7c06c9

Parent tracker: #76

Problem

The headless runtime service is mature inside one process, but the deployed chart defaults to two Maestro web replicas while runtime ownership, event replay, connections, subscriptions, and controller leases are process-local. A remote client can create a runtime on pod A, then send SSE/POST/heartbeat traffic to pod B and get stale session/connection/subscriber failures.

Evidence:

• deploy/helm/maestro/values.yaml:1 sets replicaCount: 2.
• deploy/helm/maestro/templates/deployment.yaml:1-48 has basic probes/env/resources but no sticky routing, preStop/drain, runtime-owner registry, Redis/Postgres runtime routing, or ingress/SSE settings.
• src/server/headless-runtime-service.ts:2144-2253 stores all runtimes in an in-memory Map and cleans them up from the local process.
• src/server/handlers/headless-sessions.ts:437-452 resolves /api/headless/sessions/:id by looking up the runtime in the local process map.
• packages/tui-rs/src/headless/remote_transport.rs:990-1062 bootstraps and subscribes over HTTP, and 1176-1234 retries message POSTs, but there is no server-side redirect/routing contract for runtime owner changes.
• Platform's agentruntime.v1 proto already models worker leases, checkpoints, waits, heartbeats, and stream cursors (proto/agentruntime/v1/runtime.proto:13-74, 215-327).

Scope

• Decide the near-term deployment rule: set Maestro web/headless replicas to 1 for hosted remote mode, or require ingress sticky sessions for /api/headless/* and /api/chat/ws.
• Add a durable runtime-owner/session-routing record keyed by scope_key + session_id, including owner instance id, lease/heartbeat, created/updated timestamps, and routing metadata.
• Add a server response contract for runtime_owned_elsewhere so clients can reconnect to the correct owner or the ingress can route deterministically.
• Define whether event fanout/replay stays local with sticky sessions or moves to Redis/NATS/Postgres-backed broker state.
• Align runtime cleanup/dispose with pod shutdown and Platform AgentRun lease semantics so active runs are cancelled, drained, or re-claimed intentionally.

Acceptance Criteria

• With more than one web replica, a remote TUI can create a session, subscribe, send prompts/messages, heartbeat, disconnect, and reconnect without random 404/403 stale runtime failures.
• With no sticky routing configured, the chart refuses multi-replica remote mode or documents/probes the unsupported configuration clearly.
• Graceful shutdown drains or transfers active headless sessions before pod termination.
• Tests or smoke scripts cover simulated owner mismatch/reconnect and stale connection recovery.

Related: #76, #72, evalops/platform#671, evalops/platform#780, evalops/platform#765.

---

Runtime Parity / Conformance Links

• Define Rust core runtime parity contract for hosted Maestro #84 — Rust core runtime parity decision for hosted Maestro.
• Add headless runtime conformance suite for Rust parity #85 — headless runtime conformance suite that should gate Rust-hosted runner parity.

Use #85 as the protocol-level acceptance bar: lifecycle, subscribe/events/messages/heartbeat/disconnect, viewer/controller roles, permission request/response, utility operations, workspace-root enforcement, cursor replay/reset, and drain/resume behavior should match across the current TypeScript web/headless bridge and the target Rust runtime.

[haasonsaas]

Listener-generation counter pattern for clean controller handoff — openai/codex:codex-rs/app-server/src/thread_state.rs handles this with a monotonic listener_generation: u64 plus a Weak<CodexThread> — set_listener sends on the previous cancel_tx before replacing it, which guarantees the superseded controller's channel drops deterministically and no events are dual-routed.

Proposed acceptance criterion addition: "Controller lease handoff uses a monotonic generation counter stored on the AgentRun record. The superseded controller receives an explicit controller_superseded event before its subscriber channel is dropped. Lease takeover without a newer generation is rejected."

Without this, two controllers can both think they hold the lease across a replay gap, which is subtle and nearly untestable post-facto.

[haasonsaas]

PR is up at #136.

This slice gives hosted runner owner mismatches a machine-readable runtime_owned_elsewhere response instead of a plain 409 string, includes owner/session routing metadata in the TypeScript web-hosted error envelope, teaches the Rust remote transport to classify that structured code, and documents the contract for gateway/client routing.

[haasonsaas]

Merged #136 as 0cc289e7ba27becccbf66b5d1937e03d52291e26.

That completes the structured runtime_owned_elsewhere response-contract slice for #78: TypeScript web-hosted errors now include stable ownership metadata, Rust remote transport classifies the code, and the hosted-runner contract documents the envelope. Remaining pieces are the deployment guardrail, durable owner/session routing, and shutdown/drain behavior.

[haasonsaas]

PR is up at #137.

This slice makes the Helm chart fail closed for multi-replica process-local headless routing: default replicas are now 1, and must explicitly use or . Helm render tests cover the default, rejected, and sticky-session cases.

[haasonsaas]

Correction on the #137 note: the guardrail is exactly replicaCount > 1 requiring headlessRuntime.routing.mode=sticky-session or headlessRuntime.routing.mode=durable-owner. The previous comment lost those inline values to shell interpolation.