Skip to content

dev/tunnel: reconnect on disconnect (2h cap, idle timeout, network blips) #30

Description

@I-am-nothing

Goal

The current tunnel client opens the WSS once and lives until the connection drops, at which point the spawned task exits and the CLI's tunnel goes silent. AWS WS API Gateway enforces three drop conditions:

Condition Cap Frequency in practice
Total connection duration 2 hours (hard, not configurable) Once per long dev session
Idle timeout 10 minutes without traffic Mitigated by 30s heartbeat (already in place)
Network blip (laptop sleep, wifi flap, etc.) n/a Frequent in real dev

So a dev who runs `mirrorstack dev --tunnel` for >2h sees their tunnel die silently. Issue #29 surfaces the death; this issue handles the followup — actually reconnect.

Sketch

L1.5 §"Sizing — committed numbers" already specifies the contract:

Reconnect: exp backoff `1 → 2 → 4 → 8 → 16 → 30 s` (cap), give up at 5 min total

In `run_tunnel_loop`'s drop paths, instead of `return`-ing, signal the parent to retry:

```rust
// Pseudocode
loop {
let ttok = mint_token(...).await?; // POST /v1/tunnel/token
let (sink, stream) = connect(ttok).await?;
let outcome = run_tunnel_loop(sink, stream, shutdown.clone()).await;
if shutdown.is_set() || outcome.is_giveup() { return; }
backoff_then_retry(&mut delay).await;
}
```

Considerations

  1. Fresh ttok per reconnect. The original ttok is single-use and expired. The CLI must call `POST /v1/tunnel/token` again with the cached access_token. If the access_token also expired, surface a clear hint pointing at `mirrorstack login`.
  2. Re-register on each new connection. The new connection has a new `connection_id` server-side, so register must run again. `module_id` + `local_url` are unchanged.
  3. session_id changes across reconnects. The TunnelHandle's `session_id` field becomes stale. Either (a) make it a `Arc<RwLock>` updated per reconnect, or (b) drop the field and surface session_id only on first connect (simpler). dev/tunnel: surface liveness so callers can detect a dead background task #29's liveness signal is the right place to expose the current session_id.
  4. Backoff state. Reset to 1s after a successful register_ack (a successful connect counts; we don't want every blip to push us to the 30s cap). Cap at 30s, total give-up at 5 min as L1.5 says.
  5. Don't reconnect on Ctrl-C. `shutdown.notify()` from the host means "stop forever," not "reconnect."
  6. Routing-side staleness. Until the new connect completes, the platform's session table still points at the old (dead) connection_id. Inbound calls during the reconnect window 404. Acceptable for v1; a future improvement is dispatch retrying with a backoff while the CLI reconnects.

Acceptance

  • Tunnel survives a single `docker pause`/`docker unpause` of the WS endpoint (simulating a network blip)
  • After AWS forces the 2h close, CLI reconnects within ~2s and re-registers
  • 5-min total give-up emits a clear error on the host (probably via dev/tunnel: surface liveness so callers can detect a dead background task #29's liveness signal)
  • Ctrl-C still wins — no respawn loop after user stop

Deps

Estimate

~150 LoC + a unit test using mockito-style mock WS server. ~1 hour AI time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions