Large WebSocket payloads (424KB) cause backpressure and disconnections

[danshapiro]

Problem

The tabs.sync.push WebSocket message sends payloads as large as 424KB in a single frame. This causes WebSocket backpressure issues, leading to connection closures and degraded user experience, especially when combined with event loop blocking from session indexing.

Evidence from Production Logs

{"event":"ws_backpressure_close"}  // Connection killed due to slow processing
{"event":"terminal_stream_replay_miss"}  // 400+ events
{"event":"terminal_stream_gap"}  // Data loss from connection issues

Root Cause Analysis

Large Payload Sources

1. Tab state synchronization - tabs.sync.push sends the complete tab/pane state tree
2. Session data - Full session metadata for all open terminals
3. Terminal scrollback - Large terminal buffers being synced

Why 424KB is Problematic

1. WebSocket frame size - Large frames require multiple TCP packets
2. Backpressure buildup - If client can't process fast enough, server buffers grow
3. Event loop blocking - While processing large payloads, other operations queue up
4. Memory pressure - Both client and server must hold full payload in memory

Cascading Effects

When combined with session indexer blocking the event loop for 6+ seconds:

1. Server is blocked, can't drain WebSocket send buffers
2. Backpressure builds, server closes connection
3. Client reconnects, triggering full state resync
4. Large payload sent again, cycle repeats

Observed Behavior

• ws_backpressure_close events in logs
• 400+ terminal_stream_replay_miss / terminal_stream_gap events
• Terminal input lag up to 2.5 seconds
• UI freezing during sync operations

Proposed Solutions

Short-term

1. Chunk large payloads - Split payloads >64KB into multiple messages
2. Compression - Enable permessage-deflate WebSocket compression
3. Delta sync - Only send changed portions of state, not full tree
4. Lazy loading - Don't include full scrollback in sync, fetch on demand

Long-term

1. Binary protocol - Use MessagePack or Protocol Buffers for efficient encoding
2. Streaming state - Stream state updates incrementally
3. Client-driven sync - Client requests specific data rather than push-all
4. Backpressure handling - Implement proper flow control with pause/resume

Implementation Ideas

Chunked Messages

interface ChunkedMessage {
  type: 'chunked.start' | 'chunked.data' | 'chunked.end'
  messageId: string
  chunkIndex: number
  totalChunks: number
  data?: string  // base64 for binary safety
}

Delta Sync

interface TabSyncDelta {
  type: 'tabs.sync.delta'
  added: Tab[]
  removed: string[]  // tab IDs
  updated: { id: string; changes: Partial<Tab> }[]
}

Impact

• Medium - Causes intermittent disconnections and data loss
• Exacerbated by session indexer blocking
• Affects users with many tabs/panes or large terminal buffers

Related

• Session indexer blocking - makes backpressure worse
• Memory leaks - large payloads contribute to memory pressure