pi-web server becomes unresponsive over time: event-loop saturation from per-token broadcast fan-out + session-file fd leak

[ashwin-pc]

Summary

After ~24h of uptime with active streaming and several large sessions, the pi-web server (the supervised child) becomes completely unresponsive: it accepts TCP connections but never writes a response. The browser UI shows the sessions drawer stuck on "No saved sessions yet." with the /api/sessions request hanging forever, and the tunnel/proxy eventually returns a "Tunnel Timed Out" page.

This is not a memory problem — it's single-threaded event-loop / syscall saturation.

Symptoms

• Sessions drawer renders the empty placeholder ("No saved sessions yet.") because the /api/sessions fetch never resolves.
• curl to both the supervisor (127.0.0.1:8787) and the child directly (127.0.0.1:8788) time out (HTTP 000), even though both processes are alive and LISTENing.
• Connections accept but never get a response → they pile up (observed 38 ESTAB connections to the child backing up).
• Restarting the child clears it immediately (sessions persist on disk, nothing lost).

Evidence it is CPU / event loop, not memory

System:

• 494 GB RAM, ~320 GB free, no swap, no OOM.

Child process:

• RSS only ~833 MB (far below V8's default ~4 GB old-space limit). 37 GB VmSize is just reserved address space.

Main thread is saturated in the kernel:

MainThread:  utime 291,825  vs  stime 1,715,164   (~85% of CPU in kernel/syscalls)
voluntary_ctxt_switches: 31,571,312               (31.5M wakeups)
state: running / ep_poll, ~20-30% CPU sustained for the full ~24h uptime

V8Worker / libuv threads are idle — this is a main-thread saturation, which is exactly why Node accepts sockets but never responds.

Root cause (two compounding issues in server.ts)

1. Per-streaming-event fan-out is O(events x clients x messages).

   • Every pi_event (fires on every token/chunk while a turn streams) triggers two broadcast() calls.
   • broadcast() does a JSON.stringify then a .send() (a write syscall) per connected realtime client.
   • On message_end / agent_end / compaction_end it also calls sessionStats(value), which iterates the entire message branch (one affected session had 892 messages).
   • With high-frequency events x several clients x large sessions, this becomes a syscall storm (the 31.5M context switches + huge stime).

2. File-descriptor leak on session files.

   • The same session .jsonl files are held open repeatedly — observed one file open 18x, others 9x / 7x / 6x — totaling 244 open fds.
   • Session listing/reading opens files without closing them; with frequent /api/sessions + state polling over a long-lived server, fds and parse/I/O work accumulate.

Why it tips from "slow" to "fully hung"

The server runs ~20-30% kernel CPU continuously (chronic). Streaming a turn on top of large sessions + accumulated connections/fds pushes the event loop past the point where it can drain incoming HTTP requests between broadcast bursts. Requests then queue indefinitely → empty drawer, perpetually-pending /api/sessions, tunnel timeout.

Suggested fixes

• Throttle / coalesce pi_event broadcasts (batch instead of fanning out every token).
• Prune dead realtime clients; avoid the double-broadcast per event.
• Memoize or incrementally maintain sessionStats instead of re-scanning the full branch on every event.
• Close session-file handles after reading (fix the fd leak).

Workaround

Restart the supervised child (sessions persist on disk):

curl -X POST http://127.0.0.1:8787/api/restart
# or
kill -9 <child-pid>   # supervisor auto-respawns

[ashwin-pc]

New profiling data — the dominant cost is child_process.spawn, and the deeper root cause is the never-disposed liveSessions cache

I reproduced this on a long-running instance and captured a V8 CPU profile that refines the diagnosis here. The "syscall storm / high stime / millions of voluntary context switches" described above is real, but on my instance it was not primarily the broadcast fan-out — it was child_process.spawn (fork/exec/wait is a heavy kernel path, which is exactly what shows up as kernel time + context switches).

Direct evidence: 7s CPU profile (27,646 samples)

86.3%  23856 hits  spawn @ :0                         ← child_process spawn
 2.8%    772 hits  normalizeSpawnArguments @ node:child_process
 2.1%    572 hits  parseSessionEntryLine @ .../core/session-manager.js:234
 1.5%    405 hits  (garbage collector)
 0.1%     15 hits  git @ <user>/.pi/web/extensions/workspace-footer.ts:44   ← caller
 0.0%      7 hits  renderFooter @ <user>/.pi/web/extensions/workspace-footer.ts:175

Process at the time: >100% CPU with nothing actively streaming, RSS 2.1 GB, GET / taking ~8 s, GET /api/sessions not responding within 8 s. voluntary_ctxt_switches was ~78M (vs the 31.5M reported above — same failure mode, further along).

Why: per-session extension timers × an unbounded, never-disposed session cache

The trigger was an ordinary footer extension that runs a per-session setInterval (4 s) calling synchronous execFileSync("git", …) for every repo under the cwd (13 repos × ~2 git calls). Sync spawn blocks the event loop on every call. But the real amplifier is a pi-web lifecycle bug:

• liveSessions (server.ts:1231) is an unbounded cache that is never evicted, and session.dispose() is never called anywhere — grep -cE "\.dispose\(|evict|lru" server.ts → 0. ~20 endpoints call getOrCreateLiveSessionById(), each loads a full agent session and keeps it forever.
• Even deleteSessionById() (server.ts:1200-1201) only does unsubscribe() + liveSessions.delete(); it never disposes the session, so extension session_shutdown handlers (e.g. the footer's clearInterval) never fire.
• Result: every session a user ever tab-switched to stays resident and keeps running its extensions' timers. On my instance ~23 sessions in one workspace were live, each firing the 4 s git-spawn interval → hundreds of blocking spawns every few seconds → permanent event-loop saturation.

This also explains a second user-visible symptom: an edited extension shows an old version in some tabs and the new version in others. Extensions are loaded per live session in makeAgentSession() and never reloaded, so only newly-loaded sessions pick up edits.

Corroborates the fd leak reported here (worse over time)

801 open fds: 611 session .jsonl handles (51 unique files; one file open 69×), 95 sockets, 11 pipes

(matches the "one file open 18×, 244 fds total" observation — same leak, longer uptime). Two sources: live sessions never released, and createReadStream(file).pipe(res) at server.ts:144/159/350 with no res/req close handler to destroy() the stream.

Recovery

POST /api/restart dropped the fresh child from >100% CPU → 0.6% and RSS 2.1 GB → 300 MB — accumulated state, not load.

Suggested fixes (in addition to the broadcast/stats fixes above)

1. LRU eviction + disposal (primary): bound liveSessions to the N most-recently-used (PI_WEB_MAX_LIVE_SESSIONS, default ~6–8) plus any isStreaming/isCompacting session plus the home session. On eviction call unsubscribe() and session.dispose() (AgentSession.dispose() aborts work, invalidates the extension runner, and runs session_shutdown handlers — verified at dist/core/agent-session.js:472), then clean the runtimeStartedAts/runtimeLastActivityAts/toolStartedAts maps. Reload on demand when switching back. This bounds memory + fds and confines extension timers to in-use sessions.
2. Call dispose() in deleteSessionById so explicit deletes tear down extension timers/handlers.
3. Destroy read streams on abort at the three createReadStream().pipe(res) sites: const s = createReadStream(f); res.on("close", () => s.destroy()); s.pipe(res);.
4. Reload extensions when their source mtime changes so a single long-lived session doesn't serve a stale extension indefinitely.

Happy to split the liveSessions lifecycle piece into its own issue if you'd prefer to track it separately from the broadcast fan-out.