Skip to content

fix(server): terminal subprocess polling no longer floods the PID space - #6377

Open
SunkenInTime wants to merge 3 commits into
pingdotgg:mainfrom
SunkenInTime:fix/terminal-subprocess-poller-pid-churn
Open

fix(server): terminal subprocess polling no longer floods the PID space#6377
SunkenInTime wants to merge 3 commits into
pingdotgg:mainfrom
SunkenInTime:fix/terminal-subprocess-poller-pid-churn

Conversation

@SunkenInTime

@SunkenInTime SunkenInTime commented Aug 12, 2026

Copy link
Copy Markdown

What Changed

The terminal subprocess poller now takes one process-table snapshot per poll tick and derives every terminal's child state from it, instead of spawning pgrep -P <pid> plus up to two full ps -eo pid=,ppid= dumps (and a ps -p <child> -o comm=) per terminal every second.

  • POSIX: one ps -eo pid=,ppid=,comm= per tick, shared across all terminals. ps is resolved to an absolute path (/bin/ps, then /usr/bin/ps) once at startup so the spawn no longer walks PATH.
  • Windows: the existing Get-CimInstance PowerShell call is likewise taken once per tick instead of once per terminal.
  • The per-tick snapshot is skipped entirely when no session is running (unchanged), and a snapshot failure logs one warning and skips the tick.
  • Added a regression test that two running terminals derive their activity from the shared snapshot with no per-terminal pgrep/ps -p spawns.

Why

Fixes #6332. With N terminals the poller spawned up to 3 processes per terminal per second, each by bare name — so every spawn attempted one posix_spawn per PATH entry until the hit. On the reporter's machine (14 terminals, 46-entry PATH) that measured ~455 process creations/sec, 94% failing ENOENT, wrapping the macOS PID space every ~4 minutes and eventually destabilizing launchservicesd.

Cost is now O(1) spawns per tick regardless of terminal count, and the one remaining spawn is a single successful posix_spawn. This also fixes a latent double-execution of the full ps table in the pgrep-fallback path, and gives Windows the same per-terminal → per-tick reduction (one PowerShell process per second instead of one per terminal).

Behavior notes: child detection, labels (comm normalization), and the process-tree ids fed to port discovery are unchanged. pgrep is no longer used at all. The subprocessInspector test seam is preserved.

Windows benchmark

Measured on Windows 11 with 16 logical processors and about 378 visible processes, using the exact Get-CimInstance Win32_Process command and 1.5-second timeout used by the poller. These are scan-phase measurements from this host, not universal latency guarantees.

Scan phase Before: 14 terminals After: shared snapshot Improvement
PowerShell/CIM launches per tick 14 1 92.9% fewer
PowerShell CPU time 20.6s 1.8s 91.3% lower
Peak aggregate working set 906 MB 92 MB 89.9% lower
Scan-phase wall time 1.68s 1.03s 38.5% faster
Scans completed within the 1.5s timeout 0/14 1/1 Detection succeeds under load

Without enforcing the timeout, 14 simultaneous scans consumed 37.6 CPU-seconds, peaked at 1.17 GB aggregate working set, and took 3.43 seconds of wall time.

The structural launch reduction is 1 - (1 / terminal count): 75% at 4 terminals, 87.5% at 8, 92.9% at 14, and 96.7% at 30.

Checklist

  • This PR is small and focused
  • I explained what changed and why
  • I included before/after screenshots for any UI changes (n/a — no UI changes)
  • I included a video for animation/interaction changes (n/a)

Work done by Claude (Fable 5) in Claude Code, directed by @SunkenInTime.


Note

Medium Risk
Touches terminal activity labels and process IDs used for port discovery on a 1s poll loop; snapshot-skip semantics are safer than falsely marking terminals idle but change failure handling versus the old per-terminal path.

Overview
Fixes PID-space flooding (#6332) by replacing per-terminal subprocess inspection with one process-table snapshot per poll tick shared across all running sessions.

On POSIX, each tick runs a single ps -eo pid=,ppid=,comm= (no pgrep or per-child ps -p); ps is resolved once at startup to /bin/ps or /usr/bin/ps to avoid PATH-walking on every spawn. On Windows, the existing Get-CimInstance PowerShell query runs once per tick instead of once per terminal. Per-terminal state is derived in-process via deriveSubprocessInspectResult from the parsed parent/child map and command names.

Behavior change: if the snapshot fails (non-zero exit, timeout, or truncated stdout), the whole tick is skipped with a warning—terminals are not cleared to idle and last-known subprocess labels/activity are retained. TerminalSubprocessCheckError no longer includes terminalPid and reports snapshot-level failure details instead.

Regression tests cover multi-terminal shared snapshot usage and snapshot-failure retention.

Reviewed by Cursor Bugbot for commit f10cf45. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Fix terminal subprocess polling to use one shared process snapshot per tick instead of per-terminal commands

  • Previously, subprocess activity was checked per-terminal using individual pgrep/ps or PowerShell calls each poll tick, which could exhaust the PID space under load.
  • Now, TerminalManager takes a single ps/PowerShell snapshot per tick and derives per-terminal activity from the in-memory result via deriveSubprocessInspectResult.
  • If the snapshot fails (nonzero exit, timeout, or truncated output), the tick is skipped entirely and previously known subprocess state is retained — terminals do not flip to idle on transient errors.
  • On POSIX, the ps binary path is resolved once at startup (checking /bin/ps and /usr/bin/ps) to avoid repeated PATH lookups.
  • Behavioral Change: snapshot errors no longer clear subprocess activity; terminals stay in their last known state until a successful snapshot is obtained.

Macroscope summarized f10cf45.

The terminal subprocess poller ran pgrep and a full ps table dump per
terminal every second, each spawned by bare name so every call walked the
full PATH with one failed posix_spawn per directory. With many terminals
this sustains hundreds of process creations per second and can wrap the
macOS PID space (pingdotgg#6332).

Now one ps -eo pid=,ppid=,comm= snapshot (resolved to an absolute path at
startup) is taken per poll tick and every terminal derives its child state
from that table, matching the shape the Windows path already used.
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6d8fa821-f779-4657-bc24-2340e4f27481

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Aug 12, 2026
Comment thread apps/server/src/terminal/Manager.ts
Comment thread apps/server/src/terminal/Manager.ts

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2861360. Configure here.

Comment thread apps/server/src/terminal/Manager.ts
@macroscopeapp

macroscopeapp Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved f10cf45

This PR fixes PID space exhaustion by refactoring terminal subprocess polling to use one shared process table snapshot per tick instead of per-terminal spawns. The change is well-scoped to internal polling logic, includes test coverage for the new behavior, and properly handles failures by preserving prior state.

You can customize Macroscope's approvability policy. Learn more.

A failed, timed-out, or truncated ps/PowerShell snapshot was parsed into an
empty table, marking every terminal idle and clearing its registered process
ids. Fail the snapshot instead so the tick logs a warning and keeps prior
state. Flagged by Macroscope and Cursor Bugbot on the PR.

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One convention finding on the new snapshot failure path in apps/server/src/terminal/Manager.ts: the snapshot failure is modeled by fabricating an Error whose message carries the structural context, instead of putting that context on TerminalSubprocessCheckError as attributes.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/server/src/terminal/Manager.ts Outdated
Replace the manufactured Error cause with structural exitCode/timedOut/
stdoutTruncated fields on TerminalSubprocessCheckError and derive the
message from them, keeping cause for real spawn failures.

@t3-code t3-code Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed at f10cf45. no blocking findings. targeted terminal manager tests pass (54/54), server typecheck passes, and the patch merges cleanly with current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug?]: Terminal subprocess poller spawns ~455 processes/sec (bare-name pgrep/ps against long PATH), exhausting the macOS PID space

1 participant