Sample the benchmark under a real workload via the macterm CLI#141
Merged
Conversation
External apps and AI agents need to drive Macterm programmatically (#107), and the CI benchmark needs to spawn realistic workloads headlessly. Darwin notifications (BenchmarkControl) carry no payload or response, so this adds a Unix-socket control plane: newline-delimited JSON requests dispatched on the main actor into the same AppState paths the UI uses, and a bundled ArgumentParser CLI at Contents/Resources/bin/macterm (PATH-injected into every pane along with MACTERM_SOCKET). This first slice ships the protocol, server, and read-only verbs — status, project/tab/pane list, session list/info. Mutations (create, select, split, run) and the benchmark workload follow in separate PRs. Design notes drawn from cmux/Zentty/Supacode source: socket-ownership check before connect, env var as discovery hint (never a hard pin, so a stale export can't brick shells after an app restart), non-blocking connect with a short poll, error responses carrying recovery hints, and a safe-fail CLI contract (stdout only on success; exit 2 when the app is unreachable).
With the control plane in place, this teaches it to actually drive the app: project create/select (idempotent by canonical path, ProjectPath- validated, remote specs rejected until #104), tab new/select/close, pane split/focus/close/run, an equal-cells grid verb (the benchmark workhorse), session kill, and layout apply/save. Semantics worth reviewing: - New panes/tabs take a command via the layout run: path (spawn-time initial_input); pane run types into an EXISTING live shell through a new GhosttyTerminalNSView.sendText — the paste path, guarded so a missing surface is a typed no_surface error, never a silent drop. - Headless callers can't answer dialogs, so close/apply verbs return a typed busy error where the UI would stage a confirmation; --force takes the destructive path deliberately. - Every pane's env now carries MACTERM_SESSION (its restart-stable zmx session name), so macterm run inside a pane self-targets. pane close deliberately ignores that fallback: destroying "whatever pane I'm in" by omission is a footgun. docs/cli.md documents the grammar, targeting rules, wire protocol for non-CLI clients, and the same-user security posture.
The window-state benchmark measured an empty app — one idle shell — so regressions that only show under load (render churn, poll pressure, per-surface overhead) were invisible. Now each state is sampled twice: idle as before, then again as workload-* states after spawning BENCH_WORKLOAD (default 2) busy tabs through the bundled CLI (tab new --run + grid 2x2 --run, a /bin/sh date loop per pane — a real external child emitting a line a second, portable across POSIX shells and nushell). Baseline migration is seamless by construction: the idle state keys are unchanged, so their history keeps comparing, and a baseline that lacks the workload-* states renders them with absolute values, no deltas, no labels, plus an explanatory footnote — the first post-merge main run fills them in. The spawn is paced (0.5s/tab) and verified: a pane-count mismatch aborts the run rather than silently benchmarking a partial workload against a full one.
github-actions
Bot
added
area:ci
First CI run of the workload failed with 'control socket never became ready': the throwaway HOME under the runner's $TMPDIR (/var/folders/…/T/…) pushes <data-dir>/control.sock past the ~104-byte sun_path limit, so the app refuses to bind and the CLI polls a socket that never appears. /tmp keeps the whole path around 50 bytes. The readiness-poll failure now also surfaces the CLI's stderr instead of timing out silently.
…ce launch The CI workload run exposed a launch-ordering race: SwiftUI can fire the window's onAppear (which runs installResponders and attaches the control handler) BEFORE applicationDidFinishLaunching. When the server was created inside didFinishLaunching, that ordering skipped the attach — the socket then answered 'starting' forever and the benchmark workload timed out. The server object now exists from delegate init (attach-before-start is harmless); didFinishLaunching still owns start() and the env exports.
Contributor
Window-state benchmark
30s sampling window per state; CPU % is the process CPU-time delta over the window. Runs land on different shared runners, so treat small deltas as noise — 🔺/🔻 marks changes ≥25% that also clear the metric's absolute noise floor (CPU % ≥0.5, Memory (RSS MB) ≥25, CPU ms/s ≥5, Wakeups/s ≥50), and those add the The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of three PRs for #107, stacked on #140 (will retarget as the stack lands). Closes the loop with the CI benchmark (#129): the window-state benchmark measured an empty app — one idle shell — so regressions that only show under load (render churn, poll pressure, per-surface overhead) were invisible.
What changes
benchmark.py rungains--workload TABS(default off;bench.shpassesBENCH_WORKLOAD, default 2). After the three idle states are sampled exactly as before, the harness spawns TABS busy tabs through the bundledmactermCLI (tab new --run+grid 2x2 --runagainst<data-dir>/control.sock) and samples the same three states again asworkload-focused/unfocused/minimized./bin/sh -c "while :; do date; sleep 1; done"per pane — a real external child emitting a line a second ("logs trickling in"), negligible CPU of its own, and parses in POSIX shells and nushell (it's typed into the pane's shell verbatim).pane list --jsonmust show exactly1 + 4×TABSpanes or the run aborts — silently benchmarking a partial workload against a full one would poison comparisons.macterm statusuntil exit 0) exercises the socket'sstartingstate; CLI failures dump the existing diagnostics (screenshot + app log).Baseline migration — seamless by construction
The idle state keys (
focused/unfocused/minimized) are unchanged, so their baseline history keeps comparing across this PR. A baseline that lacks theworkload-*states renders them with absolute values,—deltas, no labels, and a footnote saying deltas appear once main's baseline includes a workload run. Verified against a synthesized pre-workload baseline: no spuriousbenchmark:regression/improvementverdict, note renders. The first post-merge main push fills the baseline in.Verified locally
Full
run --workload 2against the built app (short windows): all six states sample, and the workload signal is exactly what was missing — focused RSS 160 → 230 MB, wakeups 801 → 1404/s, CPU 8.0 → 12.3% with 8 busy panes on screen. Results JSON records the workload config (tabs,panes_per_tab,command) for future comparability checks. Time cost in CI: roughly doubles the sampling phase (~2 extra minutes), well inside the workflow's 30-minute timeout.🤖 Generated with Claude Code