Benchmark window-state resource usage in CI and compare PRs against main#129
Merged
Conversation
Adds a resource benchmark that measures CPU-time delta, RSS, and (best-effort via powermetrics) wakeups across three window states: focused, open-but-unfocused, and minimized. Pushes to main publish a baseline artifact; PR runs fetch the latest main baseline and post a comparison table to the job summary and an updated-in-place PR comment. CI can't script window states through AppleScript or synthetic keys (both need TCC grants a headless runner can't approve), so the app gains a benchmark-only remote control: launched with MACTERM_BENCHMARK=1 it listens for Darwin notifications (notifyutil -p), which need no permissions. Benchmark mode holds an NSProcessInfo activity assertion — otherwise App Nap suspends the occluded app and queues the control notifications undelivered — and skips the notification-permission prompt that would steal key focus mid-measurement. The harness launches via open --env rather than exec'ing the binary: launch-time activation counts as user intent, and SwiftUI doesn't create the window (or run the onAppear that wires app state) until activation is granted. App Support resolves via the user record, not $HOME, so benchmark runs point FileStorage at a throwaway directory via MACTERM_BENCHMARK_DATA_DIR to keep real app data untouched.
github-actions
Bot
added
area:state
The bench-built Release app carries the placeholder EdDSA key, so Sparkle's updater fails to start and raises an app-modal alert that blocks the launch run loop until someone clicks OK — locally that interrupts the run, and on CI nobody is there to click. Don't start the updater in benchmark mode. Also nudge activation once a second until the window exists (a single post-launch request can be denied or race session readiness on a fresh runner), and dump the app's log plus a screenshot as an artifact when the readiness marker never appears.
Contributor
Window-state benchmark
30s sampling window per state; CPU % is the process CPU-time delta over the window. Runs land on different shared runners, so treat small deltas as noise — 🔺/🔻 marks changes ≥25% that also clear the metric's absolute noise floor, and those add the No main-branch baseline found; showing absolute values only. |
A delta is significant only when it clears both ±25% and the metric's absolute noise floor — a pure percentage threshold would flag noise on tiny absolute values (minimized CPU going 0.03% → 0.05% reads as +67%). Significant deltas add a benchmark-regression or benchmark-improvement label (kept in sync on later pushes) and the PR comment gains a section explaining exactly which metrics triggered the label.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A
Benchmarkworkflow that measures Macterm's resource usage in three window states — focused, open but unfocused, and minimized — and reports how a PR changes them relative tomain:mainrun the benchmark and upload abenchmark-resultsartifact (the baseline).Locally:
mise run bench(BENCH_SECONDSto adjust the 30s default window).How
CI can't drive window states via AppleScript or synthetic keystrokes — both need TCC grants a headless runner can't approve. Instead:
BenchmarkControl(app, armed only byMACTERM_BENCHMARK=1): listens for Darwin notifications (notifyutil -p com.thdxg.macterm.bench.<cmd>, no permissions needed) —open-project(a fresh runner otherwise sits on the WelcomeView with no shell/surface),activate,minimize,restore. Holds anNSProcessInfoactivity assertion so App Nap can't suspend the app and swallow control commands — which also means the numbers measure the app's own timer/render behavior rather than OS nap throttling. Skips the notification-permission prompt (it would steal key focus mid-measurement).scripts/benchmark.py: launches viaopen --env(launch-time activation counts as user intent; SwiftUI won't create the window until activation is granted), waits for the project-open readiness marker, then per state: settle, sample CPU-time delta + median RSS over the window, with a concurrentpowermetrics --samplers taskspass (null inside VMs where it fails). Focus changes useopen(self) andopen -a Finder(steal focus); minimize goes through the notification hook.$HOME, and — because App Support resolves via the user record, not$HOME—FileStoragegains aMACTERM_BENCHMARK_DATA_DIRoverride so benchmark runs never read or write real app data.Caveats
powermetricsis best-effort on virtualized runners; the ps-based CPU/RSS numbers are the guaranteed floor.The first main-branch run after merge establishes the baseline; PRs before that report absolute values only.