macOS offscreen smoke: xmms no-exit-after-close + xwpe/xmms in-process snapshot timeout (rc=-4)

[vejeta]

Summary

On the macOS runner (macos.yml, offscreen replay smoke with
SDL_VIDEODRIVER=dummy + --in-process-snapshots), three replay legs fail with
macOS-specific runtime behavior in the shim, not replay-file, fixture, or
CI-config problems. They are currently surfaced as non-blocking smoke
(continue-on-error: true) with artifacts uploaded on failure, so they do not
gate PR #58. This issue tracks the underlying src/ fixes.

Two independent failure classes, plus a debuggability prerequisite that should be
done first so the fixes start from evidence rather than guesses.

Class A — process does not exit after close latest (xmms)

• tests/ui/replays/xmms-startup.replay:10: process still running, expected 0
  (all 3 retries)
• tests/ui/replays/xmms-window.replay:59: process still running, expected 0

Both replays end with close latest → delay 1000 → assert-exit 0. XMMS is
still alive 1s after the close request under the offscreen dummy driver. This is
the window-close → process-teardown path in the shim not completing within the
replay's timing window on macOS.

Class B — in-process snapshot round-trip times out (rc=-4)

• tests/ui/replays/xwpe-keyboard-edit.replay:14 (screenshot typed, expanded
  to snapshot .../typed.bmp) → the runner reports
  in-process snapshot ... did not produce a file within 121s, and the app log
  shows replay: line 22: snapshot .../typed.bmp failed (rc=-4).
• tests/ui/replays/xmms-window.replay:2: state-snapshot ... initial.json did not appear within 121.07s.

rc=-4 originates in src/snapshot.c:180 (waitSnapshotResult): snapshotDone
was never set, i.e. pthread_cond_timedwait hit its hardcoded 120s deadline
(src/snapshot.c:174, deadline.tv_sec += 120) because the SDL user event
pushed by snapshotRequestAndWait / stateSnapshotRequestAndWait was never
processed by the app's main thread. The round-trip requires the X client to
return to the SDL/X event loop (e.g. XNextEvent) so snapshotHandleEvent
runs; offscreen on macOS these apps do not return to the pump in time, so the
snapshot/state-snapshot event sits unserviced until the deadline.

Same signature drives the xmms-window state-snapshot timeout
(src/state-snapshot.c handler, gated by
LIBX11_COMPAT_STATE_SNAPSHOT_TIMEOUT_SEC).

Prerequisite — make these failures debuggable (do first)

The uploaded artifacts are currently not diagnosable: the "App log tail"
printed by the fail-loud snapshot gate is empty because:

1. LOG() (src/util.h:24-31) is gated on -DDEBUG_LIBX11_COMPAT and writes to
   stderr, which src/util.h:35-39 itself notes is unreliable for curses
   apps like xwpe (they take over the terminal). The smoke build is a release
   build, and there is no flag-change rebuild guard (mk/common.mk:44-47
   depends only on the .c + SDL-backend stamp), so a debug rebuild cannot be
   toggled cleanly from CI.
2. The file-based, curses-safe compatTrace() / LIBX11_COMPAT_TRACE
   (src/util.c:11-25) has no call sites in snapshot.c / state-snapshot.c
   (only events.c/window.c today).
3. The BMP snapshot.c timeout is hardcoded 120s (no env override), so a stalled
   snapshot wastes the full 120s per step instead of failing fast with context.

Proposed debuggability work (all src/ + a smoke-recipe wire-up):

• Add compatTrace(...) at the key decision points in src/snapshot.c and
  src/state-snapshot.c (request pushed, event serviced/stale, target-window
  resolution, save result, timeout) so the failure reason is captured in a file.
• Make the snapshot.c timeout env-configurable (mirror
  LIBX11_COMPAT_STATE_SNAPSHOT_TIMEOUT_SEC) so smoke can fail fast.
• Wire LIBX11_COMPAT_TRACE=<out-root>/trace-lib.log into the smoke recipes so
  the trace file lands under build/ui-smoke/** and is uploaded by the existing
  *-ui-smoke artifact step.

Scope / notes

• No replay-file, fixture, or CI-config change fixes these — verified against
  the replay grammar and the runner (scripts/run-ui-replay.py) and the shim
  source. They are shim runtime behavior on the offscreen macOS event loop.
• PR ci: add a macOS build + install + link job #58 (ci/macos-runner) deliberately keeps these legs non-blocking; once the
  Class A/B fixes land and earn a green track record, the smoke legs can be
  promoted to blocking.
• Out of scope here: the intermittent Linux thread-sanitize
  lock-order-inversion (potential-deadlock) warning seen on the branch is a
  separate concern (shim locking under TSan with halt_on_error=1), unrelated to
  the macOS snapshot/teardown paths above.

Evidence

• macOS run for commit 8c553af (branch ci/macos-runner): grace/xephem smoke
  fully green (0 replay failures, 0 uploads); xmms (6 replay failures) and xwpe
  (1: xwpe-keyboard-edit) upload *-ui-smoke artifacts. All 15 build legs pass.