Skip to content

fix(native): serialize startup frame capture to stop heap corruption on macOS 26#120

Open
JubaKitiashvili wants to merge 1 commit into
EvanBacon:mainfrom
JubaKitiashvili:fix/macos26-startup-frame-capture-race
Open

fix(native): serialize startup frame capture to stop heap corruption on macOS 26#120
JubaKitiashvili wants to merge 1 commit into
EvanBacon:mainfrom
JubaKitiashvili:fix/macos26-startup-frame-capture-race

Conversation

@JubaKitiashvili

@JubaKitiashvili JubaKitiashvili commented Jun 29, 2026

Copy link
Copy Markdown

Summary

On macOS 26 (Tahoe), serve-sim reliably crashes the host process with a heap-corruption abort shortly after a browser connects:

node(…) MallocStackLogging: BUG IN CLIENT OF LIBMALLOC: memory corruption of free block
… _xzm_xzone_malloc_freelist_outlined …  (EXC_BREAKPOINT / SIGTRAP)
… serve-sim-native.node …

The crash is intermittent (≈1 in 25 cold starts, but often on the very first browser connect) and is far more likely on the default --codec auto (H.264/AVCC) path. MJPEG-only sessions usually survive. It does not reproduce on macOS 15.

Root cause

A startup data race in FrameCapture. wireUpFramebuffer() registers the SimulatorKit screen callbacks — which dispatch captureFrame() onto captureQueue — and then calls captureFrame() directly on the caller thread:

for desc in candidates { try registerFrameCallbacks(desc: desc) }  // callbacks start firing on captureQueue

captureFrame()   // ← also runs on the caller (JS/NodeActor) thread

So two captureFrame()handleFrame() runs execute concurrently during startup. handleFrame() does the first-frame encoder setup (videoEncoder.stop() then videoEncoder.setup(...), which swaps the onEncodedFrame closure), so the two concurrent runs double-free the old closure's context. On macOS 26 the hardened "xzone" allocator detects the corrupted freelist and aborts; on older allocators the same race was benign. The AVCC path amplifies it by adding VTCompressionSession / pixel-buffer-pool / Data allocations in the same tiny window.

The observable tell is in the logs: "JPEG encoder ready" is printed twice at startup — once per concurrent handleFrame().

(Consistent with Package.swift's swiftLanguageModes: [.v5], chosen so these cross-queue captures stay warnings rather than Swift 6 errors.)

Fix

  • FrameCapture.swift — dispatch the initial captureFrame() onto captureQueue (like the callback- and idle-timer-driven captures already do), so every captureFrame() is mutually exclusive. async is safe in both the start() (caller thread) and the self-heal re-wire (already on captureQueue) paths.
  • VideoEncoder.swift — guard onEncodedFrame with an NSLock so the resolution-change swap (capture queue) can't race the read in encode() (encode queue). Defensive; same closure-double-free class on a different trigger.

2 files, +24/−2. No behavior change other than the initial frame being captured on the proper queue.

Verification (macOS 26.5.1, arm64, Xcode 26, Node 22)

Check Before After
"JPEG encoder ready" at startup logged (the race) logged
Startup-race stress (AVCC+AX burst on a freshly started server) crashes ≈1/25 0 crashes / 60 trials
New crash reports yes none
H.264/AVCC live stream in browser aborts renders, stable

Notes

This is macOS-26-specific in manifestation (the new allocator turns a latent race into a hard abort), but the race itself exists on all versions. No allocator-specific workaround is used — the fix removes the race.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved startup and frame-processing stability to prevent concurrent state updates during video capture.
    • Made frame encoding callbacks safer across threads, reducing the risk of race conditions and inconsistent behavior during streaming.

…on macOS 26

FrameCapture.wireUpFramebuffer() registered the SimulatorKit screen
callbacks (which dispatch captureFrame() onto captureQueue) and then called
captureFrame() directly on the caller thread. The two ran concurrently
during startup, so handleFrame() set up the encoders twice in parallel and
double-freed VideoEncoder.onEncodedFrame's closure context. On macOS 26 the
hardened "xzone" allocator detects the corrupted freelist and aborts with
SIGTRAP ("BUG IN CLIENT OF LIBMALLOC: memory corruption of free block").
The AVCC/H.264 path made it far more likely by adding concurrent native
allocations in the same window. The observable tell was "JPEG encoder
ready" being logged twice at startup.

- FrameCapture: dispatch the initial captureFrame() onto captureQueue so
  every capture is mutually exclusive with the callback-driven ones.
- VideoEncoder: guard onEncodedFrame with a lock so the resolution-change
  swap on the capture queue cannot race the read on the encode queue.

Verified on macOS 26.5.1 (arm64, Xcode 26, Node 22): "JPEG encoder ready"
now logs once, 0 crashes across 60 startup-race trials (baseline ~1/25),
and H.264/AVCC renders without aborting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b639ab68-9d6d-4648-b864-d370e7f0b1cf

📥 Commits

Reviewing files that changed from the base of the PR and between f94d57c and 4b48f52.

📒 Files selected for processing (2)
  • packages/serve-sim/Sources/SimNative/FrameCapture.swift
  • packages/serve-sim/Sources/SimNative/VideoEncoder.swift

📝 Walkthrough

Walkthrough

Two concurrency fixes in the SimNative layer: FrameCapture.wireUpFramebuffer() now dispatches the initial captureFrame() call asynchronously onto captureQueue, and VideoEncoder introduces an NSLock (cbLock) to guard all reads and writes of the onEncodedFrame callback in setup, encode, and stop.

Changes

Concurrency Fixes

Layer / File(s) Summary
VideoEncoder callback locking
packages/serve-sim/Sources/SimNative/VideoEncoder.swift
Adds cbLock: NSLock; setup locks around assigning onEncodedFrame, encode locks to copy it to a local before invoking, and stop locks around clearing it.
FrameCapture async startup dispatch
packages/serve-sim/Sources/SimNative/FrameCapture.swift
Replaces direct captureFrame() call in wireUpFramebuffer() with captureQueue.async { captureFrame() } to serialize it with screen-callback-driven invocations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 Hops and locks upon the queue,
No race conditions breaking through.
The callback safe, the frame in line,
Async dispatch works just fine.
Synchronized at last — divine! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main fix: serializing startup frame capture to prevent the macOS 26 heap corruption crash.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kabiroberai

Copy link
Copy Markdown
Contributor

This is likely resolved by #117

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants