Vendor LiveKit WebRTC and add WebRTC transport by sjchmiela · Pull Request #3 · expo/serve-sim

sjchmiela · 2026-06-29T15:40:43Z

Summary

This is the first grouped WebRTC PR in the new stack.

Commits:

Vendor LiveKit WebRTC framework and native build integration
Add WebRTC transport and signaling
Add WebRTC codec selection and fallback
Improve WebRTC ICE and TURN handling
Fix packaged WebRTC H264 runtime behavior for Tart VMs

Notes:

The LiveKit vendoring commit uses the final SwiftPM binary artifact approach. It does not include the earlier checked-in LiveKitWebRTC.framework source vendoring flow.
Tunnel-provider support is intentionally not included in this stack.

Validation

./node_modules/.bin/tsc --noEmit
npm exec --yes --package bun@latest -- bun run packages/serve-sim/build.ts

## Summary - Renames the package/workspace to `@expo/serve-sim` for the Expo fork

szdziedzic · 2026-07-02T09:05:55Z

+    try {
+      const body = await readRequestBody(req);
+      const offer = JSON.parse(body.toString("utf8")) as unknown;
+      const answer = this.capture.handleWebRTCOffer(offer);


This WebRTC offer path is synchronous all the way into native code. handleWebRTCOffer() calls this.capture.handleWebRTCOffer(offer) on the Node request path, and the native implementation blocks on a semaphore while WebRTC creates the answer and gathers ICE. That means one /webrtc/offer request can stall the whole server event loop for seconds, including unrelated HTTP requests, WS input, and stream fan-out. This should be exposed as an async native method, similar to the AX async methods, so signaling work does not block Node.

Addressed in 4de4d70. SimCapture.handleWebRTCOffer is now an async Node method and the JS NativeCapture/DeviceSession path awaits the returned Promise, so /webrtc/offer signaling no longer blocks the Node request path while native creates the answer / gathers ICE.

szdziedzic · 2026-07-02T09:08:44Z

 }

+export function mjpegStreamUrlFrom(config: NonNullable<Window["__SIM_PREVIEW__"]>): string {
+  return `${config.url.replace(/\/+$/, "")}/stream.mjpeg`;


This derives the MJPEG URL from config.url, but in embedded/non-proxy mode config.url is only the helper origin while config.streamUrl contains the required /helper/<udid>/stream.mjpeg path. That regresses MJPEG fallback for the Expo CLI integration path: fallback will request /stream.mjpeg at the root and get a 404/blank video. We should keep using config.streamUrl for MJPEG, or derive from streamUrl rather than url, and add a test using the inProcessServeSimState() shape.

Addressed in 76f77db. mjpegStreamUrlFrom() now derives from config.streamUrl and swaps the terminal stream endpoint to /stream.mjpeg, preserving embedded /helper/<udid>/... paths. Added a regression test using inProcessServeSimState().

szdziedzic · 2026-07-02T09:10:28Z

    case "/stream.avcc": session.handleAvcc(req, res); return true;
    case "/config": session.handleConfig(req, res); return true;
    case "/health": session.handleHealth(req, res); return true;
+    case "/webrtc/offer": void session.handleWebRTCOffer(req, res); return true;


/webrtc/offer needs an explicit CORS preflight path. The browser sends POST with Content-Type: application/json, so cross-origin helper usage will issue an OPTIONS preflight first. Today this route falls through to handleWebRTCOffer(), which tries to parse the empty OPTIONS body as an offer and returns an error instead of a 204/200 with CORS headers. This will make WebRTC fail before signaling in direct helper mode.

Addressed in 76f77db. /webrtc/offer now handles OPTIONS with a 204 and CORS headers before reading/parsing a body or starting the helper session. Added a focused CORS header test.

szdziedzic · 2026-07-02T09:12:28Z

+            setDataChannelOpen(false);
+          }
+        };
+        pc.ontrack = (event) => {


Receiving ontrack clears the failed codec state before we know any frames are actually decodable. If negotiation succeeds but the browser cannot decode the produced bitstream, the UI can end up permanently black: the connection may be connected, ontrack fires, but no frame ever renders and the codec ladder never advances. The AVCC path has a startup/no-frame watchdog; WebRTC needs a similar first-frame watchdog using requestVideoFrameCallback, loadeddata, or decoded-frame stats.

Addressed in 8930a8c. ontrack now starts a first-frame watchdog instead of clearing codec failure immediately. SimulatorView marks success only after requestVideoFrameCallback or loadeddata, and the hook fails the current codec if no decoded frame arrives.

szdziedzic · 2026-07-02T09:18:33Z

  // drop to MJPEG, which every helper serves. See avcc-fallback.ts.
  const avcc = useAvccStream();
+  const streamSettings = config.streamSettings;
+  const useWebRtcVideo = streamSettings?.transport === "webrtc";


Once transport === "webrtc", the UI stays pinned to the WebRTC path even after the codec fallback ladder is exhausted. If VP8/VP9/H264 all fail, nextWebRtcFallbackCodec() returns null, but we never drop to MJPEG as the terminal safety net. That leaves users with a dead/black stream instead of the existing reliable fallback.

I don't know if this is something we care about or overcomplication. Maybe this is too much overthinking? Maybe we should explicitly fail? I think I prefer explicit failure.

Addressed in 8930a8c. I kept the transport pinned to WebRTC and made exhaustion explicit: once the codec ladder is out of attempts, the simulator view shows a WebRTC failure instead of silently dropping to MJPEG.

szdziedzic · 2026-07-02T09:19:26Z


    private let deviceUDID: String
    private let onFrame: SimFrameCallback
+    private let webRTCPublisher = WebRTCPublisher()


nit: This eagerly constructs WebRTCPublisher for every capture engine, including pure HTTP/MJPEG sessions. That constructor builds WebRTC factory state and probes H.264 support, so the HTTP path pays WebRTC startup cost even when WebRTC is never requested. Can we lazy-create this only when the first WebRTC offer arrives?

I'm not sure if we care about it here tbh

Addressed in 4de4d70. CaptureEngine now lazy-creates WebRTCPublisher on the first WebRTC offer, so HTTP/MJPEG-only sessions do not pay the WebRTC factory/probe cost during capture startup.

szdziedzic · 2026-07-02T09:20:08Z

        }

        let h264Request = reserveH264EncodeIfNeeded()
+        let shouldSendWebRTC = webRTCPublisher.isActive


This checks webRTCPublisher.isActive on every captured frame, and isActive synchronously hops onto the publisher queue. That adds a cross-thread sync to the hot frame path even when the selected transport is HTTP/MJPEG. A cached atomic/state flag, or gating this by configured transport, would avoid the per-frame synchronization.

Addressed in 4de4d70. The frame hot path now checks a cached webRTCActive flag updated by WebRTCPublisher.onActiveChanged; it no longer calls webRTCPublisher.isActive and synchronously hops onto the publisher queue for every frame.

szdziedzic · 2026-07-02T09:22:09Z

+    private var encodedCount: Int64 = 0
+    private let stateQueue = DispatchQueue(label: "serve-sim.webrtc.h264.encoder")
+
+    init(codecInfo: LKRTCVideoCodecInfo) {


The software H.264 encoder logs the negotiated codec parameters, but the actual VideoToolbox encoder still hardcodes High profile. If the browser answers constrained-baseline/profile-level-id parameters, we may emit SPS/PPS that do not match the negotiated profile, especially on the VM path where this encoder matters most. We should either honor the negotiated profile-level-id or avoid accepting profiles the encoder will not produce.

Addressed in 4de4d70. The custom software H.264 encoder factory now filters/rejects H.264 capabilities unless profile-level-id is High profile (64...), so baseline/constrained-baseline answers are not negotiated against the VideoToolbox encoder that emits High-profile SPS/PPS.

szdziedzic

Nice work 👏

I have 2 cleanup/nit comments but it LGTM overall

szdziedzic · 2026-07-02T10:31:32Z

+        }
+    }
+
+    func handleOffer(_ request: WebRTCOfferPayload) throws -> WebRTCAnswerPayload {


The new async handleOffer(...) path appears to be the only remaining call site, so can we remove this old synchronous overload? Keeping the semaphore-based version around makes it easier for a future call site to accidentally reintroduce the JS-thread blocking behavior this update just fixed.

Addressed in 63f3ed3. Removed the old semaphore-based synchronous handleOffer(...) overload; the async path is now the only offer handling API.

szdziedzic · 2026-07-02T10:32:05Z

+        return try result!.get()
+    }
+
+    func handleOffer(_ request: WebRTCOfferPayload) async throws -> WebRTCAnswerPayload {


nit: The async native path fixes the Node event-loop blocking issue, but this continuation still depends on libwebrtc eventually invoking one of the signaling callbacks. ICE gathering has its own timeout later, but setRemoteDescription, answer, and setLocalDescription do not appear to have a native-side timeout. If one of those callbacks is dropped or stalls, the /webrtc/offer request can remain pending indefinitely. Could we add a signaling timeout/cleanup here, or otherwise guarantee these callbacks always resolve?

Addressed in 63f3ed3. Added a native-side 10s signaling timeout guarded by a one-shot completion helper, so stalled/dropped setRemoteDescription / answer / setLocalDescription callbacks reject the offer and close the active session instead of leaving /webrtc/offer pending indefinitely.

sjchmiela force-pushed the stanley/expo-serve-sim-02-tunnel-support branch from ca949d8 to 1151105 Compare June 30, 2026 08:46

sjchmiela force-pushed the stanley/expo-serve-sim-03-vendor-livekit-webrtc branch from 18472b9 to 198d472 Compare June 30, 2026 08:46

Rename package to @expo/serve-sim (#1)

7191593

## Summary - Renames the package/workspace to `@expo/serve-sim` for the Expo fork

sjchmiela force-pushed the stanley/expo-serve-sim-02-tunnel-support branch 4 times, most recently from 152c36f to ad92736 Compare June 30, 2026 12:27

sjchmiela force-pushed the stanley/expo-serve-sim-03-vendor-livekit-webrtc branch from 198d472 to 222f286 Compare June 30, 2026 12:34

sjchmiela changed the base branch from stanley/expo-serve-sim-02-tunnel-support to main June 30, 2026 12:34

sjchmiela marked this pull request as ready for review June 30, 2026 12:34

sjchmiela added 5 commits June 30, 2026 15:57

Vendor LiveKit WebRTC framework and native build integration

1f56185

Add WebRTC transport and signaling

97dcbbc

Add WebRTC codec selection and fallback

7fec26a

Improve WebRTC ICE and TURN handling

8b46cbc

Fix packaged WebRTC H264 runtime behavior for Tart VMs

64ebdfd

sjchmiela force-pushed the stanley/expo-serve-sim-03-vendor-livekit-webrtc branch from 32c5cf5 to 64ebdfd Compare June 30, 2026 14:02

sjchmiela changed the title ~~Vendor LiveKit WebRTC framework and native build integration~~ Vendor LiveKit WebRTC and add WebRTC transport Jun 30, 2026

sjchmiela added 2 commits June 30, 2026 16:48

Fix WebRTC PR build

0050425

Forward stream options to detached helpers

e3e885a

szdziedzic self-requested a review June 30, 2026 15:13

sjchmiela added 3 commits June 30, 2026 22:32

Fix WebRTC input data channel handling

fc14c97

Clean up WebRTC PR surface

589cb59

Clarify WebRTC transport naming

4d49835

sjchmiela added 3 commits June 30, 2026 23:27

Remove unused WebRTC hook state

394f4f8

Address WebRTC naming review comments

d0f1dfa

Consolidate stream settings in preview config

6ab3329

sjchmiela force-pushed the main branch from 7191593 to f94d57c Compare July 1, 2026 15:04

szdziedzic reviewed Jul 2, 2026

View reviewed changes

sjchmiela added 3 commits July 2, 2026 11:35

Fix WebRTC preflight and MJPEG fallback URLs

76f77db

Harden native WebRTC signaling

4de4d70

Add WebRTC first-frame fallback

8930a8c

sjchmiela requested a review from szdziedzic July 2, 2026 09:37

szdziedzic approved these changes Jul 2, 2026

View reviewed changes

Clean up WebRTC offer signaling

63f3ed3

Uh oh!

Conversation

sjchmiela commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szdziedzic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sjchmiela commented Jun 29, 2026 •

edited

Loading