Multicam Director mode: direct up to 4 iPhone cameras with synced capture - #203
Open
darioalessandro wants to merge 13 commits into
Open
Multicam Director mode: direct up to 4 iPhone cameras with synced capture#203darioalessandro wants to merge 13 commits into
darioalessandro wants to merge 13 commits into
Conversation
…c metadata helpers Inert groundwork for the multicam director feature (PR0 of the plan): - FeatureFlags.ENABLE_MULTICAM=false master switch; cameras advertise the new supports_multicam capability tied to the flag, so the same release that flips it on starts advertising. - FlatBufferSchemas.fbs: supports_multicam appended to CameraCapabilities (same evolution pattern as supports_focus_point), regenerated with flatc, round-tripped in RemoteCmdSerializationTests including the legacy-peer absent-field default. - CaptureSyncMetadata: the per-clip alignment record (shared director-clock anchor + captureId/sessionId + offset quality) with filename prefix, QuickTime metadata items, and deterministic JSON; unused until synced capture ships. No behavior change; full suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two behavior-preserving cuts that the multicam director needs (PR1): - MultipeerServiceDelegate.didReceiveMessage now carries the source peer. The 1:1 SessionCoordinator ignores it (its single link makes the source unambiguous); a multicam controller will route responses by it. - sendMessage/sendOrGoToScanning accept an explicit peer list (nil = all connected, unchanged), and the monitor's frame ack now addresses only the camera whose frame was consumed — with several cameras, a broadcast ack would advance every camera's credit window on one camera's frame. Single-cam behavior is identical (one connected peer makes both forms the same). New test proves the ack targets only the sending peer with two peers connected; full suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR2 of the multicam plan. The synced-shutter foundation: the director will schedule captures on each camera's own clock, which needs a measured per-camera offset. - Wire: ClockSyncPing (action 25, carries director clock at send) answered with a CameraStateResponse echoing t0 plus the camera clock at receipt. Appended-field schema evolution; round-trip tested. Only ever sent to peers advertising supports_multicam. - The camera answers pings in the nonisolated delegate callback, off the actor inbox, so queued state-machine work cannot smear the timestamp (same pacing precedent as frame acks). Pong is addressed to the pinging peer only. - ClockOffsetEstimator: pure NTP-style min-RTT-of-5 window; offset = cameraClock − (t0 + rtt/2); rejects stale pongs; reset() for background/reconnect invalidation. SyncClock supplies the monotonic ms clock both sides read. Inert in production until a director sends pings. Full suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The director engine and screen, all behind ENABLE_MULTICAM (off), so every single-camera path stays byte-identical. Engine (new, unit-tested): - MulticamController: a sibling actor of SessionCoordinator, reached only for a multicam director session. Owns the transport as its delegate after the scanner hands it off; keeps a CameraLink per camera keyed by MCPeerID. Per-camera capability handshake, per-camera frame routing with a per-source RequestFrame ack (Seam B), focused-peer command addressing, a per-camera clock-sync loop (ClockSyncPing/Pong -> ClockOffsetEstimator, ~30s + on foreground), keep-browsing + re-invite of a dropped camera (its lane degrades to .reconnecting; the others are untouched), and logical camera removal. The camera side is unchanged: a camera can't tell a multicam director from a single monitor. - CameraLink: per-camera status/capabilities/clock estimate. UI (new): - MulticamViewController hosts MulticamView in focus mode: the focused camera fills the viewfinder (reusing the 1:1 LiveFrameView) with a floating strip of the other cameras' live thumbnails; tap to refocus. Each lane has its own FrameDisplayModel + FrameStreamReceiver, so a frame from camera B never re-renders camera A's tile. Per-tile reconnecting scrim. Grid mode is a later PR. - MulticamViewModel/CameraLane reconcile controller snapshots while preserving lane instances (and their live streams). Scanner (flag-gated, additive): - With ENABLE_MULTICAM and the monitor role, the scanner accumulates connected cameras instead of auto-advancing on the first connect, and shows "Start (N)". One camera runs the classic MonitorViewController unchanged; two or more hand the live transport to a MulticamController and push the director. The SessionCoordinator seam is a single flag-defaulted-false collecting mode; every non-multicam path is untouched. Tests: MulticamControllerTests (handshake, per-lane frame routing + source- only ack, focused-only commands, disconnect isolation, browser re-invite, clock offset, removal) and MulticamViewModelTests (lane reconcile/focus). Full suite green (668 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Schedule-at-timestamp synced stills across the rig, behind ENABLE_MULTICAM. Wire (additive, same evolution pattern): - ScheduledCapture (action 26): director -> camera, carrying the shutter instant in the camera's own SyncClock domain (director applied the offset), the shared director-clock anchor (the alignment key stamped into each clip), capture id / session id, and the camera's 1-based index. - CameraStateResponse gains capture_id_echo for the ack. RemoteCmd ScheduledCapture / ScheduledCaptureAck, both dispatch switches, round-trips. Camera side (SessionCoordinator, single-cam untouched): - On ScheduledCapture: ack (or nack if the fire time is > 1s past) immediately, then schedule the shutter. The delay runs OFF the actor and only enqueues FireScheduledCapture, so the capture is pulled by the message pump in order and never races a state transition (the flagged concern). The photo saves locally as today, additionally stamped with CaptureSyncMetadata: EXIF UserComment JSON + DateTimeOriginal/SubSec, and a shared RS_<sess>_<cap>_cam<k> originalFilename so any editor can group and align the angles. A stamping failure never costs the user the photo. Director side (MulticamController): - capturePhoto(): picks fireAt = now + 150ms, sends per-camera ScheduledCapture with fireAt + lane.offset to every linked multicam lane; if any offset is missing, falls back to a plain TakePic fan-out under the same shot id. Aggregate state capturingPhoto(captureId, acksRemaining); best-effort policy — per-lane 3s ack timeout marks a silent camera failed; returns to monitoring when all lanes acked/nacked/timed out. Per-lane outcome (captured/failed) surfaced to the tile badge. UI: the MulticamView shutter (reusing the 1:1 ShutterButton + activity ring) fires capturePhoto(); tiles show a captured/failed badge. Tests: serialization round-trips; MulticamControllerTests (per-lane offsets => different fire instants, non-multicam lane excluded, ack aggregation, timeout completion, fallback); camera-side (past-fire nack, valid fire acks immediately then pulls the shutter). Full suite green (677 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix (from PR4 review): CaptureSyncMetadata.stamped derived EXIF DateTimeOriginal from anchorMillis, which is monotonic SyncClock uptime, so photos got ~1970 dates. DateTimeOriginal/SubSec now come from the camera's wall clock at capture (capturedAt); anchorMillis stays an opaque alignment key inside the UserComment JSON (and the QuickTime keys for video). Wire (additive): ScheduledStartRecording (27), ScheduledStopRecording (28), reusing the PR4 capture params; ScheduledRecordingAck echoes the capture id and carries isStop so the director routes start vs stop acks. Both dispatch switches + round-trips. Camera side (single-cam untouched): scheduled start/stop fire at their instants via the same off-actor-delay -> pump-message pattern as PR4 (no actor/sleep race). The recording pipeline runs as today (local save, no auto-transfer in multicam) plus QuickTime metadata items on the AVAssetWriter and an RS_<sess>_<cap>_cam<k>.mov filename. Resilient camera (gated hard on inMulticamSession): the latch is set only by an incoming scheduled multicam command and cleared on session teardown, so with ENABLE_MULTICAM off no camera advertises multicam, no director sends scheduled commands, and the latch never sets -> single-cam byte-identical. When set, a mid-recording director drop keeps the clip rolling (the rest of the rig is still recording) and enters the reconnect path instead of stopping. Proven both ways. Director side (MulticamController): startRecording/stopRecording with the same per-lane offset scheduling; aggregate recording/stoppingRecording states; per-lane REC badge; shutter gains a photo/video mode toggle and a record/stop button reusing the 1:1 ShutterButton. Tests: serialization round-trips; per-lane offsets give distinct start AND stop fire times with matching clip lengths; start acks mark lanes recording and stop returns to monitoring; multicam disconnect keeps recording, single-cam still stops; scheduled recording stamps metadata; EXIF date is wall-clock not the anchor. Full suite green (686 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 11, 2026
Contributor
📊 Test Coverage Report✅ 715/727 tests passed · ⏭️ 12 skipped
🎥 Capture & session core
📁 Full report — 96 files
Generated from |
Tiered previews so N camera streams fit the aggregate bandwidth + decode budget, plus a grid "monitor wall". All behind ENABLE_MULTICAM. Wire (additive): SetStreamProfile (action 29, params max_long_edge / bitrate_kbps / fps). Both dispatch switches + round-trip. Camera side (single-cam untouched): FrameStreamer gains a mutable active profile that defaults to today's full peer values — a streamer that is never re-profiled behaves identically. applyProfile (capture-queue confined via FrameStreamingCoordinator -> CameraRig -> CameraControlling) rebuilds the still chain and drops the video encoder so it is rebuilt at the new resolution on the next frame — the same drop-and-rebuild path the failure fallback already uses, so the new encoder's first frame is a keyframe the monitor re-syncs on. SetStreamProfile is handled in inCamera and inCameraRecordingVideo (the preview keeps streaming while recording). Director side (MulticamController): the focused lane gets StreamProfile .focused (1200/1.2Mbps/30), the rest .thumbnail (640/500kbps/20). Profiles are pushed when a lane goes live and re-tiered on focus switch, de-duped via CameraLink.lastSentProfile so nothing is re-sent when the tier is unchanged. Per-peer keyframe/stall routing was already per-peer from PR3. Grid UI: MultiCamChrome pure layout policy (near-square columns; toggle only when >1 camera) with unit tests. MulticamView gains a focus/grid toggle and a LazyVGrid of the same per-lane tiles; tapping a grid tile focuses it and returns to focus mode. Per-camera controls hide in grid; the capture cluster stays. Tests: SetStreamProfile round-trip; FrameStreamer rebuilds the encoder on profile change (and not on a no-op change); director tiers focused vs thumbnail and re-tiers on focus switch without redundant sends; MultiCamChrome columns/toggle. Full suite green (694). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StoreManager.maxCameras() { hasFullAccess() ? 4 : 2 } beside the other gates,
with StoreManagerTests coverage. The director's own entitlement is what
counts (checked locally, like every existing gate).
Enforced at invite time in BOTH paths:
- Scanner multi-select: a connect past the cap routes to the paywall instead
of inviting (only when collecting; the count stays 0 otherwise, so
single-cam is untouched).
- In-session AddCameraSheet: the director keeps browsing; discovered-but-not-
joined cameras are surfaced (MulticamController tracks an available set and
publishes it) and listed in a sheet reached from an "Add camera" tile at the
end of the strip. Tapping the tile at the cap opens the paywall; below the
cap it opens the sheet. inviteCamera() invites a chosen peer.
The paywall is the existing SettingsView sheet in both places — no bespoke
multicam paywall. Its Pro footer gains "Direct up to 4 cameras at once" (only
when the flag is on). Role picker reframes the monitor role as
"Director" / "Control one or more iPhone cameras" under the flag; single-cam
keeps "Remote" / "Control the shutter".
Tests: maxCameras 2 free / 4 pro (mode + subscription); a discovered peer
becomes available without auto-joining; inviteCamera invites it (and ignores
an undiscovered one); an available peer clears once it joins the rig. Full
suite green (700).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every multicam UI string translated across all 15 shipped locales (da, de-DE, en, es-MX, fr-FR, hi, it, ja, ko, ms, pt-BR, ru, tr, vi, zh-Hans), faithful to each file's existing register: Start (%d), Add camera, Add, Searching for cameras…, Director, Control one or more iPhone cameras, Direct up to 4 cameras at once, RECONNECTING RECONNECTING was previously unlocalized (the key fell back to itself in every language, including the 1:1 monitor's reconnect chip) — now properly translated everywhere. Start (%d) existed only in en; filled in the other 14. All 15 files lint clean (plutil). Deviation from the brief's suggested string list: "%d of %d cameras", "Connected to director", and "Unlock 4 cameras with Pro" are NOT added — the UI landed without them (the count lives in "Start (%d)"; the paywall is the existing Settings sheet, not a bespoke CTA; there is no camera-side director badge in v1). Only the eight strings actually referenced in code are shipped, so there are no dead keys. Full suite green (700). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dario found the collecting scanner confusing — discovered rows didn't read as selectable. Reworked to the Photos/Mail edit-mode idiom (the shape Final Cut Camera's Add Angles uses). - Each discovered-camera row gets a leading selection circle: `circle` (secondary) unselected → spinner connecting → `checkmark.circle.fill` (accent) selected, with an accent row border when selected. Tapping the row toggles: unselected → invite/connect; selected → deselect. Deselect is logical (the QUIC transport has no per-peer teardown), honored by an effective-peer set the count, handoff and scanner all read; a still-connected camera re-selects instantly without re-inviting. The cap is enforced at selection time — an over-cap unselected row shows a lock and routes to the existing paywall. - "Connect All" row at the top of the list invites every discovered, not-yet-selected camera up to maxCameras(); hidden once all are selected. The over-cap remainder keeps the per-row lock. - Start (N) stays the bottom CTA; its count is the effective selection, so selection state and the count are always consistent. - Single-camera / flag-off scanner is byte-identical (all of this is gated on ENABLE_MULTICAM && monitor role; the coordinator's toggle handler is a no-op unless collecting). - New string "Connect All" fanned out to all 15 locales. Wire: UICmd.ToggleMulticamCamera (in-process). SessionCoordinator gains an effective-peer set (connected − deselected) driving multicamConnectedCount, detach and promote. Tests: VM row-state transitions + Connect-All cap math; coordinator select→connect→deselect→reselect round trip. Full suite green (703). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dario's device pass found tapping a row connected immediately. Reworked to
true two-phase selection: tap picks (zero network), a bottom CTA connects the
chosen set.
Semantics (monitor role, flag on):
- Tap = pure selection. toggleMulticamSelection flips a checkmark and fires
NO invite; the view model is entirely network-free. Spinner/connected/failed
states appear only during the connect phase.
- "Select All" (renamed from "Connect All") picks up to maxCameras(), pure.
- Bottom CTA is "Connect (N)" (renamed from "Start (N)"): disabled at N=0,
fires one invite per selected camera, shows per-row spinners, and hands off
once every invite settles — 1 connected → classic monitor, ≥2 → director.
A row that fails after the retry is marked; the rig proceeds with whoever
connected; all-fail returns to the scanner with the existing error alert.
- Per-peer connect retry/timeout reuses the 1:1 20s-invite / retry-once
pattern, tracked per camera (multicamInviteAttempts) instead of the single
`link`.
- Single-cam / flag-off scanner byte-identical.
Dead code removed (the logical-removal model only applied post-connect, which
no longer happens in the scanner): UICmd.ToggleMulticamCamera, the coordinator
deselected-set + effectiveMulticamPeers, and the VM's updateMulticamSelection
/ peersToConnectAll / multicamCollectedCount. Handoff decision extracted to a
pure MulticamHandoff.decide. Strings: dropped "Start (%d)" and "Connect All",
added "Connect (%d)" and "Select All" across all 15 locales.
Parametrized screen tests (the point): a 0…10-discovered × {cap 2,4} × k
sweep asserts selecting k rows yields zero invites, the CTA is disabled iff
k==0 with label k, and Connect invites exactly the k selected; plus
connecting→connected/failed transitions plus handoff destination, cap locking
beyond max, empty state, and a 3-selected/1-fails partial. A coordinator-level
guard asserts selection sends zero invitePeer on the fake transport (the
regression). Full suite green (709).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"Framing belongs to a camera; the shot belongs to the rig." Focused-camera controls (zoom/focus/flash/lens) are unchanged; a new rig-scope tray adds one self-timer and rig-wide quality, reachable from both focus and grid modes. Quality — the intersection model (Apple's hybrid, decided with Dario): - RigQualityMenu: a pure, unit-tested type that intersects every connected lane's current-camera VideoQualityCapabilities (resolution×fps matrix) and PhotoQualityCapabilities (HEIF/HDR). An option is offered only when every camera supports it; non-intersection options are listed greyed, naming the camera(s) that block them. - Manual selection within the intersection is FIRST-CLASS: tap 1080p30, tap 4K30, each fans SetVideoQuality to every lane. "Automatic" (best-in- intersection: highest res then fps, floor 1080p30) is the reset at the top, not a mode you must leave. HEIF/HDR the same via SetPhotoQuality. - Late joiner / device switch that can't match the running rig setting badges its tile and offers a re-match (re-run Automatic) — the rig is never silently changed. Timer: one director countdown; each tick fans TimerCountdown to every camera (subjects see it) and expiry triggers the existing synced capture/record path. No per-camera timers. The controller tracks the active rig quality, computes the RigSettingsSnapshot (picker options + blockers + active labels) for the tray, and publishes it on caps change. New strings → 15 locales. Tests: RigQualityMenu intersection math (homogeneous/heterogeneous/empty→floor, photo intersection, late-joiner, Automatic order); controller fan-out sends the chosen quality to every lane; manual toggle fans out each time; Automatic picks best-in-intersection; late joiner flagged; timer counts down, fans out, and fires the synced capture. Full suite green (723). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every take now gathers to the director, while each camera keeps its own copy. Photos (inline return): a scheduled ScheduledCapture now stamps the still once (EXIF sync JSON + wall-clock DateTimeOriginal), saves that stamped image locally under RS_<sess>_<cap>_cam<k>, AND returns it to the director via the existing TakePicResp media path. The director saves it to its own library under the shared RS_ name and marks the lane collected. Decision: inline return (not resource) — stills are small, so no staggering needed. Videos (resource transfer): a scheduled stop now saves locally AND pushes the clip to the director via the existing sendResource path (single-cam already did exactly this). The camera names the transfer with the RS_ filename so the director saves under the group; QuickTime sync metadata rides inside the .mov, untouched by transfer. Multicam clips are COPIED (not moved) to Photos so the temp survives the send/retry; the next recording cleans it up. Sequencing (decision): lane-index staggered send — each camera delays its transfer by (cameraIndex-1) × stagger (default 2s) so N×4K don't hit the link at once. Previews may degrade during collection (acceptable). Limitation: a fixed delay can overlap if one transfer runs long; a director-coordinated turn-taking is a follow-up. Per-peer receive routing: the resource delegate callbacks now carry the source peer (Seam A for resources; the 1:1 coordinator ignores it). The director tracks per-lane collection state (idle → transferring → collected/failed), shown as a tile badge; a failed transfer marks the lane (footage still safe on the camera) with a Retry that sends RequestVideoResend (new additive action 30), and the camera re-sends its held clip. Tests: video transfer start/finish updates lane state + saves; failed marks + retry re-requests to that peer; returned photo marks collected; scheduled capture/stop now send media to the director; RequestVideoResend round-trip. Full suite green (727). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
darioalessandro
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Remote Shutter's biggest feature since the remote itself: Director mode. One phone (the director — today's monitor role) connects up to 4 camera phones over Stormo (QUIC, direct peer-to-peer — no Wi-Fi network needed), shows live previews of every angle, and fires synchronized photo capture and video recording across all of them. Every camera saves full-res locally and the footage auto-collects to the director after each take, stamped with alignment metadata (EXIF/QuickTime + shared
RS_<session>_<shot>_cam<k>filenames) so any editor — CapCut, FCP, Resolve — can line the angles up. No bundled editor, deliberately.Free = 2 cameras, Pro = 4 (existing one-time
06purchase; gate checked on the director only, at invite time, routed through the existing paywall).How synced capture works
ClockSyncPing/Pong, min-RTT-of-5 estimator, re-synced every ~30s and on foreground) measures each camera's clock offset to ±2–5ms.UX
Compatibility & safety
ENABLE_MULTICAMis stillfalse— merging this changes nothing user-visible. The flip ships separately with the 9.1.0 release.supports_multicamcapability; 9.0.x peers pair as ordinary single cameras (no flag-day, same-major gate untouched).Test coverage
727 green across the stack: multi-peer loopback (frame routing, per-peer acks), clock-offset estimator math, scheduled-capture offset application, parametrized scanner sweep (0–10 discovered × free/Pro caps × 0–n selected, incl. no-invite-before-Connect), quality-intersection math, transfer/retry state, snapshot + chrome layout tests, wire round-trips for every new message.
Commit guide (reviewable in order)
scaffolding → transport seams → clock sync → director core + focus/strip UI → synced photo → synced video + resilient camera → grid + stream profiles → gating/paywall → localization (15 locales) → scanner multi-select → select-then-connect + parametrized tests → rig tray + quality intersection → auto-collect.
Not in this PR (ships with the release flip)
ENABLE_MULTICAM=true, 9.1.0 version bump, App Store keywords ("multicam", "multi camera recording"), director screenshots, updated06IAP description.🤖 Generated with Claude Code