diff --git a/README.md b/README.md index 6f2742b..bdfe8e8 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,14 @@ blanks the source. **Apple Log** image and grade it yourself with OBS's Apply LUT filter — the plugin decodes it faithfully and otherwise keeps its hands off. Nothing else in this space offers it. +- **Virtual green screen (beta).** Remove your background without a + green screen: the phone segments you (any camera, no depth hardware + required), composites the background to chroma green before encoding, + and the plugin auto-adds a pre-configured chroma-key filter — toggle + on the phone, keyed subject in OBS. On Face ID front cameras and + LiDAR rear Main cameras, depth sharpens the matte and a + **subject distance** cutoff (app Live screen or web panel) drops + people walking by behind you. - **Pick any lens** — Main, Ultra Wide, Telephoto, or Front — switchable live while you stream. - **Hold it however.** The app's UI follows the phone's rotation (unless diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index c34de26..af92922 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -230,3 +230,16 @@ Deliberate divergences (don't "fix" these without reading this): - **Rotation is sensor-native** (`.landscapeRight`, an effective 0°). The AVCaptureConnection docs warn per-frame rotation costs; we never rotate the stream, only the on-phone preview. +- **Green screen OFF is free, ON pays exactly once.** With the toggle + off, the capture→encode path is the pre-feature code verbatim — no + compositor branch on the hot path, no depth output attached. On, the + budget is: Vision person segmentation per frame (`.fast` above + 30 fps, `.balanced` at 30 — the dominant cost), one Metal compute + pass writing green into a pool buffer (that pass *is* the single + copy; the camera's own buffer is never written in place — it races + the encoder's async read), and depth at ~15 Hz when assist is + active. Depth delivery deactivates Center Stage and the system video + effects by iOS policy, and overload sheds frames via the existing + drop-don't-queue capture behaviour. On-device bench + thermal-soak + numbers for green-screen-ON are still to be captured and pasted + here; the OFF run must show the zero-cost claim holds. diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.md index 3491672..ab624b6 100644 --- a/docs/PROTOCOL.md +++ b/docs/PROTOCOL.md @@ -123,6 +123,7 @@ Camera remote control. Payload: UTF-8 JSON, one command per packet: { "cmd": "set_format", "resolution": "1080p", "fps": 60, "codec": "hevc" } { "cmd": "mic", "id": "builtin:2" } { "cmd": "tally", "program": true, "preview": false, "sync": "locked" } +{ "cmd": "green_screen", "on": true, "maxDistance": 2.5 } ``` `set_format` switches the capture format mid-stream; any subset of its @@ -173,6 +174,18 @@ serving as source audio (type 10). An older app ignores the command and streams continuously, which the plugin handles fine; a plugin that never sends it gets today's always-on behaviour. +`green_screen` toggles the virtual green screen and its depth cutoff; +either field may appear alone. `on` arms/disarms the segmentation + +green composite (the compositing happens **on the phone, before +encoding** — the wire carries ordinary video whose background happens +to be chroma green, so receivers need no new decode behaviour). +`maxDistance` (metres; `0` = no cutoff, otherwise 0.5–5.0) drives the +depth-assisted subject cutoff and is meaningful only while depth +assist is active — the app clamps and ignores as needed. Green screen +is SDR-only: arming it forces the Standard colour pipeline, and the +STATE fields below keep every surface honest about what's running. +Camera connections only; screen mirror ignores it. + Unknown commands are ignored, so new ones can be added compatibly. The plugin's embedded web panel (http://localhost:9980) generates these. @@ -216,6 +229,20 @@ H.264 through the ordinary `set_format` validation. Both fields are absent on SDR streams, whose snapshots are unchanged from before colour modes existed. +Green screen state rides the snapshot the same way: +`"supportsGreenScreen": true` advertises the feature (remote UIs gate +their row on it), `"greenScreen": true` appears while it is armed, +`"greenScreenDepth": true` while depth assist is actually running +(TrueDepth front / LiDAR rear Main lens, and a depth-capable format +matched — absent means segmentation-only; remote UIs key the distance +control off truthiness), and `"greenScreenMaxDistance"` carries the +cutoff in metres when one is set. Boolean snapshot fields are emitted +only when true: a receiver's STATE cache has finite room (older +plugins truncate silently past 1 KiB), so snapshot growth is real +cost — weigh every new field against it. The plugin also reacts to `"greenScreen": true` on a +camera source by auto-adding a pre-configured chroma-key filter — +once, by name, respecting a user's later deletion of it. + While the phone mic streams as the source's audio (packet type 10), the snapshot additionally carries `micEnabled: true`, the selected `mic` id, and the selectable `mics` list (`[{ "id", "name" }, …]`) for the `mic` diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index a47d23f..5cdd12d 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -17,26 +17,28 @@ Continuity Camera, iVCam/Iriun, NDI HX Camera/Larix). | P1 | Thermal & low-power adaptation | Reliability & security | Medium | | P1 | Optional pairing & encryption | Reliability & security | Medium | | P1 | Graduate the GPU decode pipeline | Performance | Small | -| P1 | Tally light on the phone | Control & workflow | Small | | P2 | Digital pan/tilt + crop | Control & workflow | Medium | -| P2 | Virtual green screen (background removal) | Video & audio quality | Medium | -| P2 | 10-bit HEVC + HDR (HLG/PQ) | Video & audio quality | Large | | P2 | Voice isolation for the phone mic | Video & audio quality | Small | | P2 | Document the control API | Control & workflow | Small | -| P3 | Apple Log capture | Video & audio quality | Small¹ | | P3 | Manual bitrate cap | Video & audio quality | Small | | P3 | Zero-copy encoder output | Performance | Medium | | P3 | iPad: keep streaming in Split View | Control & workflow | Small | | P3 | Two lenses at once (multicam) | Video & audio quality | Large | -¹ once 10-bit HEVC ships. - - **P1 — next up.** Protects or finishes what already ships, plus the one tiny-effort/large-payoff outlier. Pick from here first. - **P2 — on deck.** Clearly worth building; start once P1 is moving. - **P3 — needs a trigger.** A dependency landing, a measurement, or real user demand should promote these before anyone starts. +Recently shipped and pruned from this file (see the release notes): +tally light with customizable colours (v1.8.0), 10-bit HDR (HLG) with +the zero-copy GPU path and Apple Log (v1.9.0), virtual green screen +with depth assist and subject-distance cutoff (v1.10.0). Pairing & +encryption was explicitly deprioritized by the maintainer in 2026-07 — +revisit when remote start gets promoted or users stream on shared +networks. + ## Reliability & security ### Thermal & low-power adaptation on the phone — P1, medium (partially shipped) @@ -91,33 +93,6 @@ if you do it.* ## Video & audio quality -### 10-bit HEVC + HDR (HLG/PQ) — P2, large -iPhones capture HDR by default; the current 8-bit NV12 pipeline throws -that away, and DroidCam already ships 10-bit HDR transfer. App: 10-bit -capture format + HEVC Main-10 in `VideoEncoder`; protocol: VIDEO_CONFIG -gains colour metadata (primaries/transfer/matrix); plugin: map to -P010 + OBS's HDR colour handling (OBS 28+), including the GPU -pipeline's per-backend texture formats. Ship with a sensible SDR -default — Twitch is still SDR, so HDR mainly pays off for YouTube and -local recording. *The headline video-quality differentiator; the only -large item on the active list, and it unlocks Apple Log below.* - -### Virtual green screen — background removal without the screen — P2, medium -Camo's flagship paid feature, on the free table. Vision's person -segmentation (`VNGeneratePersonSegmentationRequest`, iOS 15 — exactly -our floor) mattes a person from plain RGB on any camera, front or rear -— no depth hardware required. The wire stays untouched: composite the -removed background to solid chroma green on the phone and let OBS's -chroma key provide the transparency. Staged: **v1** segmentation + -green composite behind an Options toggle (per-frame GPU work — quote -before/after numbers, lean on the thermal mitigation); **v2** the app -advertises the mode in STATE and the plugin auto-adds a pre-configured -chroma-key filter to the source, so the phone toggle alone yields a -keyed subject in the scene; **v3** depth-assisted edge refinement -(hair, glasses) where TrueDepth or LiDAR exists — this is where -Apple's "enhancing live video with TrueDepth" techniques earn their -keep, as an enhancer rather than a requirement. - ### Voice isolation for the phone mic — P2, small The mic features send the raw microphone feed; iOS's built-in voice-processing pipeline (echo cancellation + noise suppression, the @@ -125,13 +100,6 @@ FaceTime mic sound) is nearly free to enable. An Options toggle in the Microphone group, applied to the send-phone-mic path. *Makes the "phone as wireless mic" feature genuinely usable in untreated rooms.* -### Apple Log capture — P3, small (gated on 10-bit HEVC) -On iPhone 15 Pro and later the camera can capture in Apple Log -(`AVCaptureColorSpace.appleLog`); streamers grade it with a LUT in OBS. -Once the 10-bit path exists this is mostly a capture-format toggle plus -correct colour tagging, gated to devices that report support. Nothing -else in this market offers it. *Trigger: 10-bit HEVC shipping.* - ### Two lenses at once (multicam) — P3, large `AVCaptureMultiCamSession` (iOS 13+) can run the front and a rear camera simultaneously — one phone feeding a face cam *and* a scene cam @@ -153,17 +121,6 @@ covers most of them today.* ## Control & workflow -### Tally light on the phone — P1, small -Pro rigs treat tally as table stakes, and no phone-webcam app has it. -The plugin already tracks per-source visibility (it drives "Disconnect -when this source isn't shown anywhere") and can read program/preview -state from the frontend API; send it to the app as a new CONTROL -command (`{"cmd":"tally","state":"live|preview|idle"}` — old apps -ignore unknown commands, so it's compatible) and surface it on the -Live screen. UI_DESIGN.md §2 needs a tally state added to the status -vocabulary. For multi-phone scenes this removes the "which camera am I -on?" guesswork. *The best payoff-per-line on this list.* - ### Digital pan/tilt + crop — reframe without touching the rig — P2, medium Zoom today is the camera's own, always center-locked. An adjustable crop applied on the phone *before* encoding (Camera Hub's signature diff --git a/docs/UI_DESIGN.md b/docs/UI_DESIGN.md index 0dc6106..49c790c 100644 --- a/docs/UI_DESIGN.md +++ b/docs/UI_DESIGN.md @@ -248,6 +248,7 @@ surfaces. | Stop | `stop.fill` | Red chip; the only destructive control | | Dim (app) | `moon.fill` | App-only battery saver | | Stats (app) | `gauge` | App-only toggle; shows a health pill (`60 fps · 11.9 Mb/s · 0 dropped`, monospaced) under the status bar | +| Green screen | `person.fill.viewfinder` | **Always "Green screen"** (never "chroma key", "background removal", or "matte" in UI copy). Armed from the Setup screen; while live **with depth assist**, a "Subject distance" slider row (0.5–5.0 m, readout `2.5 m` monospaced) appears on the Live panel and the web panel, same position both surfaces. Dragging always sets a real cutoff — full-left = tightest (0.5 m); **"All" (no cutoff) is entered by tapping the readout**, and while "All" the thumb parks at the far (5.0) end. Identical on both surfaces | ### The app icon (three appearances) diff --git a/ios-app/Sources/CameraManager.swift b/ios-app/Sources/CameraManager.swift index f08a5c6..af45599 100644 --- a/ios-app/Sources/CameraManager.swift +++ b/ios-app/Sources/CameraManager.swift @@ -53,6 +53,19 @@ final class CameraManager: NSObject { private var _onSampleBuffer: ((CMSampleBuffer) -> Void)? private let callbackLock = NSLock() + /// Latest synchronized depth frame while depth assist is active + /// (green screen). Same locking pattern as `onSampleBuffer`: set on + /// the main thread, read per frame on the capture queue. Depth runs + /// slower than video, so this fires less often than `onSampleBuffer` + /// (and never when depth assist is off). + var onDepthData: ((AVDepthData) -> Void)? { + get { callbackLock.lock(); defer { callbackLock.unlock() } + return _onDepthData } + set { callbackLock.lock(); defer { callbackLock.unlock() } + _onDepthData = newValue } + } + private var _onDepthData: ((AVDepthData) -> Void)? + /// Capture was interrupted (phone call, Camera app, Split View) or /// resumed. Delivered on the main queue. var onInterruption: ((Bool) -> Void)? @@ -67,6 +80,18 @@ final class CameraManager: NSObject { private let videoQueue = DispatchQueue(label: "obscam.video") private var videoOutput: AVCaptureVideoDataOutput? + /// Depth-assist plumbing (green screen). The synchronizer must stay + /// retained in a property or synchronized delivery silently stops. + private var depthOutput: AVCaptureDepthDataOutput? + private var outputSynchronizer: AVCaptureDataOutputSynchronizer? + + /// True only when the depth path fully armed during the last + /// configure: depth sibling device selected, a depth-capable format + /// matched the EXACT requested resolution+fps, and the synchronizer + /// replaced the plain video delegate. False = segmentation-only + /// (graceful degrade). Valid once configure() has returned. + private(set) var depthAssistActive = false + override init() { super.init() // Without these observers a phone call or the Camera app grabbing @@ -165,10 +190,33 @@ final class CameraManager: NSObject { position: lens.position) } + /// The depth-registered sibling of a user-facing lens: TrueDepth for + /// the front camera, LiDAR for the rear Main (Wide) lens — Apple + /// registers their depth maps to exactly those YUV cameras. Ultra + /// Wide and Telephoto have no depth sibling, and LiDAR needs + /// iOS 15.4 (`.builtInLiDARDepthCamera` doesn't exist below that). + /// nil = depth assist unavailable; capture stays on the normal + /// device, segmentation-only. + private static func depthSiblingDevice(for lens: Lens) -> AVCaptureDevice? { + if lens.position == .front { + return AVCaptureDevice.default(.builtInTrueDepthCamera, + for: .video, position: .front) + } + if lens.position == .back, + lens.deviceType == .builtInWideAngleCamera { + if #available(iOS 15.4, *) { + return AVCaptureDevice.default(.builtInLiDARDepthCamera, + for: .video, position: .back) + } + } + return nil + } + private static func format(for device: AVCaptureDevice, resolution: Resolution, fps: Int32, - color: StreamColor) -> AVCaptureDevice.Format? { + color: StreamColor, + requireDepth: Bool = false) -> AVCaptureDevice.Format? { let target = resolution.size // Require exact dimensions and a frame-rate range covering the // requested rate; earlier formats (unbinned, video-range) win ties. @@ -176,11 +224,17 @@ final class CameraManager: NSObject { // capture BT.2020 HLG; Apple Log a 10-bit 4:2:2 (x422) format // that can capture .appleLog — nil (unsupported) if none exists. // SDR considers every format, exactly as before colour existed. + // `requireDepth` (depth assist) additionally requires a format + // that can pair with a depth stream — the depth sibling devices + // carry both depth-capable and depth-less formats. let candidates = device.formats.filter { format in let dims = CMVideoFormatDescriptionGetDimensions(format.formatDescription) guard dims.width == target.width, dims.height == target.height else { return false } + if requireDepth, format.supportedDepthDataFormats.isEmpty { + return false + } switch color { case .sdr: break @@ -252,38 +306,83 @@ final class CameraManager: NSObject { static func formatReport() -> String { var out = ["\(UIDevice.current.model) — iOS " + UIDevice.current.systemVersion, - "flags: binned / CS / P / SL / R / colour", ""] + "flags: binned / CS / P / SL / R / colour / depth", ""] for lens in availableLenses() { guard let device = device(for: lens) else { continue } out.append("== \(lens.label) ==") - for format in device.formats { - let dims = CMVideoFormatDescriptionGetDimensions( - format.formatDescription) - let maxFps = format.videoSupportedFrameRateRanges - .map(\.maxFrameRate).max() ?? 0 - var flags = [format.isVideoBinned ? "b" : "-", - format.isCenterStageSupported ? "CS" : "--", - format.isPortraitEffectSupported ? "P" : "-"] - if #available(iOS 16.0, *) { - flags.append(format.isStudioLightSupported ? "SL" : "--") - } else { - flags.append("?") - } - if #available(iOS 17.0, *) { - flags.append(format.reactionEffectsSupported ? "R" : "-") - } else { - flags.append("?") - } - flags.append(colourFlags(format)) - out.append(String(format: "%5dx%-5d fps<=%-3.0f %@", - dims.width, dims.height, maxFps, - flags.joined(separator: " "))) - } + appendFormatRows(of: device, to: &out) + out.append("") + } + // The depth-sibling devices (green screen's depth assist) carry + // their own format tables, different from the plain lenses above. + // Dump them under their own headers so device reports answer + // which resolution+fps combos can actually carry depth. + var depthSiblings: [(String, AVCaptureDevice)] = [] + if let trueDepth = AVCaptureDevice.default(.builtInTrueDepthCamera, + for: .video, + position: .front) { + depthSiblings.append(("Front TrueDepth (depth sibling)", + trueDepth)) + } + if #available(iOS 15.4, *), + let lidar = AVCaptureDevice.default(.builtInLiDARDepthCamera, + for: .video, + position: .back) { + depthSiblings.append(("Rear LiDAR (depth sibling)", lidar)) + } + for (title, device) in depthSiblings { + out.append("== \(title) ==") + appendFormatRows(of: device, to: &out) out.append("") } return out.joined(separator: "\n") } + /// One diagnostics row per capture format of `device` — shared by + /// the user-facing lens dumps and the depth-sibling dumps above. + private static func appendFormatRows(of device: AVCaptureDevice, + to out: inout [String]) { + for format in device.formats { + let dims = CMVideoFormatDescriptionGetDimensions( + format.formatDescription) + let maxFps = format.videoSupportedFrameRateRanges + .map(\.maxFrameRate).max() ?? 0 + var flags = [format.isVideoBinned ? "b" : "-", + format.isCenterStageSupported ? "CS" : "--", + format.isPortraitEffectSupported ? "P" : "-"] + if #available(iOS 16.0, *) { + flags.append(format.isStudioLightSupported ? "SL" : "--") + } else { + flags.append("?") + } + if #available(iOS 17.0, *) { + flags.append(format.reactionEffectsSupported ? "R" : "-") + } else { + flags.append("?") + } + flags.append(colourFlags(format)) + flags.append(depthFlags(format)) + out.append(String(format: "%5dx%-5d fps<=%-3.0f %@", + dims.width, dims.height, maxFps, + flags.joined(separator: " "))) + } + } + + /// Depth column of the diagnostics table: the largest depth map this + /// video format can pair with ("D320x240"), or "-" when it supports + /// no depth at all. This is the per-device truth behind which + /// combos can run depth assist, the same way the colour column is + /// for HDR/Log. + private static func depthFlags(_ format: AVCaptureDevice.Format) -> String { + let dims = format.supportedDepthDataFormats.map { + CMVideoFormatDescriptionGetDimensions($0.formatDescription) + } + guard let best = dims.max(by: { + $0.width * $0.height < $1.width * $1.height + }) else { return "-" } + return "D\(best.width)x\(best.height)" + } + /// Colour column of the diagnostics table: the format's bit depth /// plus the HDR/Log colour spaces it can capture. This is the /// per-device truth behind the colour picker's HDR and Apple Log @@ -416,11 +515,12 @@ final class CameraManager: NSObject { resolution: Resolution, fps: Int32, lockFrameRate: Bool = true, - color: StreamColor = .sdr) throws -> StreamColor { + color: StreamColor = .sdr, + wantsDepth: Bool = false) throws -> StreamColor { try sessionQueue.sync { try configureOnQueue(lens: lens, resolution: resolution, fps: fps, lockFrameRate: lockFrameRate, - color: color) + color: color, wantsDepth: wantsDepth) } } @@ -428,7 +528,8 @@ final class CameraManager: NSObject { resolution: Resolution, fps: Int32, lockFrameRate: Bool, - color: StreamColor) throws -> StreamColor { + color: StreamColor, + wantsDepth: Bool) throws -> StreamColor { let position = lens.position session.beginConfiguration() defer { session.commitConfiguration() } @@ -436,6 +537,12 @@ final class CameraManager: NSObject { session.inputs.forEach(session.removeInput) session.outputs.forEach(session.removeOutput) + // Reset depth plumbing from any previous configuration; the + // depth block below re-arms it only when everything lines up. + depthOutput = nil + outputSynchronizer = nil + depthAssistActive = false + // Format is chosen manually below; presets can't express 4K60. session.sessionPreset = .inputPriority @@ -451,12 +558,28 @@ final class CameraManager: NSObject { session.automaticallyConfiguresCaptureDeviceForWideColor = false } - guard let device = Self.device(for: lens) else { + guard var device = Self.device(for: lens) else { throw NSError(domain: "CameraManager", code: 1, userInfo: [NSLocalizedDescriptionKey: "Camera not available"]) } - guard let format = Self.format(for: device, resolution: resolution, - fps: fps, color: color) else { + // Depth assist (green screen): swap to the depth-registered + // sibling device (TrueDepth / LiDAR) — the plain lenses have no + // depth formats at all — but ONLY when the sibling has a + // depth-capable format at the EXACT requested resolution+fps. + // Otherwise stay on the normal device with depth off: + // resolution/fps are NEVER changed to chase depth. + var depthPinnedFormat: AVCaptureDevice.Format? + if wantsDepth, let sibling = Self.depthSiblingDevice(for: lens), + let siblingFormat = Self.format(for: sibling, + resolution: resolution, + fps: fps, color: color, + requireDepth: true) { + device = sibling + depthPinnedFormat = siblingFormat + } + guard let format = depthPinnedFormat + ?? Self.format(for: device, resolution: resolution, + fps: fps, color: color) else { throw NSError(domain: "CameraManager", code: 4, userInfo: [NSLocalizedDescriptionKey: "\(resolution.rawValue) at \(fps) fps is not supported by the \(lens.label) camera"]) @@ -496,6 +619,30 @@ final class CameraManager: NSObject { if lockFrameRate { device.activeVideoMaxFrameDuration = frameDuration } + if depthPinnedFormat != nil { + // Largest DepthFloat16 entry: meters directly, mapping to an + // r16Float texture. Must come from the ACTIVE format's own + // supportedDepthDataFormats — anything else throws an + // (uncatchable) NSException. If no Float16 entry exists the + // system default depth format applies; the compositor + // converts defensively either way. + let depthFormats = format.supportedDepthDataFormats.filter { + CMFormatDescriptionGetMediaSubType($0.formatDescription) + == kCVPixelFormatType_DepthFloat16 + } + if let best = depthFormats.max(by: { + let a = CMVideoFormatDescriptionGetDimensions($0.formatDescription) + let b = CMVideoFormatDescriptionGetDimensions($1.formatDescription) + return a.width * a.height < b.width * b.height + }) { + device.activeDepthDataFormat = best + } + // ~15 Hz depth is plenty for a distance cutoff and keeps the + // added thermal/power load small. Set AFTER the formats — + // changing either resets this duration. + device.activeDepthDataMinFrameDuration = CMTime(value: 1, + timescale: 15) + } device.unlockForConfiguration() activeDevice = device @@ -515,7 +662,12 @@ final class CameraManager: NSObject { let output = AVCaptureVideoDataOutput() output.alwaysDiscardsLateVideoFrames = true - output.setSampleBufferDelegate(self, queue: videoQueue) + if depthPinnedFormat == nil { + // The pre-existing (and only) delivery path whenever depth is + // off; the synchronizer below REPLACES it in depth mode (the + // two must never both be active). + output.setSampleBufferDelegate(self, queue: videoQueue) + } guard session.canAddOutput(output) else { throw NSError(domain: "CameraManager", code: 3, @@ -524,6 +676,31 @@ final class CameraManager: NSObject { session.addOutput(output) videoOutput = output + if depthPinnedFormat != nil { + let depth = AVCaptureDepthDataOutput() + // Filtered depth avoids NaN holes punched through the + // subject (the compositor's gate is NaN-safe regardless, but + // filtering gives the smoother matte); late maps are + // dropped, never queued. + depth.isFilteringEnabled = true + depth.alwaysDiscardsLateDepthData = true + if session.canAddOutput(depth) { + session.addOutput(depth) + // One time-matched callback delivers video + depth on + // the same capture queue. + let synchronizer = AVCaptureDataOutputSynchronizer( + dataOutputs: [output, depth]) + synchronizer.setDelegate(self, queue: videoQueue) + depthOutput = depth + outputSynchronizer = synchronizer + depthAssistActive = true + } else { + // Depth output refused: degrade to segmentation-only via + // the ordinary delegate path. + output.setSampleBufferDelegate(self, queue: videoQueue) + } + } + // Video-range in every pipeline; only the depth/chroma differ. // Apple Log delivers x422 (the Log formats' native subtype); // VideoToolbox accepts x422 into a Main10 HEVC session and @@ -554,7 +731,8 @@ final class CameraManager: NSObject { return try configureOnQueue(lens: lens, resolution: resolution, fps: fps, lockFrameRate: lockFrameRate, - color: .sdr) + color: .sdr, + wantsDepth: wantsDepth) } output.videoSettings = [ kCVPixelBufferPixelFormatTypeKey as String: outputPixelFormat @@ -576,6 +754,18 @@ final class CameraManager: NSObject { connection.isVideoMirrored = true } } + // Mirror the video connection's geometry onto the depth stream + // (when supported) so the depth map stays registered to the + // video buffer instead of arriving rotated/flipped. + if let depthConnection = depthOutput?.connection(with: .depthData) { + if depthConnection.isVideoOrientationSupported { + depthConnection.videoOrientation = .landscapeRight + } + if position == .front, + depthConnection.isVideoMirroringSupported { + depthConnection.isVideoMirrored = true + } + } return color } @@ -596,6 +786,12 @@ final class CameraManager: NSObject { } func setZoom(_ factor: CGFloat) { + // TODO(green screen depth assist): whether the streamed depth map + // tracks videoZoomFactor crops is unconfirmed — under zoom the + // compositor's depth gate may misalign with the video. Zoom is + // deliberately left alone here (no clamping, no depth teardown); + // `depthAssistActive` reports the state and the gate simply may + // be off until this is verified on a real device. withLockedDevice { device in let clamped = max(device.minAvailableVideoZoomFactor, min(factor, maxZoomFactor)) @@ -781,3 +977,44 @@ extension CameraManager: AVCaptureVideoDataOutputSampleBufferDelegate { onSampleBuffer?(sampleBuffer) } } + +/// Depth mode only: the synchronizer replaces the plain sample-buffer +/// delegate above and delivers time-matched video + depth in one +/// callback on the same capture queue. +extension CameraManager: AVCaptureDataOutputSynchronizerDelegate { + func dataOutputSynchronizer( + _ synchronizer: AVCaptureDataOutputSynchronizer, + didOutput synchronizedDataCollection: AVCaptureSynchronizedDataCollection + ) { + // Resolve the outputs from the CALLBACK's own synchronizer, never + // from self.videoOutput/self.depthOutput: those properties are + // rewritten on sessionQueue during a reconfigure while this + // callback runs on the capture queue — an unsynchronized + // cross-queue read (and a callback holding the OLD synchronizer + // could misresolve against the NEW outputs). Everything below is + // callback-local state, so a mid-reconfigure delivery is safe. + var syncedDepth: AVCaptureSynchronizedDepthData? + var syncedVideo: AVCaptureSynchronizedSampleBufferData? + for output in synchronizer.dataOutputs { + switch synchronizedDataCollection.synchronizedData(for: output) { + case let depth as AVCaptureSynchronizedDepthData: + syncedDepth = depth + case let video as AVCaptureSynchronizedSampleBufferData: + syncedVideo = video + default: + break + } + } + // Depth first, so the compositor holds the freshest map when the + // matching video frame lands below. Depth legitimately runs + // slower than video — many callbacks carry no depth at all, and + // the consumer keeps reusing the last map. + if let syncedDepth, !syncedDepth.depthDataWasDropped { + onDepthData?(syncedDepth.depthData) + } + guard let syncedVideo, !syncedVideo.sampleBufferWasDropped else { + return + } + onSampleBuffer?(syncedVideo.sampleBuffer) + } +} diff --git a/ios-app/Sources/ContentView.swift b/ios-app/Sources/ContentView.swift index 2bfbaf7..7286b4c 100644 --- a/ios-app/Sources/ContentView.swift +++ b/ios-app/Sources/ContentView.swift @@ -301,8 +301,14 @@ struct ContentView: View { Text("Apple Log").tag(StreamColor.log) } } + // Green screen forces Standard: disabled, not hidden — + // a vanished row reads as a lost feature, a greyed one + // as a constraint (the Documentation sheet explains). + .disabled(streamer.greenScreenEnabled) } + Toggle("Green screen", isOn: $streamer.greenScreenEnabled) + if streamer.cameraPermissionDenied || streamer.micPermissionDenied { Button("Camera access denied — open Settings") { if let url = URL(string: UIApplication.openSettingsURLString) { diff --git a/ios-app/Sources/DocumentationView.swift b/ios-app/Sources/DocumentationView.swift index e111f9e..3d6c0bc 100644 --- a/ios-app/Sources/DocumentationView.swift +++ b/ios-app/Sources/DocumentationView.swift @@ -22,6 +22,14 @@ struct DocumentationView: View { Text("Color") } + Section { + Text("**Green screen** keeps you in the picture and paints everything else solid green before the video leaves the phone — turning it on sets Color to Standard. The **LensLink Camera** source in OBS adds a ready-tuned filter that keys the green out; it's added once, so deleting or re-tuning it in OBS sticks.") + Text("**Depth assist** sharpens the cutout with real depth — the front camera on Face ID phones, or the rear Main lens on Pro (LiDAR) phones. Other lenses use shape detection alone, which keeps every person in frame. It also turns off Center Stage and the other system video effects.") + Text("**Subject distance** appears on the Live screen while depth assist runs: anything farther than the distance becomes background — the way to drop a passer-by behind you. Tap the readout to go back to **All** (no limit).") + } header: { + Text("Green screen") + } + Section { Text("**Send phone mic** makes this phone the camera's audio in OBS — a wireless mic.") Text("**Auto lip-sync** sends the mic only as a timing reference for aligning your real microphone; it's never heard.") diff --git a/ios-app/Sources/FrameCompositor.swift b/ios-app/Sources/FrameCompositor.swift new file mode 100644 index 0000000..9f646d9 --- /dev/null +++ b/ios-app/Sources/FrameCompositor.swift @@ -0,0 +1,512 @@ +import Foundation +import AVFoundation +import Metal +import Vision + +/// Virtual green screen: keeps the person (Vision segmentation, optionally +/// gated by a depth-based max-subject-distance cutoff) and paints +/// everything else chroma green in one Metal compute pass, so the wire +/// carries ordinary video and OBS chroma-keys it exactly like a physical +/// green screen. +/// +/// App target ONLY — deliberately absent from `LensLinkBroadcast.sources` +/// (ios-app/project.yml is an explicit list), so the broadcast extension +/// never links Vision or Metal. +/// +/// THREADING: `composite(sampleBuffer:)`, `updateDepth(_:)` and +/// `maxDistance` are all called from CameraManager's capture queue +/// ("obscam.video") only — no locking needed. Segmentation runs +/// synchronously on that queue by design: when a frame takes too long the +/// capture output sheds the next one (alwaysDiscardsLateVideoFrames) +/// instead of building a queue — docs/PERFORMANCE.md's drop-don't-queue +/// rule. +final class FrameCompositor { + + /// Person segmentation exists on every OS this app runs on + /// (VNGeneratePersonSegmentationRequest is iOS 15.0+, the deployment + /// floor). Kept as an explicit gate for symmetry with + /// `hdrAvailable` / `appleLogCaptureAvailable`, so a future + /// capability split or floor change has one place to land. + static var supportsSegmentation: Bool { true } + + /// Max subject distance in meters: anything farther is treated as + /// background even where the person mask disagrees (iOS 15's + /// segmentation is ONE mask covering all people, so this is the only + /// tool against a passer-by behind the subject). 0 disables the + /// cutoff. Capture queue only. + var maxDistance: Float = 0 + + // MARK: - Members (all capture-queue confined after init) + + private let device: MTLDevice + private let commandQueue: MTLCommandQueue + private let pipeline: MTLComputePipelineState + private let textureCache: CVMetalTextureCache + /// Metal validation requires every declared texture argument bound + /// even when the kernel's runtime branch never samples it; this 1×1 + /// NaN texel stands in when no depth map exists (NaN reads as "no + /// depth opinion" → gate 1 in the kernel, so even a stray sample is + /// harmless). + private let dummyDepthTexture: MTLTexture + + /// ONE stateful request for the stream's life: Vision smooths + /// temporal changes between frames only when the same request + /// instance sees the sequence in order (it is a VNStatefulRequest). + private let request: VNGeneratePersonSegmentationRequest + private let sequenceHandler = VNSequenceRequestHandler() + + /// App-owned output pool — the capture buffer must never be written + /// in place (the hardware encoder still reads the previous frame + /// asynchronously), so the composite pass IS the one copy. + private var pool: CVPixelBufferPool? + private var poolWidth = 0 + private var poolHeight = 0 + /// Buffers from one pool share a format description; recreated only + /// when CMVideoFormatDescriptionMatchesImageBuffer says otherwise. + private var cachedFormatDescription: CMVideoFormatDescription? + + /// Depth runs slower than video (~15 Hz vs 30/60), so the newest map + /// is kept and reused until the next one lands. + private var latestDepthPixelBuffer: CVPixelBuffer? + + private var loggedStages = Set() + private var loggedMaskDims = false + + // MARK: - Init + + /// - Parameter targetFps: the stream's configured frame rate. It + /// picks the segmentation model once — the request is stateful and + /// lives for the whole stream: `.fast` is Apple's "streaming" + /// model (the only safe choice above 30 fps), `.balanced` the + /// 30 fps video model (WWDC21 session 10040). + /// - Returns: nil when Metal is unavailable or the kernel fails to + /// compile — the caller then streams uncomposited frames. + /// The Metal objects are process-wide constants (the shader source + /// never varies — only the Vision request depends on init + /// parameters), so they compile ONCE per process: a fresh compositor + /// is created on every (re)configure while armed, and recompiling + /// tens of milliseconds of MSL on the main thread per lens switch + /// was a per-reconfigure UI hitch for nothing. + private struct SharedMetal { + let device: MTLDevice + let queue: MTLCommandQueue + let pipeline: MTLComputePipelineState + } + + private static let sharedMetal: SharedMetal? = { + guard let metalDevice = MTLCreateSystemDefaultDevice(), + let queue = metalDevice.makeCommandQueue() else { + print("FrameCompositor: Metal unavailable — green screen disabled") + return nil + } + // The kernel ships as Swift-embedded MSL compiled here once per + // process via makeLibrary(source:) — keeping the shader next to + // the code that dispatches it and avoiding .metal/metallib build + // plumbing in project.yml. + guard let library = try? metalDevice.makeLibrary( + source: shaderSource, options: nil), + let function = library.makeFunction(name: "greenScreenComposite"), + let pipeline = try? metalDevice.makeComputePipelineState( + function: function) else { + print("FrameCompositor: shader compile failed — green screen disabled") + return nil + } + return SharedMetal(device: metalDevice, queue: queue, + pipeline: pipeline) + }() + + init?(targetFps: Int32) { + guard let shared = Self.sharedMetal else { return nil } + let metalDevice = shared.device + let queue = shared.queue + let pipeline = shared.pipeline + var cache: CVMetalTextureCache? + CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice, + nil, &cache) + guard let cache else { + print("FrameCompositor: texture cache creation failed — green screen disabled") + return nil + } + let descriptor = MTLTextureDescriptor.texture2DDescriptor( + pixelFormat: .r16Float, width: 1, height: 1, mipmapped: false) + descriptor.usage = .shaderRead + guard let dummy = metalDevice.makeTexture(descriptor: descriptor) else { + print("FrameCompositor: dummy texture creation failed — green screen disabled") + return nil + } + var nan: UInt16 = 0x7E00 // IEEE 754 half-precision NaN + dummy.replace(region: MTLRegionMake2D(0, 0, 1, 1), mipmapLevel: 0, + withBytes: &nan, + bytesPerRow: MemoryLayout.size) + + let request = VNGeneratePersonSegmentationRequest() + request.qualityLevel = targetFps > 30 ? .fast : .balanced + // OneComponent8 maps straight onto an r8Unorm texture. + request.outputPixelFormat = kCVPixelFormatType_OneComponent8 + + self.device = metalDevice + self.commandQueue = queue + self.pipeline = pipeline + self.textureCache = cache + self.dummyDepthTexture = dummy + self.request = request + } + + // MARK: - Depth + + /// Store the newest depth frame (capture queue only). Converts to + /// DepthFloat16 defensively — CameraManager pins that format, so the + /// conversion is normally a no-op — because the kernel samples the + /// map as r16Float meters. + func updateDepth(_ depth: AVDepthData) { + let wanted = kCVPixelFormatType_DepthFloat16 + let data = depth.depthDataType == wanted + ? depth : depth.converting(toDepthDataType: wanted) + latestDepthPixelBuffer = data.depthDataMap + } + + // MARK: - Composite + + /// Capture queue only. Returns: + /// - a NEW sample buffer with the person kept and everything else + /// painted chroma green (source pts/duration/attachments carried + /// over verbatim, so timesync/lip-sync never notice), or + /// - the ORIGINAL buffer when any stage fails — fail-open: a Vision + /// or Metal hiccup must never blank the stream, or + /// - nil when the output pool is exhausted (every in-flight buffer + /// is still referenced downstream): the caller drops the frame, + /// per the repo-wide drop-don't-queue rule. + func composite(sampleBuffer: CMSampleBuffer) -> CMSampleBuffer? { + guard let source = CMSampleBufferGetImageBuffer(sampleBuffer) else { + return sampleBuffer + } + // Green screen is SDR-only; the composite green constants are + // BT.709 video-range and the kernel reads 8-bit biplanar 420v. + guard CVPixelBufferGetPixelFormatType(source) + == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange else { + logOnce("pixel-format", + "FrameCompositor: non-420v input — green screen is SDR-only; passing frames through") + return sampleBuffer + } + + // 1. Person mask. Synchronous on the capture queue: overload + // sheds the NEXT frame via alwaysDiscardsLateVideoFrames rather + // than queueing work. Buffers are upright sensor-native + // landscape (the connection sets .landscapeRight and mirrors the + // front camera), so .up keeps the mask in buffer coordinates. + do { + try sequenceHandler.perform([request], on: source, + orientation: .up) + } catch { + logOnce("vision", + "FrameCompositor: segmentation failed (\(error.localizedDescription)) — passing frames through") + return sampleBuffer + } + guard let maskBuffer = request.results?.first?.pixelBuffer else { + logOnce("vision-results", + "FrameCompositor: segmentation returned no mask — passing frames through") + return sampleBuffer + } + + let width = CVPixelBufferGetWidth(source) + let height = CVPixelBufferGetHeight(source) + if !loggedMaskDims { + loggedMaskDims = true + // Mask resolution per quality level is undocumented — log the + // observed value once (it feeds the feather-quality picture). + print("FrameCompositor: mask " + + "\(CVPixelBufferGetWidth(maskBuffer))x\(CVPixelBufferGetHeight(maskBuffer))" + + " for \(width)x\(height) input") + } + + // 2. Output buffer from our pool (rebuilt on dimension change). + if pool == nil || width != poolWidth || height != poolHeight { + rebuildPool(width: width, height: height) + } + guard let pool else { return sampleBuffer } // rebuild already logged + var allocated: CVPixelBuffer? + let aux = [kCVPixelBufferPoolAllocationThresholdKey as String: 6] + let poolStatus = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes( + kCFAllocatorDefault, pool, aux as CFDictionary, &allocated) + if poolStatus == kCVReturnWouldExceedAllocationThreshold { + // All in-flight buffers still referenced downstream — the + // GPU/encoder is behind. Drop, don't queue. + return nil + } + guard poolStatus == kCVReturnSuccess, let output = allocated else { + logOnce("pool-alloc", + "FrameCompositor: pool allocation failed (\(poolStatus)) — passing frames through") + return sampleBuffer + } + + // 3. GPU composite. + guard runKernel(source: source, mask: maskBuffer, output: output) else { + logOnce("metal", + "FrameCompositor: Metal composite failed — passing frames through") + return sampleBuffer + } + + // 4. Wrap for the encoder. Attachments (colour primaries / + // transfer / matrix — the BT.709 tags) must travel with the new + // buffer or colours shift on the wire. + CVBufferPropagateAttachments(source, output) + if let cached = cachedFormatDescription, + !CMVideoFormatDescriptionMatchesImageBuffer(cached, + imageBuffer: output) { + cachedFormatDescription = nil + } + if cachedFormatDescription == nil { + var created: CMVideoFormatDescription? + CMVideoFormatDescriptionCreateForImageBuffer( + allocator: kCFAllocatorDefault, imageBuffer: output, + formatDescriptionOut: &created) + cachedFormatDescription = created + } + guard let formatDescription = cachedFormatDescription else { + logOnce("format-description", + "FrameCompositor: format description creation failed — passing frames through") + return sampleBuffer + } + // Source timing verbatim: pts drives the wire timestamps. + var timing = CMSampleTimingInfo( + duration: CMSampleBufferGetDuration(sampleBuffer), + presentationTimeStamp: + CMSampleBufferGetPresentationTimeStamp(sampleBuffer), + decodeTimeStamp: .invalid) + var wrapped: CMSampleBuffer? + let status = CMSampleBufferCreateReadyWithImageBuffer( + allocator: kCFAllocatorDefault, + imageBuffer: output, + formatDescription: formatDescription, + sampleTiming: &timing, + sampleBufferOut: &wrapped) + guard status == noErr, let wrapped else { + logOnce("sample-buffer", + "FrameCompositor: sample buffer wrap failed (\(status)) — passing frames through") + return sampleBuffer + } + return wrapped + } + + // MARK: - Metal pass + + private func runKernel(source: CVPixelBuffer, mask: CVPixelBuffer, + output: CVPixelBuffer) -> Bool { + // The CVMetalTextures must outlive the GPU pass (releasing one + // recycles its backing); collected here and held past the wait. + var retained: [CVMetalTexture] = [] + guard let srcY = makeTexture(from: source, pixelFormat: .r8Unorm, + planeIndex: 0, retaining: &retained), + let srcCbCr = makeTexture(from: source, pixelFormat: .rg8Unorm, + planeIndex: 1, retaining: &retained), + let maskTexture = makeTexture(from: mask, pixelFormat: .r8Unorm, + planeIndex: 0, retaining: &retained), + let outY = makeTexture(from: output, pixelFormat: .r8Unorm, + planeIndex: 0, retaining: &retained), + let outCbCr = makeTexture(from: output, pixelFormat: .rg8Unorm, + planeIndex: 1, retaining: &retained) + else { return false } + + // Depth is optional per frame: no map yet, no cutoff configured, + // or a failed wrap all degrade to segmentation-only (gate = 1). + var depthTexture = dummyDepthTexture + var hasDepth: Float = 0 + if maxDistance > 0, let depthBuffer = latestDepthPixelBuffer { + if let wrappedDepth = makeTexture(from: depthBuffer, + pixelFormat: .r16Float, + planeIndex: 0, + retaining: &retained) { + depthTexture = wrappedDepth + hasDepth = 1 + } else { + logOnce("depth-wrap", + "FrameCompositor: depth texture wrap failed — segmentation-only") + } + } + + guard let commandBuffer = commandQueue.makeCommandBuffer(), + let encoder = commandBuffer.makeComputeCommandEncoder() else { + return false + } + encoder.setComputePipelineState(pipeline) + encoder.setTexture(srcY, index: 0) + encoder.setTexture(srcCbCr, index: 1) + encoder.setTexture(maskTexture, index: 2) + encoder.setTexture(depthTexture, index: 3) + encoder.setTexture(outY, index: 4) + encoder.setTexture(outCbCr, index: 5) + var params = SIMD2(maxDistance, hasDepth) + encoder.setBytes(¶ms, length: MemoryLayout>.size, + index: 0) + + // One thread per 2×2 luma quad. Plain dispatchThreadgroups with + // an in-kernel bounds check — non-uniform threadgroups need + // A11+, and iOS 15 still runs on A9/A10. + let chromaWidth = CVPixelBufferGetWidthOfPlane(output, 1) + let chromaHeight = CVPixelBufferGetHeightOfPlane(output, 1) + let threadsPerGroup = MTLSize(width: 16, height: 16, depth: 1) + let groups = MTLSize(width: (chromaWidth + 15) / 16, + height: (chromaHeight + 15) / 16, + depth: 1) + encoder.dispatchThreadgroups(groups, + threadsPerThreadgroup: threadsPerGroup) + encoder.endEncoding() + commandBuffer.commit() + // Blocking (~0.5 ms) is the simplest correct ordering before the + // encoder reads the buffer; overload sheds upstream via + // alwaysDiscardsLateVideoFrames. An MTLSharedEvent pipeline is a + // later optimization only if measurements demand it. + commandBuffer.waitUntilCompleted() + withExtendedLifetime(retained) {} + return commandBuffer.error == nil + } + + private func makeTexture(from buffer: CVPixelBuffer, + pixelFormat: MTLPixelFormat, + planeIndex: Int, + retaining retained: inout [CVMetalTexture]) + -> MTLTexture? { + let planar = CVPixelBufferIsPlanar(buffer) + let width = planar + ? CVPixelBufferGetWidthOfPlane(buffer, planeIndex) + : CVPixelBufferGetWidth(buffer) + let height = planar + ? CVPixelBufferGetHeightOfPlane(buffer, planeIndex) + : CVPixelBufferGetHeight(buffer) + var cvTexture: CVMetalTexture? + let status = CVMetalTextureCacheCreateTextureFromImage( + kCFAllocatorDefault, textureCache, buffer, nil, pixelFormat, + width, height, planeIndex, &cvTexture) + guard status == kCVReturnSuccess, let cvTexture, + let texture = CVMetalTextureGetTexture(cvTexture) else { + return nil + } + retained.append(cvTexture) + return texture + } + + // MARK: - Pool + + private func rebuildPool(width: Int, height: Int) { + let attributes: [String: Any] = [ + kCVPixelBufferPixelFormatTypeKey as String: + kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, + kCVPixelBufferWidthKey as String: width, + kCVPixelBufferHeightKey as String: height, + // IOSurface backing keeps the VideoToolbox handoff zero-copy + // and lets the texture cache wrap the planes for writing. + kCVPixelBufferIOSurfacePropertiesKey as String: [:], + kCVPixelBufferMetalCompatibilityKey as String: true, + ] + var created: CVPixelBufferPool? + let status = CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, + attributes as CFDictionary, + &created) + pool = status == kCVReturnSuccess ? created : nil + poolWidth = width + poolHeight = height + cachedFormatDescription = nil + if pool == nil { + logOnce("pool-create", + "FrameCompositor: pool creation failed (\(status)) — passing frames through") + } + } + + // MARK: - Logging + + /// Capture queue only. Each distinct failure stage logs once — a + /// per-frame print at 60 fps would itself be a performance bug. + private func logOnce(_ stage: String, _ message: String) { + guard loggedStages.insert(stage).inserted else { return } + print(message) + } + + // MARK: - Kernel source + + private static let shaderSource = """ + #include + using namespace metal; + + // Chroma green, BT.709 limited (video) range — THE single tunable + // set of green constants for the whole feature. + // sRGB #00B140 (0, 177, 64) is the industry chroma-key green; it + // survives 4:2:0 subsampling + H.264/HEVC quantization better than + // primary green (0, 255, 0). Derivation with BT.709 coefficients + // (Kr = 0.2126, Kg = 0.7152, Kb = 0.0722): + // Y' = 0.5146, Cb' = -0.1420, Cr' = -0.3267 + // Y = 16 + 219 * Y' = 129 + // Cb = 128 + 224 * Cb' = 96 + // Cr = 128 + 224 * Cr' = 55 + // Normalized for r8Unorm / rg8Unorm writes: + constant float GREEN_Y = 129.0 / 255.0; + constant float GREEN_CB = 96.0 / 255.0; + constant float GREEN_CR = 55.0 / 255.0; + + // Depth gate feather, meters: alpha ramps 1 -> 0 over + // [maxDistance - FEATHER, maxDistance]. + constant float DEPTH_FEATHER_M = 0.15; + + struct Params { + float maxDistance; // meters; <= 0 disables the depth cutoff + float hasDepth; // > 0.5 when the depth texture is real + }; + + // One thread per 2x2 luma quad (grid = chroma-plane dimensions). + kernel void greenScreenComposite( + texture2d srcY [[texture(0)]], + texture2d srcCbCr [[texture(1)]], + texture2d mask [[texture(2)]], + texture2d depth [[texture(3)]], + texture2d outY [[texture(4)]], + texture2d outCbCr [[texture(5)]], + constant Params ¶ms [[buffer(0)]], + uint2 gid [[thread_position_in_grid]]) + { + const uint chromaW = outCbCr.get_width(); + const uint chromaH = outCbCr.get_height(); + if (gid.x >= chromaW || gid.y >= chromaH) + return; + + constexpr sampler bilinear(coord::normalized, + address::clamp_to_edge, + filter::linear); + + const uint lumaW = outY.get_width(); + const uint lumaH = outY.get_height(); + const float2 lumaSize = float2(lumaW, lumaH); + + float alphaSum = 0.0; + for (uint i = 0; i < 4; ++i) { + const uint2 px = uint2(min(2 * gid.x + (i & 1u), lumaW - 1), + min(2 * gid.y + (i >> 1), lumaH - 1)); + // The mask is far smaller than the frame (~256 wide): this + // bilinear upsample IS the edge feather. + const float2 uv = (float2(px) + 0.5) / lumaSize; + float alpha = mask.sample(bilinear, uv).r; + + // Depth gate: 1 inside the cutoff, ramping to 0 at it. + // NaN (depth holes / no reading) must NEVER classify as + // background — every comparison against NaN is false, so + // test isnan() FIRST and fall through to "no depth + // opinion" (gate stays 1). + if (params.hasDepth > 0.5 && params.maxDistance > 0.0) { + const float d = depth.sample(bilinear, uv).r; + if (!isnan(d)) { + alpha *= 1.0 - smoothstep( + params.maxDistance - DEPTH_FEATHER_M, + params.maxDistance, d); + } + } + + const float y = srcY.read(px).r; + outY.write(float4(mix(GREEN_Y, y, alpha)), px); + alphaSum += alpha; + } + + const float alphaMean = alphaSum * 0.25; + const float2 cbcr = srcCbCr.read(gid).rg; + const float2 outC = mix(float2(GREEN_CB, GREEN_CR), cbcr, alphaMean); + outCbCr.write(float4(outC, 0.0, 0.0), gid); + } + """ +} diff --git a/ios-app/Sources/Streamer.swift b/ios-app/Sources/Streamer.swift index 93c2759..9424c56 100644 --- a/ios-app/Sources/Streamer.swift +++ b/ios-app/Sources/Streamer.swift @@ -92,7 +92,8 @@ final class Streamer: ObservableObject { resolution: resolution, fps: Int32(fps), lockFrameRate: !allowVideoEffects, - color: activeColor) + color: activeColor, + wantsDepth: greenScreenEnabled) } catch { // Never swallow this: a failed reconfigure leaves the session // without input/output — a black stream labelled "Live". @@ -110,6 +111,13 @@ final class Streamer: ObservableObject { formatChanged || oldEncoder.codec != activeCodec || oldEncoder.color != color { rebuildEncoder(codec: activeCodec, color: color) + } else if greenScreenEnabled, let encoder { + // Lens-only reconfigure while green screen is on: the + // encoder survives, but the compositor must be rebuilt + // (fresh depth state for the new lens geometry) and depth + // availability re-read — Ultra Wide/Telephoto have no + // depth sibling. Green screen OFF skips this entirely. + wireCameraToEncoder(encoder, color: color) } encoder?.requestKeyframe() @@ -140,9 +148,7 @@ final class Streamer: ObservableObject { newEncoder.onEncodedFrame = { [weak self] frame in self?.client.sendVideoFrame(frame) } - camera.onSampleBuffer = { [weak newEncoder] sampleBuffer in - newEncoder?.encode(sampleBuffer) - } + wireCameraToEncoder(newEncoder, color: color) encoder = newEncoder client.sendVideoConfig(codec: activeCodec, width: size.width, height: size.height, @@ -151,6 +157,56 @@ final class Streamer: ObservableObject { startAdaptiveBitrate( target: resolution.bitrate(for: activeCodec, color: color)) } + + /// Points capture output at `encoder`, routed through the green + /// screen compositor when armed. The arm condition is green screen + /// ON *and* `color == .sdr` — configure()'s RETURN, not the ask: + /// the capture degrade path can flip a 10-bit request to SDR, and + /// green screen's BT.709 green is only correct on the SDR path. + /// Disarmed keeps the literal pre-green-screen closure, so OFF adds + /// zero work to the per-frame hot path. Called after every + /// camera.configure at both closure-assignment sites (start / + /// rebuildEncoder) plus the lens-only reconfigure while armed. + private func wireCameraToEncoder(_ encoder: VideoEncoder, + color: StreamColor) { + if greenScreenEnabled, color == .sdr, + FrameCompositor.supportsSegmentation { + // A fresh instance per (re)configure: the compositor's + // Vision request and cached depth map are stream-shaped + // state, and a lens/format change must not leak a stale + // depth map into the new geometry. nil = Metal/shader + // unavailable — stream uncomposited (the compositor logs). + compositor = FrameCompositor(targetFps: Int32(fps)) + } else { + compositor = nil + } + greenScreenDistance.value = Float(greenScreenMaxDistance) + greenScreenDepthActive = compositor != nil && camera.depthAssistActive + if let compositor { + camera.onSampleBuffer = { [weak encoder, compositor] sampleBuffer in + // Fail-open is the compositor's contract: any + // Vision/Metal failure returns the original buffer + // untouched; nil only on pool exhaustion — drop the + // frame, never queue. + if let output = compositor.composite(sampleBuffer: sampleBuffer) { + encoder?.encode(output) + } + } + camera.onDepthData = { [compositor, + distance = greenScreenDistance] depth in + // Capture queue: the compositor's properties are + // confined there, so the cutoff crosses over on this + // ~15 Hz depth path, not per video frame. + compositor.maxDistance = distance.value + compositor.updateDepth(depth) + } + } else { + camera.onSampleBuffer = { [weak encoder] sampleBuffer in + encoder?.encode(sampleBuffer) + } + camera.onDepthData = nil + } + } @Published var codec: VideoCodec { didSet { UserDefaults.standard.set(codec.rawValue, forKey: "videoCodec") @@ -174,6 +230,16 @@ final class Streamer: ObservableObject { if colorSetting != .sdr && codec != .hevc { codec = .hevc } + // Green screen is SDR-only (its composite paints BT.709 + // video-range green): choosing HDR or Apple Log turns it + // off — the mirror of greenScreenEnabled forcing Standard. + // No re-entry: that didSet only ever writes + // `colorSetting = .sdr`, which this `!= .sdr` branch never + // matches, so the codec/colour/green-screen triangle + // terminates. + if colorSetting != .sdr && greenScreenEnabled { + greenScreenEnabled = false + } // The capture pixel format and the encoder profile both // follow the colour, so a live stream rebuilds like a // format change. @@ -182,6 +248,55 @@ final class Streamer: ObservableObject { } } } + /// Virtual green screen: the phone segments the person (optionally + /// depth-gated) and paints everything else chroma green *before* + /// encoding, so the wire carries ordinary video and OBS keys it + /// like a physical green screen (docs/PROTOCOL.md §7–8). SDR-only: + /// enabling forces Standard colour, and picking a non-SDR colour + /// turns this off (see colorSetting's didSet for why the pair + /// terminates). + @Published var greenScreenEnabled: Bool { + didSet { + UserDefaults.standard.set(greenScreenEnabled, + forKey: "greenScreen") + if greenScreenEnabled && colorSetting != .sdr { + // Forcing Standard runs colorSetting's didSet, which + // reconfigures a live stream itself (with green screen + // already on) — the else-if keeps one toggle to one + // reconfigure. + colorSetting = .sdr + } else if isStreaming && greenScreenEnabled != oldValue + && colorSetting == .sdr { + // Arm/disarm changes the capture wiring and the depth + // request (wantsDepth), so a live change rebuilds like + // a format change. The colorSetting == .sdr guard skips + // the redundant rebuild when this didSet was ENTERED + // from colorSetting.didSet (a non-SDR pick disarming + // green screen): that caller reconfigures for the new + // colour right after we return — without the guard, one + // user action would tear the session down twice. + reconfigureLiveCapture(formatChanged: true) + } + } + } + /// Depth-assisted max subject distance in metres: anything farther + /// is background even where the person mask disagrees. 0 = no + /// cutoff ("All"); otherwise the slider's 0.5–5.0 range. A live + /// change only updates the compositor's shader parameter (via + /// `greenScreenDistance`, on the capture queue) — no reconfigure. + @Published var greenScreenMaxDistance: Double { + didSet { + UserDefaults.standard.set(greenScreenMaxDistance, + forKey: "greenScreenMaxDistance") + greenScreenDistance.value = Float(greenScreenMaxDistance) + scheduleStateSend() + } + } + /// True while the live stream is green-screen composited *and* + /// depth assist actually armed (TrueDepth front / LiDAR rear Main + /// lens with a depth-capable format). The Live screen's Subject + /// distance row keys off this; segmentation-only streams hide it. + @Published private(set) var greenScreenDepthActive = false /// The colour pipeline the next capture/encode will use — the single /// source of truth every configure/encoder/config-send site reads. var activeColor: StreamColor { color(for: resolution, fps: fps) } @@ -472,6 +587,25 @@ final class Streamer: ObservableObject { case .log: state["color"] = StreamColor.log.rawValue } + // Green screen (docs/PROTOCOL.md §8): the support flag is + // always advertised (remote UIs gate their row on it); the + // live keys appear only while actually armed — the compositor, + // not the setting, is the truth — so a snapshot with green + // screen off matches today's apart from supportsGreenScreen. + state["supportsGreenScreen"] = FrameCompositor.supportsSegmentation + if compositor != nil { + state["greenScreen"] = true + // Only when true: absent reads as false to every truthiness + // consumer, and each byte matters — an OLD plugin's STATE + // cache is 1024 bytes with silent truncation, so the armed + // snapshot should grow as little as possible. + if camera.depthAssistActive { + state["greenScreenDepth"] = true + } + if greenScreenMaxDistance > 0 { + state["greenScreenMaxDistance"] = greenScreenMaxDistance + } + } // Mic picker (docs/PROTOCOL.md §8): only while the phone mic is // live as the source's audio — remote UIs key their row off // micEnabled, and the list is only meaningful with capture up. @@ -627,6 +761,17 @@ final class Streamer: ObservableObject { private let client = StreamClient() private var audioReference: AudioReference? + /// The green screen compositor — exists exactly while armed (green + /// screen ON and the colour capture actually delivered is SDR), + /// created by `wireCameraToEncoder` and released on disarm/stop. + /// nil with green screen on means arming failed (no Metal) and the + /// stream runs uncomposited — the STATE snapshot stays honest by + /// keying off this, not the setting. + private var compositor: FrameCompositor? + /// Bridges the main-actor `greenScreenMaxDistance` onto the capture + /// queue, where the compositor's `maxDistance` is confined. + private let greenScreenDistance = GreenScreenDistanceCell() + init() { let defaults = UserDefaults.standard resolution = CameraManager.Resolution( @@ -655,7 +800,24 @@ final class Streamer: ObservableObject { } // didSet doesn't run during init; enforce the HEVC-only rule here // (stored defaults could disagree after an edit or migration). - colorSetting = storedCodec == .hevc ? storedColor : .sdr + let resolvedColor: StreamColor = + storedCodec == .hevc ? storedColor : .sdr + colorSetting = resolvedColor + // didSet doesn't run during init; enforce the SDR-only green + // screen rule here too. When the stored defaults disagree + // (green screen saved on alongside a non-SDR colour), the + // COLOUR wins and green screen loads off — deliberately: + // silently swapping the colour pipeline would change the look + // of every frame on the wire, while a dropped green screen is + // one visible toggle away. + greenScreenEnabled = resolvedColor == .sdr + && defaults.bool(forKey: "greenScreen") + // Out-of-range persisted cutoffs (an edited plist) reset to + // "no cutoff" rather than clamping to an edge the user never + // chose. + let storedDistance = defaults.double(forKey: "greenScreenMaxDistance") + greenScreenMaxDistance = + (0.5...5.0).contains(storedDistance) ? storedDistance : 0 dimWhileStreaming = defaults.object(forKey: "dimWhileStreaming") as? Bool ?? true allowVideoEffects = defaults.bool(forKey: "allowVideoEffects") sendAudioReference = defaults.bool(forKey: "sendAudioReference") @@ -915,6 +1077,20 @@ final class Streamer: ObservableObject { fps: Int32(fps), lens: lens) { selectedLens = lens } + case "green_screen": + // Either field may arrive alone (docs/PROTOCOL.md §7). The + // toggle goes through the didSet chain (SDR forcing + live + // reconfigure); the cutoff clamps to 0 (no cutoff) or the + // slider's 0.5–5.0 m. Screen-mirror connections never reach + // here — the broadcast extension runs its own process + // without a Streamer. + if let on = command["on"] as? Bool, on != greenScreenEnabled { + greenScreenEnabled = on + } + if let distance = (command["maxDistance"] as? NSNumber)?.doubleValue { + greenScreenMaxDistance = + distance <= 0 ? 0 : min(max(distance, 0.5), 5.0) + } default: break } @@ -1010,7 +1186,8 @@ final class Streamer: ObservableObject { resolution: resolution, fps: Int32(fps), lockFrameRate: !allowVideoEffects, - color: activeColor) + color: activeColor, + wantsDepth: greenScreenEnabled) encoder = VideoEncoder(codec: activeCodec, width: size.width, height: size.height, fps: Int32(fps), @@ -1026,9 +1203,10 @@ final class Streamer: ObservableObject { encoder.onEncodedFrame = { [weak self] frame in self?.client.sendVideoFrame(frame) } - camera.onSampleBuffer = { [weak encoder] sampleBuffer in - encoder?.encode(sampleBuffer) - } + // encoder.color is configure()'s returned colour verbatim (the + // encoder was just built from it) — the green screen arm check + // keys off the return, not the ask. + wireCameraToEncoder(encoder, color: encoder.color) self.encoder = encoder isStreaming = true @@ -1201,6 +1379,11 @@ final class Streamer: ObservableObject { client.disconnect() camera.stop() camera.onSampleBuffer = nil + camera.onDepthData = nil + // The compositor holds Metal resources and a frame pool — + // stream-lifetime state, released with the stream. + compositor = nil + greenScreenDepthActive = false encoder?.stop() encoder = nil audioReference?.stop() @@ -1287,3 +1470,28 @@ final class Streamer: ObservableObject { status = currentStatus } } + +/// A locked Float cell bridging the main-actor `greenScreenMaxDistance` +/// onto the capture queue: the compositor's `maxDistance` is +/// capture-queue-confined by contract, so the armed depth closure +/// re-reads this cell there instead of touching main-actor state. File +/// scope on purpose — nested inside Streamer it would inherit +/// @MainActor and defeat the point. Same NSLock-per-callback pattern as +/// CameraManager's closure properties. +private final class GreenScreenDistanceCell { + private let lock = NSLock() + private var stored: Float = 0 + + var value: Float { + get { + lock.lock() + defer { lock.unlock() } + return stored + } + set { + lock.lock() + stored = newValue + lock.unlock() + } + } +} diff --git a/ios-app/Sources/StreamingView.swift b/ios-app/Sources/StreamingView.swift index 189b2b5..e93cb26 100644 --- a/ios-app/Sources/StreamingView.swift +++ b/ios-app/Sources/StreamingView.swift @@ -314,6 +314,8 @@ struct StreamingView: View { whiteBalanceRow + greenScreenRow + micRow HStack(spacing: Theme.Space.m) { @@ -465,6 +467,59 @@ struct StreamingView: View { } } + /// Green screen subject distance (docs/UI_DESIGN.md §5): anything + /// farther than this is background even where the person shape + /// disagrees. Shown only while the stream is composited *with* + /// depth assist running — without depth the cutoff can do nothing, + /// and screen mirror never shows this screen at all. Standard + /// slider-row anatomy (§4); the readout doubles as the "All" + /// affordance: it shows the cutoff and taps back to no-cutoff + /// (model value 0), while "All" itself is plain text. + @ViewBuilder + private var greenScreenRow: some View { + if streamer.greenScreenDepthActive { + HStack(spacing: Theme.Space.m) { + Image(systemName: "person.fill.viewfinder") + Slider(value: subjectDistanceBinding, + in: 0.5...5.0, step: 0.1) { editing in + if editing { touched() } + } + Image(systemName: "person.2.fill") + if streamer.greenScreenMaxDistance > 0 { + Button { + touched() + streamer.greenScreenMaxDistance = 0 + } label: { + Text(String(format: "%.1f m", + streamer.greenScreenMaxDistance)) + .font(.caption.monospacedDigit()) + .frame(width: 44, alignment: .trailing) + } + } else { + Text("All") + .font(.caption.monospacedDigit()) + .frame(width: 44, alignment: .trailing) + } + } + } + } + + /// The slider's projection of the 0-means-All model value: while + /// "All", the thumb parks at the far end (everything in reach is + /// subject); any drag sets a real cutoff. + private var subjectDistanceBinding: Binding { + Binding( + get: { + streamer.greenScreenMaxDistance > 0 + ? streamer.greenScreenMaxDistance : 5.0 + }, + set: { value in + // Re-round despite the slider's step: float dust would + // jitter the readout and the STATE snapshot. + streamer.greenScreenMaxDistance = (value * 10).rounded() / 10 + }) + } + /// Mic picker: which microphone feeds OBS. Shown only while the phone /// mic is being sent as the source's audio (Options → Send phone mic); /// switching is live — the tap re-installs on the new input. diff --git a/obs-plugin/src/ios-camera-source.c b/obs-plugin/src/ios-camera-source.c index 11812f7..d36312b 100644 --- a/obs-plugin/src/ios-camera-source.c +++ b/obs-plugin/src/ios-camera-source.c @@ -62,6 +62,13 @@ static void gpu_pipeline_free(struct ios_camera_source *s); /* Built-in OBS "Video Delay (Async)" filter. */ #define ASYNC_DELAY_FILTER_ID "async_delay_filter" #define ASYNC_DELAY_FILTER_NAME "LensLink auto lip-sync delay" + +/* Built-in OBS "Chroma Key" filter (v2), auto-added once when the app + * reports its virtual green screen active. The persisted flag remembers + * the add so a user's deletion of the filter is respected forever. */ +#define CHROMA_KEY_FILTER_ID "chroma_key_filter_v2" +#define CHROMA_KEY_FILTER_NAME "LensLink green screen key" +#define S_GS_FILTER_ADDED "gs_filter_added" #define MODE_DIAL "dial" #define MODE_USB "usb" #define T_(s) obs_module_text(s) @@ -193,8 +200,10 @@ struct ios_camera_source { int control_count; /* Latest camera-state JSON reported by the app (for remote UIs). - * Guarded by status_mutex. */ - char device_state[1024]; + * Guarded by status_mutex. Sized well past today's biggest snapshot + * (mics/resolutions/lens lists + green screen): silent truncation + * here breaks the web panel's JSON.parse. */ + char device_state[2048]; /* Health mirrors for the frontend UI (the live counters belong to * the dial thread's client_state; these are 1 Hz copies). Guarded @@ -806,6 +815,59 @@ static void set_video_delay(struct ios_camera_source *s, int delay_ms) obs_data_release(settings); } +/* + * The app reports its virtual green screen active: make sure a chroma-key + * filter is keying the synthetic green background. Runs on the dial-loop + * thread — filter work from here is the set_video_delay precedent above + * (OBS guards the filter list internally). The filter is added ONCE per + * source, pre-configured for the app's uniform chroma green (#00B140, + * softened only by 4:2:0 encoding — tighter similarity than the stock + * defaults). After that it's the user's: an existing filter is never + * retuned or re-enabled, and the persisted "added once" flag means a + * deleted filter stays deleted, forever. + */ +static void ensure_chroma_key_filter(struct ios_camera_source *s) +{ + obs_source_t *existing = obs_source_get_filter_by_name( + s->source, CHROMA_KEY_FILTER_NAME); + if (existing) { + obs_source_release(existing); + return; + } + + obs_data_t *settings = obs_source_get_settings(s->source); + if (obs_data_get_bool(settings, S_GS_FILTER_ADDED)) { + /* Flag set + filter absent = the user deleted it. */ + obs_data_release(settings); + return; + } + + obs_data_t *fs = obs_data_create(); + obs_data_set_string(fs, "key_color_type", "green"); + obs_data_set_int(fs, "similarity", 300); + obs_data_set_int(fs, "smoothness", 70); + obs_data_set_int(fs, "spill", 100); + /* NB: "opacity" is a double in v2 (0.0–1.0) — leave its default. */ + obs_source_t *filter = obs_source_create_private( + CHROMA_KEY_FILTER_ID, CHROMA_KEY_FILTER_NAME, fs); + if (filter) { + obs_source_filter_add(s->source, filter); + obs_source_release(filter); + blog(LOG_INFO, + "[lenslink] green screen active — added a " + "pre-configured chroma-key filter"); + /* Remember the add on the source's live settings object. + * Deliberately NO obs_source_update: from this thread it + * would re-enter ios_camera_update synchronously (and + * pthread_join this very thread if connection settings had + * changed). The settings object is shared with the source, + * so the flag persists in the scene collection anyway. */ + obs_data_set_bool(settings, S_GS_FILTER_ADDED, true); + } + obs_data_release(fs); + obs_data_release(settings); +} + static int cmp_i64(const void *a, const void *b) { int64_t x = *(const int64_t *)a, y = *(const int64_t *)b; @@ -1590,19 +1652,23 @@ static void extract_json_string(const char *json, const char *key, char *out, } /* True only for a bare-boolean "key": true (same best-effort spirit as - * extract_json_string). Absent key or any other value reads as false. */ + * extract_json_string). Absent key or any other value reads as false. + * The pattern includes the ADJACENT colon: user-controlled string + * values in the snapshot (Bluetooth mic names, say) can contain the + * literal key text, and scanning to the *next* colon from a value hit + * would read some other key's value. Quote+key+quote+colon cannot occur + * inside JSON string content (an embedded quote is always escaped), so + * this anchors to real keys only. JSONSerialization emits compact + * "key":value with no space before the colon. */ static bool extract_json_bool(const char *json, const char *key) { char pattern[64]; - snprintf(pattern, sizeof(pattern), "\"%s\"", key); + snprintf(pattern, sizeof(pattern), "\"%s\":", key); const char *p = strstr(json, pattern); if (!p) return false; - p = strchr(p + strlen(pattern), ':'); - if (!p) - return false; - p++; + p += strlen(pattern); while (*p == ' ' || *p == '\t') p++; return strncmp(p, "true", 4) == 0; @@ -1953,7 +2019,18 @@ static bool handle_packet(struct ios_camera_source *s, struct client_state *c, : sizeof(s->device_state) - 1; memcpy(s->device_state, payload, n); s->device_state[n] = 0; + bool green_screen = + extract_json_bool(s->device_state, "greenScreen"); + /* Green screen is SDR-only; the snapshot carries "hdr" (HLG) + * or "color" (Apple Log) only while a 10-bit pipeline runs. + * The app enforces the exclusivity — this is belt and braces + * so a keyed filter is never added over a stream whose green + * isn't chroma green. */ + bool ten_bit = extract_json_bool(s->device_state, "hdr") || + strstr(s->device_state, "\"color\":") != NULL; pthread_mutex_unlock(&s->status_mutex); + if (green_screen && !ten_bit && !s->is_screen_source) + ensure_chroma_key_filter(s); break; } case OBSC_PKT_TIMESYNC_RESP: @@ -2763,6 +2840,9 @@ static void ios_camera_get_defaults(obs_data_t *settings) obs_data_set_default_bool(settings, S_AUTO_VIDEO_DELAY, false); obs_data_set_default_bool(settings, S_DEACTIVATE_HIDDEN, false); obs_data_set_default_bool(settings, S_AUTO_START, true); + /* Bookkeeping, not a user control: set (never cleared) after the + * green-screen chroma-key filter has been auto-added once. */ + obs_data_set_default_bool(settings, S_GS_FILTER_ADDED, false); } static bool add_audio_source(void *data, obs_source_t *source) diff --git a/obs-plugin/src/web-control.c b/obs-plugin/src/web-control.c index 7d4722a..4a942db 100644 --- a/obs-plugin/src/web-control.c +++ b/obs-plugin/src/web-control.c @@ -157,6 +157,22 @@ static const char control_page[] = "style='display:none'>" "" "Auto white balance" + /* Green screen: hidden until the app's STATE advertises support + * (supportsGreenScreen). The subject-distance slider appears only + * while depth assist is actually running (greenScreenDepth). Same + * anatomy as the app's Live row (UI_DESIGN §5): with no cutoff the + * thumb parks at the far (5.0) end and the readout shows "All"; + * tapping the readout returns to "All" (sends 0). The slider only + * ever sends real cutoffs (0.5-5.0) — 0 comes from the readout. */ + "" /* Mic picker: shown while the app streams its mic as the source's * audio (STATE micEnabled) — mirrors the app's mic row. */ "