Virtual green screen: segmentation, depth assist, and subject-distance cutoff

README.md

@@ -85,6 +85,14 @@ blanks the source.
   **Apple Log** image and grade it yourself with OBS's Apply LUT filter
   — the plugin decodes it faithfully and otherwise keeps its hands off.
   Nothing else in this space offers it.
+- **Virtual green screen (beta).** Remove your background without a
+  green screen: the phone segments you (any camera, no depth hardware
+  required), composites the background to chroma green before encoding,
+  and the plugin auto-adds a pre-configured chroma-key filter — toggle
+  on the phone, keyed subject in OBS. On Face ID front cameras and
+  LiDAR rear Main cameras, depth sharpens the matte and a
+  **subject distance** cutoff (app Live screen or web panel) drops
+  people walking by behind you.
 - **Pick any lens** — Main, Ultra Wide, Telephoto, or Front — switchable
   live while you stream.
 - **Hold it however.** The app's UI follows the phone's rotation (unless

docs/PERFORMANCE.md

@@ -230,3 +230,16 @@ Deliberate divergences (don't "fix" these without reading this):
 - **Rotation is sensor-native** (`.landscapeRight`, an effective 0°).
   The AVCaptureConnection docs warn per-frame rotation costs; we never
   rotate the stream, only the on-phone preview.
+- **Green screen OFF is free, ON pays exactly once.** With the toggle
+  off, the capture→encode path is the pre-feature code verbatim — no
+  compositor branch on the hot path, no depth output attached. On, the
+  budget is: Vision person segmentation per frame (`.fast` above
+  30 fps, `.balanced` at 30 — the dominant cost), one Metal compute
+  pass writing green into a pool buffer (that pass *is* the single
+  copy; the camera's own buffer is never written in place — it races
+  the encoder's async read), and depth at ~15 Hz when assist is
+  active. Depth delivery deactivates Center Stage and the system video
+  effects by iOS policy, and overload sheds frames via the existing
+  drop-don't-queue capture behaviour. On-device bench + thermal-soak
+  numbers for green-screen-ON are still to be captured and pasted
+  here; the OFF run must show the zero-cost claim holds.

docs/PROTOCOL.md

@@ -123,6 +123,7 @@ Camera remote control. Payload: UTF-8 JSON, one command per packet:
 { "cmd": "set_format", "resolution": "1080p", "fps": 60, "codec": "hevc" }
 { "cmd": "mic", "id": "builtin:2" }
 { "cmd": "tally", "program": true, "preview": false, "sync": "locked" }
+{ "cmd": "green_screen", "on": true, "maxDistance": 2.5 }
 ```
 
 `set_format` switches the capture format mid-stream; any subset of its
@@ -173,6 +174,18 @@ serving as source audio (type 10). An older app ignores the command and
 streams continuously, which the plugin handles fine; a plugin that never
 sends it gets today's always-on behaviour.
 
+`green_screen` toggles the virtual green screen and its depth cutoff;
+either field may appear alone. `on` arms/disarms the segmentation +
+green composite (the compositing happens **on the phone, before
+encoding** — the wire carries ordinary video whose background happens
+to be chroma green, so receivers need no new decode behaviour).
+`maxDistance` (metres; `0` = no cutoff, otherwise 0.5–5.0) drives the
+depth-assisted subject cutoff and is meaningful only while depth
+assist is active — the app clamps and ignores as needed. Green screen
+is SDR-only: arming it forces the Standard colour pipeline, and the
+STATE fields below keep every surface honest about what's running.
+Camera connections only; screen mirror ignores it.
+
 Unknown commands are ignored, so new ones can be added compatibly. The
 plugin's embedded web panel (http://localhost:9980) generates these.
 
@@ -216,6 +229,20 @@ H.264 through the ordinary `set_format` validation. Both fields are
 absent on SDR streams, whose snapshots are unchanged from before
 colour modes existed.
 
+Green screen state rides the snapshot the same way:
+`"supportsGreenScreen": true` advertises the feature (remote UIs gate
+their row on it), `"greenScreen": true` appears while it is armed,
+`"greenScreenDepth": true` while depth assist is actually running
+(TrueDepth front / LiDAR rear Main lens, and a depth-capable format
+matched — absent means segmentation-only; remote UIs key the distance
+control off truthiness), and `"greenScreenMaxDistance"` carries the
+cutoff in metres when one is set. Boolean snapshot fields are emitted
+only when true: a receiver's STATE cache has finite room (older
+plugins truncate silently past 1 KiB), so snapshot growth is real
+cost — weigh every new field against it. The plugin also reacts to `"greenScreen": true` on a
+camera source by auto-adding a pre-configured chroma-key filter —
+once, by name, respecting a user's later deletion of it.
+
 While the phone mic streams as the source's audio (packet type 10), the
 snapshot additionally carries `micEnabled: true`, the selected `mic` id,
 and the selectable `mics` list (`[{ "id", "name" }, …]`) for the `mic`

docs/ROADMAP.md

@@ -17,26 +17,28 @@ Continuity Camera, iVCam/Iriun, NDI HX Camera/Larix).
 | P1 | Thermal & low-power adaptation | Reliability & security | Medium |
 | P1 | Optional pairing & encryption | Reliability & security | Medium |
 | P1 | Graduate the GPU decode pipeline | Performance | Small |
-| P1 | Tally light on the phone | Control & workflow | Small |
 | P2 | Digital pan/tilt + crop | Control & workflow | Medium |
-| P2 | Virtual green screen (background removal) | Video & audio quality | Medium |
-| P2 | 10-bit HEVC + HDR (HLG/PQ) | Video & audio quality | Large |
 | P2 | Voice isolation for the phone mic | Video & audio quality | Small |
 | P2 | Document the control API | Control & workflow | Small |
-| P3 | Apple Log capture | Video & audio quality | Small¹ |
 | P3 | Manual bitrate cap | Video & audio quality | Small |
 | P3 | Zero-copy encoder output | Performance | Medium |
 | P3 | iPad: keep streaming in Split View | Control & workflow | Small |
 | P3 | Two lenses at once (multicam) | Video & audio quality | Large |
 
-¹ once 10-bit HEVC ships.
-
 - **P1 — next up.** Protects or finishes what already ships, plus the
   one tiny-effort/large-payoff outlier. Pick from here first.
 - **P2 — on deck.** Clearly worth building; start once P1 is moving.
 - **P3 — needs a trigger.** A dependency landing, a measurement, or
   real user demand should promote these before anyone starts.
 
+Recently shipped and pruned from this file (see the release notes):
+tally light with customizable colours (v1.8.0), 10-bit HDR (HLG) with
+the zero-copy GPU path and Apple Log (v1.9.0), virtual green screen
+with depth assist and subject-distance cutoff (v1.10.0). Pairing &
+encryption was explicitly deprioritized by the maintainer in 2026-07 —
+revisit when remote start gets promoted or users stream on shared
+networks.
+
 ## Reliability & security
 
 ### Thermal & low-power adaptation on the phone — P1, medium (partially shipped)
@@ -91,47 +93,13 @@ if you do it.*
 
 ## Video & audio quality
 
-### 10-bit HEVC + HDR (HLG/PQ) — P2, large
-iPhones capture HDR by default; the current 8-bit NV12 pipeline throws
-that away, and DroidCam already ships 10-bit HDR transfer. App: 10-bit
-capture format + HEVC Main-10 in `VideoEncoder`; protocol: VIDEO_CONFIG
-gains colour metadata (primaries/transfer/matrix); plugin: map to
-P010 + OBS's HDR colour handling (OBS 28+), including the GPU
-pipeline's per-backend texture formats. Ship with a sensible SDR
-default — Twitch is still SDR, so HDR mainly pays off for YouTube and
-local recording. *The headline video-quality differentiator; the only
-large item on the active list, and it unlocks Apple Log below.*
-
-### Virtual green screen — background removal without the screen — P2, medium
-Camo's flagship paid feature, on the free table. Vision's person
-segmentation (`VNGeneratePersonSegmentationRequest`, iOS 15 — exactly
-our floor) mattes a person from plain RGB on any camera, front or rear
-— no depth hardware required. The wire stays untouched: composite the
-removed background to solid chroma green on the phone and let OBS's
-chroma key provide the transparency. Staged: **v1** segmentation +
-green composite behind an Options toggle (per-frame GPU work — quote
-before/after numbers, lean on the thermal mitigation); **v2** the app
-advertises the mode in STATE and the plugin auto-adds a pre-configured
-chroma-key filter to the source, so the phone toggle alone yields a
-keyed subject in the scene; **v3** depth-assisted edge refinement
-(hair, glasses) where TrueDepth or LiDAR exists — this is where
-Apple's "enhancing live video with TrueDepth" techniques earn their
-keep, as an enhancer rather than a requirement.
-
 ### Voice isolation for the phone mic — P2, small
 The mic features send the raw microphone feed; iOS's built-in
 voice-processing pipeline (echo cancellation + noise suppression, the
 FaceTime mic sound) is nearly free to enable. An Options toggle in the
 Microphone group, applied to the send-phone-mic path. *Makes the
 "phone as wireless mic" feature genuinely usable in untreated rooms.*
 
-### Apple Log capture — P3, small (gated on 10-bit HEVC)
-On iPhone 15 Pro and later the camera can capture in Apple Log
-(`AVCaptureColorSpace.appleLog`); streamers grade it with a LUT in OBS.
-Once the 10-bit path exists this is mostly a capture-format toggle plus
-correct colour tagging, gated to devices that report support. Nothing
-else in this market offers it. *Trigger: 10-bit HEVC shipping.*
-
 ### Two lenses at once (multicam) — P3, large
 `AVCaptureMultiCamSession` (iOS 13+) can run the front and a rear
 camera simultaneously — one phone feeding a face cam *and* a scene cam
@@ -153,17 +121,6 @@ covers most of them today.*
 
 ## Control & workflow
 
-### Tally light on the phone — P1, small
-Pro rigs treat tally as table stakes, and no phone-webcam app has it.
-The plugin already tracks per-source visibility (it drives "Disconnect
-when this source isn't shown anywhere") and can read program/preview
-state from the frontend API; send it to the app as a new CONTROL
-command (`{"cmd":"tally","state":"live|preview|idle"}` — old apps
-ignore unknown commands, so it's compatible) and surface it on the
-Live screen. UI_DESIGN.md §2 needs a tally state added to the status
-vocabulary. For multi-phone scenes this removes the "which camera am I
-on?" guesswork. *The best payoff-per-line on this list.*
-
 ### Digital pan/tilt + crop — reframe without touching the rig — P2, medium
 Zoom today is the camera's own, always center-locked. An adjustable
 crop applied on the phone *before* encoding (Camera Hub's signature

docs/UI_DESIGN.md

@@ -248,6 +248,7 @@ surfaces.
 | Stop        | `stop.fill`                                | Red chip; the only destructive control |
 | Dim (app)   | `moon.fill`                                | App-only battery saver |
 | Stats (app) | `gauge`                                    | App-only toggle; shows a health pill (`60 fps · 11.9 Mb/s · 0 dropped`, monospaced) under the status bar |
+| Green screen | `person.fill.viewfinder`                  | **Always "Green screen"** (never "chroma key", "background removal", or "matte" in UI copy). Armed from the Setup screen; while live **with depth assist**, a "Subject distance" slider row (0.5–5.0 m, readout `2.5 m` monospaced) appears on the Live panel and the web panel, same position both surfaces. Dragging always sets a real cutoff — full-left = tightest (0.5 m); **"All" (no cutoff) is entered by tapping the readout**, and while "All" the thumb parks at the far (5.0) end. Identical on both surfaces |
 
 ### The app icon (three appearances)