Virtual green screen: segmentation, depth assist, and subject-distance cutoff

[MyNamesEMurray]

What & why

The roadmap's virtual green screen, all three planned stages shipped together:

Stage 1 — background removal on any camera. A Green screen toggle on the main screen (Camera & color section). The phone runs Vision person segmentation per frame (.fast above 30 fps, .balanced at 30, one stateful request per stream for temporal smoothing) and composites the background to BT.709 chroma green (#00B140) in a single Metal compute pass — the bilinear mask upsample doubles as the edge feather. The wire carries ordinary video; no protocol change to decode. Off is free: the hot path with the toggle off is the pre-feature closure verbatim (now a PERFORMANCE.md invariant). Every failure fails open to the uncomposited frame — a Vision hiccup can never blank a stream.

Stage 2 — depth assist + auto chroma key. On Face ID front cameras and LiDAR rear Main cameras, capture swaps to the depth-sibling device and synchronizes a ~15 Hz depth stream that gates the matte (NaN-safe), fixing segmentation's classic failures: posters/monitors keyed as people, low light, held objects. The plugin reacts to the STATE snapshot by auto-adding a pre-configured chroma-key filter to the source — once, by name, respecting later user deletion via a persisted per-source flag — so the phone toggle alone yields a keyed subject in OBS. Degrades silently to segmentation-only when the lens/format can't carry depth (never changes resolution/fps to chase it).

Stage 3 — subject distance. A Subject distance cutoff (0.5–5.0 m, or All) on the app's Live screen and the web panel — same anatomy both surfaces: dragging always sets a real cutoff, tapping the readout returns to All. With one semantic mask covering everyone (per-person masks need iOS 17), the depth cutoff is the only iOS 15 tool that drops people walking by behind you. Rides the existing CONTROL/STATE machinery (green_screen command; supportsGreenScreen/greenScreen/greenScreenDepth/greenScreenMaxDistance fields, spec'd in PROTOCOL.md).

Green screen is SDR-only and forces Standard colour through the same exclusivity pattern the codec uses. Depth delivery deactivates Center Stage/system video effects by iOS policy (documented in-app). STATE caches grow 1 KiB → 2 KiB on the plugin side; the app emits true-only booleans to minimize snapshot growth for old plugins.

Adversarial review pass (three independent lenses) caught and this PR includes fixes for: a cross-queue data race in the depth synchronizer delegate, a double live-reconfigure when switching to HDR/Log with green screen on, substring JSON probes spoofable by Bluetooth mic names (now colon-anchored), a panel/app slider-convention mismatch, and a per-arm Metal shader recompile (now process-wide static).

How it was tested

• Swift sources parse clean (syntax-check.sh); full type check is CI's macOS job — first compile of Vision/Metal/depth API usage, watch that job.
• Plugin builds clean under -Wall -Wextra -Werror; the real-HTTP webtest harness passes all checks against the edited web-control.c, and the served panel JS parses (node --check) with the slider logic verified by a value-table smoke test.
• Needs on-device validation (nothing here can run Vision/Metal/depth): green screen on front + rear cameras, the auto-added filter appearing in OBS, depth assist on a LiDAR phone (Main lens), the Subject distance control from both surfaces, an SDR regression pass with the toggle off, and a thermal soak — plus the formatReport diagnostics dump, which now includes a depth column to answer which resolution/fps combos carry depth per device.

🤖 Generated with Claude Code

https://claude.ai/code/session_015dWUcqAkNNuNELnMPS4RdT

---

Generated by Claude Code