feat(compositor): add source dimensions, layout re-emission, and client aspect-fit prediction#185
Conversation
…nt aspect-fit prediction Phase 1 of compositor view-data causal consistency: Rust: - Add source_width/source_height to ResolvedLayer (populated from input slot's latest frame in resolve_scene) - Track per-slot source dimensions in InputSlot.last_source_dims - Re-emit layout (set layer_configs_dirty) when source resolution changes at runtime (e.g. camera switch), ensuring clients learn about resolution changes within one tick - Add tests for source dims population TypeScript: - Materialize stub LayerState in mapServerLayers for server-only layers (auto-PiP) with default config values — precondition for client-side prediction - Port fitRectPreservingAspect from Rust to TypeScript with matching test vectors (4:3→16:9 pillarbox, 16:9→4:3 letterbox, exact match, zero inputs) - Add SourceDimsMap ref in useServerLayoutSync to store per-layer source frame dimensions from server view data (prediction state separation) - Regenerate TS types via just gen-types - Add integration test for auto-PiP layer materialization Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
In Monitor view, auto-PiP layers materialized by mapServerLayers from server view data were being dropped when the sync-from-props effect ran, because parseLayers(params) doesn't include server-only layers. Now the sync-from-props effect preserves server-only layers from current state that have no counterpart in parsed params. Extends the auto-PiP integration test to verify stubs survive a params echo-back. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
| opacity: DEFAULT_OPACITY, | ||
| zIndex: DEFAULT_Z_INDEX, |
There was a problem hiding this comment.
🚩 Auto-PiP stub z_index defaults to 0, mismatching server's idx-based z_index
When mapServerLayers creates a stub for a server-only auto-PiP layer at compositorServerSync.ts:90, it sets zIndex: DEFAULT_Z_INDEX (which is 0). However, on the server side (mod.rs:183), auto-PiP layers get z_index = idx as i32 (e.g. 1 for the second slot). This means the client's layer list ordering might not match the server's draw order. This is cosmetic since actual compositing happens server-side, but it could cause the layer panel to show incorrect stacking order for auto-PiP layers.
Was this helpful? React with 👍 or 👎 to provide feedback.
Debug
There was a problem hiding this comment.
Acknowledged — this is cosmetic since the server controls actual draw order. The stub's zIndex only affects the client's layer panel ordering. The server view data provides the actual geometry (including correct stacking via z_index = idx), so the visual compositing is correct. Could improve this in a future pass by including z_index in the ResolvedLayer view data, but for now it doesn't affect behavior.
…ized back Add serverOnly flag to LayerState, set it on stubs materialized by mapServerLayers for auto-PiP layers. serializeLayers() skips these entries, preventing explicit config from being sent to the server which would disable aspect-fit on the auto-PiP branch in resolve_scene(). Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Only preserve layers with serverOnly flag during params sync, preventing removed user layers from lingering. - Add serverOnly field to layerEqual() so flag transitions trigger proper atom updates. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…it config When another client adds explicit config for an auto-PiP layer, the serverOnly flag must be cleared so serializeLayers includes user edits. Added test verifying the flag transition. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Summary
Phase 1 of the compositor view-data causal consistency design. This PR adds the foundation for client-side geometry prediction by:
Rust (server):
source_width/source_heighttoResolvedLayer, populated from each input slot's latest frame inresolve_scene()InputSlot.last_source_dimsand re-emitting layout (layer_configs_dirty = true) when source resolution changes at runtime (e.g. camera switch from 1080p→720p). This ensures clients learn about resolution changes within one compositing tick instead of waiting for an unrelated config edit.TypeScript (client):
mapServerLayers()now materializes a stubLayerStatewith server geometry + default config values for server-only layers (auto-PiP layers with no explicit config in params). This is a precondition for client-side prediction — you can't predict geometry for a layer that doesn't exist in client state.serverOnly: trueand no counterpart in parsed params).serverOnlyflag is cleared when a layer gains explicit config in params (e.g. another client adds config for an auto-PiP layer), soserializeLayerscorrectly includes user edits.layerEqual()incompositorAtoms.tsnow compares theserverOnlyfield so flag transitions propagate to atoms.fitRectPreservingAspectfrom Rust to TypeScript (~15 lines of pure math). JSMath.round()and Rust.round()produce matching results for the non-negative values this function rounds, so predictions match the server exactly.SourceDimsMapref inuseServerLayoutSyncto store per-layer source frame dimensions from server view data, keeping prediction inputs separate fromLayerState(prediction state separation per the design doc).just gen-types.Benchmark results (compositor_pipeline, 300 frames @ 1280×720):
Review & Testing Checklist for Human
source_width/source_heightappear in view data JSON when a live pipeline is running with a compositor node (check WS messages or view-data inspector)fitRectPreservingAspecttest vectors match the Rust implementation — the 3 core cases (4:3→16:9, 16:9→4:3, exact match) plus zero-input edge casesserverOnlyflag is correctly cleared when a user adds explicit config for an auto-PiP layer (covered by new test)Notes
_sender/_revecho gating) and Phase 3 (TuneNodeSilent removal) will follow as separate PRs.fitRectPreservingAspectfunction is exported but not yet called in any prediction path — Phase 2 will wire it into the commit adapter for zero-latency feedback on auto-PiP geometry changes.SourceDimsMapis populated from view data but not yet consumed for prediction — Phase 2's stale view-data gate will use it.source_width/source_heightfields use#[serde(skip_serializing_if = "Option::is_none")]to avoid breaking clients that don't expect them.Link to Devin session: https://staging.itsdev.in/sessions/c3bbffbb30594d16845774b5bca4e32f
Requested by: @streamer45