Skip to content

feat(compositor): add source dimensions, layout re-emission, and client aspect-fit prediction#185

Open
staging-devin-ai-integration[bot] wants to merge 5 commits intomainfrom
devin/1774300965-compositor-causal-consistency-phase1
Open

feat(compositor): add source dimensions, layout re-emission, and client aspect-fit prediction#185
staging-devin-ai-integration[bot] wants to merge 5 commits intomainfrom
devin/1774300965-compositor-causal-consistency-phase1

Conversation

@staging-devin-ai-integration
Copy link
Contributor

@staging-devin-ai-integration staging-devin-ai-integration bot commented Mar 23, 2026

Summary

Phase 1 of the compositor view-data causal consistency design. This PR adds the foundation for client-side geometry prediction by:

Rust (server):

  • Adding source_width/source_height to ResolvedLayer, populated from each input slot's latest frame in resolve_scene()
  • Tracking per-slot source dimensions in InputSlot.last_source_dims and re-emitting layout (layer_configs_dirty = true) when source resolution changes at runtime (e.g. camera switch from 1080p→720p). This ensures clients learn about resolution changes within one compositing tick instead of waiting for an unrelated config edit.

TypeScript (client):

  • mapServerLayers() now materializes a stub LayerState with server geometry + default config values for server-only layers (auto-PiP layers with no explicit config in params). This is a precondition for client-side prediction — you can't predict geometry for a layer that doesn't exist in client state.
  • Server-only layer stubs are preserved across params sync cycles in Monitor view (sync-from-props effect appends layers from current state that have serverOnly: true and no counterpart in parsed params).
  • serverOnly flag is cleared when a layer gains explicit config in params (e.g. another client adds config for an auto-PiP layer), so serializeLayers correctly includes user edits.
  • layerEqual() in compositorAtoms.ts now compares the serverOnly field so flag transitions propagate to atoms.
  • Ported fitRectPreservingAspect from Rust to TypeScript (~15 lines of pure math). JS Math.round() and Rust .round() produce matching results for the non-negative values this function rounds, so predictions match the server exactly.
  • Added SourceDimsMap ref in useServerLayoutSync to store per-layer source frame dimensions from server view data, keeping prediction inputs separate from LayerState (prediction state separation per the design doc).
  • Regenerated TS types via just gen-types.

Benchmark results (compositor_pipeline, 300 frames @ 1280×720):

wall-clock   : 1.426s  (min=1.342s  max=1.524s  σ=0.0919s)
throughput   : 210.4 fps
per-frame    : 4.75 ms/frame

Review & Testing Checklist for Human

  • Verify source_width/source_height appear in view data JSON when a live pipeline is running with a compositor node (check WS messages or view-data inspector)
  • Verify that changing camera resolution at runtime (if possible) triggers a view-data re-emission without any config edit
  • Verify that auto-PiP layers (second+ inputs with no explicit rect config) now show up in the Monitor view's layer list and survive params echo-backs
  • Verify fitRectPreservingAspect test vectors match the Rust implementation — the 3 core cases (4:3→16:9, 16:9→4:3, exact match) plus zero-input edge cases
  • Verify that serverOnly flag is correctly cleared when a user adds explicit config for an auto-PiP layer (covered by new test)

Notes

  • This is Phase 1 of 3. Phase 2 (per-node config revision system with _sender/_rev echo gating) and Phase 3 (TuneNodeSilent removal) will follow as separate PRs.
  • The fitRectPreservingAspect function is exported but not yet called in any prediction path — Phase 2 will wire it into the commit adapter for zero-latency feedback on auto-PiP geometry changes.
  • SourceDimsMap is populated from view data but not yet consumed for prediction — Phase 2's stale view-data gate will use it.
  • The source_width/source_height fields use #[serde(skip_serializing_if = "Option::is_none")] to avoid breaking clients that don't expect them.

Link to Devin session: https://staging.itsdev.in/sessions/c3bbffbb30594d16845774b5bca4e32f
Requested by: @streamer45


Staging: Open in Devin

…nt aspect-fit prediction

Phase 1 of compositor view-data causal consistency:

Rust:
- Add source_width/source_height to ResolvedLayer (populated from input
  slot's latest frame in resolve_scene)
- Track per-slot source dimensions in InputSlot.last_source_dims
- Re-emit layout (set layer_configs_dirty) when source resolution changes
  at runtime (e.g. camera switch), ensuring clients learn about resolution
  changes within one tick
- Add tests for source dims population

TypeScript:
- Materialize stub LayerState in mapServerLayers for server-only layers
  (auto-PiP) with default config values — precondition for client-side
  prediction
- Port fitRectPreservingAspect from Rust to TypeScript with matching test
  vectors (4:3→16:9 pillarbox, 16:9→4:3 letterbox, exact match, zero
  inputs)
- Add SourceDimsMap ref in useServerLayoutSync to store per-layer source
  frame dimensions from server view data (prediction state separation)
- Regenerate TS types via just gen-types
- Add integration test for auto-PiP layer materialization

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
@staging-devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

staging-devin-ai-integration[bot]

This comment was marked as resolved.

In Monitor view, auto-PiP layers materialized by mapServerLayers from
server view data were being dropped when the sync-from-props effect ran,
because parseLayers(params) doesn't include server-only layers. Now the
sync-from-props effect preserves server-only layers from current state
that have no counterpart in parsed params.

Extends the auto-PiP integration test to verify stubs survive a params
echo-back.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Copy link
Contributor Author

@staging-devin-ai-integration staging-devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

View 4 additional findings in Devin Review.

Staging: Open in Devin
Debug

Playground

Comment on lines +89 to +90
opacity: DEFAULT_OPACITY,
zIndex: DEFAULT_Z_INDEX,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Auto-PiP stub z_index defaults to 0, mismatching server's idx-based z_index

When mapServerLayers creates a stub for a server-only auto-PiP layer at compositorServerSync.ts:90, it sets zIndex: DEFAULT_Z_INDEX (which is 0). However, on the server side (mod.rs:183), auto-PiP layers get z_index = idx as i32 (e.g. 1 for the second slot). This means the client's layer list ordering might not match the server's draw order. This is cosmetic since actual compositing happens server-side, but it could cause the layer panel to show incorrect stacking order for auto-PiP layers.

Staging: Open in Devin

Was this helpful? React with 👍 or 👎 to provide feedback.

Debug

Playground

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — this is cosmetic since the server controls actual draw order. The stub's zIndex only affects the client's layer panel ordering. The server view data provides the actual geometry (including correct stacking via z_index = idx), so the visual compositing is correct. Could improve this in a future pass by including z_index in the ResolvedLayer view data, but for now it doesn't affect behavior.

…ized back

Add serverOnly flag to LayerState, set it on stubs materialized by
mapServerLayers for auto-PiP layers. serializeLayers() skips these
entries, preventing explicit config from being sent to the server which
would disable aspect-fit on the auto-PiP branch in resolve_scene().

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
staging-devin-ai-integration[bot]

This comment was marked as resolved.

- Only preserve layers with serverOnly flag during params sync,
  preventing removed user layers from lingering.
- Add serverOnly field to layerEqual() so flag transitions trigger
  proper atom updates.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
staging-devin-ai-integration[bot]

This comment was marked as resolved.

…it config

When another client adds explicit config for an auto-PiP layer, the
serverOnly flag must be cleared so serializeLayers includes user edits.
Added test verifying the flag transition.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants