Add Dynamic Parameter Rendering

[leszko]

Add support for defining parts of the frontend in plugins.

Demo

demo_dynamic.mp4

Sample plugin

It's simplest to understand how it works by looking at a sample plugin: https://github.com/leszko/scope-plugin-ui

1. Look at schema.py, it introduces custom UI fields, e.g.

    config_slider: int = Field(
        default=50,
        ge=0,
        le=100,
        description="Integer 0–100 in Settings",
        json_schema_extra=ui_field_config(
            order=3, is_load_param=True, label="Slider"
        ),
    )

2. Install the plugin with DAYDREAM_SCOPE_PREVIEW=1 uv run daydream-scope install git+https://github.com/leszko/scope-plugin-ui
3. Look the fields rendered in UI

What and how to define

Any field in schema.py that includes json_schema_extra will be rendered in the UI.
You can define the following keys inside json_schema_extra to control how the field is displayed:

• order (int) — Controls the rendering order in the UI (lower numbers appear first).
• category (str) — Determines where the field is displayed. Possible values:
  • input
  • configuration (default)
• component (str) — The name of a custom or complex UI component to use for this field.
• label (str) — The label shown in the UI. If not provided, the field’s description is used.
• modes (list[str]) — Specifies which modes the field is visible in.
  Defaults to ["text", "video"].

Resources:

• Original RFC: Notion
• Discord discussions: thread

Non-Goals

• Complex components can be improved, but I'll leave it for now, better to send a separate PR for that; the intention of this PR is to have support for primitive fields for plugins
• Complex components and their usage in SettingsPanel, I also leave it as a separate PR
• UI Schema JSON returned from the Server API endpoint (for now let's just use the same schema endpoint)

[yondonfu]

This looks pretty good! I tested install of the sample plugin and confirmed that the schema driven fields show up in InputAndControls + Settings panel and that updating the fields in the former trigger parameter update sends across the WebRTC data channel during runtime.

There seems to be a stray box by default when I open up the app and look at the default LongLive settings.

[leszko]

This looks pretty good! I tested install of the sample plugin and confirmed that the schema driven fields show up in InputAndControls + Settings panel and that updating the fields in the former trigger parameter update sends across the WebRTC data channel during runtime.

There seems to be a stray box by default when I open up the app and look at the default LongLive settings.

Ahh, sorry, my bad I haven't removed this part form the PR after testing. Removed now.

[yondonfu]

Doesn't need to happen immediately, but for my own understanding will we be able to slim down BasePipelineConfig too? It seems to have a lot of stuff.

[leszko]

Doesn't need to happen immediately, but for my own understanding will we be able to slim down BasePipelineConfig too? It seems to have a lot of stuff.

Yes, agree. The same actually with some parts of the frontend, like StreamPage.tsx, schemaSettings.tsx, and SettingsPanel.tsx. I have a feeling there is too much happening there. But let's keep it as a separate PR. This is only have a lot of changes, I don't want to introduce additional refactor here.

[yondonfu]

LGTM!