Skip to content

Proper video support for plugins, with example plugin #169

@staging-devin-ai-integration

Description

Summary

Add first-class video frame support to the plugin SDK, enabling plugins to receive, process, and emit video data. This should include an example plugin demonstrating video processing capabilities.

Motivation

Currently, plugin development is primarily oriented around audio and metadata. As StreamKit's compositor and video pipeline matures, plugins need the ability to work with video frames — for use cases like custom video filters, ML-based video processing (e.g. background removal, style transfer, object detection overlays), frame analysis, and more.

Areas to Explore

  • Plugin SDK video API: extending the WIT interface and native plugin SDK to support video frame input/output with appropriate pixel format handling.
  • Frame passing: efficient zero-copy or pooled-buffer strategies for passing video frames across the plugin boundary (especially for WASM plugins where memory is sandboxed).
  • Pixel format support: which formats to expose (e.g. NV12, I420, RGBA) and whether format conversion should happen at the plugin boundary or within the plugin.
  • Example plugin: a concrete reference implementation — e.g. a simple video filter, watermark overlay, or frame transformation — to validate the API and serve as documentation.
  • Integration with FramePool: ensuring video buffers follow StreamKit's existing resource recycling patterns.

Requirements

  • Plugin must have a just target for building.
  • Model/asset download support (if applicable) so the plugin works out of the box from the marketplace once published.

Scope

This is a high-level roadmap item for investigation and design. The goal is to define the video plugin API, validate it with an example, and ensure it integrates cleanly with the existing plugin infrastructure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions