Skip to content

architecture_section.md

Latest commit

 

History

History
69 lines (53 loc) · 3.15 KB

architecture_section.md

File metadata and controls

69 lines (53 loc) · 3.15 KB

Briefly about architecture of Vulkan Media Player

The Vulkan Media Player plugin follows a modular architecture designed for efficient hardware-accelerated media playback on Linux platforms. The system is built around Unreal Engine's Media Framework and leverages Vulkan Video API for GPU-accelerated video decoding.

Core Components

Factory Module (VulkanMediaPlayerFactory)

  • Registers the player with Unreal's Media Framework
  • Handles URL validation and player creation
  • Supports MP4 files on Linux, LinuxARM64, and Windows (for testing)

Main Player (FVulkanMediaPlayer)

  • Implements Unreal's IMediaPlayer interface
  • Manages playback state, seeking, and rate control
  • Coordinates between demuxer, decoder, and tracks manager

MP4 Demuxer (FMP4Demuxer)

  • Parses MP4 containers using libmp4v2
  • Extracts encoded video (H.264/H.265) and audio (AAC) packets
  • Runs on a background thread to avoid blocking the main thread
  • Handles seeking and timestamp normalization

Video Decoder (FVulkanVideoDecoderV2)

  • Hardware-accelerated H.264/H.265 decoding using Vulkan Video API
  • Creates its own Vulkan logical device with video extensions
  • Implements proper DPB (Decoded Picture Buffer) management
  • Supports B-frame reordering for correct display order
  • Converts AVCC format (MP4) to Annex B format (Vulkan requirement)

Audio Decoder (FAACAudioDecoder)

  • AAC audio decoding using fdk-aac library
  • Converts encoded AAC frames to PCM audio samples

Video Decoder Worker (FVideoDecoderWorker)

  • Background thread that orchestrates the decoding pipeline
  • Reads packets from demuxer, decodes with Vulkan decoder
  • Performs GPU-based NV12 to RGB conversion
  • Manages frame buffering and B-frame reordering
  • Implements on-demand decoding with time-based buffering

Tracks Manager (FVulkanMediaTracks)

  • Manages video and audio tracks
  • Queues decoded samples for delivery to the engine
  • Implements Unreal's IMediaSamples and IMediaTracks interfaces

Data Flow

  1. Demuxing: MP4 file → Demuxer extracts encoded packets → Queued packets
  2. Video Decoding: Encoded packets → Vulkan Video Decoder → GPU textures (NV12 format)
  3. Color Conversion: NV12 textures → GPU shader → RGB textures
  4. Sample Delivery: RGB textures → Tracks Manager → Unreal Engine Media Framework
  5. Audio Decoding: AAC packets → fdk-aac decoder → PCM samples → Audio system

Key Features

  • Hardware Acceleration: Leverages GPU video decoding capabilities via Vulkan Video API
  • Threaded Architecture: Demuxer and decoder run on separate threads for smooth playback
  • B-frame Support: Proper frame reordering using Picture Order Count (POC)
  • GPU-only Path: Decoded frames stay on GPU throughout the pipeline (no CPU roundtrip)
  • On-demand Decoding: Time-based buffering prevents unnecessary decoding ahead of playback
  • DPB Management: Efficient reference frame handling for H.264 decoding

Dependencies

  • Vulkan Video API: Extensions for hardware video decoding
  • libmp4v2: MP4 container parsing
  • fdk-aac: AAC audio decoding
  • Unreal Engine Media Framework: Integration with engine's media system