Add Multimedia Support (V4L2, ALSA, GStreamer)

[mihai-chiorean]

Add Multimedia Support (V4L2, ALSA, GStreamer)

Description

Add comprehensive multimedia support to EdgeOS for audio/video capture, playback, and processing capabilities on Raspberry Pi 5 and Jetson Orin Nano platforms.

Purpose

Enable EdgeOS to handle multimedia workloads including:

• Video capture from CSI/USB cameras
• Audio input/output via ALSA
• Media processing pipelines with GStreamer
• Hardware-accelerated video encoding/decoding
• Platform-specific optimizations (RPi VC4/V3D, NVIDIA NVENC/NVDEC)

Recipe Structure

meta-edgeos/
├── recipes-multimedia/
│   ├── gstreamer/
│   │   ├── gstreamer1.0_%.bbappend
│   │   ├── gstreamer1.0-plugins-good_%.bbappend
│   │   └── gstreamer1.0-plugins-bad_%.bbappend
│   ├── nvidia-multimedia/ (Jetson only)
│   │   ├── nvidia-l4t-multimedia_%.bbappend
│   │   └── deepstream_%.bbappend
│   ├── v4l-utils/
│   │   └── v4l-utils_%.bbappend
│   └── alsa/
│       └── alsa-utils_%.bbappend
└── recipes-core/
    └── packagegroups/
        ├── packagegroup-edgeos-multimedia.bb
        ├── packagegroup-edgeos-multimedia-rpi.bb
        └── packagegroup-edgeos-multimedia-jetson.bb

Platform-Specific Implementation

Common Components (Both Platforms)

# packagegroup-edgeos-multimedia.bb
RDEPENDS:${PN} = " \
    v4l-utils \
    libv4l \
    media-ctl \
    alsa-utils \
    alsa-lib \
    alsa-state \
    alsa-plugins \
    gstreamer1.0 \
    gstreamer1.0-plugins-base \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad \
    gstreamer1.0-tools \
"

# Platform-specific packages
RDEPENDS:${PN}:append:raspberrypi5 = " packagegroup-edgeos-multimedia-rpi"
RDEPENDS:${PN}:append:jetson-orin-nano = " packagegroup-edgeos-multimedia-jetson"

Raspberry Pi 5 Specific

# packagegroup-edgeos-multimedia-rpi.bb
RDEPENDS:${PN} = " \
    libraspberrypi \
    libraspberrypi-bin \
    rpicam-apps \
    libcamera \
    libcamera-tools \
    gstreamer1.0-omx \
    kernel-module-bcm2835-v4l2 \
    kernel-module-bcm2835-codec \
    kernel-module-vc4 \
"

Jetson Orin Nano Specific

# packagegroup-edgeos-multimedia-jetson.bb
RDEPENDS:${PN} = " \
    nvidia-l4t-multimedia \
    nvidia-l4t-multimedia-utils \
    nvidia-l4t-camera \
    nvidia-l4t-gstreamer \
    gstreamer1.0-plugins-nvvideo4linux2 \
    gstreamer1.0-plugins-nvarguscamerasrc \
    gstreamer1.0-plugins-nvvidconv \
    gstreamer1.0-plugins-nvjpeg \
    deepstream-6.0 \
    cuda-toolkit \
    tegra-libraries-multimedia \
    tegra-libraries-camera \
    argus-samples \
    v4l2loopback-kernel-module \
"

Configuration

Raspberry Pi 5 Configuration

# /boot/config.txt
dtoverlay=vc4-kms-v3d
camera_auto_detect=1
display_auto_detect=1
gpu_mem=256

# Kernel modules
CONFIG_VIDEO_BCM2835=m
CONFIG_VIDEO_CODEC_BCM2835=m

Jetson Orin Nano Configuration

# Power mode for maximum performance
sudo nvpmodel -m 0  # MAXN mode
sudo jetson_clocks

# Device tree configuration
# Handled by JetPack flash process

# Kernel modules
CONFIG_VIDEO_TEGRA=y
CONFIG_VIDEO_TEGRA_VI=y
CONFIG_TEGRA_CAMERA=y

Test Commands

Common Tests (Both Platforms)

# V4L2 - List video devices
v4l2-ctl --list-devices

# ALSA - List audio devices
aplay -l
arecord -l

# GStreamer - Basic test
gst-launch-1.0 videotestsrc ! autovideosink
gst-launch-1.0 audiotestsrc ! autoaudiosink

Raspberry Pi 5 Specific Tests

# libcamera test
libcamera-hello --list-cameras
rpicam-hello --timeout 5000

# RPi Camera capture with libcamera
rpicam-jpeg -o test.jpg

# GStreamer with V4L2
gst-launch-1.0 v4l2src device=/dev/video0 ! \
    video/x-raw,width=1920,height=1080,framerate=30/1 ! \
    v4l2h264enc ! h264parse ! mp4mux ! \
    filesink location=test.mp4

# Hardware-accelerated decode
gst-launch-1.0 filesrc location=test.mp4 ! \
    qtdemux ! h264parse ! v4l2h264dec ! \
    autovideosink

Jetson Orin Nano Specific Tests

# NVIDIA capture test
nvgstcapture-1.0

# List Argus cameras
argus_camera --device-list

# GStreamer with NVIDIA plugins
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! \
    'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! \
    nvvidconv ! xvimagesink

# Hardware-accelerated encode with NVENC
gst-launch-1.0 nvarguscamerasrc ! \
    'video/x-raw(memory:NVMM),width=1920,height=1080' ! \
    nvv4l2h264enc bitrate=8000000 ! h264parse ! \
    mp4mux ! filesink location=test.mp4

# Hardware-accelerated decode with NVDEC
gst-launch-1.0 filesrc location=test.mp4 ! \
    qtdemux ! h264parse ! nvv4l2decoder ! \
    nvvidconv ! autovideosink

# DeepStream test
deepstream-app -c /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/source1_csi_dec_infer_resnet_int8.txt

# CUDA-accelerated processing
gst-launch-1.0 nvarguscamerasrc ! \
    'video/x-raw(memory:NVMM)' ! \
    nvivafilter cuda-process=true ! \
    nvvidconv ! autovideosink

Performance Validation

Raspberry Pi 5

# Check hardware acceleration
vcgencmd get_mem gpu
vcgencmd codec_enabled H264

# Monitor GPU usage
vcgencmd measure_temp
vcgencmd measure_clock v3d

Jetson Orin Nano

# Monitor GPU/encoder usage
tegrastats

# Check NVENC/NVDEC sessions
nvidia-smi dmon -s pucvmet

# Verify CUDA availability
deviceQuery

Acceptance Criteria

Common Requirements

• V4L2 devices enumerated at /dev/video*
• USB cameras detected and functional
• ALSA audio devices available
• Audio playback works through HDMI/jack
• GStreamer pipelines functional
• No conflicts with existing USB gadget functionality

Raspberry Pi 5 Specific

• CSI camera detected via libcamera
• rpicam-apps functional
• V4L2 M2M codec working
• VC4/V3D acceleration active
• 1080p30 capture working

Jetson Orin Nano Specific

• CSI cameras detected via Argus API
• nvarguscamerasrc plugin working
• NVENC/NVDEC hardware encoding/decoding
• CUDA processing in GStreamer pipelines
• DeepStream SDK functional
• 4K30 capture/encode capability
• Multiple camera support (up to 4)

Performance Requirements

Raspberry Pi 5

• 1080p30 capture and encode
• H.264/H.265 hardware decode up to 4K60
• <20ms audio latency
• 1080p30 simultaneous encode/decode

Jetson Orin Nano

• 4K30 capture from CSI
• 4K30 H.264/H.265 encode (NVENC)
• Multiple 1080p streams decode (NVDEC)
• <10ms glass-to-glass latency possible
• AI inference at 30fps on video stream

Machine Configuration

# local.conf additions
# For Raspberry Pi 5
MACHINE ?= "raspberrypi5"
GPU_MEM = "256"
VIDEO_CAMERA = "1"

# For Jetson Orin Nano
MACHINE ?= "jetson-orin-nano"
CUDA_VERSION = "11.4"
JETPACK_VERSION = "5.1.2"

References

Raspberry Pi

• Raspberry Pi Camera Documentation
• libcamera and libcamera-apps
• V4L2 M2M Codec

Jetson Orin Nano

• NVIDIA Jetson Linux Multimedia
• Argus Camera API
• GStreamer Plugin Guide
• DeepStream SDK
• JetPack SDK

Common

• GStreamer Documentation
• V4L2 API Documentation
• ALSA Project