diff --git a/components/rtsp/README.md b/components/rtsp/README.md old mode 100644 new mode 100755 index e87ca83f6..e6af54e4c --- a/components/rtsp/README.md +++ b/components/rtsp/README.md @@ -2,8 +2,11 @@ [![Badge](https://components.espressif.com/components/espp/rtsp/badge.svg)](https://components.espressif.com/components/espp/rtsp) -The `rtsp` component provides various classes for implementing both sides of an -RTSP stream for transmitting MJPEG video data. +The `rtsp` component provides a flexible, multi-codec RTSP streaming framework +for ESP32 devices. It supports MJPEG, H.264, and generic audio codecs through +an extensible packetizer/depacketizer architecture. The component handles only +RTP packet splitting and reassembly — encoding and decoding of media data is +performed externally. **Table of Contents** @@ -11,6 +14,7 @@ RTSP stream for transmitting MJPEG video data. - [RTSP (Real-Time Streaming Protocol) Component](#rtsp-real-time-streaming-protocol-component) - [RTSP Client](#rtsp-client) - [RTSP Server](#rtsp-server) + - [Packetizers and Depacketizers](#packetizers-and-depacketizers) - [Testing and Utilities](#testing-and-utilities) - [Example](#example) @@ -18,78 +22,89 @@ RTSP stream for transmitting MJPEG video data. ## RTSP Client -The `RtspClient` class provides an interface to an RTSP server. It is used to -send RTSP requests and receive RTSP responses. It also provides an interface -to the RTP and RTCP sessions that are created as a result of the RTSP -interactions. +The `RtspClient` class connects to an RTSP server and receives media streams +over RTP/UDP. It dispatches incoming RTP packets to codec-specific +depacketizers based on payload type. -The `RtspClient` currently only supports MJPEG streams, since the ESP32 does -not have a hardware decoder for H.264 or H.265. +For **backward compatibility**, setting the `on_jpeg_frame` callback +automatically creates an `MjpegDepacketizer` for MJPEG streams (payload type +26). For generic multi-track use, applications can use the `on_frame` +callback and inspect parsed SDP metadata through `tracks()`. -Additionally the client currently only supports UDP transport for RTP and RTCP -packets. TCP transport is not supported. +The client supports: -The user can register a callback function to be notified when new, complete JPEG -frames are received. The callback function is called with a pointer to the JPEG -frame. +* generic `on_frame(track_id, data)` callbacks for multi-track sessions +* parsed SDP track metadata including media type, payload type, codec name, + sample rate, channel count, and resolved control path +* automatic depacketizer selection for MJPEG, H.264, and generic payloads + discovered during `DESCRIBE` +* an `on_connection_lost` callback for reconnect / rediscovery workflows when + the RTSP control socket or RTP stream disappears after playback starts +* custom depacketizers via `add_depacketizer()` for additional payload types ## RTSP Server -The `RtspServer` class provides an implementation of an RTSP server. It is used -to receive RTSP requests and send RTSP responses. It is designed to allow the -user to send JPEG frames to the server, which will then send them to the client -over RTP/UDP. +The `RtspServer` class accepts RTSP connections and streams media over RTP/UDP. +It supports multiple media tracks, each with its own codec-specific packetizer, +SSRC, and sequence numbering. -The server currently only supports MJPEG streams, since the ESP32 does not have -a hardware encoder for H.264 or H.265. +For **backward compatibility**, calling `send_frame(const JpegFrame&)` lazily +creates a default MJPEG track. For other codecs, register tracks via +`add_track()` and send frames with `send_frame(track_id, data)`. -Additionally, the server currently only supports UDP transport for RTP and RTCP -packets. TCP transport is not supported. +The server also exposes helpers that are useful for embedded capture loops: + +* configurable accept, session-dispatch, and per-session control task stack + sizes +* `has_active_sessions()` to avoid capturing when no client is actively playing +* `get_capture_cooldown()` and `get_recommended_capture_period()` so an + application can slow capture when RTP backpressure is observed +* a legacy MJPEG `send_frame(std::span)` path that preserves the + older wire format for existing MJPEG-only users + +## Packetizers and Depacketizers + +The packetizer/depacketizer abstraction allows the server and client to support +multiple media codecs without changing the RTSP core: + +- **MJPEG** (`MjpegPacketizer` / `MjpegDepacketizer`) — RFC 2435 JPEG over RTP +- **H.264** (`H264Packetizer` / `H264Depacketizer`) — RFC 6184 with FU-A fragmentation +- **Generic** (`GenericPacketizer` / `GenericDepacketizer`) — MTU chunking for + audio or other pre-encoded payloads, with frame reconstruction based on RTP + marker / timestamp boundaries + +Custom packetizers can be created by subclassing `RtpPacketizer` or +`RtpDepacketizer`. ## Testing and Utilities -We have a few ways for testing the RTSP code: -- **ESPP Python Library**: - [`espp/lib`](https://github.com/esp-cpp/espp/tree/main/lib) contains the code - and scripts needed to build the ESPP Python library, which can be used to test - the RTSP server and client using Python scripts. There are existing RTSP - python tests within - [`espp/python`](https://github.com/esp-cpp/espp/tree/main/python) which use - this lib and provide various RTSP client / server functionality. -- **Pure Python**: The [`rtsp/python`](./python) folder contains pure Python - scripts that can be used to connect to and display the RTSP stream from an - RTSP server, using both mDNS and directly by IP address. - -### ESPP Python Library - -The [`espp/lib`](https://github.com/esp-cpp/espp/tree/main/lib) folder contains -the code and scripts needed to build the ESPP Python library, which can be used -to test the RTSP server and client functionality. This library provides a Python -interface to the RTSP server and client classes, allowing you to easily send and -receive RTSP requests and responses, as well as send and receive JPEG frames -over RTP/UDP. - -The [`espp/python`](https://github.com/esp-cpp/espp/tree/main/python) folder -contains the actual python scripts which run espp c++ code (via `pybind11`). See -the `README` in that directory for more information on how to use those scripts. - -### Pure Python - -To facilitate testing and debugging of the RTSP code, there are some -[python](./python) scripts in this folder which provide various mechanisms for -connecting to and displaying the RTSP stream from an RTSP server, using both -mDNS and directly by IP address. These scripts can be used to test the RTSP -server and client implementations. - -See [python/README.md](./python/README.md) for more information on how to use -these scripts. +There are several ways to exercise the RTSP stack: + +* **ESPP Python library**: build the host library from + [`lib`](https://github.com/esp-cpp/espp/tree/main/lib) with `./build.sh` to + expose the RTSP client/server classes to Python. +* **Python harness scripts**: + [`python`](https://github.com/esp-cpp/espp/tree/main/python) contains wrapper + and multi-track scripts for exercising legacy MJPEG flows, generic multi-track + flows, live microphone audio, and end-to-end host validation. +* **Embedded examples and downstream apps**: the component example plus + [`camera-streamer`](https://github.com/esp-cpp/camera-streamer) and + [`camera-display`](https://github.com/esp-cpp/camera-display) cover practical + server/client integrations. + +See [`python/README.md`](../../python/README.md) for more information on the +host-side scripts. ## Example -The [example](./example) shows the use of the `espp::RtspServer` and -`espp::RtspClient` classes provided by the `rtsp` component for performing -streaming of JPEG images (`espp::JpegFrame`) using the `MJPEG` format over the -Real Time Streaming Protocol (RTSP) / Real Time Protocol (RTP) packets. +The [example](./example) demonstrates several RTSP usage patterns selected via +menuconfig, including: + +* legacy MJPEG server + client behavior on the same device +* server-only MJPEG streaming +* client-only MJPEG reception from a remote RTSP server +* startup API tests for packetizers, depacketizers, and server/client setup +* multi-track streaming with MJPEG video plus generic audio For more complete example use, see the [camera-streamer](https://github.com/esp-cpp/camera-streamer) and diff --git a/components/rtsp/example/README.md b/components/rtsp/example/README.md index 99198bd06..63ec21671 100644 --- a/components/rtsp/example/README.md +++ b/components/rtsp/example/README.md @@ -1,9 +1,9 @@ # RTSP Example -This example shows the use of the `espp::RtspServer` and `espp::RtspClient` -classes provided by the `rtsp` component for performing streaming of JPEG images -(`espp::JpegFrame`) using the `MJPEG` format over the Real Time Streaming -Protocol (RTSP) / Real Time Protocol (RTP) packets. +This example demonstrates several `rtsp` component workflows, ranging from the +legacy MJPEG API to the newer multi-track server API. The example can run as a +server, client, or combined self-check depending on the selected menuconfig +mode. For more complete example use, see the [camera-streamer](https://github.com/esp-cpp/camera-streamer) and @@ -13,26 +13,37 @@ For more complete example use, see the ### Hardware Required -This example is designed to be run on an M5Stack ESP32 Timer Cam module (server) or a -ESP32-S3-Box (client). +Any supported ESP32 board with Wi-Fi can run this example. The bundled test +content uses an embedded JPEG image and synthetic audio data, so no camera, +microphone, or display hardware is required. ### Configure the project -``` +```sh idf.py menuconfig ``` -You need to configure the `WiFi` network in the `RTSP Example Configuration` -menuconfig and you can optionally configure the `RTSP Server Port` (default -8554). +The `RTSP Example Configuration` menu includes: + +* `Example Mode` + * `Legacy MJPEG (server + client)` - runs the original MJPEG server and + client on the same device + * `MJPEG Server only` - serves the embedded JPEG on `/mjpeg/1` + * `MJPEG Client only` - connects to a remote MJPEG RTSP server + * `Multi-track server (MJPEG + audio)` - serves MJPEG video on track 0 and + generic `L16/16000` mono audio on track 1 at `/stream` +* `Run API tests at startup` - exercises packetizers, depacketizers, and basic + client/server construction before networking starts +* Wi-Fi credentials and retry count +* `RTSP Server Port` +* `Remote RTSP server address` when client-only mode is selected ### Build and Flash -NOTE: this example is designed to be modified into client only or server-only operation depending on the hardware it is deployed to. - -Build the project and flash it to the board, then run monitor tool to view serial output: +Build the project and flash it to the board, then run monitor tool to view +serial output: -``` +```sh idf.py -p PORT flash monitor ``` @@ -40,19 +51,19 @@ idf.py -p PORT flash monitor (To exit the serial monitor, type ``Ctrl-]``.) -See the Getting Started Guide for full steps to configure and use ESP-IDF to build projects. +See the Getting Started Guide for full steps to configure and use ESP-IDF to +build projects. ## Example Output -Example showing the server (camera), the client (handheld display), and the python client, with both clients connected simultaneously: - -https://user-images.githubusercontent.com/213467/236601258-c334e1ba-5e18-4452-b48d-e792ec2ed4fb.mp4 - -Screenshot showing the server running for a very long time (all day, see timestamp in log > 33,000 seconds): -![all_day_test](https://user-images.githubusercontent.com/213467/236601320-0d9139d7-0333-4c63-b26f-da4078e141b7.png) +The serial log reports the selected mode, the local IP address after Wi-Fi +connects, and the RTSP URI for server modes. Typical output includes: -Screenshot showing the received framerate on the camera-display: -CleanShot 2023-05-06 at 10 27 21@2x +* `All API tests passed!` when startup API tests are enabled +* `RTSP URI: rtsp://:8554/mjpeg/1` in legacy and server-only modes +* `RTSP URI: rtsp://:8554/stream` in multi-track mode +* `Got JPEG frame: x` in client-only mode +* `Streaming MJPEG video (track 0) + audio (track 1)...` in multi-track mode -Screenshot of main code for camera-streamer output: -CleanShot 2023-05-06 at 10 29 31@2x +For host-side end-to-end testing, build the host library in `lib/` with +`./build.sh` and use the scripts in the repository-root `python/` directory. diff --git a/components/rtsp/example/main/Kconfig.projbuild b/components/rtsp/example/main/Kconfig.projbuild old mode 100644 new mode 100755 index 0d545fed7..eb9541633 --- a/components/rtsp/example/main/Kconfig.projbuild +++ b/components/rtsp/example/main/Kconfig.projbuild @@ -6,6 +6,55 @@ menu "RTSP Example Configuration" help The port number of the RTSP server. + choice RTSP_EXAMPLE_MODE + prompt "Example Mode" + default RTSP_EXAMPLE_MODE_LEGACY + help + Select which RTSP example mode to run. + + config RTSP_EXAMPLE_MODE_LEGACY + bool "Legacy MJPEG (server + client)" + help + Runs the original MJPEG server and client on the same device. + The server streams a JPEG image and the client receives it. + Uses the backward-compatible JpegFrame API. + + config RTSP_EXAMPLE_MODE_SERVER_ONLY + bool "MJPEG Server only" + help + Runs only the RTSP server streaming MJPEG frames. + Connect an external RTSP client (e.g. VLC) to view the stream. + + config RTSP_EXAMPLE_MODE_CLIENT_ONLY + bool "MJPEG Client only" + help + Runs only the RTSP client connecting to a remote RTSP server. + Requires an external RTSP server to be running. + + config RTSP_EXAMPLE_MODE_MULTITRACK + bool "Multi-track server (MJPEG + audio)" + help + Runs a multi-track RTSP server with an MJPEG video track and + a generic audio track, demonstrating the add_track() API + and generic send_frame(track_id, data) API. + + endchoice + + config RTSP_EXAMPLE_RUN_API_TESTS + bool "Run API tests at startup" + default y + help + Run offline API tests for packetizers, depacketizers, and + server/client construction before starting network operations. + This exercises the component APIs to catch breaking changes. + + config RTSP_CLIENT_SERVER_ADDRESS + string "Remote RTSP server address (client-only mode)" + default "192.168.1.100" + depends on RTSP_EXAMPLE_MODE_CLIENT_ONLY + help + IP address of the remote RTSP server to connect to in client-only mode. + config ESP_WIFI_SSID string "WiFi SSID" default "myssid" diff --git a/components/rtsp/example/main/rtsp_example.cpp b/components/rtsp/example/main/rtsp_example.cpp index e14f100a9..806c3441d 100644 --- a/components/rtsp/example/main/rtsp_example.cpp +++ b/components/rtsp/example/main/rtsp_example.cpp @@ -13,16 +13,472 @@ #include "rtsp_client.hpp" #include "rtsp_server.hpp" +// Packetizer/depacketizer headers +#include "generic_depacketizer.hpp" +#include "generic_packetizer.hpp" +#include "h264_depacketizer.hpp" +#include "h264_packetizer.hpp" +#include "mjpeg_depacketizer.hpp" +#include "mjpeg_packetizer.hpp" + #include "jpeg_image.hpp" using namespace std::chrono_literals; using namespace std::placeholders; +#if CONFIG_RTSP_EXAMPLE_RUN_API_TESTS + +/// Run offline API tests that exercise packetizer/depacketizer classes and +/// server/client construction. These tests run before networking and catch +/// breaking API changes at build/boot time. +static bool run_api_tests(espp::Logger &logger) { + bool all_passed = true; + int tests_run = 0; + int tests_passed = 0; + + auto check = [&](bool condition, const std::string &test_name) { + tests_run++; + if (condition) { + tests_passed++; + logger.info("[PASS] {}", test_name); + } else { + all_passed = false; + logger.error("[FAIL] {}", test_name); + } + }; + + logger.info("=== Running RTSP component API tests ==="); + + // ---- MjpegPacketizer tests ---- + { + //! [mjpeg_packetizer_test] + espp::MjpegPacketizer mjpeg_packer({.max_payload_size = 1000}); + check(mjpeg_packer.get_payload_type() == 26, "MjpegPacketizer: payload type is 26"); + check(mjpeg_packer.get_clock_rate() == 90000, "MjpegPacketizer: clock rate is 90000"); + check(!mjpeg_packer.get_sdp_media_line().empty(), "MjpegPacketizer: SDP media line non-empty"); + check(!mjpeg_packer.get_sdp_media_attributes().empty(), + "MjpegPacketizer: SDP attributes non-empty"); + + // Packetize the embedded JPEG image + std::span jpeg_data_span(reinterpret_cast(jpeg_data), + sizeof(jpeg_data)); + auto chunks = mjpeg_packer.packetize(jpeg_data_span); + check(!chunks.empty(), "MjpegPacketizer: produced chunks from JPEG data"); + check(chunks.back().marker, "MjpegPacketizer: last chunk has marker bit set"); + + // Verify chunk sizes respect MTU + bool all_chunks_within_mtu = true; + for (auto &c : chunks) { + if (c.data.size() > 1000 + 200) { // payload + MJPEG header overhead + all_chunks_within_mtu = false; + } + } + check(all_chunks_within_mtu, "MjpegPacketizer: chunk sizes within expected bounds"); + //! [mjpeg_packetizer_test] + } + + // ---- MjpegDepacketizer tests ---- + { + //! [mjpeg_depacketizer_test] + espp::MjpegDepacketizer mjpeg_depacker(espp::MjpegDepacketizer::Config{}); + + bool generic_frame_received = false; + bool jpeg_frame_received = false; + + mjpeg_depacker.set_frame_callback([&](std::vector &&data) { + generic_frame_received = true; + logger.info("MjpegDepacketizer: generic callback got {} bytes", data.size()); + }); + mjpeg_depacker.set_jpeg_frame_callback([&](std::shared_ptr frame) { + jpeg_frame_received = true; + logger.info("MjpegDepacketizer: JPEG callback got {}x{} frame", frame->get_width(), + frame->get_height()); + }); + + // Create RTP packets from the MJPEG packetizer output and feed them + // through the depacketizer to test the full round-trip + espp::MjpegPacketizer mjpeg_packer({.max_payload_size = 1000}); + std::span jpeg_data_span(reinterpret_cast(jpeg_data), + sizeof(jpeg_data)); + auto chunks = mjpeg_packer.packetize(jpeg_data_span); + + uint16_t seq = 0; + for (auto &chunk : chunks) { + espp::RtpPacket pkt(chunk.data.size()); + pkt.set_version(2); + pkt.set_payload_type(26); + pkt.set_sequence_number(seq++); + pkt.set_timestamp(0); + pkt.set_ssrc(12345); + pkt.set_marker(chunk.marker); + pkt.set_payload(std::span(chunk.data)); + pkt.serialize(); + mjpeg_depacker.process_packet(pkt); + } + + check(generic_frame_received, "MjpegDepacketizer: generic frame callback invoked"); + check(jpeg_frame_received, "MjpegDepacketizer: JPEG frame callback invoked"); + //! [mjpeg_depacketizer_test] + } + + // ---- H264Packetizer tests ---- + { + //! [h264_packetizer_test] + // Synthetic SPS and PPS (minimal valid-ish NAL units) + std::vector sps = {0x67, 0x42, 0xC0, 0x1E, 0xD9, 0x00, 0xA0, 0x47, 0xFE, 0xC8}; + std::vector pps = {0x68, 0xCE, 0x38, 0x80}; + + espp::H264Packetizer h264_packer({ + .max_payload_size = 1400, + .payload_type = 96, + .profile_level_id = "42C01E", + .packetization_mode = 1, + .sps = sps, + .pps = pps, + }); + + check(h264_packer.get_payload_type() == 96, "H264Packetizer: payload type is 96"); + check(h264_packer.get_clock_rate() == 90000, "H264Packetizer: clock rate is 90000"); + + auto sdp_attrs = h264_packer.get_sdp_media_attributes(); + check(sdp_attrs.find("H264/90000") != std::string::npos, + "H264Packetizer: SDP contains H264/90000"); + check(sdp_attrs.find("profile-level-id=42C01E") != std::string::npos, + "H264Packetizer: SDP contains profile-level-id"); + check(sdp_attrs.find("sprop-parameter-sets=") != std::string::npos, + "H264Packetizer: SDP contains SPS/PPS base64"); + + // Create a synthetic H.264 access unit in Annex B format: + // Start code + SPS + Start code + PPS + Start code + small IDR slice + std::vector annex_b_frame; + // SPS NAL + annex_b_frame.insert(annex_b_frame.end(), {0x00, 0x00, 0x00, 0x01}); + annex_b_frame.insert(annex_b_frame.end(), sps.begin(), sps.end()); + // PPS NAL + annex_b_frame.insert(annex_b_frame.end(), {0x00, 0x00, 0x00, 0x01}); + annex_b_frame.insert(annex_b_frame.end(), pps.begin(), pps.end()); + // IDR slice NAL (type 5) — fill with dummy data + annex_b_frame.insert(annex_b_frame.end(), {0x00, 0x00, 0x00, 0x01}); + annex_b_frame.push_back(0x65); // NAL header: type=5 (IDR) + for (int i = 0; i < 100; i++) { + annex_b_frame.push_back(static_cast(i & 0xFF)); + } + + auto chunks = + h264_packer.packetize(std::span(annex_b_frame.data(), annex_b_frame.size())); + check(!chunks.empty(), "H264Packetizer: produced chunks from Annex B data"); + check(chunks.back().marker, "H264Packetizer: last chunk has marker bit"); + + // With small NALs, all should be single NAL mode (no FU-A needed) + check(chunks.size() == 3, "H264Packetizer: 3 chunks for SPS+PPS+IDR"); + //! [h264_packetizer_test] + } + + // ---- H264Depacketizer tests ---- + { + //! [h264_depacketizer_test] + espp::H264Depacketizer h264_depacker(espp::H264Depacketizer::Config{}); + + bool frame_received = false; + size_t frame_size = 0; + + h264_depacker.set_frame_callback([&](std::vector &&data) { + frame_received = true; + frame_size = data.size(); + logger.info("H264Depacketizer: got frame of {} bytes", data.size()); + // Verify Annex B start codes are present + if (data.size() >= 4) { + bool has_start_code = + (data[0] == 0x00 && data[1] == 0x00 && data[2] == 0x00 && data[3] == 0x01); + logger.info("H264Depacketizer: Annex B start code present: {}", has_start_code); + } + }); + + // Create synthetic single NAL packets and feed them + std::vector sps = {0x67, 0x42, 0xC0, 0x1E, 0xD9}; + std::vector pps = {0x68, 0xCE, 0x38, 0x80}; + std::vector idr = {0x65, 0x01, 0x02, 0x03, 0x04}; + + auto make_rtp = [](const std::vector &payload, int pt, uint16_t seq, bool marker) { + espp::RtpPacket pkt(payload.size()); + pkt.set_version(2); + pkt.set_payload_type(pt); + pkt.set_sequence_number(seq); + pkt.set_timestamp(0); + pkt.set_ssrc(54321); + pkt.set_marker(marker); + pkt.set_payload(std::span(payload)); + pkt.serialize(); + return pkt; + }; + + h264_depacker.process_packet(make_rtp(sps, 96, 0, false)); + h264_depacker.process_packet(make_rtp(pps, 96, 1, false)); + h264_depacker.process_packet(make_rtp(idr, 96, 2, true)); // marker = end of AU + + check(frame_received, "H264Depacketizer: frame callback invoked"); + // Expected: 3 NALs with start codes = 3*(4) + 5+4+5 = 26 bytes + check(frame_size > 0, "H264Depacketizer: frame has data"); + //! [h264_depacketizer_test] + } + + // ---- H264 FU-A round-trip test ---- + { + //! [h264_fua_roundtrip_test] + // Create a large NAL that requires FU-A fragmentation + std::vector large_nal; + large_nal.push_back(0x65); // IDR slice (type 5) + for (int i = 0; i < 3000; i++) { + large_nal.push_back(static_cast(i & 0xFF)); + } + + // Annex B frame with one large NAL + std::vector annex_b; + annex_b.insert(annex_b.end(), {0x00, 0x00, 0x00, 0x01}); + annex_b.insert(annex_b.end(), large_nal.begin(), large_nal.end()); + + espp::H264Packetizer packer({.max_payload_size = 1000, + .payload_type = 96, + .profile_level_id = {}, + .packetization_mode = 1, + .sps = {}, + .pps = {}}); + auto chunks = packer.packetize(std::span(annex_b)); + check(chunks.size() > 1, "H264 FU-A: large NAL produces multiple chunks"); + + // Feed chunks through depacketizer + espp::H264Depacketizer depacker(espp::H264Depacketizer::Config{}); + bool fua_frame_received = false; + std::vector received_frame; + depacker.set_frame_callback([&](std::vector &&data) { + fua_frame_received = true; + received_frame = std::move(data); + }); + + uint16_t seq = 0; + for (auto &chunk : chunks) { + espp::RtpPacket pkt(chunk.data.size()); + pkt.set_version(2); + pkt.set_payload_type(96); + pkt.set_sequence_number(seq++); + pkt.set_timestamp(0); + pkt.set_ssrc(99999); + pkt.set_marker(chunk.marker); + pkt.set_payload(std::span(chunk.data)); + pkt.serialize(); + depacker.process_packet(pkt); + } + + check(fua_frame_received, "H264 FU-A: round-trip frame received"); + // Output should be: start code (4) + original NAL (3001) + check(received_frame.size() == 4 + large_nal.size(), + "H264 FU-A: round-trip frame size matches"); + // Verify the NAL content matches (skip start code) + if (received_frame.size() >= 4 + large_nal.size()) { + bool content_matches = + std::equal(large_nal.begin(), large_nal.end(), received_frame.begin() + 4); + check(content_matches, "H264 FU-A: round-trip content matches"); + } + //! [h264_fua_roundtrip_test] + } + + // ---- GenericPacketizer tests ---- + { + //! [generic_packetizer_test] + espp::GenericPacketizer generic_packer({ + .max_payload_size = 500, + .payload_type = 97, + .clock_rate = 48000, + .encoding_name = "opus", + .channels = 2, + .fmtp = {}, + .media_type = espp::MediaType::AUDIO, + }); + + check(generic_packer.get_payload_type() == 97, "GenericPacketizer: payload type is 97"); + check(generic_packer.get_clock_rate() == 48000, "GenericPacketizer: clock rate is 48000"); + + auto sdp_line = generic_packer.get_sdp_media_line(); + check(sdp_line.find("m=audio") != std::string::npos, + "GenericPacketizer: SDP media line is audio"); + + auto sdp_attrs = generic_packer.get_sdp_media_attributes(); + check(sdp_attrs.find("opus/48000/2") != std::string::npos, + "GenericPacketizer: SDP has encoding/rate/channels"); + + // Packetize 1200 bytes of synthetic audio + std::vector audio_data(1200, 0xAB); + auto chunks = + generic_packer.packetize(std::span(audio_data.data(), audio_data.size())); + check(chunks.size() == 3, "GenericPacketizer: 1200 bytes @ 500 MTU = 3 chunks"); + check(chunks.back().marker, "GenericPacketizer: last chunk has marker"); + check(!chunks.front().marker, "GenericPacketizer: first chunk has no marker"); + //! [generic_packetizer_test] + } + + // ---- GenericDepacketizer round-trip test ---- + { + //! [generic_depacketizer_test] + espp::GenericDepacketizer generic_depacker(espp::GenericDepacketizer::Config{}); + + bool audio_frame_received = false; + size_t audio_frame_size = 0; + + generic_depacker.set_frame_callback([&](std::vector &&data) { + audio_frame_received = true; + audio_frame_size = data.size(); + }); + + // Packetize and depacketize audio data + espp::GenericPacketizer generic_packer({.max_payload_size = 500, + .payload_type = 97, + .clock_rate = 48000, + .encoding_name = "L16", + .channels = 1, + .fmtp = {}, + .media_type = espp::MediaType::AUDIO}); + std::vector audio_data(1200, 0xCD); + auto chunks = + generic_packer.packetize(std::span(audio_data.data(), audio_data.size())); + + uint16_t seq = 0; + for (auto &chunk : chunks) { + espp::RtpPacket pkt(chunk.data.size()); + pkt.set_version(2); + pkt.set_payload_type(97); + pkt.set_sequence_number(seq++); + pkt.set_timestamp(1000); + pkt.set_ssrc(77777); + pkt.set_marker(chunk.marker); + pkt.set_payload(std::span(chunk.data)); + pkt.serialize(); + generic_depacker.process_packet(pkt); + } + + check(audio_frame_received, "GenericDepacketizer: frame callback invoked"); + check(audio_frame_size == 1200, "GenericDepacketizer: round-trip frame size matches"); + //! [generic_depacketizer_test] + } + + // ---- RtspServer construction and track API tests ---- + { + //! [rtsp_server_api_test] + espp::RtspServer server({ + .server_address = "127.0.0.1", + .port = 8554, + .path = "/test", + .log_level = espp::Logger::Verbosity::WARN, + }); + + // Test add_track with MJPEG packetizer + auto mjpeg_packer = std::make_shared( + espp::MjpegPacketizer::Config{.max_payload_size = 1400}); + server.add_track(espp::RtspServer::TrackConfig{.track_id = 0, .packetizer = mjpeg_packer}); + + // Test add_track with H264 packetizer + auto h264_packer = std::make_shared(espp::H264Packetizer::Config{ + .max_payload_size = 1400, + .payload_type = 96, + .profile_level_id = "42C01E", + .sps = {0x67, 0x42, 0xC0, 0x1E}, + .pps = {0x68, 0xCE, 0x38, 0x80}, + }); + server.add_track(espp::RtspServer::TrackConfig{.track_id = 1, .packetizer = h264_packer}); + + // Test add_track with generic audio packetizer + auto audio_packer = std::make_shared(espp::GenericPacketizer::Config{ + .max_payload_size = 1400, + .payload_type = 97, + .clock_rate = 48000, + .encoding_name = "opus", + .channels = 2, + .fmtp = {}, + .media_type = espp::MediaType::AUDIO, + }); + server.add_track(espp::RtspServer::TrackConfig{.track_id = 2, .packetizer = audio_packer}); + + check(true, "RtspServer: construction with multi-track succeeded"); + + // Test set_session_log_level + server.set_session_log_level(espp::Logger::Verbosity::DEBUG); + check(true, "RtspServer: set_session_log_level succeeded"); + //! [rtsp_server_api_test] + } + + // ---- RtspClient construction and depacketizer API tests ---- + { + //! [rtsp_client_api_test] + // Test backward-compatible construction with on_jpeg_frame + bool jpeg_cb_set = false; + espp::RtspClient client({ + .server_address = "127.0.0.1", + .rtsp_port = 8554, + .path = "/test", + .on_jpeg_frame = [&jpeg_cb_set](std::shared_ptr) { jpeg_cb_set = true; }, + .log_level = espp::Logger::Verbosity::WARN, + }); + + check(true, "RtspClient: construction with on_jpeg_frame succeeded"); + + // Test add_depacketizer for H264 + auto h264_depacker = std::make_shared(espp::H264Depacketizer::Config{}); + h264_depacker->set_frame_callback([](std::vector &&data) { + // would receive Annex B H.264 access units + }); + client.add_depacketizer(96, h264_depacker); + + // Test add_depacketizer for generic audio + auto audio_depacker = + std::make_shared(espp::GenericDepacketizer::Config{}); + audio_depacker->set_frame_callback([](std::vector &&data) { + // would receive reassembled audio frames + }); + client.add_depacketizer(97, audio_depacker); + + check(true, "RtspClient: add_depacketizer for H264 and audio succeeded"); + //! [rtsp_client_api_test] + } + + // ---- Legacy JpegFrame send_frame backward-compat test ---- + { + //! [legacy_send_frame_test] + espp::RtspServer server({ + .server_address = "127.0.0.1", + .port = 8556, + .path = "/mjpeg/1", + .max_data_size = 1000, + .log_level = espp::Logger::Verbosity::WARN, + }); + + // send_frame(JpegFrame) should lazily create MJPEG track 0 + std::span jpeg_data_span(reinterpret_cast(jpeg_data), + sizeof(jpeg_data)); + espp::JpegFrame frame(jpeg_data_span); + server.send_frame(frame); + + check(true, "Legacy send_frame(JpegFrame): lazy MJPEG track creation succeeded"); + //! [legacy_send_frame_test] + } + + logger.info("=== API tests complete: {}/{} passed ===", tests_passed, tests_run); + return all_passed; +} + +#endif // CONFIG_RTSP_EXAMPLE_RUN_API_TESTS + extern "C" void app_main(void) { espp::Logger logger({.tag = "main", .level = espp::Logger::Verbosity::INFO}); logger.info("Starting RTSP example!"); +#if CONFIG_RTSP_EXAMPLE_RUN_API_TESTS + if (!run_api_tests(logger)) { + logger.error("API tests FAILED — check output above"); + } else { + logger.info("All API tests passed!"); + } +#endif + std::string ip_address; espp::WifiSta wifi_sta({.ssid = CONFIG_ESP_WIFI_SSID, .password = CONFIG_ESP_WIFI_PASSWORD, @@ -38,16 +494,20 @@ extern "C" void app_main(void) { std::this_thread::sleep_for(100ms); } - //! [rtsp_server_example] - const int server_port = CONFIG_RTSP_SERVER_PORT; - const std::string server_uri = fmt::format("rtsp://{}:{}/mjpeg/1", ip_address, server_port); + // -------------------------------------------------------------------------- + // Mode: Legacy MJPEG (server + client on the same device) + // -------------------------------------------------------------------------- +#if CONFIG_RTSP_EXAMPLE_MODE_LEGACY - logger.info("Starting RTSP Server on port {}", server_port); + //! [rtsp_server_example] + const std::string server_uri = + fmt::format("rtsp://{}:{}/mjpeg/1", ip_address, CONFIG_RTSP_SERVER_PORT); + logger.info("Starting RTSP Server on port {}", CONFIG_RTSP_SERVER_PORT); logger.info("RTSP URI: {}", server_uri); espp::RtspServer rtsp_server({ .server_address = ip_address, - .port = server_port, + .port = CONFIG_RTSP_SERVER_PORT, .path = "/mjpeg/1", .log_level = espp::Logger::Verbosity::INFO, }); @@ -64,7 +524,7 @@ extern "C" void app_main(void) { //! [rtsp_client_example] espp::RtspClient rtsp_client({ - .server_address = ip_address, // string of the form {}.{}.{}.{} + .server_address = ip_address, .rtsp_port = CONFIG_RTSP_SERVER_PORT, .path = "/mjpeg/1", .on_jpeg_frame = @@ -78,7 +538,6 @@ extern "C" void app_main(void) { std::error_code ec; do { - // clear the error code ec.clear(); rtsp_client.connect(ec); if (ec) { @@ -104,9 +563,153 @@ extern "C" void app_main(void) { } //! [rtsp_client_example] - // now that both client and server are up, send frames forever while (true) { rtsp_server.send_frame(jpeg_frame); std::this_thread::sleep_for(100ms); } + + // -------------------------------------------------------------------------- + // Mode: Server only (MJPEG) + // -------------------------------------------------------------------------- +#elif CONFIG_RTSP_EXAMPLE_MODE_SERVER_ONLY + + const std::string server_uri = + fmt::format("rtsp://{}:{}/mjpeg/1", ip_address, CONFIG_RTSP_SERVER_PORT); + logger.info("Starting RTSP Server (server-only mode) on port {}", CONFIG_RTSP_SERVER_PORT); + logger.info("RTSP URI: {}", server_uri); + + espp::RtspServer rtsp_server({ + .server_address = ip_address, + .port = CONFIG_RTSP_SERVER_PORT, + .path = "/mjpeg/1", + .log_level = espp::Logger::Verbosity::INFO, + }); + rtsp_server.start(); + + std::span frame_data(reinterpret_cast(jpeg_data), + sizeof(jpeg_data)); + espp::JpegFrame jpeg_frame(frame_data); + + logger.info("Streaming JPEG frame ({}x{}), connect with an RTSP client...", + jpeg_frame.get_width(), jpeg_frame.get_height()); + + while (true) { + rtsp_server.send_frame(jpeg_frame); + std::this_thread::sleep_for(100ms); + } + + // -------------------------------------------------------------------------- + // Mode: Client only + // -------------------------------------------------------------------------- +#elif CONFIG_RTSP_EXAMPLE_MODE_CLIENT_ONLY + + logger.info("Starting RTSP Client (client-only mode)"); + logger.info("Connecting to {}:{}", CONFIG_RTSP_CLIENT_SERVER_ADDRESS, CONFIG_RTSP_SERVER_PORT); + + espp::RtspClient rtsp_client({ + .server_address = CONFIG_RTSP_CLIENT_SERVER_ADDRESS, + .rtsp_port = CONFIG_RTSP_SERVER_PORT, + .path = "/mjpeg/1", + .on_jpeg_frame = + [&logger](std::shared_ptr jpeg_frame) { + logger.info("Got JPEG frame: {}x{}", jpeg_frame->get_width(), jpeg_frame->get_height()); + }, + .log_level = espp::Logger::Verbosity::INFO, + }); + + std::error_code ec; + + do { + ec.clear(); + rtsp_client.connect(ec); + if (ec) { + logger.error("Error connecting: {}", ec.message()); + logger.info("Retrying in 1s..."); + std::this_thread::sleep_for(1s); + } + } while (ec); + + rtsp_client.describe(ec); + if (ec) { + logger.error("Error describing: {}", ec.message()); + } + + rtsp_client.setup(ec); + if (ec) { + logger.error("Error setting up: {}", ec.message()); + } + + rtsp_client.play(ec); + if (ec) { + logger.error("Error playing: {}", ec.message()); + } + + logger.info("Client is playing, receiving frames..."); + while (true) { + std::this_thread::sleep_for(1s); + } + + // -------------------------------------------------------------------------- + // Mode: Multi-track server (MJPEG video + generic audio) + // -------------------------------------------------------------------------- +#elif CONFIG_RTSP_EXAMPLE_MODE_MULTITRACK + + //! [rtsp_server_multitrack_example] + const std::string server_uri = + fmt::format("rtsp://{}:{}/stream", ip_address, CONFIG_RTSP_SERVER_PORT); + logger.info("Starting multi-track RTSP Server on port {}", CONFIG_RTSP_SERVER_PORT); + logger.info("RTSP URI: {}", server_uri); + + espp::RtspServer rtsp_server({ + .server_address = ip_address, + .port = CONFIG_RTSP_SERVER_PORT, + .path = "/stream", + .log_level = espp::Logger::Verbosity::INFO, + }); + + // Track 0: MJPEG video + auto mjpeg_packetizer = std::make_shared( + espp::MjpegPacketizer::Config{.max_payload_size = 1400}); + rtsp_server.add_track( + espp::RtspServer::TrackConfig{.track_id = 0, .packetizer = mjpeg_packetizer}); + + // Track 1: Generic audio (e.g., 16-bit PCM at 16kHz) + auto audio_packetizer = std::make_shared(espp::GenericPacketizer::Config{ + .max_payload_size = 1400, + .payload_type = 97, + .clock_rate = 16000, + .encoding_name = "L16", + .channels = 1, + .media_type = espp::MediaType::AUDIO, + }); + rtsp_server.add_track( + espp::RtspServer::TrackConfig{.track_id = 1, .packetizer = audio_packetizer}); + + rtsp_server.start(); + + // Prepare video frame + std::span jpeg_raw(reinterpret_cast(jpeg_data), + sizeof(jpeg_data)); + + // Prepare synthetic audio frame (320 samples of 16-bit PCM = 640 bytes = 20ms @ 16kHz) + std::vector audio_frame(640, 0x00); + for (size_t i = 0; i < audio_frame.size(); i += 2) { + // Generate a simple 440Hz sine-like pattern for audibility + uint16_t sample = static_cast(128 * (i % 73)); // pseudo pattern + audio_frame[i] = sample & 0xFF; + audio_frame[i + 1] = (sample >> 8) & 0xFF; + } + + logger.info("Streaming MJPEG video (track 0) + audio (track 1)..."); + + while (true) { + // Send video frame using generic send_frame(track_id, data) + rtsp_server.send_frame(0, jpeg_raw); + // Send audio frame + rtsp_server.send_frame(1, std::span(audio_frame.data(), audio_frame.size())); + std::this_thread::sleep_for(100ms); + } + //! [rtsp_server_multitrack_example] + +#endif // CONFIG_RTSP_EXAMPLE_MODE_* } diff --git a/components/rtsp/idf_component.yml b/components/rtsp/idf_component.yml old mode 100644 new mode 100755 index 507995fda..b3ca0a91c --- a/components/rtsp/idf_component.yml +++ b/components/rtsp/idf_component.yml @@ -12,10 +12,13 @@ tags: - cpp - Component - RTSP + - RTP - TCP - UDP - Video + - Audio - MJPEG + - H264 - Client - Server dependencies: diff --git a/components/rtsp/include/generic_depacketizer.hpp b/components/rtsp/include/generic_depacketizer.hpp new file mode 100644 index 000000000..c2cdc8126 --- /dev/null +++ b/components/rtsp/include/generic_depacketizer.hpp @@ -0,0 +1,51 @@ +#pragma once + +#include +#include + +#include "rtp_depacketizer.hpp" + +namespace espp { + +/// A generic RTP depacketizer that reassembles media frames from incoming RTP +/// packets. It accumulates payload data until a packet with the marker bit set +/// is received, then delivers the complete frame via the frame callback. If a +/// packet arrives with a different RTP timestamp than the current accumulation +/// buffer, the old buffer is discarded and a new one is started. +/// +/// This is suitable for audio codecs (PCM, G.711, Opus, etc.) or any payload +/// format that uses simple marker-based framing. +/// +/// \section generic_depacketizer_ex1 Example +/// \snippet generic_depacketizer_example.cpp generic_depacketizer example +class GenericDepacketizer : public RtpDepacketizer { +public: + /// Configuration for GenericDepacketizer. + struct Config { + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct a GenericDepacketizer. + /// @param config The configuration for this depacketizer. + explicit GenericDepacketizer(const Config &config); + + /// Destructor. + ~GenericDepacketizer() override = default; + + /// Process an incoming RTP packet. + /// Payload data is accumulated until a packet with the marker bit set is + /// received. At that point the assembled frame is delivered via the frame + /// callback and the buffer is reset. + /// @param packet The RTP packet to process. + void process_packet(const RtpPacket &packet) override; + +protected: + /// Reset the accumulation buffer. + void reset_buffer(); + + std::vector buffer_; ///< Accumulation buffer for frame data + int current_timestamp_{-1}; ///< RTP timestamp of the current buffer + bool has_timestamp_{false}; ///< Whether the buffer has a valid timestamp +}; + +} // namespace espp diff --git a/components/rtsp/include/generic_packetizer.hpp b/components/rtsp/include/generic_packetizer.hpp new file mode 100644 index 000000000..1ba7cee7e --- /dev/null +++ b/components/rtsp/include/generic_packetizer.hpp @@ -0,0 +1,70 @@ +#pragma once + +#include +#include + +#include "rtp_packetizer.hpp" + +namespace espp { + +/// A generic RTP packetizer suitable for audio codecs (PCM, G.711, Opus, etc.) +/// or any pre-formatted data that simply needs MTU-based chunking. It splits +/// frame data into chunks of at most max_payload_size bytes and marks the last +/// chunk with the RTP marker bit. +/// +/// \section generic_packetizer_ex1 Example +/// \snippet generic_packetizer_example.cpp generic_packetizer example +class GenericPacketizer : public RtpPacketizer { +public: + /// Configuration for GenericPacketizer. + struct Config { + size_t max_payload_size{1400}; ///< Maximum payload bytes per RTP packet + int payload_type{96}; ///< RTP payload type number + uint32_t clock_rate{48000}; ///< Clock rate in Hz for RTP timestamps + std::string encoding_name{"L16"}; ///< Encoding name for SDP rtpmap line + int channels{1}; ///< Number of audio channels + std::string fmtp; ///< Optional format parameters for SDP fmtp line + espp::MediaType media_type{espp::MediaType::AUDIO}; ///< Media type for the SDP m= line + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct a GenericPacketizer. + /// @param config The configuration for this packetizer. + explicit GenericPacketizer(const Config &config); + + /// Destructor. + ~GenericPacketizer() override = default; + + /// Split frame data into RTP payload chunks of at most max_payload_size. + /// The last (or only) chunk has its marker flag set. + /// @param frame_data The raw frame bytes to packetize. + /// @return A vector of RtpPayloadChunk ready to be wrapped in RTP packets. + std::vector packetize(std::span frame_data) override; + + /// Get the RTP payload type number. + /// @return The configured RTP payload type. + int get_payload_type() const override; + + /// Get the RTP clock rate. + /// @return The configured clock rate in Hz. + uint32_t get_clock_rate() const override; + + /// Generate the SDP media-level attribute lines for this codec. + /// Produces an a=rtpmap line and, if fmtp is non-empty, an a=fmtp line. + /// @return A string containing the SDP a= lines. + std::string get_sdp_media_attributes() const override; + + /// Generate the SDP m= line for this codec. + /// @return A string such as "m=audio 0 RTP/AVP 96". + std::string get_sdp_media_line() const override; + +protected: + int payload_type_; ///< RTP payload type number + uint32_t clock_rate_; ///< Clock rate in Hz + std::string encoding_name_; ///< Encoding name for SDP + int channels_; ///< Number of audio channels + std::string fmtp_; ///< Optional format parameters + espp::MediaType media_type_; ///< Media type (audio/video) +}; + +} // namespace espp diff --git a/components/rtsp/include/h264_depacketizer.hpp b/components/rtsp/include/h264_depacketizer.hpp new file mode 100755 index 000000000..6fc553adf --- /dev/null +++ b/components/rtsp/include/h264_depacketizer.hpp @@ -0,0 +1,63 @@ +#pragma once + +#include +#include +#include +#include + +#include "rtp_depacketizer.hpp" +#include "rtp_packet.hpp" + +namespace espp { + +/// @brief RTP depacketizer for H.264 video per RFC 6184. +/// +/// Reassembles H.264 access units from incoming RTP packets. Supports: +/// - **Single NAL unit** packets (NAL type 1–23) +/// - **STAP-A** aggregation packets (NAL type 24) +/// - **FU-A** fragmentation packets (NAL type 28) +/// +/// When the RTP marker bit is set, the accumulated NAL units are delivered +/// as one Annex B byte-stream (each NAL prefixed with 0x00 0x00 0x00 0x01) +/// via the frame callback set with set_frame_callback(). +/// +/// \section h264_depacketizer_ex1 Example +/// \snippet h264_depacketizer_example.cpp h264_depacketizer example +class H264Depacketizer : public RtpDepacketizer { +public: + /// Configuration for the H264Depacketizer. + struct Config { + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct an H264Depacketizer. + /// @param config The configuration for the depacketizer. + explicit H264Depacketizer(const Config &config); + + /// Destructor. + ~H264Depacketizer() override = default; + + /// Process an incoming RTP packet containing H.264 payload. + /// + /// Handles single NAL, STAP-A, and FU-A packet types. NAL units are + /// buffered until the RTP marker bit indicates the end of an access unit, + /// at which point the complete Annex B frame is delivered via the callback. + /// + /// @param packet The RTP packet to process. + void process_packet(const RtpPacket &packet) override; + +protected: + /// Deliver the buffered NAL units as a single Annex B frame. + void deliver_frame(); + + /// Buffer of accumulated NAL units for the current access unit. + std::vector> nal_buffer_; + + /// In-progress NAL unit being assembled from FU-A fragments. + std::vector fua_buffer_; + + /// Whether we are currently assembling FU-A fragments. + bool fua_in_progress_{false}; +}; + +} // namespace espp diff --git a/components/rtsp/include/h264_packetizer.hpp b/components/rtsp/include/h264_packetizer.hpp new file mode 100644 index 000000000..5d277b04d --- /dev/null +++ b/components/rtsp/include/h264_packetizer.hpp @@ -0,0 +1,100 @@ +#pragma once + +#include +#include +#include +#include + +#include "rtp_packetizer.hpp" + +namespace espp { + +/// @brief RTP packetizer for H.264 video per RFC 6184. +/// +/// Accepts H.264 access units in Annex B byte-stream format (NAL units +/// separated by 0x00000001 or 0x000001 start codes) and produces a sequence +/// of RTP payload chunks suitable for transmission. +/// +/// Supports two NAL-unit packetization strategies: +/// - **Single NAL unit mode** — NAL fits within max_payload_size. +/// - **FU-A fragmentation** — NAL exceeds max_payload_size (packetization_mode >= 1). +/// +/// @note This class does not manage RTP headers (sequence numbers, timestamps, +/// SSRC). The caller wraps each returned chunk into an RtpPacket. +/// +/// \section h264_packetizer_ex1 Example +/// \snippet h264_packetizer_example.cpp h264_packetizer example +class H264Packetizer : public RtpPacketizer { +public: + /// Configuration for the H264Packetizer. + struct Config { + size_t max_payload_size{1400}; ///< Maximum payload bytes per RTP packet + int payload_type{96}; ///< Dynamic RTP payload type (typically 96–127). + std::string profile_level_id; ///< H.264 profile-level-id hex string, e.g. "42C01E". + int packetization_mode{1}; ///< 0 = single NAL only, 1 = non-interleaved (FU-A allowed). + std::vector sps; ///< Sequence Parameter Set raw bytes (without start code). + std::vector pps; ///< Picture Parameter Set raw bytes (without start code). + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct an H264Packetizer. + /// @param config The configuration for the packetizer. + explicit H264Packetizer(const Config &config); + + /// Destructor. + ~H264Packetizer() override = default; + + /// Packetize a complete H.264 access unit (Annex B format). + /// + /// The input may contain multiple NAL units separated by 3-byte or 4-byte + /// start codes. Each NAL is individually packetized (single NAL or FU-A). + /// The marker bit is set on the last chunk of the last NAL unit in the + /// access unit. + /// + /// @param frame_data Raw Annex B byte-stream of one access unit. + /// @return Vector of RTP payload chunks ready for transmission. + std::vector packetize(std::span frame_data) override; + + /// Packetize a single pre-parsed NAL unit (no start code prefix). + /// + /// @param nal_data The raw NAL unit bytes (including NAL header byte). + /// @param is_last_nal If true, the marker bit is set on the last chunk. + /// @return Vector of RTP payload chunks for this NAL. + std::vector packetize_nal(std::span nal_data, + bool is_last_nal = true); + + /// Update the SPS and PPS used for SDP generation. + /// @param sps Sequence Parameter Set raw bytes. + /// @param pps Picture Parameter Set raw bytes. + void set_sps_pps(std::span sps, std::span pps); + + /// Get the RTP payload type. + /// @return The dynamic payload type configured for H.264. + int get_payload_type() const override; + + /// Get the RTP clock rate for H.264 video. + /// @return 90000 (fixed for H.264). + uint32_t get_clock_rate() const override; + + /// Get the SDP attribute lines for H.264. + /// @return SDP a= lines (rtpmap and fmtp) without trailing CRLF. + std::string get_sdp_media_attributes() const override; + + /// Get the SDP m= media line for H.264. + /// @return SDP m= line without trailing CRLF. + std::string get_sdp_media_line() const override; + +protected: + /// Parse Annex B byte-stream into individual NAL units. + /// @param data The Annex B data to parse. + /// @return Vector of spans, each pointing to a NAL unit (without start code). + static std::vector> parse_annex_b(std::span data); + + int payload_type_; + std::string profile_level_id_; + int packetization_mode_; + std::vector sps_; + std::vector pps_; +}; + +} // namespace espp diff --git a/components/rtsp/include/jpeg_header.hpp b/components/rtsp/include/jpeg_header.hpp old mode 100644 new mode 100755 index 895f5d938..133d6df77 --- a/components/rtsp/include/jpeg_header.hpp +++ b/components/rtsp/include/jpeg_header.hpp @@ -1,10 +1,13 @@ #pragma once +#include #include #include #include #include +#include "format.hpp" + namespace espp { /// A class to generate a JPEG header for a given image size and quantization tables. /// The header is generated once and then cached for future use. @@ -50,6 +53,9 @@ class JpegHeader { /// @note This is the size of the serialized JPEG header, not the image size. size_t size() const { return data_.size(); } + /// Returns whether this header parsed or serialized successfully. + bool is_valid() const { return valid_; } + /// Get the JPEG header data. /// @return The JPEG header data. std::span get_data() const { @@ -71,6 +77,10 @@ class JpegHeader { static constexpr int SOF0_SIZE = 19; static constexpr int DQT_HEADER_SIZE = 5; + bool has_bytes(size_t offset, size_t count) const { + return offset <= data_.size() && count <= data_.size() - offset; + } + // JFIF APP0 Marker for version 1.2 with 72 DPI and no thumbnail static constexpr uint8_t JFIF_APP0_DATA[] = { 0xFF, 0xE0, // APP0 marker @@ -616,9 +626,15 @@ class JpegHeader { // add the SOS marker memcpy(data_.data() + offset, SOS, sizeof(SOS)); // offset += sizeof(SOS); + valid_ = true; } - uint16_t get_marker(size_t offset) const { return (data_[offset] << 8) | data_[offset + 1]; } + uint16_t get_marker(size_t offset) const { + if (!has_bytes(offset, 2)) { + return 0; + } + return (data_[offset] << 8) | data_[offset + 1]; + } static constexpr uint16_t SOI_MARKER = 0xFFD8; static constexpr uint16_t APP0_MARKER = 0xFFE0; @@ -630,10 +646,11 @@ class JpegHeader { void parse() { constexpr size_t debug_data_size = 50; + valid_ = false; // parse the jpeg header from the data_ vector int offset = 0; // check the SOI marker - if (get_marker(offset) != SOI_MARKER) { + if (!has_bytes(offset, 2) || get_marker(offset) != SOI_MARKER) { std::span debug_data(data_.data() + offset, std::min(data_.size(), debug_data_size)); fmt::print("Invalid SOI marker: {::02x}\n", debug_data); @@ -643,6 +660,10 @@ class JpegHeader { // check the JFIF APP0 marker if (get_marker(offset) == APP0_MARKER) { + if (!has_bytes(offset, JFIF_APP0_SIZE)) { + fmt::print("Incomplete JFIF APP0 marker\n"); + return; + } if (memcmp(data_.data() + offset, JFIF_APP0_DATA, JFIF_APP0_CMP_LEN) != 0) { std::span debug_data(data_.data() + offset, std::min(data_.size(), debug_data_size)); @@ -655,12 +676,20 @@ class JpegHeader { size_t num_huffman_tables = 0; size_t num_dqt_tables = 0; bool found_sos = false; - while (!found_sos && offset < data_.size()) { + while (!found_sos && has_bytes(offset, 2)) { uint16_t marker = (data_[offset] << 8) | data_[offset + 1]; offset += 2; if (marker == DQT_MARKER) { + if (!has_bytes(offset, 3)) { + fmt::print("Incomplete DQT marker\n"); + return; + } // next two bytes are length, so read them out and then skip ahead size_t dqt_length = (data_[offset] << 8) | data_[offset + 1]; + if (dqt_length < 3 || !has_bytes(offset, dqt_length)) { + fmt::print("Invalid DQT marker length: {}\n", dqt_length); + return; + } size_t dqt_table_id = data_[offset + 2]; size_t dqt_table_length = dqt_length - 3; // length includes the marker and table ID const char *table_ptr = (const char *)data_.data() + offset + 3; @@ -679,21 +708,46 @@ class JpegHeader { offset += dqt_length; num_dqt_tables++; } else if (marker == DHT_MARKER) { + if (!has_bytes(offset, 2)) { + fmt::print("Incomplete DHT marker\n"); + return; + } size_t huffman_tables_length = (data_[offset] << 8) | data_[offset + 1]; + if (huffman_tables_length < 2 || !has_bytes(offset, huffman_tables_length)) { + fmt::print("Invalid DHT marker length: {}\n", huffman_tables_length); + return; + } offset += huffman_tables_length; // skip the length and the table num_huffman_tables++; } else if (marker == SOF0_MARKER) { + if (!has_bytes(offset, 7)) { + fmt::print("Incomplete SOF0 marker\n"); + return; + } size_t sof0_length = (data_[offset] << 8) | data_[offset + 1]; + if (sof0_length < 7 || !has_bytes(offset, sof0_length)) { + fmt::print("Invalid SOF0 marker length: {}\n", sof0_length); + return; + } [[maybe_unused]] uint8_t precision = data_[offset + 2]; height_ = (data_[offset + 3] << 8) | data_[offset + 4]; width_ = (data_[offset + 5] << 8) | data_[offset + 6]; offset += sof0_length; } else if (marker == SOS_MARKER) { + if (!has_bytes(offset, 2)) { + fmt::print("Incomplete SOS marker\n"); + return; + } size_t sos_length = (data_[offset] << 8) | data_[offset + 1]; + if (sos_length < 2 || !has_bytes(offset, sos_length)) { + fmt::print("Invalid SOS marker length: {}\n", sos_length); + return; + } offset += sos_length; found_sos = true; } else { fmt::print("Unknown marker: {:04x}\n", marker); + return; } } @@ -705,8 +759,14 @@ class JpegHeader { fmt::print("Did not find SOS marker\n"); return; } + if (width_ <= 0 || height_ <= 0 || q0_table_.empty() || q1_table_.empty()) { + fmt::print("Incomplete JPEG header: width={}, height={}, q0={}, q1={}\n", width_, height_, + q0_table_.size(), q1_table_.size()); + return; + } data_.resize(offset); + valid_ = true; } int width_{0}; @@ -715,5 +775,6 @@ class JpegHeader { std::string_view q1_table_; std::vector data_; + bool valid_{false}; }; } // namespace espp diff --git a/components/rtsp/include/mjpeg_depacketizer.hpp b/components/rtsp/include/mjpeg_depacketizer.hpp new file mode 100755 index 000000000..cfb8474e3 --- /dev/null +++ b/components/rtsp/include/mjpeg_depacketizer.hpp @@ -0,0 +1,61 @@ +#pragma once + +#include +#include +#include + +#include "jpeg_frame.hpp" +#include "rtp_depacketizer.hpp" +#include "rtp_jpeg_packet.hpp" + +namespace espp { + +/// MJPEG depacketizer that reassembles JPEG frames from RTP packets. +/// +/// This class receives individual RTP packets containing RFC 2435 MJPEG +/// payloads, reassembles the scan data fragments, reconstructs the JPEG +/// header from the MJPEG header fields, and delivers complete JPEG frames +/// through callbacks. +class MjpegDepacketizer : public RtpDepacketizer { +public: + /// Callback type for receiving complete JPEG frames as JpegFrame objects. + using jpeg_frame_callback_t = std::function)>; + + /// Configuration for the MJPEG depacketizer. + struct Config { + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct an MJPEG depacketizer. + /// @param config Configuration for the depacketizer. + explicit MjpegDepacketizer(const Config &config) + : RtpDepacketizer({.log_level = config.log_level}, "MjpegDepacketizer") {} + + /// Process an incoming RTP packet containing MJPEG data. + /// @param packet The RTP packet to process. + /// @note Packets are parsed as RtpJpegPacket. When a complete frame is + /// assembled (marker bit set and no missing sequence numbers), both + /// the generic frame callback and the JPEG frame callback are invoked. + void process_packet(const RtpPacket &packet) override; + + /// Set callback for receiving complete JPEG frames. + /// @param cb Callback receiving a shared pointer to the completed JpegFrame. + void set_jpeg_frame_callback(jpeg_frame_callback_t cb); + +private: + /// Check if there are gaps in the received sequence numbers. + /// @return true if there are missing sequence numbers, false otherwise. + bool has_missing_sequence_numbers(); + + bool assembling_frame_{false}; + bool raw_jpeg_mode_{false}; + std::vector scan_buffer_; + int frame_width_{0}; + int frame_height_{0}; + std::vector q0_table_; + std::vector q1_table_; + std::vector received_sequence_numbers_; + jpeg_frame_callback_t on_jpeg_frame_; +}; + +} // namespace espp diff --git a/components/rtsp/include/mjpeg_packetizer.hpp b/components/rtsp/include/mjpeg_packetizer.hpp new file mode 100644 index 000000000..183802bbd --- /dev/null +++ b/components/rtsp/include/mjpeg_packetizer.hpp @@ -0,0 +1,49 @@ +#pragma once + +#include "jpeg_header.hpp" +#include "rtp_packetizer.hpp" + +namespace espp { + +/// MJPEG packetizer that fragments JPEG frames into RFC 2435 RTP payloads. +/// +/// This class takes complete JPEG frames and produces RTP payload chunks +/// suitable for MJPEG streaming. Each chunk contains an RFC 2435 MJPEG +/// header, and the first chunk additionally includes quantization tables. +class MjpegPacketizer : public RtpPacketizer { +public: + /// Configuration for the MJPEG packetizer. + struct Config { + size_t max_payload_size{1400}; ///< Maximum payload bytes per RTP packet + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct an MJPEG packetizer. + /// @param config Configuration for the packetizer. + explicit MjpegPacketizer(const Config &config) + : RtpPacketizer({.max_payload_size = config.max_payload_size, .log_level = config.log_level}, + "MjpegPacketizer") {} + + /// Packetize a complete JPEG frame into RFC 2435 RTP payload chunks. + /// @param frame_data Raw JPEG data including the JPEG header. + /// @return Vector of payload chunks ready to be wrapped in RTP packets. + std::vector packetize(std::span frame_data) override; + + /// Get the RTP payload type for MJPEG. + /// @return 26 (static JPEG payload type). + int get_payload_type() const override; + + /// Get the RTP clock rate for MJPEG. + /// @return 90000 Hz. + uint32_t get_clock_rate() const override; + + /// Get the SDP media attributes for MJPEG. + /// @return SDP rtpmap attribute string. + std::string get_sdp_media_attributes() const override; + + /// Get the SDP media line for MJPEG. + /// @return SDP media description line. + std::string get_sdp_media_line() const override; +}; + +} // namespace espp diff --git a/components/rtsp/include/rtp_depacketizer.hpp b/components/rtsp/include/rtp_depacketizer.hpp new file mode 100755 index 000000000..9ee28073b --- /dev/null +++ b/components/rtsp/include/rtp_depacketizer.hpp @@ -0,0 +1,50 @@ +#pragma once + +#include +#include +#include +#include + +#include "base_component.hpp" +#include "rtp_packet.hpp" + +namespace espp { + +/// Abstract base class for reassembling media frames from incoming RTP packets. +/// Concrete depacketizers (e.g. MJPEG, H.264) override process_packet() to +/// accumulate payload data and invoke the frame callback when a complete frame +/// has been assembled. +class RtpDepacketizer : public BaseComponent { +public: + /// Callback type invoked when a complete frame has been reassembled. + /// The frame data is moved into the callback to avoid copies. + using frame_callback_t = std::function &&)>; + + /// Configuration for RtpDepacketizer. + struct Config { + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct an RtpDepacketizer. + /// @param config The configuration for this depacketizer. + /// @param name A human-readable name used for logging. + explicit RtpDepacketizer(const Config &config, const std::string &name) + : BaseComponent(name, config.log_level) {} + + /// Destructor. + virtual ~RtpDepacketizer() = default; + + /// Process an incoming RTP packet, accumulating payload data. + /// When a complete frame is assembled the frame callback is invoked. + /// @param packet The RTP packet to process. + virtual void process_packet(const RtpPacket &packet) = 0; + + /// Set the callback for completed frames. + /// @param cb The callback to invoke when a full frame is ready. + void set_frame_callback(frame_callback_t cb) { on_frame_ = std::move(cb); } + +protected: + frame_callback_t on_frame_; ///< Callback invoked with each reassembled frame +}; + +} // namespace espp diff --git a/components/rtsp/include/rtp_jpeg_packet.hpp b/components/rtsp/include/rtp_jpeg_packet.hpp old mode 100644 new mode 100755 index 0e63ee59e..28e23e8fc --- a/components/rtsp/include/rtp_jpeg_packet.hpp +++ b/components/rtsp/include/rtp_jpeg_packet.hpp @@ -1,7 +1,10 @@ #pragma once +#include #include +#include "format.hpp" + #include "rtp_packet.hpp" namespace espp { diff --git a/components/rtsp/include/rtp_packetizer.hpp b/components/rtsp/include/rtp_packetizer.hpp new file mode 100644 index 000000000..d61cf56f7 --- /dev/null +++ b/components/rtsp/include/rtp_packetizer.hpp @@ -0,0 +1,60 @@ +#pragma once + +#include +#include +#include +#include + +#include "base_component.hpp" +#include "rtp_types.hpp" + +namespace espp { + +/// Abstract base class for splitting media frames into RTP payload chunks. +/// Concrete packetizers (e.g. MJPEG, H.264) override the pure-virtual methods +/// to produce codec-specific payloads. The RTSP server wraps each returned +/// RtpPayloadChunk with an RTP header before sending. +class RtpPacketizer : public BaseComponent { +public: + /// Configuration for RtpPacketizer. + struct Config { + size_t max_payload_size{1400}; ///< Maximum payload bytes per RTP packet + espp::Logger::Verbosity log_level{espp::Logger::Verbosity::WARN}; ///< Log verbosity level + }; + + /// Construct an RtpPacketizer. + /// @param config The configuration for this packetizer. + /// @param name A human-readable name used for logging. + explicit RtpPacketizer(const Config &config, const std::string &name) + : BaseComponent(name, config.log_level) + , max_payload_size_(config.max_payload_size) {} + + /// Destructor. + virtual ~RtpPacketizer() = default; + + /// Packetize a complete media frame into RTP payload chunks. + /// @param frame_data The raw frame bytes to packetize. + /// @return A vector of RtpPayloadChunk ready to be wrapped in RTP packets. + virtual std::vector packetize(std::span frame_data) = 0; + + /// Get the RTP payload type number for this codec. + /// @return The RTP payload type (e.g. 26 for MJPEG, 96 for dynamic). + virtual int get_payload_type() const = 0; + + /// Get the RTP clock rate for timestamp calculation. + /// @return The clock rate in Hz (e.g. 90000 for video, 8000 for audio). + virtual uint32_t get_clock_rate() const = 0; + + /// Generate the SDP media-level attributes for this codec. + /// @return A string containing SDP a= lines (without trailing CRLF). + virtual std::string get_sdp_media_attributes() const = 0; + + /// Generate the SDP m= line for this codec. + /// @return A string containing the SDP m= line (without trailing CRLF). + virtual std::string get_sdp_media_line() const = 0; + +protected: + size_t max_payload_size_; ///< Maximum payload bytes per RTP packet +}; + +} // namespace espp diff --git a/components/rtsp/include/rtp_types.hpp b/components/rtsp/include/rtp_types.hpp new file mode 100755 index 000000000..950434d95 --- /dev/null +++ b/components/rtsp/include/rtp_types.hpp @@ -0,0 +1,21 @@ +#pragma once + +#include +#include + +namespace espp { + +/// Describes a media type for RTSP tracks. +enum class MediaType { + VIDEO, ///< Video media (MJPEG, H264, etc.) + AUDIO ///< Audio media (PCM, Opus, AAC, etc.) +}; + +/// Represents one RTP payload chunk ready to be wrapped in an RtpPacket. +/// Packetizers produce these; the server wraps them with RTP headers. +struct RtpPayloadChunk { + std::vector data; ///< The payload data for this chunk + bool marker{false}; ///< Set on last chunk of a frame/access unit +}; + +} // namespace espp diff --git a/components/rtsp/include/rtsp_client.hpp b/components/rtsp/include/rtsp_client.hpp old mode 100644 new mode 100755 index 9f7798b5d..42cac114e --- a/components/rtsp/include/rtsp_client.hpp +++ b/components/rtsp/include/rtsp_client.hpp @@ -2,6 +2,8 @@ #include "socket_msvc.hpp" +#include +#include #include #include #include @@ -13,6 +15,8 @@ #include "udp_socket.hpp" #include "jpeg_frame.hpp" +#include "mjpeg_depacketizer.hpp" +#include "rtp_depacketizer.hpp" namespace espp { @@ -30,9 +34,25 @@ namespace espp { /// \snippet rtsp_example.cpp rtsp_client_example class RtspClient : public BaseComponent { public: + struct TrackInfo { + int track_id{0}; + int payload_type{0}; + int clock_rate{0}; + int channels{1}; + std::string media_type; + std::string encoding_name; + std::string control_path; + }; + /// Function type for the callback to call when a JPEG frame is received typedef std::function jpeg_frame)> jpeg_frame_callback_t; + /// Generic frame callback — called for any track/codec with raw frame data + using frame_callback_t = std::function &&data)>; + + /// Callback invoked when the RTSP server disappears after playback starts. + using disconnect_callback_t = std::function; + /// Configuration for the RTSP client struct Config { std::string server_address; ///< The server IP Address to connect to @@ -40,8 +60,20 @@ class RtspClient : public BaseComponent { std::string path{"/mjpeg/1"}; ///< The path to the RTSP stream on the server. Will be appended ///< to the server address and port to form the full path of the ///< form "rtsp://:" - espp::RtspClient::jpeg_frame_callback_t - on_jpeg_frame; ///< The callback to call when a JPEG frame is received + + /// Generic frame callback for any codec (track_id, raw frame data) + frame_callback_t on_frame{nullptr}; + + /// JPEG-specific frame callback (backward compatible). + /// If set and no depacketizer is registered for PT 26, an MjpegDepacketizer + /// is automatically created. + jpeg_frame_callback_t on_jpeg_frame{nullptr}; + + /// Called once if the client loses the server after playback starts. + /// This callback is intended for applications that want to stop playback + /// and re-enter service discovery or reconnect logic automatically. + disconnect_callback_t on_connection_lost{nullptr}; + espp::Logger::Verbosity log_level = espp::Logger::Verbosity::INFO; ///< The verbosity of the logger }; @@ -112,6 +144,13 @@ class RtspClient : public BaseComponent { void setup(size_t rtp_port, size_t rtcp_port, const std::chrono::duration &receive_timeout, std::error_code &ec); + /// Register a depacketizer for a specific RTP payload type. + /// When RTP packets with this payload type are received, they are + /// dispatched to the registered depacketizer. + /// @param payload_type The RTP payload type (e.g., 26 for MJPEG, 96 for H264) + /// @param depacketizer The depacketizer to handle packets of this type + void add_depacketizer(int payload_type, std::shared_ptr depacketizer); + /// Play the RTSP stream /// Sends the PLAY request to the RTSP server and parses the response. /// \param ec The error code to set if an error occurs @@ -127,7 +166,17 @@ class RtspClient : public BaseComponent { /// \param ec The error code to set if an error occurs void teardown(std::error_code &ec); + /// Get the parsed SDP track descriptions from the most recent DESCRIBE call. + /// \return The ordered set of discovered media tracks. + const std::vector &tracks() const { return tracks_; } + protected: + void reset_transport_state(); + void start_monitor_task(); + void stop_monitor_task(); + bool monitor_task_fn(std::mutex &m, std::condition_variable &cv, bool &task_notified); + void notify_connection_lost(std::string_view reason); + /// Parse the RTSP response /// \note Parses response data for the following fields: /// - Status code @@ -156,11 +205,12 @@ class RtspClient : public BaseComponent { std::error_code &ec); /// Handle an RTP packet - /// \note Parses the RTP packet and appends it to the current JPEG frame. - /// \note If the packet is the last fragment of the JPEG frame, the frame is sent to the - /// on_jpeg_frame callback. \note This function is called by the RTP socket task. \param data The - /// data to handle \param sender_info The sender info \return Optional data to send back to the - /// sender + /// \note Parses the RTP packet header, determines the payload type, and + /// dispatches to the appropriate registered depacketizer. + /// \note This function is called by the RTP socket task. + /// \param data The data to handle + /// \param sender_info The sender info + /// \return Optional data to send back to the sender std::optional> handle_rtp_packet(std::vector &data, const espp::Socket::Info &sender_info); @@ -172,6 +222,7 @@ class RtspClient : public BaseComponent { /// \return Optional data to send back to the sender std::optional> handle_rtcp_packet(std::vector &data, const espp::Socket::Info &sender_info); + std::string server_address_; int rtsp_port_; @@ -180,12 +231,27 @@ class RtspClient : public BaseComponent { espp::UdpSocket rtcp_socket_; jpeg_frame_callback_t on_jpeg_frame_{nullptr}; + frame_callback_t on_frame_{nullptr}; + disconnect_callback_t on_connection_lost_{nullptr}; + std::unordered_map> depacketizers_; + + std::unique_ptr monitor_task_; + std::chrono::steady_clock::duration rtp_receive_timeout_{std::chrono::seconds(15)}; + std::chrono::steady_clock::duration initial_rtp_receive_timeout_{std::chrono::seconds(15)}; + std::atomic play_started_tick_{0}; + std::atomic last_rtp_packet_tick_{0}; + std::atomic playing_{false}; + std::atomic disconnecting_{false}; + std::atomic connection_lost_reported_{false}; int cseq_ = 0; int video_port_ = 0; int video_payload_type_ = 0; std::string path_; + std::string base_path_; std::string session_id_; + std::vector tracks_; + std::unordered_map payload_type_to_track_id_; }; } // namespace espp diff --git a/components/rtsp/include/rtsp_server.hpp b/components/rtsp/include/rtsp_server.hpp index 5b0c1607b..dcad63e57 100644 --- a/components/rtsp/include/rtsp_server.hpp +++ b/components/rtsp/include/rtsp_server.hpp @@ -2,7 +2,9 @@ #include "socket_msvc.hpp" +#include #include +#include #include #include #include @@ -19,9 +21,12 @@ #include "udp_socket.hpp" #include "jpeg_frame.hpp" +#include "mjpeg_packetizer.hpp" #include "rtcp_packet.hpp" #include "rtp_jpeg_packet.hpp" #include "rtp_packet.hpp" +#include "rtp_packetizer.hpp" +#include "rtp_types.hpp" #include "rtsp_session.hpp" @@ -38,6 +43,14 @@ class RtspServer : public BaseComponent { public: /// @brief Configuration for the RTSP server struct Config { +#if defined(ESP_PLATFORM) + static constexpr size_t default_accept_task_stack_size_bytes = 4 * 1024; + static constexpr size_t default_session_task_stack_size_bytes = 4 * 1024; +#else + static constexpr size_t default_accept_task_stack_size_bytes = 6 * 1024; + static constexpr size_t default_session_task_stack_size_bytes = 6 * 1024; +#endif + std::string server_address; ///< The ip address of the server int port; ///< The port to listen on std::string path; ///< The path to the RTSP stream @@ -48,6 +61,20 @@ class RtspServer : public BaseComponent { ///< properly. espp::Logger::Verbosity log_level = espp::Logger::Verbosity::WARN; ///< The log level for the RTSP server + size_t accept_task_stack_size_bytes = + default_accept_task_stack_size_bytes; ///< RTSP accept-task stack size, in bytes + size_t session_task_stack_size_bytes = + default_session_task_stack_size_bytes; ///< RTSP session-dispatch task stack size, in bytes + size_t control_task_stack_size_bytes = + RtspSession::Config::default_control_task_stack_size_bytes; ///< Per-session RTSP + ///< control-task stack size, in + ///< bytes + }; + + /// Configuration for a media track to be registered with the server + struct TrackConfig { + int track_id{0}; ///< Track identifier + std::shared_ptr packetizer; ///< Codec-specific packetizer }; /// @brief Construct an RTSP server @@ -74,31 +101,104 @@ class RtspServer : public BaseComponent { /// Stops the accept task, session task, and closes the RTSP socket void stop(); - /// @brief Send a frame over the RTSP connection - /// Converts the full JPEG frame into a series of simplified RTP/JPEG - /// packets and stores it to be sent over the RTP socket, but does not - /// actually send it - /// @note Overwrites any existing frame that has not been sent - /// @param frame The frame to send + /// @brief Register a media track with the server. + /// Each track has its own packetizer, SSRC, and sequence number. + /// @param config Track configuration including the packetizer. + void add_track(const TrackConfig &config); + + /// @brief Returns true when at least one session is actively playing. + /// @return True if an active RTSP session is ready to receive RTP packets. + bool has_active_sessions(); + + /// @brief Returns how long capture should wait before queueing another frame. + /// @return Remaining RTP backpressure cooldown, or zero if sending may resume. + std::chrono::milliseconds get_capture_cooldown(); + + /// @brief Returns the minimum recommended period between captured frames. + /// @return Recommended capture period based on recent RTP backpressure history. + std::chrono::milliseconds get_recommended_capture_period(); + + /// @brief Send a frame on a specific track. + /// The track's packetizer splits the frame into RTP payload chunks, + /// which are then wrapped with RTP headers and queued for delivery. + /// @note Overwrites any existing pending packets for this track. + /// @param track_id The track to send on. + /// @param frame_data Raw encoded frame data. + void send_frame(int track_id, std::span frame_data); + + /// @brief Send a JPEG frame over the RTSP connection (backward compatible). + /// If no tracks have been added, lazily creates a default MJPEG track on + /// track 0. Uses the legacy RtpJpegPacket packetization to preserve the + /// exact wire format for existing MJPEG users. + /// @note Overwrites any existing frame that has not been sent. + /// @param frame The frame to send. void send_frame(const espp::JpegFrame &frame); + /// @brief Send raw JPEG bytes over the default MJPEG track. + /// Uses the legacy MJPEG RTP packetization path without copying the frame + /// into an intermediate JpegFrame object. + /// @note Overwrites any existing frame that has not been sent. + /// @param frame_data Complete JPEG bytes, including header and EOI marker. + void send_frame(std::span frame_data); + protected: + /// Per-track state holding packetizer, RTP sequencing, and pending packets + struct TrackState { + struct PacketBatch { + std::vector> packets; + size_t count{0}; + }; + + int track_id{0}; + std::shared_ptr packetizer; + uint32_t ssrc{0}; + uint16_t sequence_number{0}; + std::mutex packets_mutex; + std::shared_ptr pending_batch; + std::shared_ptr recycled_batch; + }; + bool accept_task_function(std::mutex &m, std::condition_variable &cv, bool &task_notified); bool session_task_function(std::mutex &m, std::condition_variable &cv, bool &task_notified); - - uint32_t ssrc_; ///< the ssrc (synchronization source identifier) for the RTP packets - uint16_t sequence_number_{0}; ///< the sequence number for the RTP packets + void reap_closed_sessions(); + + /// Generate combined SDP from all registered tracks. + /// @param session_path Full RTSP URL path + /// @param session_id The session ID + /// @param server_address The server address (ip:port) + /// @return SDP body string + std::string generate_sdp(const std::string &session_path, uint32_t session_id, + const std::string &server_address) const; + + /// Generate a random SSRC value + /// @return A random 32-bit SSRC + static uint32_t generate_ssrc() { +#if defined(ESP_PLATFORM) + return esp_random(); +#else + static std::random_device rd; + static std::mt19937 gen(rd()); + static std::uniform_int_distribution dis; + return dis(gen); +#endif + } std::string server_address_; ///< the address of the server int port_; ///< the port of the RTSP server std::string path_; ///< the path of the RTSP server, e.g. rtsp:://\:\/\ + std::chrono::microseconds accept_timeout_{std::chrono::seconds(5)}; espp::TcpSocket rtsp_socket_; size_t max_data_size_; - std::mutex rtp_packets_mutex_; - std::vector> rtp_packets_; + std::vector> tracks_; + bool default_mjpeg_track_created_{false}; + std::chrono::steady_clock::time_point backpressure_until_{}; + size_t consecutive_backpressure_failures_{0}; + size_t accept_task_stack_size_bytes_; + size_t session_task_stack_size_bytes_; + size_t control_task_stack_size_bytes_; espp::Logger::Verbosity session_log_level_{espp::Logger::Verbosity::WARN}; std::mutex session_mutex_; diff --git a/components/rtsp/include/rtsp_session.hpp b/components/rtsp/include/rtsp_session.hpp index cf7c83c0c..f47b0d6bb 100644 --- a/components/rtsp/include/rtsp_session.hpp +++ b/components/rtsp/include/rtsp_session.hpp @@ -2,9 +2,12 @@ #include "socket_msvc.hpp" +#include #include +#include #include #include +#include #include #if defined(ESP_PLATFORM) @@ -26,12 +29,43 @@ namespace espp { /// session id and sends frame data over RTP and RTCP to the client class RtspSession : public BaseComponent { public: + /// Represents one media track within an RTSP session + struct Track { + int track_id{0}; ///< Track identifier (matches trackID=N in SDP) + std::string control_path; ///< Control path suffix (e.g., "trackID=0") + espp::UdpSocket rtp_socket; ///< RTP socket for this track + espp::UdpSocket rtcp_socket; ///< RTCP socket for this track + int client_rtp_port{0}; ///< Client's RTP port + int client_rtcp_port{0}; ///< Client's RTCP port + bool setup_complete{false}; ///< Whether SETUP has been completed for this track + + Track() + : rtp_socket({.log_level = espp::Logger::Verbosity::WARN}) + , rtcp_socket({.log_level = espp::Logger::Verbosity::WARN}) {} + }; + /// Configuration for the RTSP session struct Config { +#if defined(ESP_PLATFORM) + static constexpr size_t default_control_task_stack_size_bytes = 4 * 1024; +#else + static constexpr size_t default_control_task_stack_size_bytes = 6 * 1024; +#endif + std::string server_address; ///< The address of the server std::string rtsp_path; ///< The RTSP path of the session std::chrono::duration receive_timeout = std::chrono::seconds(5); ///< The timeout for receiving data. Should be > 0. + size_t control_task_stack_size_bytes = + default_control_task_stack_size_bytes; ///< RTSP control-task stack size, in bytes + /// SDP generator callback. If set, called during DESCRIBE to produce the SDP body. + /// If not set, a default MJPEG SDP is generated for backward compatibility. + /// @param session_path Full RTSP path (e.g., "rtsp://ip:port/path") + /// @param session_id The session ID + /// @param server_address The server address with port + std::function + sdp_generator; espp::Logger::Verbosity log_level = espp::Logger::Verbosity::WARN; ///< The log level of the session }; @@ -77,12 +111,35 @@ class RtspSession : public BaseComponent { /// and close the connection void teardown(); - /// Send an RTP packet to the client + /// Send an RTP packet on a specific track + /// @param track_id The track to send on + /// @param packet The RTP packet to send + /// @return True if the packet was sent successfully, false otherwise + bool send_rtp_packet(int track_id, const espp::RtpPacket &packet); + + /// Send a serialized RTP packet on a specific track. + /// @param track_id The track to send on + /// @param packet_data Serialized RTP packet bytes + /// @return True if the packet was sent successfully, false otherwise + bool send_rtp_packet(int track_id, std::span packet_data); + + /// Send an RTP packet to the client (backward compat — sends on default track 0) /// @param packet The RTP packet to send /// @return True if the packet was sent successfully, false otherwise bool send_rtp_packet(const espp::RtpPacket &packet); - /// Send an RTCP packet to the client + /// Send a serialized RTP packet to the client (default track 0). + /// @param packet_data Serialized RTP packet bytes + /// @return True if the packet was sent successfully, false otherwise + bool send_rtp_packet(std::span packet_data); + + /// Send an RTCP packet on a specific track + /// @param track_id The track to send on + /// @param packet The RTCP packet to send + /// @return True if the packet was sent successfully, false otherwise + bool send_rtcp_packet(int track_id, const espp::RtcpPacket &packet); + + /// Send an RTCP packet to the client (backward compat — sends on default track 0) /// @param packet The RTCP packet to send /// @return True if the packet was sent successfully, false otherwise bool send_rtcp_packet(const espp::RtcpPacket &packet); @@ -184,8 +241,7 @@ class RtspSession : public BaseComponent { int &client_rtp_port, int &client_rtcp_port); std::shared_ptr control_socket_; - espp::UdpSocket rtp_socket_; - espp::UdpSocket rtcp_socket_; + std::unordered_map> tracks_; uint32_t session_id_; bool closed_ = false; @@ -195,8 +251,8 @@ class RtspSession : public BaseComponent { std::string rtsp_path_; std::string client_address_; - int client_rtp_port_; - int client_rtcp_port_; + + std::function sdp_generator_; std::unique_ptr control_task_; }; diff --git a/components/rtsp/src/generic_depacketizer.cpp b/components/rtsp/src/generic_depacketizer.cpp new file mode 100644 index 000000000..49f7f3ba5 --- /dev/null +++ b/components/rtsp/src/generic_depacketizer.cpp @@ -0,0 +1,42 @@ +#include "generic_depacketizer.hpp" + +using namespace espp; + +GenericDepacketizer::GenericDepacketizer(const Config &config) + : RtpDepacketizer({.log_level = config.log_level}, "GenericDepacketizer") {} + +void GenericDepacketizer::process_packet(const RtpPacket &packet) { + auto payload = packet.get_payload(); + int timestamp = packet.get_timestamp(); + + // If the timestamp changed, discard the old buffer and start fresh + if (has_timestamp_ && timestamp != current_timestamp_) { + logger_.warn("Timestamp changed ({} -> {}), discarding {} buffered bytes", current_timestamp_, + timestamp, buffer_.size()); + reset_buffer(); + } + + current_timestamp_ = timestamp; + has_timestamp_ = true; + + // Append this packet's payload to the buffer + buffer_.insert(buffer_.end(), payload.begin(), payload.end()); + + logger_.debug("Buffered {} bytes (total: {}, marker: {})", payload.size(), buffer_.size(), + packet.get_marker()); + + // If the marker bit is set, deliver the complete frame + if (packet.get_marker()) { + if (on_frame_) { + logger_.debug("Delivering frame of {} bytes", buffer_.size()); + on_frame_(std::move(buffer_)); + } + reset_buffer(); + } +} + +void GenericDepacketizer::reset_buffer() { + buffer_.clear(); + has_timestamp_ = false; + current_timestamp_ = -1; +} diff --git a/components/rtsp/src/generic_packetizer.cpp b/components/rtsp/src/generic_packetizer.cpp new file mode 100644 index 000000000..948a31279 --- /dev/null +++ b/components/rtsp/src/generic_packetizer.cpp @@ -0,0 +1,73 @@ +#include "generic_packetizer.hpp" + +#include + +#include "format.hpp" + +using namespace espp; + +GenericPacketizer::GenericPacketizer(const Config &config) + : RtpPacketizer({.max_payload_size = config.max_payload_size, .log_level = config.log_level}, + "GenericPacketizer") + , payload_type_(config.payload_type) + , clock_rate_(config.clock_rate) + , encoding_name_(config.encoding_name) + , channels_(config.channels) + , fmtp_(config.fmtp) + , media_type_(config.media_type) {} + +std::vector GenericPacketizer::packetize(std::span frame_data) { + std::vector chunks; + if (frame_data.empty()) { + return chunks; + } + if (max_payload_size_ == 0) { + logger_.error("max_payload_size is 0; cannot packetize {} bytes", frame_data.size()); + return chunks; + } + + size_t offset = 0; + size_t remaining = frame_data.size(); + + while (remaining > 0) { + size_t chunk_size = std::min(remaining, max_payload_size_); + bool is_last = (chunk_size == remaining); + + RtpPayloadChunk chunk; + chunk.data.assign(frame_data.data() + offset, frame_data.data() + offset + chunk_size); + chunk.marker = is_last; + chunks.push_back(std::move(chunk)); + + offset += chunk_size; + remaining -= chunk_size; + } + + logger_.debug("Packetized {} bytes into {} chunk(s)", frame_data.size(), chunks.size()); + return chunks; +} + +int GenericPacketizer::get_payload_type() const { return payload_type_; } + +uint32_t GenericPacketizer::get_clock_rate() const { return clock_rate_; } + +std::string GenericPacketizer::get_sdp_media_attributes() const { + std::string attrs; + + if (media_type_ == MediaType::VIDEO || channels_ <= 1) { + attrs += fmt::format("a=rtpmap:{} {}/{}", payload_type_, encoding_name_, clock_rate_); + } else { + attrs += + fmt::format("a=rtpmap:{} {}/{}/{}", payload_type_, encoding_name_, clock_rate_, channels_); + } + + if (!fmtp_.empty()) { + attrs += fmt::format("\r\na=fmtp:{} {}", payload_type_, fmtp_); + } + + return attrs; +} + +std::string GenericPacketizer::get_sdp_media_line() const { + const char *media = (media_type_ == MediaType::AUDIO) ? "audio" : "video"; + return fmt::format("m={} 0 RTP/AVP {}", media, payload_type_); +} diff --git a/components/rtsp/src/h264_depacketizer.cpp b/components/rtsp/src/h264_depacketizer.cpp new file mode 100644 index 000000000..0115788f3 --- /dev/null +++ b/components/rtsp/src/h264_depacketizer.cpp @@ -0,0 +1,124 @@ +#include "h264_depacketizer.hpp" + +#include + +using namespace espp; + +/// Annex B start code: 0x00 0x00 0x00 0x01 +static constexpr uint8_t kStartCode[] = {0x00, 0x00, 0x00, 0x01}; +static constexpr size_t kStartCodeSize = sizeof(kStartCode); + +H264Depacketizer::H264Depacketizer(const Config &config) + : RtpDepacketizer({.log_level = config.log_level}, "H264Depacketizer") {} + +void H264Depacketizer::process_packet(const RtpPacket &packet) { + auto payload = packet.get_payload(); + if (payload.empty()) { + logger_.warn("Received RTP packet with empty payload"); + return; + } + + uint8_t first_byte = payload[0]; + uint8_t nal_type = first_byte & 0x1F; + + if (nal_type >= 1 && nal_type <= 23) { + // Single NAL unit packet — the entire payload is one NAL. + logger_.debug("Single NAL: type={}, size={}", nal_type, payload.size()); + nal_buffer_.emplace_back(payload.begin(), payload.end()); + + } else if (nal_type == 24) { + // STAP-A — aggregation packet containing multiple NAL units. + // Format: STAP-A header (1 byte) + { NAL size (2 bytes, big-endian) + NAL data } ... + logger_.debug("STAP-A: payload size={}", payload.size()); + size_t offset = 1; // skip STAP-A header byte + while (offset + 2 <= payload.size()) { + uint16_t nal_size = (static_cast(payload[offset]) << 8) | + static_cast(payload[offset + 1]); + offset += 2; + + if (offset + nal_size > payload.size()) { + logger_.warn("STAP-A: NAL size {} exceeds remaining payload at offset {}", nal_size, + offset); + break; + } + + nal_buffer_.emplace_back(payload.data() + offset, payload.data() + offset + nal_size); + offset += nal_size; + } + + } else if (nal_type == 28) { + // FU-A — fragmentation unit. + if (payload.size() < 2) { + logger_.warn("FU-A: payload too short ({})", payload.size()); + return; + } + + uint8_t fu_indicator = payload[0]; + uint8_t fu_header = payload[1]; + bool start_bit = (fu_header & 0x80) != 0; + bool end_bit = (fu_header & 0x40) != 0; + uint8_t original_nal_type = fu_header & 0x1F; + + if (start_bit) { + // Start of a new fragmented NAL unit. + // Reconstruct the NAL header from the FU indicator NRI and the FU header type. + uint8_t reconstructed_header = (fu_indicator & 0xE0) | original_nal_type; + fua_buffer_.clear(); + fua_buffer_.push_back(reconstructed_header); + fua_buffer_.insert(fua_buffer_.end(), payload.data() + 2, payload.data() + payload.size()); + fua_in_progress_ = true; + logger_.debug("FU-A start: nal_type={}, fragment_size={}", original_nal_type, + payload.size() - 2); + } else if (fua_in_progress_) { + // Continuation or end fragment. + fua_buffer_.insert(fua_buffer_.end(), payload.data() + 2, payload.data() + payload.size()); + logger_.debug("FU-A {}: fragment_size={}, total={}", end_bit ? "end" : "mid", + payload.size() - 2, fua_buffer_.size()); + } else { + logger_.warn("FU-A: received continuation/end fragment without start"); + return; + } + + if (end_bit) { + // FU-A reassembly complete — move into the NAL buffer. + nal_buffer_.push_back(std::move(fua_buffer_)); + fua_buffer_.clear(); + fua_in_progress_ = false; + } + + } else { + logger_.warn("Unsupported NAL unit type: {}", nal_type); + return; + } + + // If the RTP marker bit is set, the access unit is complete. + if (packet.get_marker()) { + deliver_frame(); + } +} + +void H264Depacketizer::deliver_frame() { + if (nal_buffer_.empty()) { + logger_.debug("Marker set but no NAL units buffered"); + return; + } + + // Build Annex B frame: each NAL prefixed with 0x00 0x00 0x00 0x01. + size_t total_size = std::accumulate( + nal_buffer_.begin(), nal_buffer_.end(), size_t{0}, + [](size_t acc, const auto &nal) { return acc + kStartCodeSize + nal.size(); }); + + std::vector frame; + frame.reserve(total_size); + for (const auto &nal : nal_buffer_) { + frame.insert(frame.end(), kStartCode, kStartCode + kStartCodeSize); + frame.insert(frame.end(), nal.begin(), nal.end()); + } + + logger_.debug("Delivering frame: {} NAL(s), {} bytes", nal_buffer_.size(), frame.size()); + nal_buffer_.clear(); + + if (on_frame_) { + on_frame_(std::move(frame)); + } +} diff --git a/components/rtsp/src/h264_packetizer.cpp b/components/rtsp/src/h264_packetizer.cpp new file mode 100644 index 000000000..c95be08b5 --- /dev/null +++ b/components/rtsp/src/h264_packetizer.cpp @@ -0,0 +1,256 @@ +#include "h264_packetizer.hpp" + +#include +#include + +namespace { + +/// Base64 encoding table. +static constexpr char kBase64Table[] = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + +/// Encode binary data to base64 string. +/// @param data The binary data to encode. +/// @return The base64-encoded string. +std::string base64_encode(std::span data) { + std::string result; + result.reserve(((data.size() + 2) / 3) * 4); + + size_t i = 0; + while (i + 2 < data.size()) { + uint32_t triple = (static_cast(data[i]) << 16) | + (static_cast(data[i + 1]) << 8) | + static_cast(data[i + 2]); + result.push_back(kBase64Table[(triple >> 18) & 0x3F]); + result.push_back(kBase64Table[(triple >> 12) & 0x3F]); + result.push_back(kBase64Table[(triple >> 6) & 0x3F]); + result.push_back(kBase64Table[triple & 0x3F]); + i += 3; + } + + if (i + 1 == data.size()) { + uint32_t val = static_cast(data[i]) << 16; + result.push_back(kBase64Table[(val >> 18) & 0x3F]); + result.push_back(kBase64Table[(val >> 12) & 0x3F]); + result.push_back('='); + result.push_back('='); + } else if (i + 2 == data.size()) { + uint32_t val = + (static_cast(data[i]) << 16) | (static_cast(data[i + 1]) << 8); + result.push_back(kBase64Table[(val >> 18) & 0x3F]); + result.push_back(kBase64Table[(val >> 12) & 0x3F]); + result.push_back(kBase64Table[(val >> 6) & 0x3F]); + result.push_back('='); + } + + return result; +} + +} // namespace + +using namespace espp; + +H264Packetizer::H264Packetizer(const Config &config) + : RtpPacketizer({.max_payload_size = config.max_payload_size, .log_level = config.log_level}, + "H264Packetizer") + , payload_type_(config.payload_type) + , profile_level_id_(config.profile_level_id) + , packetization_mode_(config.packetization_mode) + , sps_(config.sps) + , pps_(config.pps) {} + +std::vector H264Packetizer::packetize(std::span frame_data) { + auto nals = parse_annex_b(frame_data); + if (nals.empty()) { + logger_.warn("No NAL units found in frame data ({} bytes)", frame_data.size()); + return {}; + } + + std::vector chunks; + for (size_t i = 0; i < nals.size(); ++i) { + bool is_last = (i == nals.size() - 1); + auto nal_chunks = packetize_nal(nals[i], is_last); + chunks.insert(chunks.end(), std::make_move_iterator(nal_chunks.begin()), + std::make_move_iterator(nal_chunks.end())); + } + + logger_.debug("Packetized {} NAL unit(s) into {} chunk(s)", nals.size(), chunks.size()); + return chunks; +} + +std::vector H264Packetizer::packetize_nal(std::span nal_data, + bool is_last_nal) { + constexpr size_t kFuHeaderSize = 2; + std::vector chunks; + + if (nal_data.empty()) { + return chunks; + } + + if (nal_data.size() <= max_payload_size_) { + // Single NAL unit mode — payload is the raw NAL bytes. + RtpPayloadChunk chunk; + chunk.data.assign(nal_data.begin(), nal_data.end()); + chunk.marker = is_last_nal; + chunks.push_back(std::move(chunk)); + logger_.debug("Single NAL: type={}, size={}", nal_data[0] & 0x1F, nal_data.size()); + } else { + // FU-A fragmentation (only if packetization_mode >= 1). + if (packetization_mode_ < 1) { + logger_.warn("NAL unit ({} bytes) exceeds max_payload_size ({}) but " + "packetization_mode is 0; dropping", + nal_data.size(), max_payload_size_); + return chunks; + } + if (max_payload_size_ <= kFuHeaderSize) { + logger_.error("NAL unit ({} bytes) exceeds max_payload_size ({}) but FU-A " + "fragmentation requires room for the 2-byte FU header plus " + "fragment payload; dropping", + nal_data.size(), max_payload_size_); + return chunks; + } + + uint8_t nal_header = nal_data[0]; + uint8_t nri = nal_header & 0x60; // NRI bits (bits 5-6) + uint8_t nal_type = nal_header & 0x1F; // NAL unit type (bits 0-4) + uint8_t fu_indicator = nri | 28; // FU-A type = 28 + + // Fragment the NAL body (everything after the NAL header byte). + // Each FU-A packet has: FU indicator (1) + FU header (1) + fragment data. + size_t max_fragment_size = max_payload_size_ - kFuHeaderSize; + const uint8_t *body = nal_data.data() + 1; + size_t body_size = nal_data.size() - 1; + size_t offset = 0; + + while (offset < body_size) { + size_t fragment_size = std::min(max_fragment_size, body_size - offset); + bool is_start = (offset == 0); + bool is_end = (offset + fragment_size >= body_size); + + uint8_t fu_header = nal_type; + if (is_start) { + fu_header |= 0x80; // S bit + } + if (is_end) { + fu_header |= 0x40; // E bit + } + + RtpPayloadChunk chunk; + chunk.data.reserve(2 + fragment_size); + chunk.data.push_back(fu_indicator); + chunk.data.push_back(fu_header); + chunk.data.insert(chunk.data.end(), body + offset, body + offset + fragment_size); + chunk.marker = is_last_nal && is_end; + chunks.push_back(std::move(chunk)); + + offset += fragment_size; + } + + logger_.debug("FU-A: type={}, size={}, fragments={}", nal_type, nal_data.size(), chunks.size()); + } + + return chunks; +} + +void H264Packetizer::set_sps_pps(std::span sps, std::span pps) { + sps_.assign(sps.begin(), sps.end()); + pps_.assign(pps.begin(), pps.end()); +} + +int H264Packetizer::get_payload_type() const { return payload_type_; } + +uint32_t H264Packetizer::get_clock_rate() const { return 90000; } + +std::string H264Packetizer::get_sdp_media_attributes() const { + // a=rtpmap:{pt} H264/90000 + std::string attrs = "a=rtpmap:" + std::to_string(payload_type_) + " H264/90000"; + + // a=fmtp:{pt} packetization-mode={mode};profile-level-id={profile} + attrs += "\r\na=fmtp:" + std::to_string(payload_type_) + + " packetization-mode=" + std::to_string(packetization_mode_); + + if (!profile_level_id_.empty()) { + attrs += ";profile-level-id=" + profile_level_id_; + } + + if (!sps_.empty() && !pps_.empty()) { + std::string sps_b64 = base64_encode(std::span(sps_)); + std::string pps_b64 = base64_encode(std::span(pps_)); + attrs += ";sprop-parameter-sets=" + sps_b64 + "," + pps_b64; + } + + return attrs; +} + +std::string H264Packetizer::get_sdp_media_line() const { + return "m=video 0 RTP/AVP " + std::to_string(payload_type_); +} + +std::vector> H264Packetizer::parse_annex_b(std::span data) { + std::vector> nals; + + if (data.size() < 4) { + return nals; + } + + // Find all start code positions. Start codes are 0x000001 (3-byte) or + // 0x00000001 (4-byte). + std::vector nal_starts; + size_t i = 0; + while (i + 2 < data.size()) { + if (data[i] == 0x00 && data[i + 1] == 0x00) { + if (i + 3 < data.size() && data[i + 2] == 0x00 && data[i + 3] == 0x01) { + // 4-byte start code + nal_starts.push_back(i + 4); + i += 4; + continue; + } else if (data[i + 2] == 0x01) { + // 3-byte start code + nal_starts.push_back(i + 3); + i += 3; + continue; + } + } + ++i; + } + + // Extract NAL units between start codes. + for (size_t n = 0; n < nal_starts.size(); ++n) { + size_t start = nal_starts[n]; + size_t end; + if (n + 1 < nal_starts.size()) { + // Find the beginning of the next start code (rewind past the 0x00 bytes). + end = nal_starts[n + 1]; + // The start code for the next NAL begins with 0x00 0x00 ..., so rewind. + while (end > start && data[end - 1] == 0x00) { + // We need to be careful: the 0x00 bytes before the next start code + // belong to the start code, not the NAL. But we stored nal_starts as + // the position AFTER the start code. So the start code for nal n+1 + // starts at some position before nal_starts[n+1]. We need to find + // where the 0x00 0x00 [0x00] 0x01 pattern begins. + break; + } + // Recompute: the start code at nal_starts[n+1] was either 3-byte or 4-byte. + // We need to find where the 0x00 bytes begin before nal_starts[n+1]. + size_t sc_start = nal_starts[n + 1]; + // sc_start points right after the start code. Determine start code length. + if (sc_start >= 4 && data[sc_start - 4] == 0x00 && data[sc_start - 3] == 0x00 && + data[sc_start - 2] == 0x00 && data[sc_start - 1] == 0x01) { + end = sc_start - 4; + } else if (sc_start >= 3 && data[sc_start - 3] == 0x00 && data[sc_start - 2] == 0x00 && + data[sc_start - 1] == 0x01) { + end = sc_start - 3; + } else { + end = sc_start; + } + } else { + end = data.size(); + } + + if (end > start) { + nals.push_back(data.subspan(start, end - start)); + } + } + + return nals; +} diff --git a/components/rtsp/src/mjpeg_depacketizer.cpp b/components/rtsp/src/mjpeg_depacketizer.cpp new file mode 100755 index 000000000..4182073d4 --- /dev/null +++ b/components/rtsp/src/mjpeg_depacketizer.cpp @@ -0,0 +1,100 @@ +#include "mjpeg_depacketizer.hpp" + +#include + +namespace espp { + +void MjpegDepacketizer::process_packet(const RtpPacket &packet) { + RtpJpegPacket jpeg_packet(packet.get_data()); + auto frag_offset = jpeg_packet.get_offset(); + auto jpeg_data = jpeg_packet.get_jpeg_data(); + + if (frag_offset == 0) { + // Start of a new frame — reset state and store MJPEG header info + scan_buffer_.clear(); + received_sequence_numbers_.clear(); + + frame_width_ = jpeg_packet.get_width(); + frame_height_ = jpeg_packet.get_height(); + if (jpeg_packet.has_q_tables()) { + auto q0 = jpeg_packet.get_q_table(0); + auto q1 = jpeg_packet.get_q_table(1); + q0_table_.assign(q0.begin(), q0.end()); + q1_table_.assign(q1.begin(), q1.end()); + } + + raw_jpeg_mode_ = jpeg_data.size() >= 2 && jpeg_data[0] == 0xff && jpeg_data[1] == 0xd8; + scan_buffer_.assign(jpeg_data.begin(), jpeg_data.end()); + assembling_frame_ = true; + } else if (assembling_frame_) { + // Continuation fragment — place scan data at the correct offset + size_t required_size = frag_offset + jpeg_data.size(); + if (required_size > scan_buffer_.size()) { + scan_buffer_.resize(required_size, 0); + } + std::copy(jpeg_data.begin(), jpeg_data.end(), scan_buffer_.begin() + frag_offset); + } else { + // No frame in progress — discard this fragment + logger_.debug("Ignoring packet with offset {} — no frame in progress", frag_offset); + return; + } + + received_sequence_numbers_.push_back(jpeg_packet.get_sequence_number()); + + if (jpeg_packet.get_marker()) { + // Frame is complete + assembling_frame_ = false; + bool has_missing_packets = has_missing_sequence_numbers(); + + logger_.debug("Frame complete: {}x{}, scan={} bytes", frame_width_, frame_height_, + scan_buffer_.size()); + + std::vector jpeg_bytes; + if (raw_jpeg_mode_) { + jpeg_bytes = scan_buffer_; + } else { + // Reconstruct the full JPEG: header + scan data + JpegHeader header(frame_width_, frame_height_, std::span(q0_table_), + std::span(q1_table_)); + auto header_data = header.get_data(); + jpeg_bytes.reserve(header_data.size() + scan_buffer_.size()); + jpeg_bytes.insert(jpeg_bytes.end(), header_data.begin(), header_data.end()); + jpeg_bytes.insert(jpeg_bytes.end(), scan_buffer_.begin(), scan_buffer_.end()); + } + + if (has_missing_packets) { + logger_.debug("Delivering best-effort MJPEG frame with missing RTP packets"); + } + + if (on_jpeg_frame_) { + auto frame = std::make_shared(jpeg_bytes); + on_jpeg_frame_(frame); + } + + if (on_frame_) { + on_frame_(std::move(jpeg_bytes)); + } + } +} + +void MjpegDepacketizer::set_jpeg_frame_callback(jpeg_frame_callback_t cb) { + on_jpeg_frame_ = std::move(cb); +} + +bool MjpegDepacketizer::has_missing_sequence_numbers() { + if (received_sequence_numbers_.size() <= 1) { + return false; + } + std::sort(received_sequence_numbers_.begin(), received_sequence_numbers_.end()); + int prev_seq = received_sequence_numbers_[0]; + for (size_t i = 1; i < received_sequence_numbers_.size(); ++i) { + int curr_seq = received_sequence_numbers_[i]; + if (curr_seq != (prev_seq + 1) % 65536) { + return true; + } + prev_seq = curr_seq; + } + return false; +} + +} // namespace espp diff --git a/components/rtsp/src/mjpeg_packetizer.cpp b/components/rtsp/src/mjpeg_packetizer.cpp new file mode 100644 index 000000000..0507e1bc3 --- /dev/null +++ b/components/rtsp/src/mjpeg_packetizer.cpp @@ -0,0 +1,141 @@ +#include "mjpeg_packetizer.hpp" + +#include +#include + +namespace espp { + +static constexpr size_t MJPEG_HEADER_SIZE = 8; +static constexpr size_t QUANT_HEADER_SIZE = 4; +static constexpr size_t NUM_Q_TABLES = 2; +static constexpr size_t Q_TABLE_SIZE = 64; +static constexpr size_t FIRST_PACKET_OVERHEAD = + MJPEG_HEADER_SIZE + QUANT_HEADER_SIZE + NUM_Q_TABLES * Q_TABLE_SIZE; +static constexpr size_t OTHER_PACKET_OVERHEAD = MJPEG_HEADER_SIZE; +static constexpr int MAX_RFC2435_DIMENSION = 255 * 8; + +std::vector MjpegPacketizer::packetize(std::span frame_data) { + if (frame_data.size() < 2) { + logger_.warn("Skipping short JPEG frame ({} bytes)", frame_data.size()); + return {}; + } + // Parse the JPEG header to extract width, height, and quantization tables + JpegHeader header(frame_data); + int width = header.get_width(); + int height = header.get_height(); + auto q0 = header.get_quantization_table(0); + auto q1 = header.get_quantization_table(1); + if (!header.is_valid() || width <= 0 || height <= 0 || width > MAX_RFC2435_DIMENSION || + height > MAX_RFC2435_DIMENSION || q0.size() != Q_TABLE_SIZE || q1.size() != Q_TABLE_SIZE) { + logger_.warn("Skipping invalid JPEG frame: valid={}, size={} bytes, {}x{}, q0={}, q1={}", + header.is_valid(), frame_data.size(), width, height, q0.size(), q1.size()); + return {}; + } + + // Keep the original JPEG bytes in the payload so the depacketizer can + // preserve encoder-specific headers/tables when needed. + size_t header_size = header.size(); + auto payload_data = frame_data; + size_t data_size = payload_data.size(); + + logger_.debug("Packetizing JPEG frame: {}x{}, header={} bytes, scan={} bytes", width, height, + header_size, data_size); + + // Calculate how much scan data fits in each packet + if (max_payload_size_ < FIRST_PACKET_OVERHEAD) { + logger_.error("max_payload_size ({}) is smaller than MJPEG first-packet overhead ({})", + max_payload_size_, FIRST_PACKET_OVERHEAD); + return {}; + } + size_t first_data_capacity = max_payload_size_ - FIRST_PACKET_OVERHEAD; + size_t other_data_capacity = + max_payload_size_ > OTHER_PACKET_OVERHEAD ? max_payload_size_ - OTHER_PACKET_OVERHEAD : 0; + size_t first_chunk_size = std::min(data_size, first_data_capacity); + size_t remaining = data_size - first_chunk_size; + size_t num_additional = (other_data_capacity > 0 && remaining > 0) + ? (remaining + other_data_capacity - 1) / other_data_capacity + : 0; + size_t total_packets = 1 + num_additional; + + std::vector chunks; + chunks.reserve(total_packets); + + size_t data_offset = 0; + + for (size_t i = 0; i < total_packets; i++) { + RtpPayloadChunk chunk; + + if (i == 0) { + // First packet: MJPEG header + Q table header + Q tables + scan data + chunk.data.resize(FIRST_PACKET_OVERHEAD + first_chunk_size); + size_t pos = 0; + + // MJPEG header (8 bytes per RFC 2435) + chunk.data[pos++] = 0; // type_specific + chunk.data[pos++] = 0; // fragment offset [23:16] + chunk.data[pos++] = 0; // fragment offset [15:8] + chunk.data[pos++] = 0; // fragment offset [7:0] + chunk.data[pos++] = 0; // fragment type + chunk.data[pos++] = 128; // q (128 = includes Q tables) + chunk.data[pos++] = static_cast(width / 8); // width / 8 + chunk.data[pos++] = static_cast(height / 8); // height / 8 + + // Quantization table header (4 bytes) + chunk.data[pos++] = 0; // MBZ + chunk.data[pos++] = 0; // precision + chunk.data[pos++] = 0; // length (high byte) + chunk.data[pos++] = static_cast(NUM_Q_TABLES * Q_TABLE_SIZE); // length (low byte) + + // Quantization tables (2 x 64 bytes) + std::memcpy(chunk.data.data() + pos, q0.data(), Q_TABLE_SIZE); + pos += Q_TABLE_SIZE; + std::memcpy(chunk.data.data() + pos, q1.data(), Q_TABLE_SIZE); + pos += Q_TABLE_SIZE; + + // Scan data + std::memcpy(chunk.data.data() + pos, payload_data.data() + data_offset, first_chunk_size); + data_offset += first_chunk_size; + } else { + // Subsequent packets: MJPEG header + scan data (no Q tables) + size_t chunk_data_size = std::min(remaining, other_data_capacity); + remaining -= chunk_data_size; + + chunk.data.resize(OTHER_PACKET_OVERHEAD + chunk_data_size); + size_t pos = 0; + + // MJPEG header (8 bytes) + chunk.data[pos++] = 0; // type_specific + chunk.data[pos++] = static_cast((data_offset >> 16) & 0xff); // offset [23:16] + chunk.data[pos++] = static_cast((data_offset >> 8) & 0xff); // offset [15:8] + chunk.data[pos++] = static_cast(data_offset & 0xff); // offset [7:0] + chunk.data[pos++] = 0; // fragment type + chunk.data[pos++] = 96; // q (no Q tables) + chunk.data[pos++] = static_cast(width / 8); // width / 8 + chunk.data[pos++] = static_cast(height / 8); // height / 8 + + // Scan data + std::memcpy(chunk.data.data() + pos, payload_data.data() + data_offset, chunk_data_size); + data_offset += chunk_data_size; + } + + // Last packet gets the marker bit + if (i == total_packets - 1) { + chunk.marker = true; + } + + chunks.push_back(std::move(chunk)); + } + + logger_.debug("Produced {} payload chunks", chunks.size()); + return chunks; +} + +int MjpegPacketizer::get_payload_type() const { return 26; } + +uint32_t MjpegPacketizer::get_clock_rate() const { return 90000; } + +std::string MjpegPacketizer::get_sdp_media_line() const { return "m=video 0 RTP/AVP 26"; } + +std::string MjpegPacketizer::get_sdp_media_attributes() const { return "a=rtpmap:26 JPEG/90000"; } + +} // namespace espp diff --git a/components/rtsp/src/rtsp_client.cpp b/components/rtsp/src/rtsp_client.cpp index 38fca8e31..6c2d0a44c 100644 --- a/components/rtsp/src/rtsp_client.cpp +++ b/components/rtsp/src/rtsp_client.cpp @@ -1,7 +1,164 @@ #include "rtsp_client.hpp" +#include +#include +#include +#include + +#include "generic_depacketizer.hpp" +#include "h264_depacketizer.hpp" + using namespace espp; +namespace { +constexpr auto monitor_poll_interval = std::chrono::milliseconds(250); + +auto now_tick() { return std::chrono::steady_clock::now().time_since_epoch().count(); } + +std::string trim_copy(std::string_view value) { + auto is_space = [](unsigned char c) { return std::isspace(c) != 0; }; + while (!value.empty() && is_space(static_cast(value.front()))) { + value.remove_prefix(1); + } + while (!value.empty() && is_space(static_cast(value.back()))) { + value.remove_suffix(1); + } + return std::string(value); +} + +std::string normalize_rtsp_path(std::string_view path) { + if (path.empty()) { + return "/"; + } + if (path.starts_with("rtsp://")) { + return std::string(path); + } + + while (!path.empty() && path.front() == '/') { + path.remove_prefix(1); + } + if (path.empty()) { + return "/"; + } + return "/" + std::string(path); +} + +std::string make_rtsp_url(std::string_view server_address, int port, std::string_view path) { + if (path.starts_with("rtsp://")) { + return std::string(path); + } + return "rtsp://" + std::string(server_address) + ":" + std::to_string(port) + + normalize_rtsp_path(path); +} + +std::string get_response_header(std::string_view response, std::string_view header_name) { + auto header = std::string(header_name) + ": "; + auto pos = response.find(header); + if (pos == std::string_view::npos) { + return {}; + } + pos += header.size(); + auto end = response.find("\r\n", pos); + if (end == std::string_view::npos) { + return {}; + } + return trim_copy(response.substr(pos, end - pos)); +} + +std::string get_response_body(std::string_view response) { + auto body_pos = response.find("\r\n\r\n"); + if (body_pos == std::string_view::npos) { + return {}; + } + return std::string(response.substr(body_pos + 4)); +} + +std::string get_rtsp_origin(std::string_view url) { + if (!url.starts_with("rtsp://")) { + return {}; + } + auto path_pos = url.find('/', 7); + if (path_pos == std::string_view::npos) { + return std::string(url); + } + return std::string(url.substr(0, path_pos)); +} + +std::string join_rtsp_url(std::string_view base, std::string_view suffix) { + auto trimmed_suffix = trim_copy(suffix); + if (trimmed_suffix.empty()) { + return std::string(base); + } + if (trimmed_suffix.starts_with("rtsp://")) { + return trimmed_suffix; + } + if (trimmed_suffix.front() == '/') { + auto origin = get_rtsp_origin(base); + if (!origin.empty()) { + return origin + trimmed_suffix; + } + return trimmed_suffix; + } + + std::string joined(base); + while (!joined.empty() && joined.back() == '/') { + joined.pop_back(); + } + return joined + "/" + trimmed_suffix; +} + +bool iequals(std::string_view lhs, std::string_view rhs) { + if (lhs.size() != rhs.size()) { + return false; + } + for (size_t i = 0; i < lhs.size(); ++i) { + if (std::tolower(static_cast(lhs[i])) != + std::tolower(static_cast(rhs[i]))) { + return false; + } + } + return true; +} + +bool parse_decimal_int(std::string_view text, int &value) { + if (text.empty()) { + return false; + } + int parsed = 0; + for (char c : text) { + if (c < '0' || c > '9') { + return false; + } + int digit = c - '0'; + if (parsed > (std::numeric_limits::max() - digit) / 10) { + return false; + } + parsed = parsed * 10 + digit; + } + value = parsed; + return true; +} + +int parse_track_id(std::string_view control_path, int fallback_track_id) { + auto pos = control_path.find("trackID="); + if (pos == std::string_view::npos) { + return fallback_track_id; + } + pos += 8; + auto end = control_path.find_first_not_of("0123456789", pos); + auto id_string = control_path.substr(pos, end == std::string_view::npos ? end : end - pos); + if (id_string.empty()) { + return fallback_track_id; + } + int track_id = 0; + if (!parse_decimal_int(id_string, track_id)) { + return fallback_track_id; + } + return track_id; +} + +} // namespace + RtspClient::RtspClient(const Config &config) : BaseComponent("RtspClient", config.log_level) , server_address_(config.server_address) @@ -10,8 +167,28 @@ RtspClient::RtspClient(const Config &config) , rtp_socket_({.log_level = espp::Logger::Verbosity::WARN}) , rtcp_socket_({.log_level = espp::Logger::Verbosity::WARN}) , on_jpeg_frame_(config.on_jpeg_frame) + , on_frame_(config.on_frame) + , on_connection_lost_(config.on_connection_lost) , cseq_(0) - , path_("rtsp://" + server_address_ + ":" + std::to_string(rtsp_port_) + config.path) {} + , path_(make_rtsp_url(server_address_, rtsp_port_, config.path)) + , base_path_(path_) { + if (on_jpeg_frame_ || on_frame_) { + auto mjpeg_depacker = std::make_shared(MjpegDepacketizer::Config{}); + if (on_jpeg_frame_) { + mjpeg_depacker->set_jpeg_frame_callback(on_jpeg_frame_); + } + if (on_frame_) { + mjpeg_depacker->set_frame_callback([this](std::vector &&data) { + int track_id = 0; + if (auto it = payload_type_to_track_id_.find(26); it != payload_type_to_track_id_.end()) { + track_id = it->second; + } + on_frame_(track_id, std::move(data)); + }); + } + depacketizers_[26] = mjpeg_depacker; + } +} RtspClient::~RtspClient() { std::error_code ec; @@ -21,14 +198,106 @@ RtspClient::~RtspClient() { } } +void RtspClient::reset_transport_state() { + playing_ = false; + rtp_socket_.stop_receiving(); + rtcp_socket_.stop_receiving(); + rtsp_socket_.close(); + rtsp_socket_.reinit(); + session_id_.clear(); + tracks_.clear(); + payload_type_to_track_id_.clear(); + base_path_ = path_; + video_port_ = 0; + video_payload_type_ = 0; + cseq_ = 0; + play_started_tick_.store(0, std::memory_order_relaxed); + last_rtp_packet_tick_.store(0, std::memory_order_relaxed); +} + +void RtspClient::start_monitor_task() { + stop_monitor_task(); + using namespace std::placeholders; + monitor_task_ = std::make_unique(Task::Config{ + .callback = std::bind(&RtspClient::monitor_task_fn, this, _1, _2, _3), + .task_config = + { + .name = "RtspClientMon", + .stack_size_bytes = 4 * 1024, + }, + .log_level = Logger::Verbosity::WARN, + }); + monitor_task_->start(); +} + +void RtspClient::stop_monitor_task() { + if (monitor_task_ && monitor_task_->is_started()) { + monitor_task_->stop(); + } + monitor_task_.reset(); +} + +bool RtspClient::monitor_task_fn(std::mutex &m, std::condition_variable &cv, bool &task_notified) { + { + std::unique_lock lk(m); + auto stop_requested = + cv.wait_for(lk, monitor_poll_interval, [&task_notified] { return task_notified; }); + task_notified = false; + if (stop_requested) { + return true; + } + } + + if (disconnecting_.load(std::memory_order_relaxed) || !playing_.load(std::memory_order_relaxed)) { + return false; + } + + if (!rtsp_socket_.is_connected()) { + notify_connection_lost("RTSP control socket disconnected"); + return true; + } + + auto last_tick = last_rtp_packet_tick_.load(std::memory_order_relaxed); + if (last_tick == 0) { + auto play_start_tick = play_started_tick_.load(std::memory_order_relaxed); + if (play_start_tick != 0) { + auto now = std::chrono::steady_clock::now().time_since_epoch().count(); + auto timeout_ticks = initial_rtp_receive_timeout_.count(); + if (timeout_ticks > 0 && now - play_start_tick > timeout_ticks) { + notify_connection_lost("Timed out waiting for initial RTP packets"); + return true; + } + } + return false; + } + auto now = std::chrono::steady_clock::now().time_since_epoch().count(); + auto timeout_ticks = rtp_receive_timeout_.count(); + if (timeout_ticks > 0 && now - last_tick > timeout_ticks) { + notify_connection_lost("Timed out waiting for RTP packets"); + return true; + } + return false; +} + +void RtspClient::notify_connection_lost(std::string_view reason) { + if (disconnecting_.load(std::memory_order_relaxed) || + connection_lost_reported_.exchange(true, std::memory_order_relaxed)) { + return; + } + logger_.warn("{}", reason); + reset_transport_state(); + if (on_connection_lost_) { + on_connection_lost_(); + } +} + std::string RtspClient::send_request(const std::string &method, const std::string &path, const std::unordered_map &extra_headers, std::error_code &ec) { - // send the request std::string request = method + " " + path + " RTSP/1.0\r\n"; request += "CSeq: " + std::to_string(cseq_) + "\r\n"; - if (session_id_.size() > 0) { + if (!session_id_.empty()) { request += "Session: " + session_id_ + "\r\n"; } for (auto &[key, value] : extra_headers) { @@ -37,17 +306,18 @@ RtspClient::send_request(const std::string &method, const std::string &path, request += "User-Agent: rtsp-client\r\n"; request += "Accept: application/sdp\r\n"; request += "\r\n"; + std::string response; auto transmit_config = espp::TcpSocket::TransmitConfig{ .wait_for_response = true, - .response_size = 1024, + .response_size = 16 * 1024, .on_response_callback = [&response](auto &response_vector) { response.assign(response_vector.begin(), response_vector.end()); }, .response_timeout = std::chrono::seconds(5), }; - // NOTE: now this call blocks until the response is received + logger_.debug("Request:\n{}", request); if (!rtsp_socket_.transmit(request, transmit_config)) { ec = std::make_error_code(std::errc::io_error); @@ -55,12 +325,6 @@ RtspClient::send_request(const std::string &method, const std::string &path, return {}; } - // TODO: how to keep receiving until we get the full response? - // if (response.find("\r\n\r\n") != std::string::npos) { - // break; - // } - - // parse the response logger_.debug("Response:\n{}", response); if (parse_response(response, ec)) { return response; @@ -69,11 +333,15 @@ RtspClient::send_request(const std::string &method, const std::string &path, } void RtspClient::connect(std::error_code &ec) { - // exit early if error code is already set if (ec) { return; } + disconnecting_ = false; + connection_lost_reported_ = false; + playing_ = false; + play_started_tick_.store(0, std::memory_order_relaxed); + last_rtp_packet_tick_.store(0, std::memory_order_relaxed); rtsp_socket_.reinit(); auto did_connect = rtsp_socket_.connect({ .ip_address = server_address_, @@ -85,71 +353,204 @@ void RtspClient::connect(std::error_code &ec) { return; } - // send the options request send_request("OPTIONS", "*", {}, ec); } void RtspClient::disconnect(std::error_code &ec) { - // send the teardown request - teardown(ec); - rtsp_socket_.reinit(); + disconnecting_ = true; + stop_monitor_task(); + std::error_code teardown_ec; + if (rtsp_socket_.is_connected()) { + teardown(teardown_ec); + } + reset_transport_state(); + disconnecting_ = false; + connection_lost_reported_ = false; + + if (!ec && teardown_ec) { + ec = teardown_ec; + } } void RtspClient::describe(std::error_code &ec) { - // exit early if the error code is set if (ec) { return; } - // send the describe request + auto response = send_request("DESCRIBE", path_, {}, ec); if (ec) { return; } - // sdp response is of the form: - // std::regex sdp_regex("m=video (\\d+) RTP/AVP (\\d+)"); - // parse the sdp response and get the video port without using regex - // this is a very simple sdp parser that only works for this specific case - auto sdp_start = response.find("m=video"); - if (sdp_start == std::string::npos) { + + tracks_.clear(); + payload_type_to_track_id_.clear(); + base_path_ = path_; + + auto content_base = get_response_header(response, "Content-Base"); + if (!content_base.empty()) { + base_path_ = content_base; + } + + auto sdp = get_response_body(response); + if (sdp.empty()) { ec = std::make_error_code(std::errc::wrong_protocol_type); logger_.error("Invalid sdp"); return; } - auto sdp_end = response.find("\r\n", sdp_start); - if (sdp_end == std::string::npos) { - ec = std::make_error_code(std::errc::protocol_error); - logger_.error("Incomplete sdp"); - return; + + std::istringstream stream(sdp); + std::string line; + TrackInfo *current_track = nullptr; + int next_track_id = 0; + + while (std::getline(stream, line)) { + auto trimmed = trim_copy(line); + if (trimmed.empty()) { + continue; + } + + if (trimmed.starts_with("m=")) { + auto first_space = trimmed.find(' '); + auto last_space = trimmed.rfind(' '); + if (first_space == std::string::npos || last_space == std::string::npos || + first_space == last_space) { + continue; + } + + TrackInfo track; + track.media_type = trimmed.substr(2, first_space - 2); + if (!parse_decimal_int(trimmed.substr(last_space + 1), track.payload_type)) { + logger_.warn("Skipping SDP media line with invalid payload type: {}", trimmed); + continue; + } + track.track_id = next_track_id++; + tracks_.push_back(std::move(track)); + current_track = &tracks_.back(); + continue; + } + + if (trimmed.starts_with("a=control:")) { + auto control_value = trimmed.substr(10); + auto resolved_path = join_rtsp_url(base_path_, control_value); + if (current_track) { + current_track->control_path = resolved_path; + current_track->track_id = parse_track_id(resolved_path, current_track->track_id); + } else { + base_path_ = resolved_path; + } + continue; + } + + if (trimmed.starts_with("a=rtpmap:") && current_track) { + auto payload_space = trimmed.find(' '); + if (payload_space == std::string::npos) { + continue; + } + auto slash = trimmed.find('/', payload_space + 1); + if (slash == std::string::npos) { + continue; + } + int payload_type = 0; + if (!parse_decimal_int(trimmed.substr(9, payload_space - 9), payload_type)) { + logger_.warn("Skipping malformed rtpmap payload type: {}", trimmed); + continue; + } + if (payload_type != current_track->payload_type) { + continue; + } + current_track->encoding_name = trimmed.substr(payload_space + 1, slash - payload_space - 1); + auto clock_rate_start = slash + 1; + auto channels_slash = trimmed.find('/', clock_rate_start); + auto clock_rate_length = channels_slash == std::string::npos + ? std::string::npos + : channels_slash - clock_rate_start; + auto clock_rate_text = trimmed.substr(clock_rate_start, clock_rate_length); + if (!clock_rate_text.empty() && + !parse_decimal_int(clock_rate_text, current_track->clock_rate)) { + logger_.warn("Ignoring malformed clock rate in rtpmap: {}", trimmed); + } + if (channels_slash != std::string::npos) { + auto channels_text = trimmed.substr(channels_slash + 1); + if (!channels_text.empty() && !parse_decimal_int(channels_text, current_track->channels)) { + logger_.warn("Ignoring malformed channel count in rtpmap: {}", trimmed); + } + } + } } - auto sdp = response.substr(sdp_start, sdp_end - sdp_start); - auto port_start = sdp.find(" "); - if (port_start == std::string::npos) { - ec = std::make_error_code(std::errc::protocol_error); - logger_.error("Could not find port start"); + + if (tracks_.empty()) { + ec = std::make_error_code(std::errc::wrong_protocol_type); + logger_.error("Invalid sdp"); return; } - auto port_end = sdp.find(" ", port_start + 1); - if (port_end == std::string::npos) { - ec = std::make_error_code(std::errc::protocol_error); - logger_.error("Could not find port end"); - return; + + for (auto &track : tracks_) { + if (track.control_path.empty()) { + track.control_path = base_path_; + } + payload_type_to_track_id_[track.payload_type] = track.track_id; } - auto port = sdp.substr(port_start + 1, port_end - port_start - 1); - video_port_ = std::stoi(port); - logger_.debug("Video port: {}", video_port_); - auto payload_type_start = sdp.find(" ", port_end + 1); - if (payload_type_start == std::string::npos) { - ec = std::make_error_code(std::errc::protocol_error); - logger_.error("Could not find payload type start"); - return; + + video_port_ = 0; + video_payload_type_ = tracks_.front().payload_type; + if (auto video_track_it = + std::find_if(tracks_.begin(), tracks_.end(), + [](const auto &track) { return iequals(track.media_type, "video"); }); + video_track_it != tracks_.end()) { + video_payload_type_ = video_track_it->payload_type; + } + + for (const auto &track : tracks_) { + if (depacketizers_.contains(track.payload_type)) { + continue; + } + + if ((track.payload_type == 26 || iequals(track.encoding_name, "JPEG")) && + (on_jpeg_frame_ || on_frame_)) { + auto mjpeg_depacketizer = std::make_shared(MjpegDepacketizer::Config{}); + if (on_jpeg_frame_) { + mjpeg_depacketizer->set_jpeg_frame_callback(on_jpeg_frame_); + } + if (on_frame_) { + auto payload_type = track.payload_type; + mjpeg_depacketizer->set_frame_callback([this, payload_type](std::vector &&data) { + int track_id = 0; + if (auto it = payload_type_to_track_id_.find(payload_type); + it != payload_type_to_track_id_.end()) { + track_id = it->second; + } + on_frame_(track_id, std::move(data)); + }); + } + depacketizers_[track.payload_type] = std::move(mjpeg_depacketizer); + continue; + } + + if (!on_frame_) { + continue; + } + + std::shared_ptr depacketizer; + if (iequals(track.encoding_name, "H264")) { + depacketizer = std::make_shared(H264Depacketizer::Config{}); + } else { + depacketizer = std::make_shared(GenericDepacketizer::Config{}); + } + + auto payload_type = track.payload_type; + depacketizer->set_frame_callback([this, payload_type](std::vector &&data) { + int track_id = 0; + if (auto it = payload_type_to_track_id_.find(payload_type); + it != payload_type_to_track_id_.end()) { + track_id = it->second; + } + on_frame_(track_id, std::move(data)); + }); + depacketizers_[track.payload_type] = std::move(depacketizer); } - auto payload_type = sdp.substr(payload_type_start + 1, sdp.size() - payload_type_start - 1); - video_payload_type_ = std::stoi(payload_type); - logger_.debug("Video payload type: {}", video_payload_type_); } void RtspClient::setup(std::error_code &ec) { - // default to rtp and rtcp client ports 5000 and 5001 using namespace std::chrono_literals; static constexpr size_t rtp_port = 5000; static constexpr size_t rtcp_port = 5001; @@ -159,17 +560,24 @@ void RtspClient::setup(std::error_code &ec) { void RtspClient::setup(size_t rtp_port, size_t rtcp_port, const std::chrono::duration &receive_timeout, std::error_code &ec) { - // exit early if the error code is set if (ec) { return; } - // set up the transport header with the rtp and rtcp ports auto transport_header = "RTP/AVP;unicast;client_port=" + std::to_string(rtp_port) + "-" + std::to_string(rtcp_port); - // send the setup request (no response is expected) - send_request("SETUP", path_, {{"Transport", transport_header}}, ec); + if (tracks_.empty()) { + send_request("SETUP", path_, {{"Transport", transport_header}}, ec); + } else { + for (const auto &track : tracks_) { + auto setup_path = track.control_path.empty() ? base_path_ : track.control_path; + send_request("SETUP", setup_path, {{"Transport", transport_header}}, ec); + if (ec) { + return; + } + } + } if (ec) { return; } @@ -178,35 +586,45 @@ void RtspClient::setup(size_t rtp_port, size_t rtcp_port, init_rtcp(rtcp_port, receive_timeout, ec); } +void RtspClient::add_depacketizer(int payload_type, std::shared_ptr depacketizer) { + depacketizers_[payload_type] = std::move(depacketizer); +} + void RtspClient::play(std::error_code &ec) { - // exit early if the error code is set if (ec) { return; } - // send the play request - send_request("PLAY", path_, {}, ec); + send_request("PLAY", base_path_, {}, ec); + if (!ec) { + playing_ = true; + connection_lost_reported_ = false; + play_started_tick_.store(now_tick(), std::memory_order_relaxed); + last_rtp_packet_tick_.store(0, std::memory_order_relaxed); + start_monitor_task(); + } } void RtspClient::pause(std::error_code &ec) { - // exit early if the error code is set if (ec) { return; } - // send the pause request - send_request("PAUSE", path_, {}, ec); + send_request("PAUSE", base_path_, {}, ec); + if (!ec) { + playing_ = false; + stop_monitor_task(); + } } void RtspClient::teardown(std::error_code &ec) { - // exit early if the error code is set - if (ec) { + playing_ = false; + stop_monitor_task(); + if (!rtsp_socket_.is_connected()) { return; } - // send the teardown request - send_request("TEARDOWN", path_, {}, ec); + send_request("TEARDOWN", base_path_, {}, ec); } bool RtspClient::parse_response(const std::string &response_data, std::error_code &ec) { - // exit early if the error code is set if (ec) { return false; } @@ -215,41 +633,48 @@ bool RtspClient::parse_response(const std::string &response_data, std::error_cod logger_.error("Empty response"); return false; } - // RTP response is of the form: - // std::regex response_regex("RTSP/1.0 (\\d+) (.*)\r\n(.*)\r\n\r\n"); - // parse the response but don't use regex since it may be slow on embedded platforms - // make sure it matches the expected response format if (!response_data.starts_with("RTSP/1.0")) { ec = std::make_error_code(std::errc::protocol_error); logger_.error("Invalid response: '{}'", response_data); return false; } - // parse the status code and message - int status_code = std::stoi(response_data.substr(9, 3)); + + int status_code = 0; + if (response_data.size() < 12 || + !parse_decimal_int(std::string_view(response_data).substr(9, 3), status_code)) { + ec = std::make_error_code(std::errc::protocol_error); + logger_.error("Invalid RTSP status line: '{}'", response_data); + return false; + } std::string status_message = response_data.substr(13, response_data.find("\r\n") - 13); if (status_code != 200) { ec = std::make_error_code(std::errc::protocol_error); logger_.error(std::string("Request failed: ") + status_message); return false; } - // parse the session id + auto session_pos = response_data.find("Session: "); if (session_pos != std::string::npos) { session_id_ = response_data.substr(session_pos + 9, response_data.find("\r\n", session_pos) - session_pos - 9); } - // increment the cseq + cseq_++; return true; } void RtspClient::init_rtp(size_t rtp_port, const std::chrono::duration &receive_timeout, std::error_code &ec) { - // exit early if the error code is set if (ec) { return; } logger_.debug("Starting rtp socket"); + auto timeout = + std::chrono::duration_cast(receive_timeout * 3); + auto minimum_timeout = + std::chrono::duration_cast(std::chrono::seconds(15)); + rtp_receive_timeout_ = std::max(timeout, minimum_timeout); + initial_rtp_receive_timeout_ = rtp_receive_timeout_; rtp_socket_.set_receive_timeout(receive_timeout); auto rtp_task_config = espp::Task::BaseConfig{ .name = "Rtp", @@ -270,7 +695,6 @@ void RtspClient::init_rtp(size_t rtp_port, const std::chrono::duration &r void RtspClient::init_rtcp(size_t rtcp_port, const std::chrono::duration &receive_timeout, std::error_code &ec) { - // exit early if the error code is set if (ec) { return; } @@ -295,65 +719,25 @@ void RtspClient::init_rtcp(size_t rtcp_port, const std::chrono::duration std::optional> RtspClient::handle_rtp_packet(std::vector &data, const espp::Socket::Info &sender_info) { - // jpeg frame that we are building - static std::shared_ptr jpeg_frame; - logger_.debug("Got RTP packet of size: {}", data.size()); + play_started_tick_.store(0, std::memory_order_relaxed); + last_rtp_packet_tick_.store(now_tick(), std::memory_order_relaxed); - std::span packet(data.data(), data.size()); - // parse the rtp packet - RtpJpegPacket rtp_jpeg_packet(packet); - auto frag_offset = rtp_jpeg_packet.get_offset(); - if (frag_offset == 0) { - // first fragment - logger_.debug("Received first fragment, size: {}, sequence number: {}", - rtp_jpeg_packet.get_data().size(), rtp_jpeg_packet.get_sequence_number()); - if (jpeg_frame) { - // we already have a frame, this is an error - logger_.warn("Received first fragment but already have a frame"); - jpeg_frame.reset(); - } - logger_.debug("Creating new jpeg frame from rtpjpegpacket of size {} x {} pixels", - rtp_jpeg_packet.get_width(), rtp_jpeg_packet.get_height()); - jpeg_frame = std::make_shared(rtp_jpeg_packet); - logger_.debug("Created new jpeg frame, size: {} x {} pixels", jpeg_frame->get_width(), - jpeg_frame->get_height()); - } else if (jpeg_frame) { - logger_.debug("Received middle fragment, size: {}, sequence number: {}", - rtp_jpeg_packet.get_data().size(), rtp_jpeg_packet.get_sequence_number()); - // middle fragment - jpeg_frame->append(rtp_jpeg_packet); + RtpPacket packet(std::span(data.data(), data.size())); + int pt = packet.get_payload_type(); + + auto it = depacketizers_.find(pt); + if (it != depacketizers_.end()) { + it->second->process_packet(packet); } else { - // we don't have a frame to append to but we got a middle fragment - // this is an error - logger_.warn("Received middle fragment without a frame"); - return {}; + logger_.debug("No depacketizer registered for payload type {}", pt); } - // check if this is the last packet of the frame (the last packet will have - // the marker bit set) - if (jpeg_frame && jpeg_frame->is_complete()) { - // get the jpeg data - auto jpeg_data = jpeg_frame->get_data(); - logger_.debug("Received jpeg frame of size: {} B", jpeg_data.size()); - // call the on_jpeg_frame callback - if (on_jpeg_frame_) { - on_jpeg_frame_(jpeg_frame); - } - // reset the jpeg frame - jpeg_frame.reset(); - logger_.debug("Sent jpeg frame to callback, now jpeg_frame is nullptr? {}", - jpeg_frame == nullptr); - } - // return an empty vector to indicate that we don't want to send a response return {}; } std::optional> RtspClient::handle_rtcp_packet(std::vector &data, const espp::Socket::Info &sender_info) { - // receive the rtcp packet [[maybe_unused]] std::string_view packet(reinterpret_cast(data.data()), data.size()); - // TODO: parse the rtcp packet - // return an empty vector to indicate that we don't want to send a response return {}; } diff --git a/components/rtsp/src/rtsp_server.cpp b/components/rtsp/src/rtsp_server.cpp index f753a7776..af47b35e2 100644 --- a/components/rtsp/src/rtsp_server.cpp +++ b/components/rtsp/src/rtsp_server.cpp @@ -1,24 +1,104 @@ #include "rtsp_server.hpp" +#include + using namespace espp; +namespace { + +std::string normalize_rtsp_path(std::string_view path) { + if (path.empty()) { + return {}; + } + while (!path.empty() && path.front() == '/') { + path.remove_prefix(1); + } + return std::string(path); +} + +std::string make_legacy_mjpeg_sdp(std::string_view session_path, uint32_t session_id, + std::string_view server_address) { + return "v=0\r\n" + "o=- " + + std::to_string(session_id) + " 1 IN IP4 " + std::string(server_address) + + "\r\n" + "s=MJPEG Stream\r\n" + "i=MJPEG Stream\r\n" + "t=0 0\r\n" + "a=control:" + + std::string(session_path) + + "\r\n" + "a=mimetype:string;\"video/x-motion-jpeg\"\r\n" + "m=video 0 RTP/AVP 26\r\n" + "c=IN IP4 0.0.0.0\r\n" + "b=AS:256\r\n" + "a=control:" + + std::string(session_path) + + "\r\n" + "a=udp-only\r\n"; +} + +#if defined(ESP_PLATFORM) +constexpr size_t rtp_send_burst_size = 4; +constexpr auto rtp_send_burst_delay = std::chrono::milliseconds(1); +constexpr auto initial_backpressure_cooldown = std::chrono::milliseconds(100); +constexpr auto max_backpressure_cooldown = std::chrono::milliseconds(1000); +constexpr auto base_capture_period = std::chrono::milliseconds(100); +constexpr auto max_capture_period = std::chrono::milliseconds(2000); +#endif + +constexpr size_t rtp_header_size = 12; +constexpr size_t mjpeg_header_size = 8; +constexpr size_t quant_header_size = 4; +constexpr size_t num_q_tables = 2; +constexpr size_t q_table_size = 64; +constexpr size_t first_packet_overhead = + mjpeg_header_size + quant_header_size + num_q_tables * q_table_size; +constexpr size_t other_packet_overhead = mjpeg_header_size; +constexpr int max_rfc2435_dimension = 255 * 8; + +void serialize_rtp_header(std::vector &packet, int payload_type, uint16_t sequence_number, + uint32_t timestamp, uint32_t ssrc, bool marker) { + packet[0] = 0x80; // version 2 + packet[1] = static_cast((marker ? 0x80 : 0x00) | (payload_type & 0x7f)); + packet[2] = static_cast((sequence_number >> 8) & 0xff); + packet[3] = static_cast(sequence_number & 0xff); + packet[4] = static_cast((timestamp >> 24) & 0xff); + packet[5] = static_cast((timestamp >> 16) & 0xff); + packet[6] = static_cast((timestamp >> 8) & 0xff); + packet[7] = static_cast(timestamp & 0xff); + packet[8] = static_cast((ssrc >> 24) & 0xff); + packet[9] = static_cast((ssrc >> 16) & 0xff); + packet[10] = static_cast((ssrc >> 8) & 0xff); + packet[11] = static_cast(ssrc & 0xff); +} + +uint32_t get_elapsed_timestamp(uint32_t clock_rate) { + static auto start_time = std::chrono::steady_clock::now(); + auto now = std::chrono::steady_clock::now(); + auto elapsed_ms = std::chrono::duration_cast(now - start_time).count(); + return static_cast(elapsed_ms * clock_rate / 1000); +} + +} // namespace + RtspServer::RtspServer(const Config &config) : BaseComponent("RTSP Server", config.log_level) , server_address_(config.server_address) , port_(config.port) - , path_(config.path) + , path_(normalize_rtsp_path(config.path)) , rtsp_socket_({.log_level = espp::Logger::Verbosity::WARN}) - , max_data_size_(config.max_data_size) { - // generate a random ssrc -#if defined(ESP_PLATFORM) - ssrc_ = esp_random(); -#else - std::random_device rd; - std::mt19937 gen(rd()); - std::uniform_int_distribution dis; - ssrc_ = dis(gen); -#endif -} + , max_data_size_(config.max_data_size) + , accept_task_stack_size_bytes_(config.accept_task_stack_size_bytes == 0 + ? Config::default_accept_task_stack_size_bytes + : config.accept_task_stack_size_bytes) + , session_task_stack_size_bytes_(config.session_task_stack_size_bytes == 0 + ? Config::default_session_task_stack_size_bytes + : config.session_task_stack_size_bytes) + , control_task_stack_size_bytes_( + config.control_task_stack_size_bytes == 0 + ? RtspSession::Config::default_control_task_stack_size_bytes + : config.control_task_stack_size_bytes) {} RtspServer::~RtspServer() { stop(); } @@ -34,9 +114,7 @@ bool RtspServer::start(const std::chrono::duration &accept_timeout) { logger_.info("Starting RTSP server on port {}", port_); - // ensure the receive timeout is set so that the accept will not block - // indefinitely and the accept task can be stopped. - rtsp_socket_.set_receive_timeout(accept_timeout); + accept_timeout_ = std::chrono::duration_cast(accept_timeout); if (!rtsp_socket_.bind(port_)) { logger_.error("Failed to bind to port {}", port_); @@ -55,16 +133,23 @@ bool RtspServer::start(const std::chrono::duration &accept_timeout) { .task_config = { .name = "RTSP Accept Task", - .stack_size_bytes = 6 * 1024, + .stack_size_bytes = accept_task_stack_size_bytes_, }, .log_level = espp::Logger::Verbosity::WARN, }); - accept_task_->start(); + if (!accept_task_->start()) { + logger_.error("Failed to start RTSP accept task"); + accept_task_.reset(); + rtsp_socket_.close(); + return false; + } return true; } void RtspServer::stop() { logger_.info("Stopping RTSP server"); + // close the listening socket first so any blocking accept() returns + rtsp_socket_.close(); // stop the accept task if (accept_task_) { accept_task_->stop(); @@ -75,96 +160,278 @@ void RtspServer::stop() { } // clear the list of sessions sessions_.clear(); - // close the RTSP socket - rtsp_socket_.close(); } -void RtspServer::send_frame(const espp::JpegFrame &frame) { - // get the frame scan data - auto frame_header = frame.get_header(); - auto frame_data = frame.get_data(); +void RtspServer::add_track(const TrackConfig &config) { + auto state = std::make_shared(); + state->track_id = config.track_id; + state->packetizer = config.packetizer; + state->ssrc = generate_ssrc(); + tracks_.push_back(state); + logger_.info("Added track {} with SSRC {}", config.track_id, state->ssrc); +} + +bool RtspServer::has_active_sessions() { + std::lock_guard lk(session_mutex_); + for (const auto &[sid, session_ptr] : sessions_) { + if (session_ptr && session_ptr->is_active() && !session_ptr->is_closed()) { + return true; + } + } + return false; +} + +std::chrono::milliseconds RtspServer::get_capture_cooldown() { + std::lock_guard lk(session_mutex_); +#if defined(ESP_PLATFORM) + auto now = std::chrono::steady_clock::now(); + if (backpressure_until_ > now) { + return std::chrono::duration_cast(backpressure_until_ - now); + } +#endif + return std::chrono::milliseconds(0); +} + +std::chrono::milliseconds RtspServer::get_recommended_capture_period() { + std::lock_guard lk(session_mutex_); +#if defined(ESP_PLATFORM) + auto period = base_capture_period; + for (size_t i = 0; i < consecutive_backpressure_failures_; i++) { + period = std::min(period * 2, max_capture_period); + if (period == max_capture_period) { + break; + } + } + auto now = std::chrono::steady_clock::now(); + if (backpressure_until_ > now) { + period = std::max( + period, std::chrono::duration_cast(backpressure_until_ - now)); + } + return period; +#else + return std::chrono::milliseconds(0); +#endif +} + +void RtspServer::send_frame(int track_id, std::span frame_data) { + if (!has_active_sessions()) { + return; + } + + // Find the track + auto track_it = std::find_if(tracks_.begin(), tracks_.end(), + [track_id](const auto &t) { return t->track_id == track_id; }); + std::shared_ptr track = track_it != tracks_.end() ? *track_it : nullptr; + if (!track) { + logger_.error("No track with id {} found", track_id); + return; + } + + // Packetize the frame using the track's codec-specific packetizer + auto chunks = track->packetizer->packetize(frame_data); + auto timestamp = get_elapsed_timestamp(track->packetizer->get_clock_rate()); + std::shared_ptr batch; + { + std::lock_guard lock(track->packets_mutex); + if (track->recycled_batch) { + batch = std::move(track->recycled_batch); + } else { + batch = std::make_shared(); + } + } + if (batch->packets.size() < chunks.size()) { + batch->packets.resize(chunks.size()); + } + batch->count = chunks.size(); + + for (size_t i = 0; i < chunks.size(); ++i) { + auto &chunk = chunks[i]; + auto &packet = batch->packets[i]; + packet.resize(rtp_header_size + chunk.data.size()); + serialize_rtp_header(packet, track->packetizer->get_payload_type(), track->sequence_number++, + timestamp, track->ssrc, chunk.marker); + std::memcpy(packet.data() + rtp_header_size, chunk.data.data(), chunk.data.size()); + } + + { + std::lock_guard lock(track->packets_mutex); + if (!track->recycled_batch && track->pending_batch) { + track->recycled_batch = std::move(track->pending_batch); + } + track->pending_batch = std::move(batch); + } +} + +void RtspServer::send_frame(const espp::JpegFrame &frame) { send_frame(frame.get_data()); } + +void RtspServer::send_frame(std::span frame_data) { + if (!has_active_sessions()) { + return; + } + + // Lazily create default MJPEG track for backward compatibility + if (!default_mjpeg_track_created_) { + MjpegPacketizer::Config mjpeg_config; + mjpeg_config.max_payload_size = max_data_size_; + auto mjpeg_packetizer = std::make_shared(mjpeg_config); + add_track(TrackConfig{.track_id = 0, .packetizer = mjpeg_packetizer}); + default_mjpeg_track_created_ = true; + } + + // Find track 0 + auto track_it = + std::find_if(tracks_.begin(), tracks_.end(), [](const auto &t) { return t->track_id == 0; }); + std::shared_ptr track = track_it != tracks_.end() ? *track_it : nullptr; + if (!track) + return; + + if (frame_data.size() < 2) { + logger_.warn("Skipping short JPEG frame ({} bytes)", frame_data.size()); + return; + } + + // Use the legacy RtpJpegPacket-based packetization to preserve the + // exact wire format for existing MJPEG users + JpegHeader frame_header(frame_data); auto width = frame_header.get_width(); auto height = frame_header.get_height(); auto q0 = frame_header.get_quantization_table(0); auto q1 = frame_header.get_quantization_table(1); + if (!frame_header.is_valid() || width <= 0 || height <= 0 || width > max_rfc2435_dimension || + height > max_rfc2435_dimension || q0.size() != q_table_size || q1.size() != q_table_size) { + logger_.warn("Skipping invalid JPEG frame: valid={}, size={} bytes, {}x{}, q0={}, q1={}", + frame_header.is_valid(), frame_data.size(), width, height, q0.size(), q1.size()); + return; + } + // if the frame data is larger than the MTU, then we need to break it up // into multiple RTP packets - size_t num_packets = frame_data.size() / max_data_size_ + 1; + size_t num_packets = + std::max(1, (frame_data.size() + max_data_size_ - 1) / max_data_size_); logger_.debug("Frame data is {} bytes, breaking into {} packets", frame_data.size(), num_packets); + auto timestamp = get_elapsed_timestamp(90000); + std::shared_ptr batch; + { + std::lock_guard lock(track->packets_mutex); + if (track->recycled_batch) { + batch = std::move(track->recycled_batch); + } else { + batch = std::make_shared(); + } + } + if (batch->packets.size() < num_packets) { + batch->packets.resize(num_packets); + } + batch->count = num_packets; - // create num_packets RtpJpegPackets - // The first packet will have the quantization tables, and the last packet - // will have the end of image marker and the marker bit set - std::vector> packets; - packets.reserve(num_packets); for (size_t i = 0; i < num_packets; i++) { - // get the start and end indices for the current packet size_t start_index = i * max_data_size_; size_t end_index = std::min(start_index + max_data_size_, frame_data.size()); - - static const int type_specific = 0; - static const int fragment_type = 0; - int offset = start_index; - - std::unique_ptr packet; - // if this is the first packet, it has the quantization tables - if (i == 0) { - // use the original q value and include the quantization tables - packet = std::make_unique( - type_specific, fragment_type, 128, width, height, q0, q1, - frame_data.subspan(start_index, end_index - start_index)); - } else { - // use a different q value (less than 128) and don't include the - // quantization tables - packet = std::make_unique( - type_specific, offset, fragment_type, 96, width, height, - frame_data.subspan(start_index, end_index - start_index)); + size_t scan_size = end_index - start_index; + bool include_q_tables = i == 0; + size_t payload_size = + (include_q_tables ? first_packet_overhead : other_packet_overhead) + scan_size; + auto &packet = batch->packets[i]; + packet.resize(rtp_header_size + payload_size); + serialize_rtp_header(packet, 26, track->sequence_number++, timestamp, track->ssrc, + i == num_packets - 1); + + size_t pos = rtp_header_size; + packet[pos++] = 0; // type_specific + packet[pos++] = static_cast((start_index >> 16) & 0xff); + packet[pos++] = static_cast((start_index >> 8) & 0xff); + packet[pos++] = static_cast(start_index & 0xff); + packet[pos++] = 0; // fragment type + packet[pos++] = static_cast(include_q_tables ? 128 : 96); + packet[pos++] = static_cast(width / 8); + packet[pos++] = static_cast(height / 8); + + if (include_q_tables) { + packet[pos++] = 0; + packet[pos++] = 0; + packet[pos++] = 0; + packet[pos++] = static_cast(num_q_tables * q_table_size); + std::memcpy(packet.data() + pos, q0.data(), q_table_size); + pos += q_table_size; + std::memcpy(packet.data() + pos, q1.data(), q_table_size); + pos += q_table_size; } - // set the payload type to 26 (JPEG) - packet->set_payload_type(26); - // set the sequence number - packet->set_sequence_number(sequence_number_++); - // set the timestamp - static auto start_time = std::chrono::steady_clock::now(); - auto now = std::chrono::steady_clock::now(); - auto timestamp = - std::chrono::duration_cast(now - start_time).count(); - packet->set_timestamp(timestamp * 90); - - // set the ssrc - packet->set_ssrc(ssrc_); - - // auto mjpeg_header = packet->get_mjpeg_header(); - // std::vector mjpeg_vec(mjpeg_header.begin(), mjpeg_header.end()); - - // if it's the last packet, set the marker bit - if (i == num_packets - 1) { - packet->set_marker(true); + std::memcpy(packet.data() + pos, frame_data.data() + start_index, scan_size); + } + + { + std::lock_guard lock(track->packets_mutex); + if (!track->recycled_batch && track->pending_batch) { + track->recycled_batch = std::move(track->pending_batch); } + track->pending_batch = std::move(batch); + } +} - // make sure the packet header has been serialized - packet->serialize(); +std::string RtspServer::generate_sdp(const std::string &session_path, uint32_t session_id, + const std::string &server_address) const { + if (tracks_.empty()) { + return make_legacy_mjpeg_sdp(session_path, session_id, server_address); + } - // add the packet to the list of packets - packets.emplace_back(std::move(packet)); + std::string sdp; + sdp += "v=0\r\n"; + sdp += fmt::format("o=- {} 1 IN IP4 {}\r\n", session_id, server_address); + sdp += "s=RTSP Stream\r\n"; + sdp += "t=0 0\r\n"; + sdp += fmt::format("a=control:{}\r\n", session_path); + + for (const auto &track : tracks_) { + sdp += track->packetizer->get_sdp_media_line() + "\r\n"; + sdp += "c=IN IP4 0.0.0.0\r\n"; + sdp += "b=AS:256\r\n"; + sdp += track->packetizer->get_sdp_media_attributes() + "\r\n"; + sdp += fmt::format("a=control:{}/trackID={}\r\n", session_path, track->track_id); } - // now move the packets into the rtp_packets_ vector - { - std::unique_lock lock(rtp_packets_mutex_); - // move the new packets into the list - rtp_packets_ = std::move(packets); + return sdp; +} + +void RtspServer::reap_closed_sessions() { + std::lock_guard lk(session_mutex_); + for (auto it = sessions_.begin(); it != sessions_.end();) { + auto &session = it->second; + if (session->is_closed()) { + logger_.info("Removing session {}", session->get_session_id()); + it = sessions_.erase(it); + } else { + ++it; + } } } bool RtspServer::accept_task_function(std::mutex &m, std::condition_variable &cv, bool &task_notified) { + if (!rtsp_socket_.is_valid()) { + return true; + } + + auto num_ready = rtsp_socket_.select(accept_timeout_); + if (num_ready <= 0) { + if (!rtsp_socket_.is_valid()) { + return true; + } + if (task_notified) { + task_notified = false; + return true; + } + return false; + } + // accept a new connection auto control_socket = rtsp_socket_.accept(); if (!control_socket) { + if (!rtsp_socket_.is_valid()) { + return true; + } logger_.info("Failed to accept new connection"); // if we were notified, then we should stop the task if (task_notified) { @@ -176,13 +443,24 @@ bool RtspServer::accept_task_function(std::mutex &m, std::condition_variable &cv } logger_.info("Accepted new connection"); + reap_closed_sessions(); // create a new session auto session = std::make_unique( std::move(control_socket), RtspSession::Config{.server_address = fmt::format("{}:{}", server_address_, port_), .rtsp_path = path_, + .control_task_stack_size_bytes = control_task_stack_size_bytes_, + .sdp_generator = + [this](const std::string &session_path, uint32_t session_id, + const std::string &server_address) { + return generate_sdp(session_path, session_id, server_address); + }, .log_level = session_log_level_}); + if (session->is_closed()) { + logger_.error("Failed to initialize RTSP session {}", session->get_session_id()); + return false; + } // add the session to the list of sessions auto session_id = session->get_session_id(); @@ -200,11 +478,23 @@ bool RtspServer::accept_task_function(std::mutex &m, std::condition_variable &cv .task_config = { .name = "RtspSessionTask", - .stack_size_bytes = 6 * 1024, + .stack_size_bytes = session_task_stack_size_bytes_, }, .log_level = espp::Logger::Verbosity::WARN, }); - session_task_->start(); + if (!session_task_->start()) { + logger_.error("Failed to start RTSP session task"); + session_task_.reset(); + { + std::lock_guard lk(session_mutex_); + auto it = sessions_.find(session_id); + if (it != sessions_.end()) { + it->second->teardown(); + sessions_.erase(it); + } + } + return false; + } } // we do not want to stop the task return task_notified; @@ -223,50 +513,106 @@ bool RtspServer::session_task_function(std::mutex &m, std::condition_variable &c } } - // when this function returns, the vector of pointers will go out of scope - // and the pointers will be deleted (which is good because it means we - // won't send the same frame twice) - std::vector> packets; - { - // copy the rtp packets into a local vector - std::unique_lock lock(rtp_packets_mutex_); - if (rtp_packets_.empty()) { - // if there is not a new frame (no packets), then simply return - // we do not want to stop the task - return false; - } - // move the packets into the local vector - packets = std::move(rtp_packets_); + reap_closed_sessions(); + + // Collect pending packets from all tracks + struct TrackPackets { + std::shared_ptr track; + int track_id; + std::shared_ptr batch; + }; + std::vector all_track_packets; + + for (auto &track : tracks_) { + std::lock_guard lock(track->packets_mutex); + if (!track->pending_batch || track->pending_batch->count == 0) + continue; + all_track_packets.push_back({track, track->track_id, std::move(track->pending_batch)}); } + if (all_track_packets.empty()) { + // no new frames, do not stop the task + return false; + } + +#if defined(ESP_PLATFORM) + auto now = std::chrono::steady_clock::now(); + if (now < backpressure_until_) { + logger_.warn("Skipping {} pending track packet batches while recovering from RTP backpressure", + all_track_packets.size()); + return false; + } +#endif + logger_.debug("Sending frame data to clients"); // for each session in sessions_ // if the session is active // send the latest frame to the client - std::lock_guard lk(session_mutex_); - for (auto &session : sessions_) { - [[maybe_unused]] auto session_id = session.first; - auto &session_ptr = session.second; - // send the packets to the client - for (auto &packet : packets) { - // if the session is not active or is closed, then stop sending - if (!session_ptr->is_active() || session_ptr->is_closed()) { - break; + bool saw_backpressure = false; + bool sent_packets = false; + { + std::lock_guard lk(session_mutex_); + for (auto &[sid, session_ptr] : sessions_) { + if (!session_ptr->is_active() || session_ptr->is_closed()) + continue; + bool send_failed = false; +#if defined(ESP_PLATFORM) + size_t burst_packets_sent = 0; +#endif + for (auto &tp : all_track_packets) { + for (size_t i = 0; i < tp.batch->count; ++i) { + auto &packet = tp.batch->packets[i]; + if (!session_ptr->send_rtp_packet( + tp.track_id, std::span(packet.data(), packet.size()))) { + logger_.warn("Dropping remaining RTP packets for session {} after send backpressure", + session_ptr->get_session_id()); + send_failed = true; + saw_backpressure = true; + break; + } + sent_packets = true; +#if defined(ESP_PLATFORM) + if (++burst_packets_sent >= rtp_send_burst_size) { + burst_packets_sent = 0; + std::this_thread::sleep_for(rtp_send_burst_delay); + } +#endif + } + if (send_failed) + break; + } + } +#if defined(ESP_PLATFORM) + if (saw_backpressure) { + consecutive_backpressure_failures_++; + auto cooldown = initial_backpressure_cooldown; + for (size_t i = 1; i < consecutive_backpressure_failures_; i++) { + cooldown = std::min(cooldown * 2, max_backpressure_cooldown); + if (cooldown == max_backpressure_cooldown) { + break; + } + } + backpressure_until_ = std::chrono::steady_clock::now() + cooldown; + logger_.warn("Applying RTP backpressure cooldown of {} ms", cooldown.count()); + } else if (sent_packets) { + if (consecutive_backpressure_failures_ > 0) { + consecutive_backpressure_failures_--; + } + if (consecutive_backpressure_failures_ == 0) { + backpressure_until_ = {}; } - session_ptr->send_rtp_packet(*packet); } +#endif } - // loop over the sessions and erase ones which are closed - for (auto it = sessions_.begin(); it != sessions_.end();) { - auto &session = it->second; - if (session->is_closed()) { - logger_.info("Removing session {}", session->get_session_id()); - it = sessions_.erase(it); - } else { - ++it; + for (auto &tp : all_track_packets) { + std::lock_guard lock(tp.track->packets_mutex); + if (!tp.track->recycled_batch) { + tp.batch->count = 0; + tp.track->recycled_batch = std::move(tp.batch); } } + reap_closed_sessions(); // we do not want to stop the task return false; diff --git a/components/rtsp/src/rtsp_session.cpp b/components/rtsp/src/rtsp_session.cpp index c7338ddab..705cf2a2a 100644 --- a/components/rtsp/src/rtsp_session.cpp +++ b/components/rtsp/src/rtsp_session.cpp @@ -1,16 +1,50 @@ #include "rtsp_session.hpp" +#include + using namespace espp; +namespace { + +std::string make_rtsp_url(std::string_view server_address, std::string_view path) { + while (!path.empty() && path.front() == '/') { + path.remove_prefix(1); + } + if (path.empty()) { + return "rtsp://" + std::string(server_address); + } + return "rtsp://" + std::string(server_address) + "/" + std::string(path); +} + +bool parse_decimal_int(std::string_view text, int &value) { + if (text.empty()) { + return false; + } + int parsed = 0; + for (char c : text) { + if (c < '0' || c > '9') { + return false; + } + int digit = c - '0'; + if (parsed > (std::numeric_limits::max() - digit) / 10) { + return false; + } + parsed = parsed * 10 + digit; + } + value = parsed; + return true; +} + +} // namespace + RtspSession::RtspSession(std::shared_ptr control_socket, const Config &config) : BaseComponent("RtspSession", config.log_level) , control_socket_(std::move(control_socket)) - , rtp_socket_({.log_level = Logger::Verbosity::WARN}) - , rtcp_socket_({.log_level = Logger::Verbosity::WARN}) , session_id_(generate_session_id()) , server_address_(config.server_address) , rtsp_path_(config.rtsp_path) - , client_address_(control_socket_->get_remote_info().address) { + , client_address_(control_socket_->get_remote_info().address) + , sdp_generator_(config.sdp_generator) { // set the logger tag to include the session id logger_.set_tag("RtspSession " + std::to_string(session_id_)); // ensure there is a timeout on the control socket receive @@ -22,15 +56,24 @@ RtspSession::RtspSession(std::shared_ptr control_socket, const Config .task_config = { .name = "RtspSession " + std::to_string(session_id_), - .stack_size_bytes = 6 * 1024, + .stack_size_bytes = config.control_task_stack_size_bytes == 0 + ? RtspSession::Config::default_control_task_stack_size_bytes + : config.control_task_stack_size_bytes, }, .log_level = Logger::Verbosity::WARN, }); - control_task_->start(); + if (!control_task_->start()) { + logger_.error("Failed to start RTSP control task"); + teardown(); + control_socket_->close(); + } } RtspSession::~RtspSession() { teardown(); + if (control_socket_) { + control_socket_->close(); + } // stop the session task if (control_task_ && control_task_->is_started()) { logger_.info("Stopping control task"); @@ -57,22 +100,54 @@ void RtspSession::teardown() { closed_ = true; } -bool RtspSession::send_rtp_packet(const RtpPacket &packet) { - logger_.debug("Sending RTP packet"); - return rtp_socket_.send(packet.get_data(), { - .ip_address = client_address_, - .port = (size_t)client_rtp_port_, - }); +bool RtspSession::send_rtp_packet(int track_id, const RtpPacket &packet) { + return send_rtp_packet(track_id, packet.get_data()); +} + +bool RtspSession::send_rtp_packet(int track_id, std::span packet_data) { + auto it = tracks_.find(track_id); + if (it == tracks_.end() || !it->second) { + logger_.debug("Skipping RTP packet for unconfigured track {}", track_id); + return true; + } + auto &track = *it->second; + if (track.client_rtp_port <= 0) { + logger_.debug("Skipping RTP packet for track {} without RTP transport", track_id); + return true; + } + logger_.debug("Sending RTP packet on track {}", track_id); + return track.rtp_socket.send(packet_data, { + .ip_address = client_address_, + .port = (size_t)track.client_rtp_port, + }); } -bool RtspSession::send_rtcp_packet(const RtcpPacket &packet) { - logger_.debug("Sending RTCP packet"); - return rtcp_socket_.send(packet.get_data(), { - .ip_address = client_address_, - .port = (size_t)client_rtcp_port_, - }); +bool RtspSession::send_rtp_packet(const RtpPacket &packet) { return send_rtp_packet(0, packet); } + +bool RtspSession::send_rtp_packet(std::span packet_data) { + return send_rtp_packet(0, packet_data); } +bool RtspSession::send_rtcp_packet(int track_id, const RtcpPacket &packet) { + auto it = tracks_.find(track_id); + if (it == tracks_.end() || !it->second) { + logger_.debug("Skipping RTCP packet for unconfigured track {}", track_id); + return true; + } + auto &track = *it->second; + if (track.client_rtcp_port <= 0) { + logger_.debug("Skipping RTCP packet for track {} without RTCP transport", track_id); + return true; + } + logger_.debug("Sending RTCP packet on track {}", track_id); + return track.rtcp_socket.send(packet.get_data(), { + .ip_address = client_address_, + .port = (size_t)track.client_rtcp_port, + }); +} + +bool RtspSession::send_rtcp_packet(const RtcpPacket &packet) { return send_rtcp_packet(0, packet); } + bool RtspSession::send_response(int code, std::string_view message, int sequence_number, std::string_view headers, std::string_view body) { // create a response @@ -118,27 +193,33 @@ bool RtspSession::handle_rtsp_describe(std::string_view request) { // create a response int code = 200; std::string message = "OK"; - // SDP description for an MJPEG stream - std::string rtsp_path = "rtsp://" + server_address_ + "/" + rtsp_path_; - std::string body = "v=0\r\n" // version (0) - "o=- " + - std::to_string(session_id_) + " 1 IN IP4 " + server_address_ + - "\r\n" // username (none), session id, version, network type (internet), - // address type, address - "s=MJPEG Stream\r\n" // session name (can be anything) - "i=MJPEG Stream\r\n" // session name (can be anything) - "t=0 0\r\n" // start / stop - "a=control:" + - rtsp_path + - "\r\n" // the RTSP path - "a=mimetype:string;\"video/x-motion-jpeg\"\r\n" // MIME type - "m=video 0 RTP/AVP 26\r\n" // MJPEG - "c=IN IP4 0.0.0.0\r\n" // client will use the RTSP address - "b=AS:256\r\n" // 256kbps - "a=control:" + - rtsp_path + - "\r\n" - "a=udp-only\r\n"; + std::string rtsp_path = make_rtsp_url(server_address_, rtsp_path_); + + std::string body; + if (sdp_generator_) { + body = sdp_generator_(rtsp_path, session_id_, server_address_); + } else { + // Default SDP description for an MJPEG stream (backward compatibility) + body = "v=0\r\n" // version (0) + "o=- " + + std::to_string(session_id_) + " 1 IN IP4 " + server_address_ + + "\r\n" // username (none), session id, version, network type (internet), + // address type, address + "s=MJPEG Stream\r\n" // session name (can be anything) + "i=MJPEG Stream\r\n" // session name (can be anything) + "t=0 0\r\n" // start / stop + "a=control:" + + rtsp_path + + "\r\n" // the RTSP path + "a=mimetype:string;\"video/x-motion-jpeg\"\r\n" // MIME type + "m=video 0 RTP/AVP 26\r\n" // MJPEG + "c=IN IP4 0.0.0.0\r\n" // client will use the RTSP address + "b=AS:256\r\n" // 256kbps + "a=control:" + + rtsp_path + + "\r\n" + "a=udp-only\r\n"; + } std::string headers = "Content-Type: application/sdp\r\n" "Content-Base: " + @@ -163,9 +244,36 @@ bool RtspSession::handle_rtsp_setup(std::string_view request) { return handle_rtsp_invalid_request(request); } logger_.info("RTSP SETUP request"); - // save the client port numbers - client_rtp_port_ = client_rtp_port; - client_rtcp_port_ = client_rtcp_port; + + // extract track ID from the RTSP URL (e.g., "rtsp://ip:port/path/trackID=0") + int track_id = 0; // default to track 0 for backward compatibility + auto track_id_pos = rtsp_path.find("trackID="); + if (track_id_pos != std::string_view::npos) { + auto track_id_str = rtsp_path.substr(track_id_pos + 8); + auto end_pos = track_id_str.find_first_not_of("0123456789"); + if (end_pos != std::string_view::npos) { + track_id_str = track_id_str.substr(0, end_pos); + } + int parsed_track_id = 0; + if (!track_id_str.empty() && parse_decimal_int(track_id_str, parsed_track_id)) { + track_id = parsed_track_id; + } else if (!track_id_str.empty()) { + logger_.warn("Invalid track ID '{}', defaulting to track 0", track_id_str); + } + } + + // create or find the track + auto &track_ptr = tracks_[track_id]; + if (!track_ptr) { + track_ptr = std::make_unique(); + } + auto &track = *track_ptr; + track.track_id = track_id; + track.control_path = "trackID=" + std::to_string(track_id); + track.client_rtp_port = client_rtp_port; + track.client_rtcp_port = client_rtcp_port; + track.setup_complete = true; + // create a response int code = 200; std::string message = "OK"; @@ -258,7 +366,7 @@ bool RtspSession::handle_rtsp_request(std::string_view request) { } else if (method == "DESCRIBE") { return handle_rtsp_describe(request_body); } else if (method == "SETUP") { - return handle_rtsp_setup(request_body); + return handle_rtsp_setup(request); } else if (method == "PLAY") { return handle_rtsp_play(request_body); } else if (method == "PAUSE") { @@ -341,8 +449,7 @@ bool RtspSession::parse_rtsp_command_sequence(std::string_view request, int &cse return false; } // convert the cseq to an integer - cseq = std::stoi(std::string{cseq_str}); - return true; + return parse_decimal_int(cseq_str, cseq); } std::string_view RtspSession::parse_rtsp_path(std::string_view request) { @@ -393,7 +500,17 @@ bool RtspSession::parse_rtsp_setup_request(std::string_view request, std::string // parse the rtp port from the request auto client_port_index = request.find("client_port="); + if (client_port_index == std::string::npos) { + logger_.error("Failed to parse client_port from request"); + handle_rtsp_invalid_request(request); + return false; + } auto dash_index = request.find('-', client_port_index); + if (dash_index == std::string::npos || dash_index <= client_port_index + 12) { + logger_.error("Failed to parse client RTP/RTCP ports from request"); + handle_rtsp_invalid_request(request); + return false; + } std::string_view rtp_port = request.substr(client_port_index + 12, dash_index - client_port_index - 12); if (rtp_port.empty()) { @@ -402,15 +519,24 @@ bool RtspSession::parse_rtsp_setup_request(std::string_view request, std::string return false; } // parse the rtcp port from the request - std::string_view rtcp_port = - request.substr(dash_index + 1, request.find('\r', client_port_index) - dash_index - 1); + auto rtcp_end_index = request.find('\r', client_port_index); + if (rtcp_end_index == std::string::npos || rtcp_end_index <= dash_index + 1) { + logger_.error("Failed to parse client RTCP port from request"); + handle_rtsp_invalid_request(request); + return false; + } + std::string_view rtcp_port = request.substr(dash_index + 1, rtcp_end_index - dash_index - 1); if (rtcp_port.empty()) { logger_.error("Empty client RTCP port in request"); handle_rtsp_invalid_request(request); return false; } // convert the rtp and rtcp ports to integers - client_rtp_port = std::stoi(std::string{rtp_port}); - client_rtcp_port = std::stoi(std::string{rtcp_port}); + if (!parse_decimal_int(rtp_port, client_rtp_port) || + !parse_decimal_int(rtcp_port, client_rtcp_port)) { + logger_.error("Failed to parse client RTP/RTCP ports from request"); + handle_rtsp_invalid_request(request); + return false; + } return true; } diff --git a/components/socket/README.md b/components/socket/README.md index af446e3f0..f479673f8 100644 --- a/components/socket/README.md +++ b/components/socket/README.md @@ -33,6 +33,13 @@ UDP sockets provide unreliable, unordered communication over IP network sockets. UDP sockets can be used in unicast (point to point), multicast (one to many and many to one), and broadcast (one to all). +The `UdpSocket` API supports both one-shot sends and long-running receive tasks: + +* `send(...)` can optionally wait for a response with a timeout and callback +* `start_receiving(...)` starts a task that continuously receives datagrams and + can optionally send a callback-produced response +* `stop_receiving()` cleanly stops a blocked receive task during teardown + ## TCP Socket TCP sockets provide reliable, ordered communication over IP network sockets and @@ -41,11 +48,21 @@ transmission speed / bandwidth adjustment. TCP sockets cannot be used with multicast (many to one, one to many). -## Example +The `TcpSocket` API covers both client and server patterns: -The [example](./example) shows the use of the classes provided by the `socket` -component, including: +* `connect(...)` plus `transmit(...)` for client-style request/response flows +* optional blocking response waits with callback delivery +* `bind(...)`, `listen(...)`, and `accept()` for server-side flows +* `close()` / `reinit()` helpers for teardown and reconnect paths -* `UdpSocket` (as both `client` and `server`, including unicast and multicast configurations) -* `TcpSocket` (as both `client` and `server`) +## Example +The [example](./example) shows the use of the classes provided by the `socket` +component and runs a scenario-driven self-test which covers teardown, timeout, +and reconnect behavior, including: + +* `UdpSocket` as both `client` and `server`, including unicast and multicast configurations +* `TcpSocket` as both `client` and `server` +* scope-based teardown while tasks are active or blocked +* request/response callbacks and timeout handling +* reconnect behavior after TCP session shutdown diff --git a/components/socket/example/README.md b/components/socket/example/README.md index 0f16fb151..869009be6 100644 --- a/components/socket/example/README.md +++ b/components/socket/example/README.md @@ -1,10 +1,25 @@ # Socket Example -This example shows the use of the classes provided by the `socket` component, -including: - -* `UdpSocket` (as both `client` and `server`, including unicast and multicast configurations) -* `TcpSocket` (as both `client` and `server`) +This example now acts as both a usage demo and a lightweight regression test for +the `socket` component. It runs a sequence of UDP and TCP scenarios and prints a +pass/fail summary at the end. + +The covered scenarios include: + +* UDP unicast client/server messaging with scope-based teardown +* UDP request/response with a larger payload +* UDP multicast request/response +* UDP response timeout when no server is listening +* UDP blocked-receive teardown +* TCP unicast client/server messaging with scope-based teardown +* TCP request/response followed by reconnect +* TCP blocked-accept teardown +* TCP connect failure to an unused port + +At startup the example creates a small open Wi-Fi AP so the network stack is +initialized, but the actual test traffic stays local to the device using +loopback (`127.0.0.1`) and a local multicast group (`239.1.1.1`). No second +device or external server is required. ## How to use example @@ -24,4 +39,5 @@ See the Getting Started Guide for full steps to configure and use ESP-IDF to bui ## Example Output -![output](https://user-images.githubusercontent.com/213467/235313443-d745261e-3e3c-4a7e-87ae-ef525a380e21.png) +The serial log shows each scenario as it starts, a per-scenario pass/fail line, +and a final summary such as `Socket example summary: 9/9 scenarios passed`. diff --git a/components/socket/example/main/socket_example.cpp b/components/socket/example/main/socket_example.cpp index 187b36d93..1f4274731 100644 --- a/components/socket/example/main/socket_example.cpp +++ b/components/socket/example/main/socket_example.cpp @@ -1,8 +1,13 @@ #include +#include #include #include -#include +#include +#include +#include +#include #include +#include #include "logger.hpp" #include "task.hpp" @@ -11,362 +16,525 @@ #include "wifi_ap.hpp" using namespace std::chrono_literals; -using namespace std::placeholders; -extern "C" void app_main(void) { - fmt::print("Stating socket example!\n"); +namespace { +using ByteVector = std::vector; - auto test_duration = 3s; +constexpr auto kLoopbackAddress = "127.0.0.1"; +constexpr auto kMulticastGroup = "239.1.1.1"; +constexpr auto kPollInterval = 10ms; +constexpr auto kTaskInterval = 100ms; +constexpr auto kScenarioTimeout = 2500ms; +constexpr auto kSettleDelay = 50ms; +constexpr size_t kMaxPacketSize = 1536; +constexpr int kMaxConnections = 2; - // create a wifi access point here so that LwIP will be init for this example - espp::WifiAp wifi_ap({.ssid = "SocketExample", - .password = "", // no security - .log_level = espp::Logger::Verbosity::INFO}); +struct ScenarioResult { + std::string name; + bool passed; + std::string detail; +}; + +ByteVector make_payload(size_t size, uint8_t seed = 0) { + ByteVector data(size); + std::iota(data.begin(), data.end(), seed); + return data; +} + +ByteVector reversed(ByteVector data) { + std::reverse(data.begin(), data.end()); + return data; +} + +template +bool wait_until(Predicate &&predicate, std::chrono::milliseconds timeout, + std::chrono::milliseconds poll_interval = kPollInterval) { + auto deadline = std::chrono::steady_clock::now() + timeout; + while (std::chrono::steady_clock::now() < deadline) { + if (predicate()) { + return true; + } + std::this_thread::sleep_for(poll_interval); + } + return predicate(); +} + +bool wait_or_stop(std::mutex &m, std::condition_variable &cv, bool ¬ified, + std::chrono::milliseconds duration = kTaskInterval) { + std::unique_lock lock(m); + auto stop_requested = cv.wait_for(lock, duration, [¬ified] { return notified; }); + if (stop_requested) { + notified = false; + } + return stop_requested; +} + +ScenarioResult pass(std::string name, std::string detail) { + return {.name = std::move(name), .passed = true, .detail = std::move(detail)}; +} + +ScenarioResult fail(std::string name, std::string detail) { + return {.name = std::move(name), .passed = false, .detail = std::move(detail)}; +} - fmt::print(fg(fmt::terminal_color::yellow) | fmt::emphasis::bold, "Staring Basic UDP test.\n"); +void print_scenario_start(std::string_view name) { + fmt::print(fg(fmt::terminal_color::yellow) | fmt::emphasis::bold, "Running: {}\n", name); +} + +void print_scenario_result(const ScenarioResult &result) { + auto color = result.passed ? fmt::terminal_color::green : fmt::terminal_color::red; + auto status = result.passed ? "PASS" : "FAIL"; + fmt::print(fg(color) | fmt::emphasis::bold, "[{}] {}", status, result.name); + if (!result.detail.empty()) { + fmt::print(": {}", result.detail); + } + fmt::print("\n"); +} + +espp::Task::BaseConfig make_task_config(std::string_view name, size_t stack_size = 6 * 1024) { + return {.name = std::string(name), .stack_size_bytes = stack_size}; +} + +ScenarioResult run_udp_unicast_teardown_scenario() { + constexpr size_t port = 5000; + constexpr size_t expected_messages = 3; + std::atomic_size_t received_messages{0}; - // Unicast (client-server) example { //! [UDP Server example] - std::string server_address = "127.0.0.1"; - size_t port = 5000; espp::UdpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); - auto server_task_config = espp::Task::BaseConfig{ - .name = "UdpServer", - .stack_size_bytes = 6 * 1024, - }; + auto server_task_config = make_task_config("UdpServer"); auto server_config = espp::UdpSocket::ReceiveConfig{ .port = port, - .buffer_size = 1024, - .on_receive_callback = - [](auto &data, auto &source) -> auto{fmt::print("Server received: {}\n" - " from source: {}\n", - data, source); + .buffer_size = kMaxPacketSize, + .on_receive_callback = [&received_messages](const auto &data, + const auto &source) -> auto{received_messages++; + fmt::print("UDP server received {} bytes from {}\n", data.size(), source); return std::nullopt; } + , }; server_socket.start_receiving(server_task_config, server_config); //! [UDP Server example] //! [UDP Client example] -espp::UdpSocket client_socket({}); -// create thread for sending data using the socket -auto client_task_fn = [&server_address, &client_socket, &port](auto &, auto &) { - static size_t iterations = 0; - std::vector data{0, 1, 2, 3, 4}; - std::transform(data.begin(), data.end(), data.begin(), - [](const auto &d) { return d + iterations; }); - auto send_config = espp::UdpSocket::SendConfig{.ip_address = server_address, .port = port}; - client_socket.send(data, send_config); - iterations++; - std::this_thread::sleep_for(1s); - // don't want to stop the task - return false; -}; -auto client_task = - espp::Task::make_unique({.callback = client_task_fn, - .task_config = {.name = "Client Task", .stack_size_bytes = 5 * 1024}}); +espp::UdpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); +auto client_task = espp::Task::make_unique({ + .callback = [&client_socket](std::mutex &m, std::condition_variable &cv, + bool ¬ified) -> bool { + static size_t iterations = 0; + auto data = make_payload(32, static_cast(iterations * 8)); + auto send_config = espp::UdpSocket::SendConfig{ + .ip_address = kLoopbackAddress, + .port = port, + }; + client_socket.send(data, send_config); + iterations++; + return wait_or_stop(m, cv, notified); + }, + .task_config = make_task_config("UdpClient", 5 * 1024), +}); client_task->start(); //! [UDP Client example] -// now sleep for a while to let the monitor do its thing -std::this_thread::sleep_for(test_duration); -} -fmt::print(fg(fmt::terminal_color::green) | fmt::emphasis::bold, "Basic UDP test finished.\n"); -std::this_thread::sleep_for(100ms); -fmt::print(fg(fmt::terminal_color::yellow) | fmt::emphasis::bold, - "Staring UDP send waiting for response test.\n"); - -// Unicast (client-server) example with server response -{ - //! [UDP Server Response example] - std::string server_address = "127.0.0.1"; - size_t port = 5000; - espp::UdpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); - auto server_task_config = - espp::Task::BaseConfig{.name = "UdpServer", .stack_size_bytes = 6 * 1024}; - auto server_config = espp::UdpSocket::ReceiveConfig{ - .port = port, - .buffer_size = 1024, - .on_receive_callback = - [](auto &data, auto &source) -> auto{fmt::print("Server received: {}\n" - " from source: {}\n", - data, source); - // reverse the data - std::reverse(data.begin(), data.end()); - // and send it back - return data; +if (!wait_until([&received_messages] { return received_messages.load() >= expected_messages; }, + kScenarioTimeout)) { + return fail("UDP unicast teardown", + fmt::format("timed out after {} messages", received_messages.load())); } +} // namespace + +return pass("UDP unicast teardown", + fmt::format("received {} messages and exited scope cleanly", received_messages.load())); } -; + +ScenarioResult run_udp_response_scenario() { + constexpr size_t port = 5001; + auto request = make_payload(768, 0x20); + auto expected_response = reversed(request); + ByteVector response; + std::atomic_bool callback_invoked{false}; + + { + //! [UDP Server Response example] + espp::UdpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); + auto server_task_config = make_task_config("UdpResponseServer"); + auto server_config = espp::UdpSocket::ReceiveConfig{ + .port = port, .buffer_size = kMaxPacketSize, .on_receive_callback = [ + ](const auto &data, auto &) -> auto{return std::optional(reversed(data)); + } + , +}; server_socket.start_receiving(server_task_config, server_config); //! [UDP Server Response example] //! [UDP Client Response example] espp::UdpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); -// create threads -auto client_task_fn = [&server_address, &client_socket, &port](auto &, auto &) { - static size_t iterations = 0; - std::vector data{0, 1, 2, 3, 4}; - std::transform(data.begin(), data.end(), data.begin(), - [](const auto &d) { return d + iterations; }); - auto send_config = espp::UdpSocket::SendConfig{ - .ip_address = server_address, - .port = port, - .wait_for_response = true, - .response_size = 128, - .on_response_callback = [](auto &response) { fmt::print("Client received: {}\n", response); }, - }; - // NOTE: now this call blocks until the response is received - client_socket.send(data, send_config); - iterations++; - std::this_thread::sleep_for(1s); - // don't want to stop the task - return false; +auto send_config = espp::UdpSocket::SendConfig{ + .ip_address = kLoopbackAddress, + .port = port, + .wait_for_response = true, + .response_size = kMaxPacketSize, + .on_response_callback = + [&response, &callback_invoked](const auto &received_response) { + response = received_response; + callback_invoked = true; + }, + .response_timeout = 500ms, }; -auto client_task = - espp::Task::make_unique({.callback = client_task_fn, - .task_config = {.name = "Client Task", .stack_size_bytes = 5 * 1024}}); -client_task->start(); +auto ok = client_socket.send(request, send_config); //! [UDP Client Response example] -// now sleep for a while to let the monitor do its thing -std::this_thread::sleep_for(test_duration); + +if (!ok) { + return fail("UDP request/response", "client send failed"); +} +if (!callback_invoked.load()) { + return fail("UDP request/response", "response callback was not invoked"); +} } -fmt::print(fg(fmt::terminal_color::green) | fmt::emphasis::bold, - "UDP send waiting for response test finished.\n"); -std::this_thread::sleep_for(100ms); -fmt::print(fg(fmt::terminal_color::yellow) | fmt::emphasis::bold, "Staring UDP multicast test.\n"); - -// Multicast example -{ - //! [UDP Multicast Server example] - std::string multicast_group = "239.1.1.1"; - size_t port = 5000; - espp::UdpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); - auto server_task_config = espp::Task::BaseConfig{ - .name = "UdpServer", - .stack_size_bytes = 6 * 1024, - }; - auto server_config = espp::UdpSocket::ReceiveConfig{ - .port = port, - .buffer_size = 1024, - .is_multicast_endpoint = true, - .multicast_group = multicast_group, - .on_receive_callback = - [](auto &data, auto &source) -> auto{fmt::print("Server received: {}\n" - " from source: {}\n", - data, source); - // reverse the data - std::reverse(data.begin(), data.end()); - // and send it back - return data; +if (response != expected_response) { + return fail("UDP request/response", fmt::format("response mismatch ({} bytes)", response.size())); } +return pass("UDP request/response", + fmt::format("round-tripped {} bytes with reversed response", response.size())); } -; + +ScenarioResult run_udp_multicast_scenario() { + constexpr size_t port = 5002; + auto request = make_payload(64, 0x40); + auto expected_response = reversed(request); + ByteVector response; + + { + //! [UDP Multicast Server example] + espp::UdpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); + auto server_task_config = make_task_config("UdpMulticastServer"); + auto server_config = espp::UdpSocket::ReceiveConfig{ + .port = port, + .buffer_size = kMaxPacketSize, + .is_multicast_endpoint = true, + .multicast_group = kMulticastGroup, + .on_receive_callback = + [](const auto &data, auto &) -> auto{return std::optional(reversed(data)); + } + , +}; server_socket.start_receiving(server_task_config, server_config); //! [UDP Multicast Server example] //! [UDP Multicast Client example] -espp::UdpSocket client_socket({}); -// create threads -auto client_task_fn = [&client_socket, &port, &multicast_group](auto &, auto &) { - static size_t iterations = 0; - std::vector data{0, 1, 2, 3, 4}; - std::transform(data.begin(), data.end(), data.begin(), - [](const auto &d) { return d + iterations; }); - auto send_config = espp::UdpSocket::SendConfig{.ip_address = multicast_group, - .port = port, - .is_multicast_endpoint = true, - .wait_for_response = true, - .response_size = 128, - .on_response_callback = [](auto &response) { - fmt::print("Client received: {}\n", response); - }}; - // NOTE: now this call blocks until the response is received - client_socket.send(data, send_config); - iterations++; - std::this_thread::sleep_for(1s); - // don't want to stop the task - return false; +espp::UdpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); +auto send_config = espp::UdpSocket::SendConfig{ + .ip_address = kMulticastGroup, + .port = port, + .is_multicast_endpoint = true, + .wait_for_response = true, + .response_size = kMaxPacketSize, + .on_response_callback = + [&response](const auto &received_response) { response = received_response; }, + .response_timeout = 500ms, }; -auto client_task = - espp::Task::make_unique({.callback = client_task_fn, - .task_config = {.name = "Client Task", .stack_size_bytes = 5 * 1024}}); -client_task->start(); +auto ok = client_socket.send(request, send_config); //! [UDP Multicast Client example] -// now sleep for a while to let the monitor do its thing -std::this_thread::sleep_for(test_duration); + +if (!ok) { + return fail("UDP multicast request/response", "multicast send failed"); +} } -fmt::print(fg(fmt::terminal_color::green) | fmt::emphasis::bold, "UDP multicast test finished.\n"); -std::this_thread::sleep_for(100ms); -fmt::print(fg(fmt::terminal_color::yellow) | fmt::emphasis::bold, "Staring Basic TCP test.\n"); - -// Unicast (client-server) example -{ - //! [TCP Server example] - std::string server_address = "127.0.0.1"; - size_t port = 5000; - int max_connections = 1; - espp::TcpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); - server_socket.bind(port); - server_socket.listen(max_connections); - auto server_task_fn = [&server_socket](auto &m, auto &cv) -> bool { - static std::unique_ptr client_socket; - if (!client_socket) { - client_socket = server_socket.accept(); - if (client_socket) { - auto info = client_socket->get_remote_info(); - fmt::print("Server accepted connection from: {}\n", info); - } +if (response != expected_response) { + return fail("UDP multicast request/response", + fmt::format("response mismatch ({} bytes)", response.size())); +} +return pass("UDP multicast request/response", + fmt::format("received {} byte response", response.size())); +} + +ScenarioResult run_udp_timeout_scenario() { + constexpr size_t port = 5003; + espp::UdpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); + auto send_config = espp::UdpSocket::SendConfig{ + .ip_address = kLoopbackAddress, + .port = port, + .wait_for_response = true, + .response_size = 64, + .response_timeout = 150ms, + }; + + auto ok = client_socket.send(make_payload(16, 0x55), send_config); + if (ok) { + return fail("UDP response timeout", "unexpectedly received a response"); + } + return pass("UDP response timeout", "send timed out as expected with no server"); +} + +ScenarioResult run_udp_blocked_receive_teardown_scenario() { + constexpr size_t port = 5004; + { + espp::UdpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); + auto server_task_config = make_task_config("UdpBlockedServer"); + auto server_config = espp::UdpSocket::ReceiveConfig{ + .port = port, .buffer_size = kMaxPacketSize, .on_receive_callback = [ + ](auto &, auto &) -> auto{return std::nullopt; + } + , +}; +if (!server_socket.start_receiving(server_task_config, server_config)) { + return fail("UDP blocked receive teardown", "failed to start receive task"); +} +std::this_thread::sleep_for(kSettleDelay); +} +return pass("UDP blocked receive teardown", "receive task stopped while idle"); +} + +ScenarioResult run_tcp_unicast_teardown_scenario() { + constexpr size_t port = 6000; + constexpr size_t expected_messages = 3; + std::atomic_size_t received_messages{0}; + + { + //! [TCP Server example] + espp::TcpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); + if (!server_socket.bind(port) || !server_socket.listen(kMaxConnections)) { + return fail("TCP unicast teardown", "failed to bind/listen"); } - if (client_socket) { - std::vector data; - size_t max_receive_size = 1024; - if (client_socket->receive(data, max_receive_size)) { - fmt::print("Server received: {}\n", data); - } + auto server_task = espp::Task::make_unique({ + .callback = [&server_socket, &received_messages](std::mutex &m, std::condition_variable &cv, + bool ¬ified) -> bool { + static std::unique_ptr client_socket; + if (!client_socket) { + client_socket = server_socket.accept(); + if (!client_socket) { + return wait_or_stop(m, cv, notified, 10ms); + } + } + + ByteVector data; + if (client_socket->receive(data, kMaxPacketSize)) { + received_messages++; + fmt::print("TCP server received {} bytes\n", data.size()); + } else if (!client_socket->is_connected()) { + client_socket.reset(); + } + return wait_or_stop(m, cv, notified, 10ms); + }, + .task_config = make_task_config("TcpServer"), + }); + server_task->start(); + //! [TCP Server example] + + //! [TCP Client example] + espp::TcpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); + if (!client_socket.connect({.ip_address = kLoopbackAddress, .port = port})) { + server_socket.close(); + server_task->stop(); + return fail("TCP unicast teardown", "client failed to connect"); } - { - std::unique_lock lock(m); - cv.wait_for(lock, 10ms); + auto client_task = espp::Task::make_unique({ + .callback = [&client_socket](std::mutex &m, std::condition_variable &cv, + bool ¬ified) -> bool { + static size_t iterations = 0; + auto data = make_payload(24, static_cast(iterations * 5)); + client_socket.transmit(data); + iterations++; + return wait_or_stop(m, cv, notified); + }, + .task_config = make_task_config("TcpClient", 5 * 1024), + }); + client_task->start(); + //! [TCP Client example] + + if (!wait_until([&received_messages] { return received_messages.load() >= expected_messages; }, + kScenarioTimeout)) { + client_task->stop(); + client_socket.close(); + server_socket.close(); + server_task->stop(); + return fail("TCP unicast teardown", + fmt::format("timed out after {} messages", received_messages.load())); } - // don't want to stop the task - return false; - }; - auto server_task_config = espp::Task::Config{ - .callback = server_task_fn, - .task_config = - { - .name = "TcpServer", - .stack_size_bytes = 6 * 1024, - }, - }; - auto server_task = espp::Task::make_unique(server_task_config); - server_task->start(); - //! [TCP Server example] - - //! [TCP Client example] - espp::TcpSocket client_socket({}); - client_socket.connect({.ip_address = server_address, .port = port}); - // create thread for sending data using the socket - auto client_task_fn = [&server_address, &client_socket, &port](auto &, auto &) { - static size_t iterations = 0; - std::vector data{0, 1, 2, 3, 4}; - std::transform(data.begin(), data.end(), data.begin(), - [](const auto &d) { return d + iterations; }); - client_socket.transmit(data); - iterations++; - std::this_thread::sleep_for(1s); - // don't want to stop the task - return false; - }; - auto client_task = espp::Task::make_unique( - {.callback = client_task_fn, - .task_config = {.name = "Client Task", .stack_size_bytes = 5 * 1024}}); - client_task->start(); - //! [TCP Client example] - // now sleep for a while to let the monitor do its thing - std::this_thread::sleep_for(test_duration); + client_task->stop(); + client_socket.close(); + server_socket.close(); + server_task->stop(); + } + + return pass("TCP unicast teardown", fmt::format("received {} messages and exited scope cleanly", + received_messages.load())); } -fmt::print(fg(fmt::terminal_color::green) | fmt::emphasis::bold, "Basic TCP test finished.\n"); -std::this_thread::sleep_for(100ms); -fmt::print(fg(fmt::terminal_color::yellow) | fmt::emphasis::bold, - "Staring TCP send waiting for response test.\n"); - -// Unicast (client-server) example with server response -{ - //! [TCP Server Response example] - std::string server_address = "127.0.0.1"; - size_t port = 5000; - int max_connections = 1; - espp::TcpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); - server_socket.bind(port); - server_socket.listen(max_connections); - auto server_task_fn = [&server_socket](auto &m, auto &cv) -> bool { - static std::unique_ptr client_socket; - if (!client_socket) { - client_socket = server_socket.accept(); - if (client_socket) { - auto info = client_socket->get_remote_info(); - fmt::print("Server accepted connection from: {}\n", info); - } +ScenarioResult run_tcp_response_reconnect_scenario() { + constexpr size_t port = 6001; + std::atomic_size_t accepted_connections{0}; + + { + //! [TCP Server Response example] + espp::TcpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); + if (!server_socket.bind(port) || !server_socket.listen(kMaxConnections)) { + return fail("TCP response/reconnect", "failed to bind/listen"); } - if (client_socket) { - std::vector data; - size_t max_receive_size = 1024; - if (client_socket->receive(data, max_receive_size)) { - fmt::print("Server received: {}\n", data); - // reverse the data - std::reverse(data.begin(), data.end()); - // and send it back - client_socket->transmit(data); + auto server_task = espp::Task::make_unique({ + .callback = [&server_socket, &accepted_connections]( + std::mutex &m, std::condition_variable &cv, bool ¬ified) -> bool { + static std::unique_ptr client_socket; + if (!client_socket) { + client_socket = server_socket.accept(); + if (client_socket) { + accepted_connections++; + } else { + return wait_or_stop(m, cv, notified, 10ms); + } + } + + ByteVector data; + if (client_socket->receive(data, kMaxPacketSize)) { + client_socket->transmit(reversed(data)); + } else if (!client_socket->is_connected()) { + client_socket.reset(); + } + return wait_or_stop(m, cv, notified, 10ms); + }, + .task_config = make_task_config("TcpResponseServer"), + }); + server_task->start(); + auto stop_server = [&server_socket, &server_task]() { + server_socket.close(); + server_task->stop(); + }; + //! [TCP Server Response example] + + //! [TCP Client Response example] + auto exchange_once = [port](uint8_t seed, ByteVector &response) -> bool { + espp::TcpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); + if (!client_socket.connect({.ip_address = kLoopbackAddress, .port = port})) { + return false; } + auto request = make_payload(40, seed); + auto transmit_config = espp::TcpSocket::TransmitConfig{ + .wait_for_response = true, + .response_size = kMaxPacketSize, + .on_response_callback = + [&response](const auto &received_response) { response = received_response; }, + .response_timeout = 500ms, + }; + return client_socket.transmit(request, transmit_config); + }; + //! [TCP Client Response example] + + ByteVector first_response; + ByteVector second_response; + if (!exchange_once(0x10, first_response)) { + stop_server(); + return fail("TCP response/reconnect", "first request/response failed"); } - { - std::unique_lock lock(m); - cv.wait_for(lock, 10ms); + if (!wait_until([&accepted_connections] { return accepted_connections.load() >= 1; }, + kScenarioTimeout)) { + stop_server(); + return fail("TCP response/reconnect", "server never accepted the first connection"); + } + if (!exchange_once(0x30, second_response)) { + stop_server(); + return fail("TCP response/reconnect", "second request/response failed"); + } + if (!wait_until([&accepted_connections] { return accepted_connections.load() >= 2; }, + kScenarioTimeout)) { + stop_server(); + return fail("TCP response/reconnect", "server never accepted the reconnect"); } - // don't want to stop the task - return false; - }; - auto server_task_config = espp::Task::Config{ - .callback = server_task_fn, - .task_config = - { - .name = "TcpServer", - .stack_size_bytes = 6 * 1024, - }, - }; - auto server_task = espp::Task::make_unique(server_task_config); - server_task->start(); - //! [TCP Server Response example] + if (first_response != reversed(make_payload(40, 0x10))) { + stop_server(); + return fail("TCP response/reconnect", "first response mismatch"); + } + if (second_response != reversed(make_payload(40, 0x30))) { + stop_server(); + return fail("TCP response/reconnect", "second response mismatch"); + } - //! [TCP Client Response example] - espp::TcpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); - client_socket.connect({ - .ip_address = server_address, - .port = port, - }); - // create threads - auto client_task_fn = [&server_address, &client_socket, &port](auto &, auto &) { - static size_t iterations = 0; - std::vector data{0, 1, 2, 3, 4}; - std::transform(data.begin(), data.end(), data.begin(), - [](const auto &d) { return d + iterations; }); - auto transmit_config = espp::TcpSocket::TransmitConfig{ - .wait_for_response = true, - .response_size = 128, - .on_response_callback = - [](auto &response) { fmt::print("Client received: {}\n", response); }, - }; - // NOTE: now this call blocks until the response is received - client_socket.transmit(data, transmit_config); - iterations++; - std::this_thread::sleep_for(1s); - // don't want to stop the task - return false; - }; - auto client_task = espp::Task::make_unique( - {.callback = client_task_fn, - .task_config = {.name = "Client Task", .stack_size_bytes = 5 * 1024}}); - client_task->start(); - //! [TCP Client Response example] - // now sleep for a while to let the monitor do its thing - std::this_thread::sleep_for(test_duration); + stop_server(); + } + + return pass("TCP response/reconnect", + fmt::format("handled {} sequential client connections", accepted_connections.load())); } -fmt::print(fg(fmt::terminal_color::green) | fmt::emphasis::bold, - "TCP send waiting for response test finished.\n"); -std::this_thread::sleep_for(100ms); -fmt::print(fg(fmt::terminal_color::green) | fmt::emphasis::bold, "Socket example finished!\n"); +ScenarioResult run_tcp_blocked_accept_teardown_scenario() { + constexpr size_t port = 6002; + { + espp::TcpSocket server_socket({.log_level = espp::Logger::Verbosity::WARN}); + if (!server_socket.bind(port) || !server_socket.listen(kMaxConnections)) { + return fail("TCP blocked accept teardown", "failed to bind/listen"); + } + auto server_task = espp::Task::make_unique({ + .callback = [&server_socket](std::mutex &m, std::condition_variable &cv, + bool ¬ified) -> bool { + auto accepted_socket = server_socket.accept(); + if (accepted_socket) { + fmt::print("Unexpected TCP connection from {}\n", accepted_socket->get_remote_info()); + } + return wait_or_stop(m, cv, notified, 10ms); + }, + .task_config = make_task_config("TcpBlockedAccept"), + }); + server_task->start(); + std::this_thread::sleep_for(kSettleDelay); -// sleep forever -while (true) { - std::this_thread::sleep_for(1s); + server_socket.close(); + server_task->stop(); + } + return pass("TCP blocked accept teardown", "accept task stopped while idle"); } + +ScenarioResult run_tcp_connect_failure_scenario() { + constexpr size_t port = 6003; + espp::TcpSocket client_socket({.log_level = espp::Logger::Verbosity::WARN}); + if (client_socket.connect({.ip_address = kLoopbackAddress, .port = port})) { + return fail("TCP connect failure", "unexpectedly connected to an unused port"); + } + return pass("TCP connect failure", "connect failed as expected"); +} +} // namespace + +extern "C" void app_main(void) { + fmt::print("Starting socket example!\n"); + + // create a wifi access point here so that LwIP will be init for this example + espp::WifiAp wifi_ap({.ssid = "SocketExample", + .password = "", // no security + .log_level = espp::Logger::Verbosity::INFO}); + + std::vector results; + results.reserve(9); + + auto run_and_record = [&results](auto &&scenario_runner, std::string_view name) { + print_scenario_start(name); + auto result = scenario_runner(); + print_scenario_result(result); + results.push_back(std::move(result)); + std::this_thread::sleep_for(kSettleDelay); + }; + + run_and_record(run_udp_unicast_teardown_scenario, "UDP unicast teardown"); + run_and_record(run_udp_response_scenario, "UDP request/response"); + run_and_record(run_udp_multicast_scenario, "UDP multicast request/response"); + run_and_record(run_udp_timeout_scenario, "UDP response timeout"); + run_and_record(run_udp_blocked_receive_teardown_scenario, "UDP blocked receive teardown"); + run_and_record(run_tcp_unicast_teardown_scenario, "TCP unicast teardown"); + run_and_record(run_tcp_response_reconnect_scenario, "TCP response/reconnect"); + run_and_record(run_tcp_blocked_accept_teardown_scenario, "TCP blocked accept teardown"); + run_and_record(run_tcp_connect_failure_scenario, "TCP connect failure"); + + auto passed_count = std::count_if(results.begin(), results.end(), + [](const auto &result) { return result.passed; }); + fmt::print(fg(fmt::terminal_color::cyan) | fmt::emphasis::bold, + "Socket example summary: {}/{} scenarios passed\n", passed_count, results.size()); + for (const auto &result : results) { + print_scenario_result(result); + } + + while (true) { + std::this_thread::sleep_for(1s); + } } diff --git a/components/socket/include/udp_socket.hpp b/components/socket/include/udp_socket.hpp index 4cea59665..03672fe87 100644 --- a/components/socket/include/udp_socket.hpp +++ b/components/socket/include/udp_socket.hpp @@ -81,6 +81,9 @@ class UdpSocket : public Socket { */ ~UdpSocket(); + /// Stop the receive task, if one is running, and close the socket. + void stop_receiving(); + /** * @brief Send data to the endpoint specified by the send_config. * Can be configured to multicast (within send_config) and can be diff --git a/components/socket/src/socket.cpp b/components/socket/src/socket.cpp index 85e810a28..87a97c624 100644 --- a/components/socket/src/socket.cpp +++ b/components/socket/src/socket.cpp @@ -250,34 +250,28 @@ bool Socket::add_multicast_group(const std::string &multicast_group) { int Socket::select(const std::chrono::microseconds &timeout) { fd_set readfds; - fd_set writefds; fd_set exceptfds; FD_ZERO(&readfds); - FD_ZERO(&writefds); FD_ZERO(&exceptfds); FD_SET(socket_, &readfds); - FD_SET(socket_, &writefds); FD_SET(socket_, &exceptfds); int nfds = socket_ + 1; // convert timeout to timeval struct timeval tv; tv.tv_sec = std::chrono::duration_cast(timeout).count(); tv.tv_usec = std::chrono::duration_cast(timeout).count() % 1000000; - int retval = ::select(nfds, &readfds, &writefds, &exceptfds, &tv); + int retval = ::select(nfds, &readfds, nullptr, &exceptfds, &tv); if (retval < 0) { logger_.error("select failed: {}", error_string()); return -1; } if (retval == 0) { - logger_.warn("select timed out"); + logger_.debug("select timed out"); return 0; } if (FD_ISSET(socket_, &readfds)) { logger_.debug("select read"); } - if (FD_ISSET(socket_, &writefds)) { - logger_.debug("select write"); - } if (FD_ISSET(socket_, &exceptfds)) { logger_.debug("select except"); } @@ -335,19 +329,31 @@ std::string Socket::error_string(int err) const { void Socket::cleanup() { if (is_valid()) { - int status = 0; + auto socket_fd = socket_; #ifdef _MSC_VER - status = shutdown(socket_, SD_BOTH); - if (status == 0) { - closesocket(socket_); - } socket_ = INVALID_SOCKET; -#else - status = shutdown(socket_, SHUT_RDWR); - if (status == 0) { - close(socket_); + int status = shutdown(socket_fd, SD_BOTH); + if (status != 0) { + int err = WSAGetLastError(); + if (err != WSAENOTCONN && err != WSAENOTSOCK) { + logger_.debug("shutdown failed before close: {}", error_string(err)); + } + } + if (closesocket(socket_fd) != 0) { + logger_.debug("close failed: {}", error_string(WSAGetLastError())); } +#else socket_ = -1; + int status = shutdown(socket_fd, SHUT_RDWR); + if (status != 0) { + int err = errno; + if (err != ENOTCONN) { + logger_.debug("shutdown failed before close: {}", error_string(err)); + } + } + if (close(socket_fd) != 0) { + logger_.debug("close failed: {}", error_string(errno)); + } #endif logger_.info("Closed socket"); diff --git a/components/socket/src/tcp_socket.cpp b/components/socket/src/tcp_socket.cpp index d14d3d03b..26473de6e 100644 --- a/components/socket/src/tcp_socket.cpp +++ b/components/socket/src/tcp_socket.cpp @@ -21,7 +21,10 @@ void TcpSocket::reinit() { init(Type::STREAM); } -void TcpSocket::close() { ::close(socket_); } +void TcpSocket::close() { + connected_ = false; + cleanup(); +} bool TcpSocket::is_connected() const { return connected_; } diff --git a/components/socket/src/udp_socket.cpp b/components/socket/src/udp_socket.cpp index 584cce2d8..f5c7825cf 100644 --- a/components/socket/src/udp_socket.cpp +++ b/components/socket/src/udp_socket.cpp @@ -1,14 +1,48 @@ #include "udp_socket.hpp" +#include + using namespace espp; +namespace { +#ifdef _MSC_VER +int last_socket_error() { return WSAGetLastError(); } +#else +int last_socket_error() { return errno; } +#endif + +bool is_transient_send_error(int err) { +#ifdef _MSC_VER + return err == WSAEWOULDBLOCK || err == WSAENOBUFS; +#else + return err == EAGAIN || err == EWOULDBLOCK || err == ENOBUFS || err == ENOMEM; +#endif +} + +constexpr int transient_send_retry_count = 5; +constexpr auto transient_send_retry_delay = std::chrono::milliseconds(2); + +int transient_send_retry_limit(int err) { +#if defined(ESP_PLATFORM) + if (err == ENOBUFS || err == ENOMEM) { + return 0; + } +#endif + return transient_send_retry_count; +} +} // namespace + UdpSocket::UdpSocket(const UdpSocket::Config &config) : Socket(Type::DGRAM, Logger::Config{.tag = "UdpSocket", .level = config.log_level}) {} -UdpSocket::~UdpSocket() { - // we have to explicitly call cleanup here so that the server recvfrom - // will return and the task can stop. +UdpSocket::~UdpSocket() { stop_receiving(); } + +void UdpSocket::stop_receiving() { + // Close the socket first so any blocking recvfrom returns and the task can stop. cleanup(); + if (task_ && task_->is_started()) { + task_->stop(); + } } bool UdpSocket::send(const std::vector &data, const UdpSocket::SendConfig &send_config) { @@ -46,11 +80,39 @@ bool UdpSocket::send(std::span data, const UdpSocket::SendConfig auto server_address = server_info.ipv4_ptr(); logger_.info("Client sending {} bytes to {}:{}", data.size(), send_config.ip_address, send_config.port); - int num_bytes_sent = - sendto(socket_, reinterpret_cast(data.data()), data.size(), 0, - reinterpret_cast(server_address), sizeof(*server_address)); + int num_bytes_sent = -1; + for (int attempt = 0; attempt <= transient_send_retry_count; attempt++) { + num_bytes_sent = + sendto(socket_, reinterpret_cast(data.data()), data.size(), 0, + reinterpret_cast(server_address), sizeof(*server_address)); + if (num_bytes_sent >= 0) { + break; + } + int err = last_socket_error(); + int retry_limit = transient_send_retry_limit(err); + if (!is_transient_send_error(err)) { + logger_.error("Error occurred during sending {} bytes to {}:{}: {}", data.size(), + send_config.ip_address, send_config.port, error_string(err)); + return false; + } + if (attempt >= retry_limit) { +#if defined(ESP_PLATFORM) + if (err == ENOBUFS || err == ENOMEM) { + logger_.warn("Dropping UDP send of {} bytes to {}:{} due to TX backpressure: {}", + data.size(), send_config.ip_address, send_config.port, error_string(err)); + return false; + } +#endif + logger_.error("Error occurred during sending {} bytes to {}:{}: {}", data.size(), + send_config.ip_address, send_config.port, error_string(err)); + return false; + } + logger_.warn("Transient send failure sending {} bytes to {}:{} (attempt {}/{}): {}", + data.size(), send_config.ip_address, send_config.port, attempt + 1, + retry_limit + 1, error_string(err)); + std::this_thread::sleep_for(transient_send_retry_delay); + } if (num_bytes_sent < 0) { - logger_.error("Error occurred during sending: {}", error_string()); return false; } logger_.debug("Client sent {} bytes", num_bytes_sent); diff --git a/doc/Doxyfile b/doc/Doxyfile index 987e263d6..03b2b0ddc 100755 --- a/doc/Doxyfile +++ b/doc/Doxyfile @@ -316,14 +316,23 @@ INPUT = \ $(PROJECT_PATH)/components/remote_debug/include/remote_debug.hpp \ $(PROJECT_PATH)/components/rmt/include/rmt.hpp \ $(PROJECT_PATH)/components/rmt/include/rmt_encoder.hpp \ + $(PROJECT_PATH)/components/rtsp/include/generic_depacketizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/generic_packetizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/h264_depacketizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/h264_packetizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/jpeg_frame.hpp \ + $(PROJECT_PATH)/components/rtsp/include/jpeg_header.hpp \ + $(PROJECT_PATH)/components/rtsp/include/mjpeg_depacketizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/mjpeg_packetizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/rtcp_packet.hpp \ + $(PROJECT_PATH)/components/rtsp/include/rtp_depacketizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/rtp_jpeg_packet.hpp \ + $(PROJECT_PATH)/components/rtsp/include/rtp_packet.hpp \ + $(PROJECT_PATH)/components/rtsp/include/rtp_packetizer.hpp \ + $(PROJECT_PATH)/components/rtsp/include/rtp_types.hpp \ $(PROJECT_PATH)/components/rtsp/include/rtsp_client.hpp \ $(PROJECT_PATH)/components/rtsp/include/rtsp_server.hpp \ $(PROJECT_PATH)/components/rtsp/include/rtsp_session.hpp \ - $(PROJECT_PATH)/components/rtsp/include/rtcp_packet.hpp \ - $(PROJECT_PATH)/components/rtsp/include/rtp_packet.hpp \ - $(PROJECT_PATH)/components/rtsp/include/rtp_jpeg_packet.hpp \ - $(PROJECT_PATH)/components/rtsp/include/jpeg_frame.hpp \ - $(PROJECT_PATH)/components/rtsp/include/jpeg_header.hpp \ $(PROJECT_PATH)/components/runqueue/include/runqueue.hpp \ $(PROJECT_PATH)/components/rx8130ce/include/rx8130ce.hpp \ $(PROJECT_PATH)/components/serialization/include/serialization.hpp \ diff --git a/doc/en/network/socket.rst b/doc/en/network/socket.rst index 06e5823e8..cbd39bf7e 100644 --- a/doc/en/network/socket.rst +++ b/doc/en/network/socket.rst @@ -1,14 +1,68 @@ -Sockets -******* +Socket Component +**************** -The socket provides the base abstraction around the socket file descriptor and -includes some initialization, cleanup, and conversion utilities. +The network APIs provide a useful abstraction over POSIX sockets, making it +easier to build client/server communication flows and bind callback-driven +receive handlers on embedded targets. -The socket class is subclassed into UdpSocket and TcpSocket. +Currently, UDP and TCP sockets are supported. -.. ---------------------------- API Reference ---------------------------------- +Base Socket +----------- -API Reference -------------- +The base ``Socket`` abstraction wraps socket file descriptor lifecycle, +configuration helpers, and endpoint conversion utilities shared by both UDP and +TCP transports. -.. include-build-file:: inc/socket.inc + +UDP Socket +---------- + +UDP sockets provide unreliable, unordered communication over IP network +sockets. + +UDP sockets can be used in unicast (point to point), multicast (one to many and +many to one), and broadcast (one to all). + +The ``UdpSocket`` API supports both one-shot sends and long-running receive +tasks: + +- ``send(...)`` can optionally block waiting for a response with a timeout and + response callback +- ``start_receiving(...)`` starts a task that continuously receives datagrams + and optionally sends a callback-produced response +- ``stop_receiving()`` cleanly stops an in-flight receive task and closes the + socket, which is especially useful for teardown paths on embedded targets + + +TCP Socket +---------- + +TCP sockets provide reliable, ordered communication over IP network sockets and +have built-in protocols for handling packet acknowledgement as well as +transmission speed / bandwidth adjustment. + +TCP sockets cannot be used with multicast (many to one, one to many). + +The ``TcpSocket`` API covers both client and server patterns: + +- ``connect(...)`` plus ``transmit(...)`` for client-style request/response +- optional blocking response waits with callback delivery +- ``bind(...)``, ``listen(...)``, and ``accept()`` for server-side flows +- explicit ``close()`` / ``reinit()`` helpers used by teardown and reconnect + paths + + +Example +------- + +The :doc:`socket_example ` page shows the use of the +classes provided by the ``socket`` component and runs a scenario-driven +self-test which covers teardown, timeout, and reconnect behavior, including: + +* ``UdpSocket`` as both client and server, including unicast and multicast + configurations +* ``TcpSocket`` as both client and server +* scope-based UDP/TCP teardown while tasks are active or blocked +* request/response callbacks and timeout handling +* reconnect behavior after TCP session shutdown diff --git a/doc/en/network/tcp_socket.rst b/doc/en/network/tcp_socket.rst index 5c1fc1a64..77918bddc 100644 --- a/doc/en/network/tcp_socket.rst +++ b/doc/en/network/tcp_socket.rst @@ -7,6 +7,11 @@ transmission speed / bandwidth adjustment. TCP sockets cannot be used with multicast (many to one, one to many). +The implementation supports both client-style request/response flows and +server-style ``bind(...)`` / ``listen(...)`` / ``accept()`` loops. ``close()`` +and ``reinit()`` are used by the updated teardown and reconnect paths to ensure +connection state is reset cleanly. + .. ---------------------------- API Reference ---------------------------------- API Reference diff --git a/doc/en/network/udp_socket.rst b/doc/en/network/udp_socket.rst index a33fc1019..5b1a9e695 100644 --- a/doc/en/network/udp_socket.rst +++ b/doc/en/network/udp_socket.rst @@ -1,11 +1,17 @@ UDP Sockets *********** -UDP sockets provide unreliable, unordered communication over IP network sockets. +UDP sockets provide unreliable, unordered communication over IP network +sockets. UDP sockets can be used in unicast (point to point), multicast (one to many and many to one), and broadcast (one to all). +The implementation supports both blocking request/response sends and background +receive tasks. In addition to ``start_receiving(...)``, the class exposes +``stop_receiving()`` so applications can reliably stop a blocked receive task +during teardown. + .. ---------------------------- API Reference ---------------------------------- API Reference diff --git a/doc/en/rtsp.rst b/doc/en/rtsp.rst index d769bc8d0..3de33b8d6 100644 --- a/doc/en/rtsp.rst +++ b/doc/en/rtsp.rst @@ -1,38 +1,111 @@ RTSP APIs ********* +The ``rtsp`` component provides a flexible, multi-codec RTSP streaming framework +for ESP32 devices. It supports MJPEG, H.264, and generic audio codecs through +an extensible packetizer/depacketizer architecture. The component handles RTP +packet splitting and reassembly; encoding and decoding of media data is handled +externally by the application. + RTSP Client ----------- -The `RtspClient` class provides an interface to an RTSP server. It is used to -send RTSP requests and receive RTSP responses. It also provides an interface -to the RTP and RTCP sessions that are created as a result of the RTSP -interactions. +The ``RtspClient`` class connects to an RTSP server and receives media streams +over RTP/UDP. It dispatches incoming RTP packets to codec-specific +depacketizers based on payload type. -The `RtspClient` currently only supports MJPEG streams, since the ESP32 does -not have a hardware decoder for H.264 or H.265. +For backward compatibility, setting the ``on_jpeg_frame`` callback +automatically creates an ``MjpegDepacketizer`` for MJPEG streams (payload type +26). For generic multi-track use, applications can use the ``on_frame`` callback +and inspect parsed SDP metadata through ``tracks()``. -Additionally the client currently only supports UDP transport for RTP and RTCP -packets. TCP transport is not supported. +The client now supports: -The user can register a callback function to be notified when new, complete JPEG -frames are received. The callback function is called with a pointer to the JPEG -frame. +- generic ``on_frame(track_id, data)`` callbacks for multi-track sessions +- parsed SDP track metadata including media type, payload type, codec name, + sample rate, channel count, and resolved control path +- automatic depacketizer selection for MJPEG, H.264, and generic payloads + discovered during ``DESCRIBE`` +- an ``on_connection_lost`` callback for reconnect / rediscovery workflows when + the RTSP control socket or RTP stream disappears after playback starts RTSP Server ----------- -The `RtspServer` class provides an implementation of an RTSP server. It is used -to receive RTSP requests and send RTSP responses. It is designed to allow the -user to send JPEG frames to the server, which will then send them to the client -over RTP/UDP. +The ``RtspServer`` class accepts RTSP connections and streams media over +RTP/UDP. It supports multiple tracks, each with its own codec-specific +packetizer, SSRC, and sequence numbering. + +For backward compatibility, calling ``send_frame(const JpegFrame&)`` lazily +creates a default MJPEG track. For other codecs, register tracks via +``add_track()`` and send frames with ``send_frame(track_id, data)``. + +The server also exposes helpers that are useful for embedded capture loops: + +- configurable accept, session-dispatch, and per-session control task stack + sizes +- ``has_active_sessions()`` to avoid capturing when no client is actively + playing +- ``get_capture_cooldown()`` and ``get_recommended_capture_period()`` so an + application can slow capture when RTP backpressure is observed +- a legacy MJPEG ``send_frame(std::span)`` path that preserves + the older wire format for existing MJPEG-only users + + +RTP Packetizers & Depacketizers +------------------------------- + +The packetizer/depacketizer abstraction allows the server and client to support +multiple media codecs without changing the RTSP core. Concrete implementations +are provided for: + +- **MJPEG** (``MjpegPacketizer`` / ``MjpegDepacketizer``) — RFC 2435 JPEG over RTP +- **H.264** (``H264Packetizer`` / ``H264Depacketizer``) — RFC 6184 with FU-A fragmentation +- **Generic** (``GenericPacketizer`` / ``GenericDepacketizer``) — MTU chunking + for audio or other pre-encoded payloads, with frame reconstruction based on + RTP marker / timestamp boundaries + +Custom packetizers can be created by subclassing ``RtpPacketizer`` or +``RtpDepacketizer``. + + +Testing and Utilities +--------------------- + +There are several ways to exercise the RTSP stack: + +- **ESPP Python library**: + ``espp/lib`` contains the build scripts and bindings used to expose the RTSP + client / server classes to Python. +- **Python harness scripts**: + ``espp/python`` contains wrapper and multitrack scripts for exercising legacy + MJPEG flows, generic multi-track flows, live microphone audio, and end-to-end + host validation. +- **Embedded examples and downstream apps**: + the component example plus repositories such as ``camera-streamer`` and + ``camera-display`` cover practical server/client integrations. + +See ``python/README.md`` in the repository root for more information on the +host-side scripts. + + +Example +------- + +The :doc:`rtsp_example` page demonstrates several RTSP usage patterns selected +via menuconfig, including: + +- legacy MJPEG server + client behavior +- server-only MJPEG streaming +- client-only MJPEG reception +- API-level packetizer / depacketizer exercises +- multi-track streaming with MJPEG video plus generic audio -The server currently only supports MJPEG streams, since the ESP32 does not have -a hardware encoder for H.264 or H.265. +For more complete integrations, see the `camera-streamer +`_ and `camera-display +`_ repositories. -Additionally, the server currently only supports UDP transport for RTP and RTCP -packets. TCP transport is not supported. .. ------------------------------- Example ------------------------------------- @@ -48,6 +121,15 @@ API Reference .. include-build-file:: inc/rtsp_client.inc .. include-build-file:: inc/rtsp_server.inc .. include-build-file:: inc/rtsp_session.inc +.. include-build-file:: inc/rtp_packetizer.inc +.. include-build-file:: inc/rtp_depacketizer.inc +.. include-build-file:: inc/rtp_types.inc +.. include-build-file:: inc/mjpeg_packetizer.inc +.. include-build-file:: inc/mjpeg_depacketizer.inc +.. include-build-file:: inc/h264_packetizer.inc +.. include-build-file:: inc/h264_depacketizer.inc +.. include-build-file:: inc/generic_packetizer.inc +.. include-build-file:: inc/generic_depacketizer.inc .. include-build-file:: inc/rtp_packet.inc .. include-build-file:: inc/rtp_jpeg_packet.inc .. include-build-file:: inc/rtcp_packet.inc diff --git a/lib/autogenerate_bindings.py b/lib/autogenerate_bindings.py index 5615b9e5d..3d3624ec7 100644 --- a/lib/autogenerate_bindings.py +++ b/lib/autogenerate_bindings.py @@ -130,6 +130,15 @@ def autogenerate() -> None: include_dir + "joystick/include/joystick.hpp", # NOTE: this must come after socket since it depends on it! + include_dir + "rtsp/include/rtp_types.hpp", + include_dir + "rtsp/include/rtp_depacketizer.hpp", + include_dir + "rtsp/include/rtp_packetizer.hpp", + include_dir + "rtsp/include/generic_depacketizer.hpp", + include_dir + "rtsp/include/generic_packetizer.hpp", + include_dir + "rtsp/include/h264_depacketizer.hpp", + include_dir + "rtsp/include/h264_packetizer.hpp", + include_dir + "rtsp/include/mjpeg_depacketizer.hpp", + include_dir + "rtsp/include/mjpeg_packetizer.hpp", include_dir + "rtsp/include/rtp_jpeg_packet.hpp", include_dir + "rtsp/include/jpeg_frame.hpp", include_dir + "rtsp/include/jpeg_header.hpp", diff --git a/lib/espp.cmake b/lib/espp.cmake index 9375866c6..95bd2d761 100644 --- a/lib/espp.cmake +++ b/lib/espp.cmake @@ -58,6 +58,12 @@ set(ESPP_SOURCES ${ESPP_COMPONENTS}/rtsp/src/rtsp_client.cpp ${ESPP_COMPONENTS}/rtsp/src/rtsp_server.cpp ${ESPP_COMPONENTS}/rtsp/src/rtsp_session.cpp + ${ESPP_COMPONENTS}/rtsp/src/generic_depacketizer.cpp + ${ESPP_COMPONENTS}/rtsp/src/generic_packetizer.cpp + ${ESPP_COMPONENTS}/rtsp/src/h264_packetizer.cpp + ${ESPP_COMPONENTS}/rtsp/src/h264_depacketizer.cpp + ${ESPP_COMPONENTS}/rtsp/src/mjpeg_depacketizer.cpp + ${ESPP_COMPONENTS}/rtsp/src/mjpeg_packetizer.cpp ${ESPP_COMPONENTS}/task/src/task.cpp ${ESPP_COMPONENTS}/timer/src/timer.cpp ${ESPP_COMPONENTS}/socket/src/socket.cpp diff --git a/lib/include/espp.hpp b/lib/include/espp.hpp index 768e010c8..f6006e776 100644 --- a/lib/include/espp.hpp +++ b/lib/include/espp.hpp @@ -28,12 +28,21 @@ extern "C" { // TODO: these are not working // #include "hid-rp.hpp" // #include "hid-rp-gamepad.hpp" +#include "generic_depacketizer.hpp" +#include "generic_packetizer.hpp" +#include "h264_depacketizer.hpp" +#include "h264_packetizer.hpp" #include "joystick.hpp" #include "logger.hpp" #include "lowpass_filter.hpp" +#include "mjpeg_depacketizer.hpp" +#include "mjpeg_packetizer.hpp" #include "ndef.hpp" #include "pid.hpp" #include "range_mapper.hpp" +#include "rtp_depacketizer.hpp" +#include "rtp_packetizer.hpp" +#include "rtp_types.hpp" #include "rtsp_client.hpp" #include "rtsp_server.hpp" #include "serialization.hpp" diff --git a/lib/python_bindings/espp/__init__.pyi b/lib/python_bindings/espp/__init__.pyi index 2794ffa61..4bb845263 100644 --- a/lib/python_bindings/espp/__init__.pyi +++ b/lib/python_bindings/espp/__init__.pyi @@ -2158,35 +2158,6 @@ class Pid: """Auto-generated default constructor with named params""" pass - def set_config(self, config: Pid.Config, reset_state: bool = True) -> None: - """* - * @brief Change the gains and other configuration for the PID controller. - * @param config Configuration struct with new gains and sampling time. - * @param reset_state Reset / clear the PID controller state. - - """ - pass - - def clear(self) -> None: - """* - * @brief Clear the PID controller state. - - """ - pass - - def update(self, error: float) -> float: - """* - * @brief Update the PID controller with the latest error measurement, - * getting the output control signal in return. - * - * @note Tracks invocation timing to better compute time-accurate - * integral/derivative signals. - * - * @param error Latest error signal. - * @return The output control signal based on the PID state and error. - - """ - pass def __call__(self, error: float) -> float: """* @@ -3469,10 +3440,34 @@ class Joystick: #################### #################### +#################### #################### + + + +class MediaType(enum.IntEnum): + """/ Describes a media type for RTSP tracks.""" + video = enum.auto() # (= 0) #/< Video media (MJPEG, H264, etc.) + audio = enum.auto() # (= 1) #/< Audio media (PCM, Opus, AAC, etc.) + +class RtpPayloadChunk: + """/ Represents one RTP payload chunk ready to be wrapped in an RtpPacket. + / Packetizers produce these; the server wraps them with RTP headers. + """ + data: List[int] #/< The payload data for this chunk + marker: bool = bool(False) #/< Set on last chunk of a frame/access unit + def __init__(self, data: List[int] = List[int](), marker: bool = bool(False)) -> None: + """Auto-generated default constructor with named params""" + pass + + +#################### #################### + + #################### #################### + class RtpJpegPacket: """/ RTP packet for JPEG video. / The RTP payload for JPEG is defined in RFC 2435. @@ -3727,6 +3722,7 @@ class JpegFrame: #################### #################### + class JpegHeader: """/ A class to generate a JPEG header for a given image size and quantization tables. / The header is generated once and then cached for future use. @@ -4041,6 +4037,7 @@ class RtspClient: / \snippet rtsp_example.cpp rtsp_client_example """ + class Config: """/ Configuration for the RTSP client""" server_address: str #/< The server IP Address to connect to @@ -4048,14 +4045,23 @@ class RtspClient: path: str = str("/mjpeg/1") #/< The path to the RTSP stream on the server. Will be appended #/< to the server address and port to form the full path of the #/< form "rtsp://:" - on_jpeg_frame: RtspClient.jpeg_frame_callback_t #/< The callback to call when a JPEG frame is received + + #/ Generic frame callback for any codec (track_id, raw frame data) + on_frame: frame_callback_t = frame_callback_t(None) + + #/ JPEG-specific frame callback (backward compatible). + #/ If set and no depacketizer is registered for PT 26, an MjpegDepacketizer + #/ is automatically created. + on_jpeg_frame: jpeg_frame_callback_t = jpeg_frame_callback_t(None) + log_level: Logger.Verbosity = Logger.Verbosity.info #/< The verbosity of the logger def __init__( self, server_address: str = "", rtsp_port: int = int(8554), path: str = str("/mjpeg/1"), - on_jpeg_frame: RtspClient.jpeg_frame_callback_t = RtspClient.jpeg_frame_callback_t(), + on_frame: frame_callback_t = frame_callback_t(None), + on_jpeg_frame: jpeg_frame_callback_t = jpeg_frame_callback_t(None), log_level: Logger.Verbosity = Logger.Verbosity.info ) -> None: """Auto-generated default constructor with named params""" @@ -4144,6 +4150,15 @@ class RtspClient: """ pass + def add_depacketizer(self, payload_type: int, depacketizer: RtpDepacketizer) -> None: + """/ Register a depacketizer for a specific RTP payload type. + / When RTP packets with this payload type are received, they are + / dispatched to the registered depacketizer. + / @param payload_type The RTP payload type (e.g., 26 for MJPEG, 96 for H264) + / @param depacketizer The depacketizer to handle packets of this type + """ + pass + def play(self, ec: std.error_code) -> None: """/ Play the RTSP stream / Sends the PLAY request to the RTSP server and parses the response. @@ -4212,6 +4227,18 @@ class RtspServer: """Auto-generated default constructor with named params""" pass + class TrackConfig: + """/ Configuration for a media track to be registered with the server""" + track_id: int = int(0) #/< Track identifier + packetizer: RtpPacketizer #/< Codec-specific packetizer + def __init__( + self, + track_id: int = int(0), + packetizer: RtpPacketizer = RtpPacketizer() + ) -> None: + """Auto-generated default constructor with named params""" + pass + def set_session_log_level(self, log_level: Logger.Verbosity) -> None: @@ -4240,13 +4267,32 @@ class RtspServer: """ pass + def add_track(self, config: RtspServer.TrackConfig) -> None: + """/ @brief Register a media track with the server. + / Each track has its own packetizer, SSRC, and sequence number. + / @param config Track configuration including the packetizer. + """ + pass + + @overload + def send_frame(self, track_id: int, frame_data: std.span[ int]) -> None: + """/ @brief Send a frame on a specific track. + / The track's packetizer splits the frame into RTP payload chunks, + / which are then wrapped with RTP headers and queued for delivery. + / @note Overwrites any existing pending packets for this track. + / @param track_id The track to send on. + / @param frame_data Raw encoded frame data. + """ + pass + + @overload def send_frame(self, frame: JpegFrame) -> None: - """/ @brief Send a frame over the RTSP connection - / Converts the full JPEG frame into a series of simplified RTP/JPEG - / packets and stores it to be sent over the RTP socket, but does not - / actually send it - / @note Overwrites any existing frame that has not been sent - / @param frame The frame to send + """/ @brief Send a JPEG frame over the RTSP connection (backward compatible). + / If no tracks have been added, lazily creates a default MJPEG track on + / track 0. Uses the legacy RtpJpegPacket packetization to preserve the + / exact wire format for existing MJPEG users. + / @note Overwrites any existing frame that has not been sent. + / @param frame The frame to send. """ pass @@ -4268,11 +4314,30 @@ class RtspSession: """/ Class that reepresents an RTSP session, which is uniquely identified by a / session id and sends frame data over RTP and RTCP to the client """ + class Track: + """/ Represents one media track within an RTSP session""" + track_id: int = int(0) #/< Track identifier (matches trackID=N in SDP) + control_path: str #/< Control path suffix (e.g., "trackID=0") + rtp_socket: UdpSocket #/< RTP socket for this track + rtcp_socket: UdpSocket #/< RTCP socket for this track + client_rtp_port: int = int(0) #/< Client's RTP port + client_rtcp_port: int = int(0) #/< Client's RTCP port + setup_complete: bool = bool(False) #/< Whether SETUP has been completed for this track + + def __init__(self) -> None: + pass + class Config: """/ Configuration for the RTSP session""" server_address: str #/< The address of the server rtsp_path: str #/< The RTSP path of the session receive_timeout: std.chrono.duration[float] = std.chrono.seconds(5) #/< The timeout for receiving data. Should be > 0. + #/ SDP generator callback. If set, called during DESCRIBE to produce the SDP body. + #/ If not set, a default MJPEG SDP is generated for backward compatibility. + #/ @param session_path Full RTSP path (e.g., "rtsp://ip:port/path") + #/ @param session_id The session ID + #/ @param server_address The server address with port + sdp_generator: std.function[str( str session_path, int session_id, str server_address)] log_level: Logger.Verbosity = Logger.Verbosity.warn #/< The log level of the session def __init__( self, @@ -4331,15 +4396,35 @@ class RtspSession: """ pass + @overload + def send_rtp_packet(self, track_id: int, packet: RtpPacket) -> bool: + """/ Send an RTP packet on a specific track + / @param track_id The track to send on + / @param packet The RTP packet to send + / @return True if the packet was sent successfully, False otherwise + """ + pass + + @overload def send_rtp_packet(self, packet: RtpPacket) -> bool: - """/ Send an RTP packet to the client + """/ Send an RTP packet to the client (backward compat — sends on default track 0) / @param packet The RTP packet to send / @return True if the packet was sent successfully, False otherwise """ pass + @overload + def send_rtcp_packet(self, track_id: int, packet: RtcpPacket) -> bool: + """/ Send an RTCP packet on a specific track + / @param track_id The track to send on + / @param packet The RTCP packet to send + / @return True if the packet was sent successfully, False otherwise + """ + pass + + @overload def send_rtcp_packet(self, packet: RtcpPacket) -> bool: - """/ Send an RTCP packet to the client + """/ Send an RTCP packet to the client (backward compat — sends on default track 0) / @param packet The RTCP packet to send / @return True if the packet was sent successfully, False otherwise """ @@ -4352,6 +4437,499 @@ class RtspSession: #################### #################### +#################### #################### + + + + +class GenericDepacketizer: + """/ A generic RTP depacketizer that reassembles media frames from incoming RTP + / packets. It accumulates payload data until a packet with the marker bit set + / is received, then delivers the complete frame via the frame callback. If a + / packet arrives with a different RTP timestamp than the current accumulation + / buffer, the old buffer is discarded and a new one is started. + / + / This is suitable for audio codecs (PCM, G.711, Opus, etc.) or any payload + / format that uses simple marker-based framing. + / + / \section generic_depacketizer_ex1 Example + / \snippet generic_depacketizer_example.cpp generic_depacketizer example + """ + class Config: + """/ Configuration for GenericDepacketizer.""" + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + + def process_packet(self, packet: RtpPacket) -> None: + """/ Process an incoming RTP packet. + / Payload data is accumulated until a packet with the marker bit set is + / received. At that point the assembled frame is delivered via the frame + / callback and the buffer is reset. + / @param packet The RTP packet to process. + """ + pass + + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + +#################### #################### + + + + +class H264Depacketizer: + """/ @brief RTP depacketizer for H.264 video per RFC 6184. + / + / Reassembles H.264 access units from incoming RTP packets. Supports: + / - **Single NAL unit** packets (NAL type 1–23) + / - **STAP-A** aggregation packets (NAL type 24) + / - **FU-A** fragmentation packets (NAL type 28) + / + / When the RTP marker bit is set, the accumulated NAL units are delivered + / as one Annex B byte-stream (each NAL prefixed with 0x00 0x00 0x00 0x01) + / via the frame callback set with set_frame_callback(). + / + / \section h264_depacketizer_ex1 Example + / \snippet h264_depacketizer_example.cpp h264_depacketizer example + """ + class Config: + """/ Configuration for the H264Depacketizer.""" + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + + def process_packet(self, packet: RtpPacket) -> None: + """/ Process an incoming RTP packet containing H.264 payload. + / + / Handles single NAL, STAP-A, and FU-A packet types. NAL units are + / buffered until the RTP marker bit indicates the end of an access unit, + / at which point the complete Annex B frame is delivered via the callback. + / + / @param packet The RTP packet to process. + """ + pass + + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + +#################### #################### + + + + +class MjpegDepacketizer: + """/ MJPEG depacketizer that reassembles JPEG frames from RTP packets. + / + / This class receives individual RTP packets containing RFC 2435 MJPEG + / payloads, reassembles the scan data fragments, reconstructs the JPEG + / header from the MJPEG header fields, and delivers complete JPEG frames + / through callbacks. + """ + + class Config: + """/ Configuration for the MJPEG depacketizer.""" + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + def set_jpeg_frame_callback(self, cb: jpeg_frame_callback_t) -> None: + """/ Set callback for receiving complete JPEG frames. + / @param cb Callback receiving a shared pointer to the completed JpegFrame. + """ + pass + + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + +#################### #################### + + + + +class RtpDepacketizer: + """/ Abstract base class for reassembling media frames from incoming RTP packets. + / Concrete depacketizers (e.g. MJPEG, H.264) override process_packet() to + / accumulate payload data and invoke the frame callback when a complete frame + / has been assembled. + """ + + class Config: + """/ Configuration for RtpDepacketizer.""" + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + + def process_packet(self, packet: RtpPacket) -> None: + """/ Process an incoming RTP packet, accumulating payload data. + / When a complete frame is assembled the frame callback is invoked. + / @param packet The RTP packet to process. + """ + pass + + def set_frame_callback(self, cb: frame_callback_t) -> None: + """/ Set the callback for completed frames. + / @param cb The callback to invoke when a full frame is ready. + """ + pass + + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + +#################### #################### + + + + +class GenericPacketizer: + """/ A generic RTP packetizer suitable for audio codecs (PCM, G.711, Opus, etc.) + / or any pre-formatted data that simply needs MTU-based chunking. It splits + / frame data into chunks of at most max_payload_size bytes and marks the last + / chunk with the RTP marker bit. + / + / \section generic_packetizer_ex1 Example + / \snippet generic_packetizer_example.cpp generic_packetizer example + """ + class Config: + """/ Configuration for GenericPacketizer.""" + max_payload_size: int = int(1400) #/< Maximum payload bytes per RTP packet + payload_type: int = int(96) #/< RTP payload type number + clock_rate: int = int(48000) #/< Clock rate in Hz for RTP timestamps + encoding_name: str = str("L16") #/< Encoding name for SDP rtpmap line + channels: int = int(1) #/< Number of audio channels + fmtp: str #/< Optional format parameters for SDP fmtp line + media_type: MediaType = MediaType(MediaType.audio) #/< Media type for the SDP m= line + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + max_payload_size: int = int(1400), + payload_type: int = int(96), + clock_rate: int = int(48000), + encoding_name: str = str("L16"), + channels: int = int(1), + fmtp: str = "", + media_type: MediaType = MediaType(MediaType.audio), + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + + def packetize(self, frame_data: std.span[ int]) -> List[RtpPayloadChunk]: + """/ Split frame data into RTP payload chunks of at most max_payload_size. + / The last (or only) chunk has its marker flag set. + / @param frame_data The raw frame bytes to packetize. + / @return A vector of RtpPayloadChunk ready to be wrapped in RTP packets. + """ + pass + + def get_payload_type(self) -> int: + """/ Get the RTP payload type number. + / @return The configured RTP payload type. + """ + pass + + def get_clock_rate(self) -> int: + """/ Get the RTP clock rate. + / @return The configured clock rate in Hz. + """ + pass + + def get_sdp_media_attributes(self) -> str: + """/ Generate the SDP media-level attribute lines for this codec. + / Produces an a=rtpmap line and, if fmtp is non-empty, an a=fmtp line. + / @return A string containing the SDP a= lines. + """ + pass + + def get_sdp_media_line(self) -> str: + """/ Generate the SDP m= line for this codec. + / @return A string such as "m=audio 0 RTP/AVP 96". + """ + pass + + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + +#################### #################### + + + + +class H264Packetizer: + """/ @brief RTP packetizer for H.264 video per RFC 6184. + / + / Accepts H.264 access units in Annex B byte-stream format (NAL units + / separated by 0x00000001 or 0x000001 start codes) and produces a sequence + / of RTP payload chunks suitable for transmission. + / + / Supports two NAL-unit packetization strategies: + / - **Single NAL unit mode** — NAL fits within max_payload_size. + / - **FU-A fragmentation** — NAL exceeds max_payload_size (packetization_mode >= 1). + / + / @note This class does not manage RTP headers (sequence numbers, timestamps, + / SSRC). The caller wraps each returned chunk into an RtpPacket. + / + / \section h264_packetizer_ex1 Example + / \snippet h264_packetizer_example.cpp h264_packetizer example + """ + class Config: + """/ Configuration for the H264Packetizer.""" + max_payload_size: int = int(1400) #/< Maximum payload bytes per RTP packet + payload_type: int = int(96) #/< Dynamic RTP payload type (typically 96–127). + profile_level_id: str #/< H.264 profile-level-id hex string, e.g. "42C01E". + packetization_mode: int = int(1) #/< 0 = single NAL only, 1 = non-interleaved (FU-A allowed). + sps: List[int] #/< Sequence Parameter Set raw bytes (without start code). + pps: List[int] #/< Picture Parameter Set raw bytes (without start code). + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + max_payload_size: int = int(1400), + payload_type: int = int(96), + profile_level_id: str = "", + packetization_mode: int = int(1), + sps: List[int] = List[int](), + pps: List[int] = List[int](), + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + + def packetize(self, frame_data: std.span[ int]) -> List[RtpPayloadChunk]: + """/ Packetize a complete H.264 access unit (Annex B format). + / + / The input may contain multiple NAL units separated by 3-byte or 4-byte + / start codes. Each NAL is individually packetized (single NAL or FU-A). + / The marker bit is set on the last chunk of the last NAL unit in the + / access unit. + / + / @param frame_data Raw Annex B byte-stream of one access unit. + / @return Vector of RTP payload chunks ready for transmission. + """ + pass + + def packetize_nal( + self, + nal_data: std.span[ int], + is_last_nal: bool = True + ) -> List[RtpPayloadChunk]: + """/ Packetize a single pre-parsed NAL unit (no start code prefix). + / + / @param nal_data The raw NAL unit bytes (including NAL header byte). + / @param is_last_nal If True, the marker bit is set on the last chunk. + / @return Vector of RTP payload chunks for this NAL. + """ + pass + + def set_sps_pps(self, sps: std.span[ int], pps: std.span[ int]) -> None: + """/ Update the SPS and PPS used for SDP generation. + / @param sps Sequence Parameter Set raw bytes. + / @param pps Picture Parameter Set raw bytes. + """ + pass + + def get_payload_type(self) -> int: + """/ Get the RTP payload type. + / @return The dynamic payload type configured for H.264. + """ + pass + + def get_clock_rate(self) -> int: + """/ Get the RTP clock rate for H.264 video. + / @return 90000 (fixed for H.264). + """ + pass + + def get_sdp_media_attributes(self) -> str: + """/ Get the SDP attribute lines for H.264. + / @return SDP a= lines (rtpmap and fmtp) without trailing CRLF. + """ + pass + + def get_sdp_media_line(self) -> str: + """/ Get the SDP m= media line for H.264. + / @return SDP m= line without trailing CRLF. + """ + pass + + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + +#################### #################### + + + +class MjpegPacketizer: + """/ MJPEG packetizer that fragments JPEG frames into RFC 2435 RTP payloads. + / + / This class takes complete JPEG frames and produces RTP payload chunks + / suitable for MJPEG streaming. Each chunk contains an RFC 2435 MJPEG + / header, and the first chunk additionally includes quantization tables. + """ + class Config: + """/ Configuration for the MJPEG packetizer.""" + max_payload_size: int = int(1400) #/< Maximum payload bytes per RTP packet + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + max_payload_size: int = int(1400), + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + def get_payload_type(self) -> int: + """/ Get the RTP payload type for MJPEG. + / @return 26 (static JPEG payload type). + """ + pass + + def get_clock_rate(self) -> int: + """/ Get the RTP clock rate for MJPEG. + / @return 90000 Hz. + """ + pass + + def get_sdp_media_attributes(self) -> str: + """/ Get the SDP media attributes for MJPEG. + / @return SDP rtpmap attribute string. + """ + pass + + def get_sdp_media_line(self) -> str: + """/ Get the SDP media line for MJPEG. + / @return SDP media description line. + """ + pass + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + +#################### #################### + + + + +class RtpPacketizer: + """/ Abstract base class for splitting media frames into RTP payload chunks. + / Concrete packetizers (e.g. MJPEG, H.264) override the pure-virtual methods + / to produce codec-specific payloads. The RTSP server wraps each returned + / RtpPayloadChunk with an RTP header before sending. + """ + class Config: + """/ Configuration for RtpPacketizer.""" + max_payload_size: int = int(1400) #/< Maximum payload bytes per RTP packet + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) #/< Log verbosity level + def __init__( + self, + max_payload_size: int = int(1400), + log_level: Logger.Verbosity = Logger.Verbosity(Logger.Verbosity.warn) + ) -> None: + """Auto-generated default constructor with named params""" + pass + + + + def packetize(self, frame_data: std.span[ int]) -> List[RtpPayloadChunk]: + """/ Packetize a complete media frame into RTP payload chunks. + / @param frame_data The raw frame bytes to packetize. + / @return A vector of RtpPayloadChunk ready to be wrapped in RTP packets. + """ + pass + + def get_payload_type(self) -> int: + """/ Get the RTP payload type number for this codec. + / @return The RTP payload type (e.g. 26 for MJPEG, 96 for dynamic). + """ + pass + + def get_clock_rate(self) -> int: + """/ Get the RTP clock rate for timestamp calculation. + / @return The clock rate in Hz (e.g. 90000 for video, 8000 for audio). + """ + pass + + def get_sdp_media_attributes(self) -> str: + """/ Generate the SDP media-level attributes for this codec. + / @return A string containing SDP a= lines (without trailing CRLF). + """ + pass + + def get_sdp_media_line(self) -> str: + """/ Generate the SDP m= line for this codec. + / @return A string containing the SDP m= line (without trailing CRLF). + """ + pass + + def __init__(self) -> None: + """Auto-generated default constructor""" + pass + + +#################### #################### + + #################### #################### diff --git a/lib/python_bindings/pybind_espp.cpp b/lib/python_bindings/pybind_espp.cpp index 8b40a2e23..6e74b5e01 100644 --- a/lib/python_bindings/pybind_espp.cpp +++ b/lib/python_bindings/pybind_espp.cpp @@ -758,7 +758,7 @@ void py_init_module_espp(py::module &m) { "distribution.\n") .def("unmap", &espp::RangeMapper::unmap, py::arg("v"), "*\n * @brief Unmap a value \\p v from the configured output range (centered,\n * " - " default [-1,1]) back into the input distribution.\n * @param T&v Value from the " + " default [-1,1]) back into the input distribution.\n * @param v Value from the " "centered output distribution.\n * @return Value within the input distribution.\n"); auto pyClassRangeMapper_float = py::class_>( m, "RangeMapper_float", py::dynamic_attr(), @@ -2457,6 +2457,428 @@ void py_init_module_espp(py::module &m) { "alive.\n"); //////////////////// //////////////////// + //////////////////// //////////////////// + auto pyEnumMediaType = + py::enum_(m, "MediaType", py::arithmetic(), + "/ Describes a media type for RTSP tracks.") + .value("video", espp::MediaType::VIDEO, "/< Video media (MJPEG, H264, etc.)") + .value("audio", espp::MediaType::AUDIO, "/< Audio media (PCM, Opus, AAC, etc.)"); + + auto pyClassRtpPayloadChunk = + py::class_( + m, "RtpPayloadChunk", py::dynamic_attr(), + "/ Represents one RTP payload chunk ready to be wrapped in an RtpPacket.\n/ Packetizers " + "produce these; the server wraps them with RTP headers.") + .def(py::init<>( + [](std::vector data = std::vector(), bool marker = {false}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->data = data; + r_ctor_->marker = marker; + return r_ctor_; + }), + py::arg("data") = std::vector(), py::arg("marker") = bool{false}) + .def_readwrite("data", &espp::RtpPayloadChunk::data, "/< The payload data for this chunk") + .def_readwrite("marker", &espp::RtpPayloadChunk::marker, + "/< Set on last chunk of a frame/access unit"); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassRtpDepacketizer = py::class_>( + m, "RtpDepacketizer", py::dynamic_attr(), + "/ Abstract base class for reassembling media frames from incoming RTP packets.\n/ Concrete " + "depacketizers (e.g. MJPEG, H.264) override process_packet() to\n/ accumulate payload data " + "and invoke the frame callback when a complete frame\n/ has been assembled."); + + { // inner classes & enums of RtpDepacketizer + auto pyClassRtpDepacketizer_ClassConfig = + py::class_(pyClassRtpDepacketizer, "Config", + py::dynamic_attr(), + "/ Configuration for RtpDepacketizer.") + .def( + py::init<>([](espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("log_level", &espp::RtpDepacketizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of RtpDepacketizer + + pyClassRtpDepacketizer + .def( + "process_packet", &espp::RtpDepacketizer::process_packet, py::arg("packet"), + "/ Process an incoming RTP packet, accumulating payload data.\n/ When a complete frame " + "is assembled the frame callback is invoked.\n/ @param packet The RTP packet to process.") + .def("set_frame_callback", &espp::RtpDepacketizer::set_frame_callback, py::arg("cb"), + "/ Set the callback for completed frames.\n/ @param cb The callback to invoke when a " + "full frame is ready."); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassGenericDepacketizer = py::class_>( + m, "GenericDepacketizer", py::dynamic_attr(), + "/ A generic RTP depacketizer that reassembles media frames from incoming RTP\n/ packets. It " + "accumulates payload data until a packet with the marker bit set\n/ is received, then " + "delivers the complete frame via the frame callback. If a\n/ packet arrives with a different " + "RTP timestamp than the current accumulation\n/ buffer, the old buffer is discarded and a " + "new one is started.\n/\n/ This is suitable for audio codecs (PCM, G.711, Opus, etc.) or any " + "payload\n/ format that uses simple marker-based framing.\n/\n/ \\section " + "generic_depacketizer_ex1 Example\n/ \\snippet generic_depacketizer_example.cpp " + "generic_depacketizer example"); + + { // inner classes & enums of GenericDepacketizer + auto pyClassGenericDepacketizer_ClassConfig = + py::class_(pyClassGenericDepacketizer, "Config", + py::dynamic_attr(), + "/ Configuration for GenericDepacketizer.") + .def( + py::init<>([](espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("log_level", &espp::GenericDepacketizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of GenericDepacketizer + + pyClassGenericDepacketizer + .def(py::init(), + py::arg("config") = espp::GenericDepacketizer::Config{}) + .def("process_packet", &espp::GenericDepacketizer::process_packet, py::arg("packet"), + "/ Process an incoming RTP packet.\n/ Payload data is accumulated until a packet with " + "the marker bit set is\n/ received. At that point the assembled frame is delivered via " + "the frame\n/ callback and the buffer is reset.\n/ @param packet The RTP packet to " + "process."); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassH264Depacketizer = py::class_>( + m, "H264Depacketizer", py::dynamic_attr(), + "/ @brief RTP depacketizer for H.264 video per RFC 6184.\n/\n/ Reassembles H.264 access " + "units from incoming RTP packets. Supports:\n/ - **Single NAL unit** packets (NAL type " + "1–23)\n/ - **STAP-A** aggregation packets (NAL type 24)\n/ - **FU-A** fragmentation " + "packets (NAL type 28)\n/\n/ When the RTP marker bit is set, the accumulated NAL units are " + "delivered\n/ as one Annex B byte-stream (each NAL prefixed with 0x00 0x00 0x00 0x01)\n/ via " + "the frame callback set with set_frame_callback().\n/\n/ \\section h264_depacketizer_ex1 " + "Example\n/ \\snippet h264_depacketizer_example.cpp h264_depacketizer example"); + + { // inner classes & enums of H264Depacketizer + auto pyClassH264Depacketizer_ClassConfig = + py::class_(pyClassH264Depacketizer, "Config", + py::dynamic_attr(), + "/ Configuration for the H264Depacketizer.") + .def( + py::init<>([](espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("log_level", &espp::H264Depacketizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of H264Depacketizer + + pyClassH264Depacketizer + .def(py::init(), + py::arg("config") = espp::H264Depacketizer::Config{}) + .def("process_packet", &espp::H264Depacketizer::process_packet, py::arg("packet"), + "/ Process an incoming RTP packet containing H.264 payload.\n/\n/ Handles single NAL, " + "STAP-A, and FU-A packet types. NAL units are\n/ buffered until the RTP marker bit " + "indicates the end of an access unit,\n/ at which point the complete Annex B frame is " + "delivered via the callback.\n/\n/ @param packet The RTP packet to process."); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassMjpegDepacketizer = py::class_>( + m, "MjpegDepacketizer", py::dynamic_attr(), + "/ MJPEG depacketizer that reassembles JPEG frames from RTP packets.\n/\n/ This class " + "receives individual RTP packets containing RFC 2435 MJPEG\n/ payloads, reassembles the scan " + "data fragments, reconstructs the JPEG\n/ header from the MJPEG header fields, and delivers " + "complete JPEG frames\n/ through callbacks."); + + { // inner classes & enums of MjpegDepacketizer + auto pyClassMjpegDepacketizer_ClassConfig = + py::class_(pyClassMjpegDepacketizer, "Config", + py::dynamic_attr(), + "/ Configuration for the MJPEG depacketizer.") + .def( + py::init<>([](espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("log_level", &espp::MjpegDepacketizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of MjpegDepacketizer + + pyClassMjpegDepacketizer + .def(py::init(), + py::arg("config") = espp::MjpegDepacketizer::Config{}) + .def("set_jpeg_frame_callback", &espp::MjpegDepacketizer::set_jpeg_frame_callback, + py::arg("cb"), + "/ Set callback for receiving complete JPEG frames.\n/ @param cb Callback receiving a " + "shared pointer to the completed JpegFrame."); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassRtpPacketizer = py::class_>( + m, "RtpPacketizer", py::dynamic_attr(), + "/ Abstract base class for splitting media frames into RTP payload chunks.\n/ Concrete " + "packetizers (e.g. MJPEG, H.264) override the pure-virtual methods\n/ to produce " + "codec-specific payloads. The RTSP server wraps each returned\n/ RtpPayloadChunk with an RTP " + "header before sending."); + + { // inner classes & enums of RtpPacketizer + auto pyClassRtpPacketizer_ClassConfig = + py::class_(pyClassRtpPacketizer, "Config", py::dynamic_attr(), + "/ Configuration for RtpPacketizer.") + .def( + py::init<>([](size_t max_payload_size = {1400}, + espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->max_payload_size = max_payload_size; + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("max_payload_size") = size_t{1400}, + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("max_payload_size", &espp::RtpPacketizer::Config::max_payload_size, + "/< Maximum payload bytes per RTP packet") + .def_readwrite("log_level", &espp::RtpPacketizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of RtpPacketizer + + pyClassRtpPacketizer + .def("packetize", &espp::RtpPacketizer::packetize, py::arg("frame_data"), + "/ Packetize a complete media frame into RTP payload chunks.\n/ @param frame_data The " + "raw frame bytes to packetize.\n/ @return A vector of RtpPayloadChunk ready to be " + "wrapped in RTP packets.") + .def("get_payload_type", &espp::RtpPacketizer::get_payload_type, + "/ Get the RTP payload type number for this codec.\n/ @return The RTP payload type " + "(e.g. 26 for MJPEG, 96 for dynamic).") + .def("get_clock_rate", &espp::RtpPacketizer::get_clock_rate, + "/ Get the RTP clock rate for timestamp calculation.\n/ @return The clock rate in Hz " + "(e.g. 90000 for video, 8000 for audio).") + .def("get_sdp_media_attributes", &espp::RtpPacketizer::get_sdp_media_attributes, + "/ Generate the SDP media-level attributes for this codec.\n/ @return A string " + "containing SDP a= lines (without trailing CRLF).") + .def("get_sdp_media_line", &espp::RtpPacketizer::get_sdp_media_line, + "/ Generate the SDP m= line for this codec.\n/ @return A string containing the SDP m= " + "line (without trailing CRLF)."); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassGenericPacketizer = py::class_>( + m, "GenericPacketizer", py::dynamic_attr(), + "/ A generic RTP packetizer suitable for audio codecs (PCM, G.711, Opus, etc.)\n/ or any " + "pre-formatted data that simply needs MTU-based chunking. It splits\n/ frame data into " + "chunks of at most max_payload_size bytes and marks the last\n/ chunk with the RTP marker " + "bit.\n/\n/ \\section generic_packetizer_ex1 Example\n/ \\snippet " + "generic_packetizer_example.cpp generic_packetizer example"); + + { // inner classes & enums of GenericPacketizer + auto pyClassGenericPacketizer_ClassConfig = + py::class_(pyClassGenericPacketizer, "Config", + py::dynamic_attr(), + "/ Configuration for GenericPacketizer.") + .def( + py::init<>([](size_t max_payload_size = {1400}, int payload_type = {96}, + uint32_t clock_rate = {48000}, std::string encoding_name = {"L16"}, + int channels = {1}, std::string fmtp = std::string(), + espp::MediaType media_type = {espp::MediaType::AUDIO}, + espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->max_payload_size = max_payload_size; + r_ctor_->payload_type = payload_type; + r_ctor_->clock_rate = clock_rate; + r_ctor_->encoding_name = encoding_name; + r_ctor_->channels = channels; + r_ctor_->fmtp = fmtp; + r_ctor_->media_type = media_type; + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("max_payload_size") = size_t{1400}, py::arg("payload_type") = int{96}, + py::arg("clock_rate") = uint32_t{48000}, + py::arg("encoding_name") = std::string{"L16"}, py::arg("channels") = int{1}, + py::arg("fmtp") = std::string(), + py::arg("media_type") = espp::MediaType{espp::MediaType::AUDIO}, + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("max_payload_size", &espp::GenericPacketizer::Config::max_payload_size, + "/< Maximum payload bytes per RTP packet") + .def_readwrite("payload_type", &espp::GenericPacketizer::Config::payload_type, + "/< RTP payload type number") + .def_readwrite("clock_rate", &espp::GenericPacketizer::Config::clock_rate, + "/< Clock rate in Hz for RTP timestamps") + .def_readwrite("encoding_name", &espp::GenericPacketizer::Config::encoding_name, + "/< Encoding name for SDP rtpmap line") + .def_readwrite("channels", &espp::GenericPacketizer::Config::channels, + "/< Number of audio channels") + .def_readwrite("fmtp", &espp::GenericPacketizer::Config::fmtp, + "/< Optional format parameters for SDP fmtp line") + .def_readwrite("media_type", &espp::GenericPacketizer::Config::media_type, + "/< Media type for the SDP m= line") + .def_readwrite("log_level", &espp::GenericPacketizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of GenericPacketizer + + pyClassGenericPacketizer + .def(py::init(), + py::arg("config") = espp::GenericPacketizer::Config{}) + .def("packetize", &espp::GenericPacketizer::packetize, py::arg("frame_data"), + "/ Split frame data into RTP payload chunks of at most max_payload_size.\n/ The last " + "(or only) chunk has its marker flag set.\n/ @param frame_data The raw frame bytes to " + "packetize.\n/ @return A vector of RtpPayloadChunk ready to be wrapped in RTP packets.") + .def("get_payload_type", &espp::GenericPacketizer::get_payload_type, + "/ Get the RTP payload type number.\n/ @return The configured RTP payload type.") + .def("get_clock_rate", &espp::GenericPacketizer::get_clock_rate, + "/ Get the RTP clock rate.\n/ @return The configured clock rate in Hz.") + .def("get_sdp_media_attributes", &espp::GenericPacketizer::get_sdp_media_attributes, + "/ Generate the SDP media-level attribute lines for this codec.\n/ Produces an a=rtpmap " + "line and, if fmtp is non-empty, an a=fmtp line.\n/ @return A string containing the SDP " + "a= lines.") + .def("get_sdp_media_line", &espp::GenericPacketizer::get_sdp_media_line, + "/ Generate the SDP m= line for this codec.\n/ @return A string such as \"m=audio 0 " + "RTP/AVP 96\"."); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassH264Packetizer = py::class_>( + m, "H264Packetizer", py::dynamic_attr(), + "/ @brief RTP packetizer for H.264 video per RFC 6184.\n/\n/ Accepts H.264 access units in " + "Annex B byte-stream format (NAL units\n/ separated by 0x00000001 or 0x000001 start codes) " + "and produces a sequence\n/ of RTP payload chunks suitable for transmission.\n/\n/ Supports " + "two NAL-unit packetization strategies:\n/ - **Single NAL unit mode** — NAL fits within " + "max_payload_size.\n/ - **FU-A fragmentation** — NAL exceeds max_payload_size " + "(packetization_mode >= 1).\n/\n/ @note This class does not manage RTP headers (sequence " + "numbers, timestamps,\n/ SSRC). The caller wraps each returned chunk into an " + "RtpPacket.\n/\n/ \\section h264_packetizer_ex1 Example\n/ \\snippet " + "h264_packetizer_example.cpp h264_packetizer example"); + + { // inner classes & enums of H264Packetizer + auto pyClassH264Packetizer_ClassConfig = + py::class_(pyClassH264Packetizer, "Config", + py::dynamic_attr(), + "/ Configuration for the H264Packetizer.") + .def(py::init<>( + [](size_t max_payload_size = {1400}, int payload_type = {96}, + std::string profile_level_id = std::string(), int packetization_mode = {1}, + std::vector sps = std::vector(), + std::vector pps = std::vector(), + espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->max_payload_size = max_payload_size; + r_ctor_->payload_type = payload_type; + r_ctor_->profile_level_id = profile_level_id; + r_ctor_->packetization_mode = packetization_mode; + r_ctor_->sps = sps; + r_ctor_->pps = pps; + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("max_payload_size") = size_t{1400}, py::arg("payload_type") = int{96}, + py::arg("profile_level_id") = std::string(), + py::arg("packetization_mode") = int{1}, py::arg("sps") = std::vector(), + py::arg("pps") = std::vector(), + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("max_payload_size", &espp::H264Packetizer::Config::max_payload_size, + "/< Maximum payload bytes per RTP packet") + .def_readwrite("payload_type", &espp::H264Packetizer::Config::payload_type, + "/< Dynamic RTP payload type (typically 96–127).") + .def_readwrite("profile_level_id", &espp::H264Packetizer::Config::profile_level_id, + "/< H.264 profile-level-id hex string, e.g. \"42C01E\".") + .def_readwrite("packetization_mode", &espp::H264Packetizer::Config::packetization_mode, + "/< 0 = single NAL only, 1 = non-interleaved (FU-A allowed).") + .def_readwrite("sps", &espp::H264Packetizer::Config::sps, + "/< Sequence Parameter Set raw bytes (without start code).") + .def_readwrite("pps", &espp::H264Packetizer::Config::pps, + "/< Picture Parameter Set raw bytes (without start code).") + .def_readwrite("log_level", &espp::H264Packetizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of H264Packetizer + + pyClassH264Packetizer + .def(py::init(), + py::arg("config") = espp::H264Packetizer::Config{}) + .def("packetize", &espp::H264Packetizer::packetize, py::arg("frame_data"), + "/ Packetize a complete H.264 access unit (Annex B format).\n/\n/ The input may contain " + "multiple NAL units separated by 3-byte or 4-byte\n/ start codes. Each NAL is " + "individually packetized (single NAL or FU-A).\n/ The marker bit is set on the last " + "chunk of the last NAL unit in the\n/ access unit.\n/\n/ @param frame_data Raw Annex B " + "byte-stream of one access unit.\n/ @return Vector of RTP payload chunks ready for " + "transmission.") + .def("packetize_nal", &espp::H264Packetizer::packetize_nal, py::arg("nal_data"), + py::arg("is_last_nal") = true, + "/ Packetize a single pre-parsed NAL unit (no start code prefix).\n/\n/ @param nal_data " + "The raw NAL unit bytes (including NAL header byte).\n/ @param is_last_nal If True, the " + "marker bit is set on the last chunk.\n/ @return Vector of RTP payload chunks for this " + "NAL.") + .def("set_sps_pps", &espp::H264Packetizer::set_sps_pps, py::arg("sps"), py::arg("pps"), + "/ Update the SPS and PPS used for SDP generation.\n/ @param sps Sequence Parameter Set " + "raw bytes.\n/ @param pps Picture Parameter Set raw bytes.") + .def("get_payload_type", &espp::H264Packetizer::get_payload_type, + "/ Get the RTP payload type.\n/ @return The dynamic payload type configured for H.264.") + .def("get_clock_rate", &espp::H264Packetizer::get_clock_rate, + "/ Get the RTP clock rate for H.264 video.\n/ @return 90000 (fixed for H.264).") + .def("get_sdp_media_attributes", &espp::H264Packetizer::get_sdp_media_attributes, + "/ Get the SDP attribute lines for H.264.\n/ @return SDP a= lines (rtpmap and fmtp) " + "without trailing CRLF.") + .def("get_sdp_media_line", &espp::H264Packetizer::get_sdp_media_line, + "/ Get the SDP m= media line for H.264.\n/ @return SDP m= line without trailing CRLF."); + //////////////////// //////////////////// + + //////////////////// //////////////////// + auto pyClassMjpegPacketizer = py::class_>( + m, "MjpegPacketizer", py::dynamic_attr(), + "/ MJPEG packetizer that fragments JPEG frames into RFC 2435 RTP payloads.\n/\n/ This class " + "takes complete JPEG frames and produces RTP payload chunks\n/ suitable for MJPEG streaming. " + "Each chunk contains an RFC 2435 MJPEG\n/ header, and the first chunk additionally includes " + "quantization tables."); + + { // inner classes & enums of MjpegPacketizer + auto pyClassMjpegPacketizer_ClassConfig = + py::class_(pyClassMjpegPacketizer, "Config", + py::dynamic_attr(), + "/ Configuration for the MJPEG packetizer.") + .def( + py::init<>([](size_t max_payload_size = {1400}, + espp::Logger::Verbosity log_level = {espp::Logger::Verbosity::WARN}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->max_payload_size = max_payload_size; + r_ctor_->log_level = log_level; + return r_ctor_; + }), + py::arg("max_payload_size") = size_t{1400}, + py::arg("log_level") = espp::Logger::Verbosity{espp::Logger::Verbosity::WARN}) + .def_readwrite("max_payload_size", &espp::MjpegPacketizer::Config::max_payload_size, + "/< Maximum payload bytes per RTP packet") + .def_readwrite("log_level", &espp::MjpegPacketizer::Config::log_level, + "/< Log verbosity level"); + } // end of inner classes & enums of MjpegPacketizer + + pyClassMjpegPacketizer + .def(py::init(), + py::arg("config") = espp::MjpegPacketizer::Config{}) + .def("get_payload_type", &espp::MjpegPacketizer::get_payload_type, + "/ Get the RTP payload type for MJPEG.\n/ @return 26 (static JPEG payload type).") + .def("get_clock_rate", &espp::MjpegPacketizer::get_clock_rate, + "/ Get the RTP clock rate for MJPEG.\n/ @return 90000 Hz.") + .def("get_sdp_media_attributes", &espp::MjpegPacketizer::get_sdp_media_attributes, + "/ Get the SDP media attributes for MJPEG.\n/ @return SDP rtpmap attribute string.") + .def("get_sdp_media_line", &espp::MjpegPacketizer::get_sdp_media_line, + "/ Get the SDP media line for MJPEG.\n/ @return SDP media description line."); + //////////////////// //////////////////// + //////////////////// //////////////////// auto pyClassRtpJpegPacket = py::class_( @@ -2704,25 +3126,46 @@ void py_init_module_espp(py::module &m) { "rtsp_client_ex1 RtspClient Example\n/ \\snippet rtsp_example.cpp rtsp_client_example"); { // inner classes & enums of RtspClient + auto pyClassRtspClient_ClassTrackInfo = + py::class_(pyClassRtspClient, "TrackInfo", py::dynamic_attr(), + "/ Parsed SDP metadata for a discovered RTSP track") + .def(py::init<>()) + .def_readwrite("track_id", &espp::RtspClient::TrackInfo::track_id, + "/< Track identifier (matches trackID=N in SDP)") + .def_readwrite("payload_type", &espp::RtspClient::TrackInfo::payload_type, + "/< RTP payload type") + .def_readwrite("clock_rate", &espp::RtspClient::TrackInfo::clock_rate, + "/< RTP clock rate in Hz") + .def_readwrite("channels", &espp::RtspClient::TrackInfo::channels, + "/< Number of audio channels") + .def_readwrite("media_type", &espp::RtspClient::TrackInfo::media_type, + "/< SDP media type (audio/video)") + .def_readwrite("encoding_name", &espp::RtspClient::TrackInfo::encoding_name, + "/< Codec / encoding name from a=rtpmap") + .def_readwrite("control_path", &espp::RtspClient::TrackInfo::control_path, + "/< Resolved control path used for SETUP"); + auto pyClassRtspClient_ClassConfig = py::class_(pyClassRtspClient, "Config", py::dynamic_attr(), "/ Configuration for the RTSP client") .def(py::init<>([](std::string server_address = std::string(), int rtsp_port = {8554}, std::string path = {"/mjpeg/1"}, - espp::RtspClient::jpeg_frame_callback_t on_jpeg_frame = - espp::RtspClient::jpeg_frame_callback_t(), + espp::RtspClient::frame_callback_t on_frame = {nullptr}, + espp::RtspClient::jpeg_frame_callback_t on_jpeg_frame = {nullptr}, espp::Logger::Verbosity log_level = espp::Logger::Verbosity::INFO) { auto r_ctor_ = std::make_unique(); r_ctor_->server_address = server_address; r_ctor_->rtsp_port = rtsp_port; r_ctor_->path = path; + r_ctor_->on_frame = on_frame; r_ctor_->on_jpeg_frame = on_jpeg_frame; r_ctor_->log_level = log_level; return r_ctor_; }), py::arg("server_address") = std::string(), py::arg("rtsp_port") = int{8554}, py::arg("path") = std::string{"/mjpeg/1"}, - py::arg("on_jpeg_frame") = espp::RtspClient::jpeg_frame_callback_t(), + py::arg("on_frame") = espp::RtspClient::frame_callback_t{nullptr}, + py::arg("on_jpeg_frame") = espp::RtspClient::jpeg_frame_callback_t{nullptr}, py::arg("log_level") = espp::Logger::Verbosity::INFO) .def_readwrite("server_address", &espp::RtspClient::Config::server_address, "/< The server IP Address to connect to") @@ -2730,13 +3173,17 @@ void py_init_module_espp(py::module &m) { "/< The port of the RTSP server") .def_readwrite("path", &espp::RtspClient::Config::path, "/< The path to the RTSP stream on the server. Will be appended") + .def_readwrite("on_frame", &espp::RtspClient::Config::on_frame, + "/ Generic frame callback for any codec (track_id, raw frame data)") .def_readwrite("on_jpeg_frame", &espp::RtspClient::Config::on_jpeg_frame, - "/< The callback to call when a JPEG frame is received") + "/ JPEG-specific frame callback (backward compatible).\n/ If set and no " + "depacketizer is registered for PT 26, an MjpegDepacketizer\n/ is " + "automatically created.") .def_readwrite("log_level", &espp::RtspClient::Config::log_level, "/< The verbosity of the logger"); } // end of inner classes & enums of RtspClient - pyClassRtspClient.def(py::init()) + pyClassRtspClient.def(py::init(), py::arg("config")) .def("send_request", &espp::RtspClient::send_request, py::arg("method"), py::arg("path"), py::arg("extra_headers"), py::arg("ec"), "/ Send an RTSP request to the server\n/ \note This is a blocking call\n/ \note This " @@ -2764,6 +3211,10 @@ void py_init_module_espp(py::module &m) { .def("describe", &espp::RtspClient::describe, py::arg("ec"), "/ Describe the RTSP stream\n/ Sends the DESCRIBE request to the RTSP server and parses " "the response.\n/ \\param ec The error code to set if an error occurs") + .def( + "tracks", [](const espp::RtspClient &self) { return self.tracks(); }, + "/ Get the parsed SDP track descriptions from the most recent DESCRIBE call.\n/ " + "\\return The ordered set of discovered media tracks") .def("setup", py::overload_cast(&espp::RtspClient::setup), py::arg("ec"), "/ Setup the RTSP stream\n/ \note Starts the RTP and RTCP threads.\n/ Sends the SETUP " "request to the RTSP server and parses the response.\n/ \note The default ports are " @@ -2778,6 +3229,12 @@ void py_init_module_espp(py::module &m) { "port\n/ \\param rtcp_port The RTCP client port\n/ \\param receive_timeout The timeout " "for receiving RTP and RTCP packets\n/ \\param ec The error code to set if an error " "occurs") + .def("add_depacketizer", &espp::RtspClient::add_depacketizer, py::arg("payload_type"), + py::arg("depacketizer"), + "/ Register a depacketizer for a specific RTP payload type.\n/ When RTP packets with " + "this payload type are received, they are\n/ dispatched to the registered " + "depacketizer.\n/ @param payload_type The RTP payload type (e.g., 26 for MJPEG, 96 for " + "H264)\n/ @param depacketizer The depacketizer to handle packets of this type") .def("play", &espp::RtspClient::play, py::arg("ec"), "/ Play the RTSP stream\n/ Sends the PLAY request to the RTSP server and parses the " "response.\n/ \\param ec The error code to set if an error occurs") @@ -2826,9 +3283,26 @@ void py_init_module_espp(py::module &m) { "will be broken") .def_readwrite("log_level", &espp::RtspServer::Config::log_level, "/< The log level for the RTSP server"); + auto pyClassRtspServer_ClassTrackConfig = + py::class_( + pyClassRtspServer, "TrackConfig", py::dynamic_attr(), + "/ Configuration for a media track to be registered with the server") + .def(py::init<>([](int track_id = {0}, + std::shared_ptr packetizer = {nullptr}) { + auto r_ctor_ = std::make_unique(); + r_ctor_->track_id = track_id; + r_ctor_->packetizer = packetizer; + return r_ctor_; + }), + py::arg("track_id") = int{0}, + py::arg("packetizer") = std::shared_ptr{nullptr}) + .def_readwrite("track_id", &espp::RtspServer::TrackConfig::track_id, + "/< Track identifier") + .def_readwrite("packetizer", &espp::RtspServer::TrackConfig::packetizer, + "/< Codec-specific packetizer"); } // end of inner classes & enums of RtspServer - pyClassRtspServer.def(py::init()) + pyClassRtspServer.def(py::init(), py::arg("config")) .def("set_session_log_level", &espp::RtspServer::set_session_log_level, py::arg("log_level"), "/ @brief Sets the log level for the RTSP sessions created by this server\n/ @note This " "does not affect the log level of the RTSP server itself\n/ @note This does not change " @@ -2841,11 +3315,24 @@ void py_init_module_espp(py::module &m) { .def("stop", &espp::RtspServer::stop, "/ @brief Stop the FTP server\n/ Stops the accept task, session task, and closes the " "RTSP socket") - .def("send_frame", &espp::RtspServer::send_frame, py::arg("frame"), - "/ @brief Send a frame over the RTSP connection\n/ Converts the full JPEG frame into a " - "series of simplified RTP/JPEG\n/ packets and stores it to be sent over the RTP socket, " - "but does not\n/ actually send it\n/ @note Overwrites any existing frame that has not " - "been sent\n/ @param frame The frame to send"); + .def("add_track", &espp::RtspServer::add_track, py::arg("config"), + "/ @brief Register a media track with the server.\n/ Each track has its own packetizer, " + "SSRC, and sequence number.\n/ @param config Track configuration including the " + "packetizer.") + .def("send_frame", + py::overload_cast>(&espp::RtspServer::send_frame), + py::arg("track_id"), py::arg("frame_data"), + "/ @brief Send a frame on a specific track.\n/ The track's packetizer splits the frame " + "into RTP payload chunks,\n/ which are then wrapped with RTP headers and queued for " + "delivery.\n/ @note Overwrites any existing pending packets for this track.\n/ @param " + "track_id The track to send on.\n/ @param frame_data Raw encoded frame data.") + .def("send_frame", py::overload_cast(&espp::RtspServer::send_frame), + py::arg("frame"), + "/ @brief Send a JPEG frame over the RTSP connection (backward compatible).\n/ If no " + "tracks have been added, lazily creates a default MJPEG track on\n/ track 0. Uses the " + "legacy RtpJpegPacket packetization to preserve the\n/ exact wire format for existing " + "MJPEG users.\n/ @note Overwrites any existing frame that has not been sent.\n/ @param " + "frame The frame to send."); //////////////////// //////////////////// //////////////////// //////////////////// @@ -2855,6 +3342,24 @@ void py_init_module_espp(py::module &m) { "and sends frame data over RTP and RTCP to the client"); { // inner classes & enums of RtspSession + auto pyClassRtspSession_ClassTrack = + py::class_(pyClassRtspSession, "Track", py::dynamic_attr(), + "/ Represents one media track within an RTSP session") + .def_readwrite("track_id", &espp::RtspSession::Track::track_id, + "/< Track identifier (matches trackID=N in SDP)") + .def_readwrite("control_path", &espp::RtspSession::Track::control_path, + "/< Control path suffix (e.g., \"trackID=0\")") + .def_readonly("rtp_socket", &espp::RtspSession::Track::rtp_socket, + "/< RTP socket for this track") + .def_readonly("rtcp_socket", &espp::RtspSession::Track::rtcp_socket, + "/< RTCP socket for this track") + .def_readwrite("client_rtp_port", &espp::RtspSession::Track::client_rtp_port, + "/< Client's RTP port") + .def_readwrite("client_rtcp_port", &espp::RtspSession::Track::client_rtcp_port, + "/< Client's RTCP port") + .def_readwrite("setup_complete", &espp::RtspSession::Track::setup_complete, + "/< Whether SETUP has been completed for this track") + .def(py::init<>()); auto pyClassRtspSession_ClassConfig = py::class_(pyClassRtspSession, "Config", py::dynamic_attr(), "/ Configuration for the RTSP session") @@ -2879,12 +3384,14 @@ void py_init_module_espp(py::module &m) { "/< The RTSP path of the session") .def_readwrite("receive_timeout", &espp::RtspSession::Config::receive_timeout, "/< The timeout for receiving data. Should be > 0.") + .def_readwrite("sdp_generator", &espp::RtspSession::Config::sdp_generator, "") .def_readwrite("log_level", &espp::RtspSession::Config::log_level, "/< The log level of the session"); } // end of inner classes & enums of RtspSession pyClassRtspSession - .def(py::init, const espp::RtspSession::Config &>()) + .def(py::init, const espp::RtspSession::Config &>(), + py::arg("control_socket"), py::arg("config")) .def("get_session_id", &espp::RtspSession::get_session_id, "/ @brief Get the session id\n/ @return The session id") .def("is_closed", &espp::RtspSession::is_closed, @@ -2906,12 +3413,30 @@ void py_init_module_espp(py::module &m) { .def("teardown", &espp::RtspSession::teardown, "/ Teardown the session\n/ This will cause the server to stop sending frames to the " "client\n/ and close the connection") - .def("send_rtp_packet", &espp::RtspSession::send_rtp_packet, py::arg("packet"), - "/ Send an RTP packet to the client\n/ @param packet The RTP packet to send\n/ @return " - "True if the packet was sent successfully, False otherwise") - .def("send_rtcp_packet", &espp::RtspSession::send_rtcp_packet, py::arg("packet"), - "/ Send an RTCP packet to the client\n/ @param packet The RTCP packet to send\n/ " - "@return True if the packet was sent successfully, False otherwise"); + .def("send_rtp_packet", + py::overload_cast(&espp::RtspSession::send_rtp_packet), + py::arg("track_id"), py::arg("packet"), + "/ Send an RTP packet on a specific track\n/ @param track_id The track to send on\n/ " + "@param packet The RTP packet to send\n/ @return True if the packet was sent " + "successfully, False otherwise") + .def("send_rtp_packet", + py::overload_cast(&espp::RtspSession::send_rtp_packet), + py::arg("packet"), + "/ Send an RTP packet to the client (backward compat — sends on default track 0)\n/ " + "@param packet The RTP packet to send\n/ @return True if the packet was sent " + "successfully, False otherwise") + .def("send_rtcp_packet", + py::overload_cast(&espp::RtspSession::send_rtcp_packet), + py::arg("track_id"), py::arg("packet"), + "/ Send an RTCP packet on a specific track\n/ @param track_id The track to send on\n/ " + "@param packet The RTCP packet to send\n/ @return True if the packet was sent " + "successfully, False otherwise") + .def("send_rtcp_packet", + py::overload_cast(&espp::RtspSession::send_rtcp_packet), + py::arg("packet"), + "/ Send an RTCP packet to the client (backward compat — sends on default track 0)\n/ " + "@param packet The RTCP packet to send\n/ @return True if the packet was sent " + "successfully, False otherwise"); //////////////////// //////////////////// //////////////////// //////////////////// diff --git a/python/README.md b/python/README.md index 6719ed8fd..d4dfa9912 100644 --- a/python/README.md +++ b/python/README.md @@ -32,10 +32,21 @@ This section gives a brief overview of what the scripts in this folder do. incoming UDP packets and prints them to the console, while the client sends UDP packets to the server. - `rtsp_client.py` and `rtsp_server.py`: These scripts demonstrate how to use the - `espp` library to create an RTSP client and server. The server streams video - from a webcam or display capture, while the client receives the stream and - displays it in a window. Note: these scripts require additional 3rd party - libraries such as `opencv`, `mss`, and `zeroconf`. + `espp` library to create an RTSP client and server. The server streams MJPEG + video from a webcam or display capture. The camera path captures live + microphone audio by default, while display capture keeps the simulated audio + tone unless `--audio-source microphone` is selected. The client receives the + stream, validates audio delivery, plays the audio in real time, and displays + the video in a window. Note: these scripts require additional 3rd party + libraries such as `opencv`, `mss`, `zeroconf`, and `sounddevice`. +- `rtsp_client_multitrack.py` and `rtsp_server_multitrack.py`: These scripts + exercise the generic multitrack RTSP APIs. The multitrack server includes + audio by default; camera mode uses live microphone capture and the other video + sources use the simulated tone unless `--audio-source microphone` is selected. + Pass `--no-audio` to disable the audio track. The multitrack client plays + audio by default when running with a UI; use `--no-audio-playback` to disable + it or `--play-audio --headless` to exercise playback without opening the video + window. - `cobs_demo.py`: Demonstration of ESPP COBS functionality with native Python data types. Shows ESPP encoding/decoding, cross-library compatibility with the cobs-python library, and practical usage examples. Includes design differences explanation and validation @@ -62,9 +73,9 @@ README for more details. ### Install Python Requirements Some tests (e.g. `rtsp_client.py`, `rtsp_server.py`) make use of 3rd party -libraries such as `zeroconf, opencv, mss` to facilitate mDNS discovery, image -display / webcam capture, and display capture. For these tests, you will need to -install the `requirements.txt`: +libraries such as `zeroconf, opencv, mss, sounddevice` to facilitate mDNS +discovery, image display / webcam capture, display capture, and audio +playback. For these tests, you will need to install the `requirements.txt`: ```console # create the virtual environment @@ -103,4 +114,3 @@ another terminal: ```console python3 udp_server.py ``` - diff --git a/python/requirements.txt b/python/requirements.txt index 86c2d9f95..6da094518 100644 --- a/python/requirements.txt +++ b/python/requirements.txt @@ -3,3 +3,4 @@ opencv-python mss simplejpeg cobs +sounddevice diff --git a/python/rtsp_api_test.py b/python/rtsp_api_test.py new file mode 100755 index 000000000..d17c7a9a1 --- /dev/null +++ b/python/rtsp_api_test.py @@ -0,0 +1,286 @@ +"""Offline API tests for the new RTSP multi-codec packetizer/depacketizer classes. + +This script exercises the packetizer and depacketizer APIs without any network +activity, making it suitable for host-side validation of the RTSP component's +RTP layer. + +Usage: + python rtsp_api_test.py +""" + +import sys +import traceback + +from support_loader import espp + +passed = 0 +failed = 0 + + +def test(name, fn): + """Run a single test and track pass/fail.""" + global passed, failed + try: + fn() + print(f" [PASS] {name}") + passed += 1 + except Exception: + print(f" [FAIL] {name}") + traceback.print_exc() + failed += 1 + + +# --------------------------------------------------------------------------- +# MediaType enum +# --------------------------------------------------------------------------- +def test_media_type_enum(): + assert hasattr(espp, "MediaType"), "MediaType enum missing" + assert hasattr(espp.MediaType, "audio"), "MediaType.audio missing" + assert hasattr(espp.MediaType, "video"), "MediaType.video missing" + assert espp.MediaType.audio != espp.MediaType.video + + +# --------------------------------------------------------------------------- +# RtpPayloadChunk +# --------------------------------------------------------------------------- +def test_rtp_payload_chunk(): + chunk = espp.RtpPayloadChunk() + assert hasattr(chunk, "data"), "RtpPayloadChunk.data missing" + assert hasattr(chunk, "marker"), "RtpPayloadChunk.marker missing" + + +# --------------------------------------------------------------------------- +# MJPEG Packetizer +# --------------------------------------------------------------------------- +def test_mjpeg_packetizer_construction(): + cfg = espp.MjpegPacketizer.Config() + cfg.max_payload_size = 1400 + p = espp.MjpegPacketizer(cfg) + assert p.get_payload_type() == 26 + assert p.get_clock_rate() == 90000 + + +def test_mjpeg_packetizer_sdp(): + p = espp.MjpegPacketizer(espp.MjpegPacketizer.Config()) + ml = p.get_sdp_media_line() + assert "video" in ml.lower(), f"Expected 'video' in media line: {ml}" + assert "26" in ml, f"Expected PT 26 in media line: {ml}" + attrs = p.get_sdp_media_attributes() + assert "rtpmap" in attrs.lower(), f"Expected rtpmap in attrs: {attrs}" + + +def test_mjpeg_packetizer_rejects_invalid_frame(): + p = espp.MjpegPacketizer(espp.MjpegPacketizer.Config()) + chunks = p.packetize(b"\xff\xd8") + assert chunks == [], f"Expected invalid/truncated JPEG to be rejected, got {len(chunks)} chunk(s)" + + +# --------------------------------------------------------------------------- +# H264 Packetizer +# --------------------------------------------------------------------------- +def test_h264_packetizer_construction(): + cfg = espp.H264Packetizer.Config() + cfg.payload_type = 96 + cfg.profile_level_id = "42C01E" + cfg.packetization_mode = 1 + cfg.sps = [0x67, 0x42, 0xC0, 0x1E] + cfg.pps = [0x68, 0xCE, 0x38, 0x80] + p = espp.H264Packetizer(cfg) + assert p.get_payload_type() == 96 + assert p.get_clock_rate() == 90000 + + +def test_h264_packetizer_sdp(): + cfg = espp.H264Packetizer.Config() + cfg.payload_type = 96 + cfg.profile_level_id = "42C01E" + cfg.packetization_mode = 1 + cfg.sps = [0x67, 0x42, 0xC0, 0x1E] + cfg.pps = [0x68, 0xCE, 0x38, 0x80] + p = espp.H264Packetizer(cfg) + ml = p.get_sdp_media_line() + assert "96" in ml, f"Expected PT 96 in media line: {ml}" + attrs = p.get_sdp_media_attributes() + assert "H264" in attrs, f"Expected H264 in attrs: {attrs}" + + +def test_h264_packetizer_packetize(): + cfg = espp.H264Packetizer.Config() + cfg.max_payload_size = 100 + cfg.payload_type = 96 + cfg.packetization_mode = 1 + p = espp.H264Packetizer(cfg) + # Annex B access unit: start code + small NAL + nal = bytes([0x00, 0x00, 0x00, 0x01, 0x65] + [0xAB] * 50) + chunks = p.packetize(nal) + assert len(chunks) >= 1, f"Expected at least 1 chunk, got {len(chunks)}" + # Last chunk should have marker set + assert chunks[-1].marker, "Last chunk should have marker bit set" + + +def test_h264_packetizer_fua_fragmentation(): + cfg = espp.H264Packetizer.Config() + cfg.max_payload_size = 50 + cfg.payload_type = 96 + cfg.packetization_mode = 1 + p = espp.H264Packetizer(cfg) + # Large NAL that should be fragmented into FU-A + nal = bytes([0x00, 0x00, 0x00, 0x01, 0x65] + [0xCD] * 200) + chunks = p.packetize(nal) + assert len(chunks) > 1, f"Expected FU-A fragmentation (>1 chunk), got {len(chunks)}" + assert chunks[-1].marker, "Last FU-A chunk should have marker" + + +def test_h264_packetizer_rejects_too_small_fua_payload(): + cfg = espp.H264Packetizer.Config() + cfg.max_payload_size = 2 + cfg.payload_type = 96 + cfg.packetization_mode = 1 + p = espp.H264Packetizer(cfg) + nal = bytes([0x00, 0x00, 0x00, 0x01, 0x65] + [0xCD] * 10) + chunks = p.packetize(nal) + assert chunks == [], f"Expected oversized NAL to be dropped when max_payload_size is too small, got {len(chunks)} chunk(s)" + + +# --------------------------------------------------------------------------- +# Generic Packetizer (audio) +# --------------------------------------------------------------------------- +def test_generic_packetizer_construction(): + cfg = espp.GenericPacketizer.Config() + cfg.payload_type = 0 + cfg.clock_rate = 8000 + cfg.encoding_name = "PCMU" + cfg.channels = 1 + cfg.media_type = espp.MediaType.audio + p = espp.GenericPacketizer(cfg) + assert p.get_payload_type() == 0 + assert p.get_clock_rate() == 8000 + + +def test_generic_packetizer_sdp(): + cfg = espp.GenericPacketizer.Config() + cfg.payload_type = 0 + cfg.clock_rate = 8000 + cfg.encoding_name = "PCMU" + cfg.channels = 1 + cfg.media_type = espp.MediaType.audio + p = espp.GenericPacketizer(cfg) + ml = p.get_sdp_media_line() + assert "audio" in ml.lower(), f"Expected 'audio' in media line: {ml}" + attrs = p.get_sdp_media_attributes() + assert "PCMU" in attrs, f"Expected PCMU in attrs: {attrs}" + + +def test_generic_packetizer_chunking(): + cfg = espp.GenericPacketizer.Config() + cfg.max_payload_size = 100 + cfg.media_type = espp.MediaType.audio + p = espp.GenericPacketizer(cfg) + data = bytes([0x55] * 250) + chunks = p.packetize(data) + assert len(chunks) == 3, f"Expected 3 chunks for 250 bytes / 100 MTU, got {len(chunks)}" + assert chunks[-1].marker, "Last chunk should have marker" + assert not chunks[0].marker, "First chunk should not have marker" + + +# --------------------------------------------------------------------------- +# MJPEG Depacketizer +# --------------------------------------------------------------------------- +def test_mjpeg_depacketizer_construction(): + cfg = espp.MjpegDepacketizer.Config() + d = espp.MjpegDepacketizer(cfg) + # Just verify construction succeeds + assert d is not None + + +# --------------------------------------------------------------------------- +# H264 Depacketizer +# --------------------------------------------------------------------------- +def test_h264_depacketizer_construction(): + cfg = espp.H264Depacketizer.Config() + d = espp.H264Depacketizer(cfg) + assert d is not None + + +# --------------------------------------------------------------------------- +# Generic Depacketizer +# --------------------------------------------------------------------------- +def test_generic_depacketizer_construction(): + cfg = espp.GenericDepacketizer.Config() + d = espp.GenericDepacketizer(cfg) + assert d is not None + + +# --------------------------------------------------------------------------- +# Config structs — verify all fields are accessible +# --------------------------------------------------------------------------- +def test_config_fields(): + # H264Packetizer::Config + c = espp.H264Packetizer.Config() + c.max_payload_size = 1000 + c.payload_type = 97 + c.profile_level_id = "640028" + c.packetization_mode = 0 + c.sps = [1, 2, 3] + c.pps = [4, 5, 6] + assert c.max_payload_size == 1000 + assert c.payload_type == 97 + + # GenericPacketizer::Config + c2 = espp.GenericPacketizer.Config() + c2.fmtp = "mode=30" + c2.channels = 2 + c2.encoding_name = "opus" + assert c2.fmtp == "mode=30" + assert c2.channels == 2 + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- +def main(): + print("=" * 60) + print("RTSP Multi-Codec API Tests (Host-side)") + print("=" * 60) + + print("\n--- MediaType enum ---") + test("MediaType enum values", test_media_type_enum) + + print("\n--- RtpPayloadChunk ---") + test("RtpPayloadChunk construction", test_rtp_payload_chunk) + + print("\n--- MJPEG Packetizer ---") + test("Construction", test_mjpeg_packetizer_construction) + test("SDP generation", test_mjpeg_packetizer_sdp) + test("Reject invalid frame", test_mjpeg_packetizer_rejects_invalid_frame) + + print("\n--- H264 Packetizer ---") + test("Construction", test_h264_packetizer_construction) + test("SDP generation", test_h264_packetizer_sdp) + test("Packetize small NAL", test_h264_packetizer_packetize) + test("FU-A fragmentation", test_h264_packetizer_fua_fragmentation) + test("Reject undersized FU-A MTU", test_h264_packetizer_rejects_too_small_fua_payload) + + print("\n--- Generic Packetizer (Audio) ---") + test("Construction", test_generic_packetizer_construction) + test("SDP generation", test_generic_packetizer_sdp) + test("MTU chunking", test_generic_packetizer_chunking) + + print("\n--- Depacketizers ---") + test("MjpegDepacketizer construction", test_mjpeg_depacketizer_construction) + test("H264Depacketizer construction", test_h264_depacketizer_construction) + test("GenericDepacketizer construction", test_generic_depacketizer_construction) + + print("\n--- Config field access ---") + test("Config struct fields", test_config_fields) + + print("\n" + "=" * 60) + total = passed + failed + print(f"Results: {passed}/{total} passed, {failed}/{total} failed") + print("=" * 60) + + sys.exit(1 if failed > 0 else 0) + + +if __name__ == "__main__": + main() diff --git a/python/rtsp_client.py b/python/rtsp_client.py index cd18ea417..378600c45 100644 --- a/python/rtsp_client.py +++ b/python/rtsp_client.py @@ -1,139 +1,36 @@ -import sys -import time - -import socket - -import cv2 -import numpy as np - -import queue - -from zeroconf import ServiceBrowser, ServiceListener, Zeroconf - -from support_loader import espp - -class MyListener(ServiceListener): - def __init__(self): - self.service_ip = None - self.service_port = None - self.found_service = False - - def update_service(self, zc: Zeroconf, type_: str, name: str) -> None: - print(f"Service {name} updated") - - def remove_service(self, zc: Zeroconf, type_: str, name: str) -> None: - print(f"Service {name} removed") +"""Audio-enabled MJPEG RTSP client convenience wrapper. - def add_service(self, zc: Zeroconf, type_: str, name: str) -> None: - info = zc.get_service_info(type_, name) - print(f"Service {name} added, service info: {info}") - self.service_ip = socket.inet_ntoa(info.addresses[0]) - self.service_port = info.port - self.found_service = True +This preserves the original `rtsp_client.py` workflow while using the generic +multitrack client so the default camera / display path validates both video and +audio delivery. +""" -frame_queue = queue.Queue() -# This function will be called when a new JPEG frame is received -def on_receive_frame(frame): - global frame_queue - # print(f"Received frame of size {frame.get_width()}x{frame.get_height()} pixels") - buf = frame.get_data() - decoded = cv2.imdecode(np.frombuffer(buf, dtype=np.uint8), cv2.IMREAD_COLOR) - if decoded is not None: - frame_queue.put_nowait(decoded) - else: - print("Failed to decode frame") - return - -# use zeroconf to discover the rtsp service at "_rtsp._tcp.local." -service = "_rtsp._tcp.local." -zeroconf = Zeroconf() -listener = MyListener() -browser = ServiceBrowser(zeroconf, service, listener) - -print("Finding Service...\n") -while not listener.found_service: - time.sleep(0.1) - -server_address, server_port = listener.service_ip, listener.service_port -media_path = "/mjpeg/1" -rtsp_uri = f"rtsp://{server_address}:{server_port}{media_path}" -print(f"Found service at uri: {rtsp_uri}\n") -client_config = espp.RtspClient.Config( - server_address = server_address, - rtsp_port = server_port, - path = media_path, - on_jpeg_frame = on_receive_frame, # callback for received frames - log_level = espp.Logger.Verbosity.info -) -rtsp_client = espp.RtspClient(client_config) - -# initialize the RTSP client and connect to the server -ec = espp.Error() - -print("Connecting to RTSP server...") -rtsp_client.connect(ec) -if ec: - print(f"Error connecting to RTSP server: {ec}") - sys.exit(1) - -print("Describing RTSP client...") -rtsp_client.describe(ec) -if ec: - print(f"Error describing RTSP client: {ec}") - sys.exit(1) - -print("Setting up RTSP client...") -rtsp_client.setup(ec) -if ec: - print(f"Error setting up RTSP client: {ec}") - sys.exit(1) +import sys -print("Playing RTSP client...") -rtsp_client.play(ec) -if ec: - print(f"Error playing RTSP client: {ec}") - sys.exit(1) +from rtsp_client_multitrack import main as multitrack_main -# right now our client should be connected and receiving frames, calling the -# callback. -window_name = 'RTSP Client' -cv2.namedWindow(window_name, cv2.WINDOW_NORMAL) -# set the window as always on top -cv2.setWindowProperty(window_name, cv2.WND_PROP_TOPMOST, 1) +def has_option(argv, option): + return any(arg == option or arg.startswith(f"{option}=") for arg in argv) -# capture start time here so we can measure FPS and print it at the end -start_time = time.time() -num_frames = 0 -print("RTSP client is now connected and receiving frames") -print("\nPress 'q' to quit or Ctrl+C to stop the client and exit.\n") -# wait until the user presses ctrl+c to stop the client -try: - while True: - try: - frame = frame_queue.get_nowait() - cv2.imshow('RTSP Client', frame) - num_frames += 1 - except queue.Empty: - pass - key = cv2.waitKey(1) & 0xFF - if key == ord('q'): # press 'q' to quit - break -except KeyboardInterrupt: - pass -finally: - print("Stopping RTSP client...") +def build_argv(argv): + forwarded = list(argv) + expect_audio = True + if "--no-audio-check" in forwarded: + forwarded.remove("--no-audio-check") + expect_audio = False -# calculate the elapsed time and FPS -elapsed_time = time.time() - start_time -fps = num_frames / elapsed_time if elapsed_time > 0 else 0 -print(f"Captured {num_frames} frames in {elapsed_time:.2f} seconds ({fps:.2f} FPS)") + if not has_option(forwarded, "--path"): + forwarded.extend(["--path", "/mjpeg/1"]) + if not has_option(forwarded, "--service-name"): + forwarded.extend(["--service-name", "python rtsp server"]) + if not has_option(forwarded, "--require-decoded-video"): + forwarded.append("--require-decoded-video") + if expect_audio and not has_option(forwarded, "--expect-audio"): + forwarded.append("--expect-audio") + return forwarded -# stop the client and unregister the service -rtsp_client.teardown(ec) -if ec: - print(f"Error tearing down RTSP client: {ec}") -zeroconf.close() -sys.exit(0) +if __name__ == "__main__": + sys.exit(multitrack_main(build_argv(sys.argv[1:]))) diff --git a/python/rtsp_client_multitrack.py b/python/rtsp_client_multitrack.py new file mode 100755 index 000000000..d06517cdf --- /dev/null +++ b/python/rtsp_client_multitrack.py @@ -0,0 +1,446 @@ +"""Multi-track RTSP client that works with rtsp_server_multitrack.py. + +Discovers an RTSP server via mDNS, connects, and receives frames. +Demonstrates the new generic frame callback and depacketizer APIs. + +Usage: + # Connect to any discovered RTSP server (auto-detect codec from SDP) + python rtsp_client_multitrack.py + + # Specify the server path + python rtsp_client_multitrack.py --path /stream +""" + +import sys +import time +import queue +import argparse +import socket + +import cv2 +import numpy as np + +from zeroconf import ServiceBrowser, ServiceListener, Zeroconf +from support_loader import espp + +try: + import sounddevice as sd +except ImportError: + sd = None + + +class MyListener(ServiceListener): + def __init__(self, service_name=None): + self.service_ip = None + self.service_port = None + self.found_service = False + self.service_name = service_name + + def update_service(self, zc, type_, name): + pass + + def remove_service(self, zc, type_, name): + print(f"Service {name} removed") + + def add_service(self, zc, type_, name): + if self.service_name and name != f"{self.service_name}._rtsp._tcp.local.": + return + info = zc.get_service_info(type_, name) + print(f"Service {name} added, service info: {info}") + self.service_ip = socket.inet_ntoa(info.addresses[0]) + self.service_port = info.port + self.found_service = True + + +frame_queue = queue.Queue() +audio_frame_count = 0 +audio_valid_frame_count = 0 +audio_invalid_frame_count = 0 +video_frame_count = 0 +decoded_video_frame_count = 0 +display_video_frames = True +expected_audio_bytes = 320 +min_audio_peak = 0 +audio_player = None +audio_sample_rate = 8000 +audio_channels = 1 + + +class AudioPlayer: + def __init__(self, sample_rate=8000, channels=1, bytes_per_sample=2, block_samples=160, + device=None): + self.sample_rate = sample_rate + self.channels = channels + self.bytes_per_sample = bytes_per_sample + self.block_samples = block_samples + self.device = device + self.frame_queue = queue.Queue(maxsize=128) + self.pending_bytes = bytearray() + self.stream = None + self.started = False + self.underflow_count = 0 + self.dropped_frames = 0 + + def start(self): + if sd is None: + print("Audio playback unavailable: install sounddevice") + return False + try: + self.stream = sd.RawOutputStream( + samplerate=self.sample_rate, + channels=self.channels, + dtype='int16', + blocksize=self.block_samples, + device=self.device, + callback=self._callback, + ) + self.stream.start() + self.started = True + print(f"Audio playback enabled at {self.sample_rate} Hz") + return True + except Exception as exc: + print(f"Audio playback unavailable: {exc}") + self.stream = None + return False + + def stop(self): + if self.stream is not None: + try: + self.stream.stop() + finally: + self.stream.close() + self.stream = None + self.started = False + + def enqueue(self, frame_bytes): + if not self.started: + return + try: + self.frame_queue.put_nowait(frame_bytes) + except queue.Full: + try: + self.frame_queue.get_nowait() + except queue.Empty: + # Another consumer/drain path already emptied the queue. + pass + self.dropped_frames += 1 + try: + self.frame_queue.put_nowait(frame_bytes) + except queue.Full: + self.dropped_frames += 1 + + def _callback(self, outdata, frames, time_info, status): + del frames, time_info + if status.output_underflow: + self.underflow_count += 1 + + bytes_needed = len(outdata) + while len(self.pending_bytes) < bytes_needed: + try: + self.pending_bytes.extend(self.frame_queue.get_nowait()) + except queue.Empty: + break + + available = min(len(self.pending_bytes), bytes_needed) + if available: + outdata[:available] = self.pending_bytes[:available] + del self.pending_bytes[:available] + if available < bytes_needed: + outdata[available:bytes_needed] = b'\x00' * (bytes_needed - available) + self.underflow_count += 1 + + +def as_frame_bytes(data): + if isinstance(data, bytes): + return data + if isinstance(data, bytearray): + return bytes(data) + if isinstance(data, memoryview): + return data.tobytes() + return bytes(data) + + +def validate_audio_frame(frame_bytes): + global min_audio_peak, expected_audio_bytes + if len(frame_bytes) != expected_audio_bytes: + return False, f"unexpected audio frame size: {len(frame_bytes)}" + if len(frame_bytes) % 2 != 0: + return False, f"audio frame size is not 16-bit aligned: {len(frame_bytes)}" + samples = np.frombuffer(frame_bytes, dtype=' 0 and peak < min_audio_peak: + return False, f"audio peak too small: {peak}" + return True, f"{samples.size} samples, peak {peak}" + + +def error_message(ec): + if ec is None or not bool(ec): + return "" + return str(ec) + + +def on_receive_jpeg_frame(frame): + """Legacy JPEG frame callback.""" + global video_frame_count, decoded_video_frame_count + buf = as_frame_bytes(frame.get_data()) + video_frame_count += 1 + decoded = cv2.imdecode(np.frombuffer(buf, dtype=np.uint8), cv2.IMREAD_COLOR) + if decoded is not None: + decoded_video_frame_count += 1 + if display_video_frames: + frame_queue.put_nowait(decoded) + + +def on_receive_generic_frame(track_id, data): + """Generic frame callback for any track/codec.""" + global audio_frame_count, audio_valid_frame_count, audio_invalid_frame_count + global video_frame_count, decoded_video_frame_count + global audio_player + frame_bytes = as_frame_bytes(data) + if track_id == 0: + video_frame_count += 1 + if frame_bytes.startswith(b'\xff\xd8'): + decoded = cv2.imdecode(np.frombuffer(frame_bytes, dtype=np.uint8), cv2.IMREAD_COLOR) + if decoded is not None: + decoded_video_frame_count += 1 + if display_video_frames: + frame_queue.put_nowait(decoded) + else: + print(f"[Track {track_id}] JPEG frame: {len(frame_bytes)} bytes (decode failed)") + elif track_id == 1: + audio_frame_count += 1 + valid, detail = validate_audio_frame(frame_bytes) + if valid: + audio_valid_frame_count += 1 + if audio_player is not None: + audio_player.enqueue(frame_bytes) + else: + audio_invalid_frame_count += 1 + print(f"[Track {track_id}] Invalid audio frame: {detail}") + if audio_frame_count % 50 == 0: + print(f"[Track {track_id}] Audio frames received: {audio_frame_count} ({detail})") + else: + print(f"[Track {track_id}] Frame: {len(frame_bytes)} bytes") + + +def main(argv=None): + global audio_frame_count, audio_valid_frame_count, audio_invalid_frame_count + global video_frame_count, decoded_video_frame_count + global display_video_frames, expected_audio_bytes, min_audio_peak, audio_player + global audio_sample_rate, audio_channels + + parser = argparse.ArgumentParser(description='Multi-track RTSP Client') + parser.add_argument('--path', type=str, default='/stream', + help='RTSP stream path (default: /stream)') + parser.add_argument('--service-name', type=str, default=None, + help='Only connect to this mDNS service name') + parser.add_argument('--discovery-timeout', type=float, default=10.0, + help='Seconds to wait for mDNS discovery before failing') + parser.add_argument('--duration', type=float, default=0.0, + help='Auto-stop after this many seconds (default: run until quit)') + parser.add_argument('--headless', action='store_true', + help='Do not create an OpenCV window') + parser.add_argument('--expect-audio', action='store_true', + help='Fail if valid audio frames are not received') + parser.add_argument('--require-decoded-video', action='store_true', + help='Fail if no JPEG video frames are successfully decoded') + parser.add_argument('--min-video-frames', type=int, default=1, + help='Minimum number of video frames expected before success') + parser.add_argument('--min-audio-frames', type=int, default=1, + help='Minimum number of valid audio frames expected before success') + parser.add_argument('--expected-audio-bytes', type=int, default=None, + help='Expected bytes per audio frame (default: derive from SDP or fall back to 320)') + parser.add_argument('--min-audio-peak', type=int, default=0, + help='Minimum absolute sample peak required to treat audio as valid') + parser.add_argument('--play-audio', dest='play_audio', action='store_true', + help='Play the audio track in real time') + parser.add_argument('--no-audio-playback', dest='play_audio', action='store_false', + help='Disable real-time audio playback') + parser.add_argument('--audio-device', type=int, default=None, + help='Optional sounddevice output device index') + parser.add_argument('--legacy', action='store_true', + help='Use legacy JPEG-only mode (on_jpeg_frame callback)') + parser.set_defaults(play_audio=None) + args = parser.parse_args(argv) + display_video_frames = not args.headless + if args.play_audio is None: + args.play_audio = not args.headless + expected_audio_bytes = args.expected_audio_bytes + min_audio_peak = args.min_audio_peak + audio_frame_count = 0 + audio_valid_frame_count = 0 + audio_invalid_frame_count = 0 + video_frame_count = 0 + decoded_video_frame_count = 0 + audio_player = None + audio_sample_rate = 8000 + audio_channels = 1 + + # Discover RTSP server via mDNS + print("Discovering RTSP service via mDNS...") + zeroconf = Zeroconf() + listener = MyListener(service_name=args.service_name) + browser = ServiceBrowser(zeroconf, "_rtsp._tcp.local.", listener) + discovery_start = time.monotonic() + while not listener.found_service: + if args.discovery_timeout > 0 and (time.monotonic() - discovery_start) >= args.discovery_timeout: + print(f"Error: timed out after {args.discovery_timeout:.1f}s waiting for RTSP service discovery") + browser.cancel() + zeroconf.close() + if audio_player is not None: + audio_player.stop() + return 1 + time.sleep(0.1) + + server_address = listener.service_ip + server_port = listener.service_port + rtsp_uri = f"rtsp://{server_address}:{server_port}{args.path}" + print(f"Found service at: {rtsp_uri}\n") + + # Create RTSP client with appropriate callback mode + if args.legacy: + print("Using legacy JPEG-only mode (on_jpeg_frame)") + client_config = espp.RtspClient.Config( + server_address=server_address, + rtsp_port=server_port, + path=args.path, + on_jpeg_frame=on_receive_jpeg_frame, + log_level=espp.Logger.Verbosity.info + ) + else: + print("Using generic multi-track mode (on_frame)") + client_config = espp.RtspClient.Config( + server_address=server_address, + rtsp_port=server_port, + path=args.path, + on_frame=on_receive_generic_frame, + log_level=espp.Logger.Verbosity.info + ) + + rtsp_client = espp.RtspClient(client_config) + ec = espp.Error() + + # Connect and setup + print("Connecting...") + rtsp_client.connect(ec) + connect_error = error_message(ec) + if connect_error: + print(f"Error connecting: {connect_error}") + return 1 + + print("Describing...") + rtsp_client.describe(ec) + describe_error = error_message(ec) + if describe_error: + print(f"Error describing: {describe_error}") + return 1 + + tracks = rtsp_client.tracks() + audio_track = next((track for track in tracks if track.media_type.lower() == 'audio'), None) + if audio_track is not None: + audio_sample_rate = audio_track.clock_rate or audio_sample_rate + audio_channels = audio_track.channels or audio_channels + if args.expected_audio_bytes is None: + expected_audio_bytes = max(2 * audio_channels, + (audio_sample_rate * audio_channels * 2) // 50) + print(f"Discovered audio track: PT={audio_track.payload_type}, " + f"{audio_track.encoding_name}/{audio_sample_rate}/{audio_channels}, " + f"expecting {expected_audio_bytes} bytes per frame") + elif expected_audio_bytes is None: + expected_audio_bytes = 320 + + if args.play_audio and audio_track is not None: + audio_player = AudioPlayer( + sample_rate=audio_sample_rate, + channels=audio_channels, + block_samples=max(1, expected_audio_bytes // max(1, 2 * audio_channels)), + device=args.audio_device, + ) + if not audio_player.start(): + audio_player = None + + print("Setting up...") + rtsp_client.setup(ec) + setup_error = error_message(ec) + if setup_error: + print(f"Error setting up: {setup_error}") + return 1 + + print("Playing...") + rtsp_client.play(ec) + play_error = error_message(ec) + if play_error: + print(f"Error playing: {play_error}") + return 1 + + window_name = 'RTSP Multi-Track Client' + if not args.headless: + cv2.namedWindow(window_name, cv2.WINDOW_NORMAL) + cv2.setWindowProperty(window_name, cv2.WND_PROP_TOPMOST, 1) + + start_time = time.time() + print("Receiving frames... Press 'q' to quit.\n") + try: + while True: + try: + frame = frame_queue.get_nowait() + if not args.headless: + cv2.imshow(window_name, frame) + except queue.Empty: + # No decoded frame is ready yet from the non-blocking queue. + pass + if args.duration > 0 and (time.time() - start_time) >= args.duration: + break + if not args.headless: + key = cv2.waitKey(1) & 0xFF + if key == ord('q'): + break + else: + time.sleep(0.01) + except KeyboardInterrupt: + # Ctrl+C should fall through to the normal teardown path below. + pass + + elapsed = time.time() - start_time + fps = video_frame_count / elapsed if elapsed > 0 else 0 + print(f"\nVideo: {video_frame_count} frames in {elapsed:.1f}s ({fps:.1f} FPS)") + print(f"Decoded video frames: {decoded_video_frame_count}") + print(f"Audio: {audio_frame_count} frames received, {audio_valid_frame_count} valid, " + f"{audio_invalid_frame_count} invalid") + + rtsp_client.teardown(ec) + browser.cancel() + zeroconf.close() + if audio_player is not None: + audio_player.stop() + print(f"Audio playback underflows: {audio_player.underflow_count}, " + f"dropped frames: {audio_player.dropped_frames}") + + failures = [] + if video_frame_count < args.min_video_frames: + failures.append( + f"expected at least {args.min_video_frames} video frames, got {video_frame_count}" + ) + if args.require_decoded_video and decoded_video_frame_count == 0: + failures.append("expected at least one decoded JPEG video frame") + if args.expect_audio and audio_valid_frame_count < args.min_audio_frames: + failures.append( + f"expected at least {args.min_audio_frames} valid audio frames, " + f"got {audio_valid_frame_count}" + ) + if audio_invalid_frame_count > 0: + failures.append(f"received {audio_invalid_frame_count} invalid audio frames") + + if failures: + for failure in failures: + print(f"[FAIL] {failure}") + return 1 + + print("[PASS] Multitrack RTSP client validation succeeded") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/python/rtsp_multitrack_test.py b/python/rtsp_multitrack_test.py new file mode 100644 index 000000000..7e9d06b32 --- /dev/null +++ b/python/rtsp_multitrack_test.py @@ -0,0 +1,123 @@ +"""Automated end-to-end multitrack RTSP tests for video + audio paths. + +Runs the multitrack RTSP server and client in subprocesses using synthetic video +so the test can execute without a camera or display. Each case validates that +the client receives both video and audio frames. +""" + +import subprocess +import sys +import time +import uuid + + +TEST_CASES = [ + { + "name": "Default MJPEG wrapper + audio", + "service_name": "python rtsp wrapper test", + "port": 8553, + "server_script": "rtsp_server.py", + "client_script": "rtsp_client.py", + "server_args": ["--video-source", "synthetic"], + "client_args": [], + }, + { + "name": "MJPEG default + audio", + "service_name": "python rtsp multitrack mjpeg test", + "port": 8554, + "server_script": "rtsp_server_multitrack.py", + "client_script": "rtsp_client_multitrack.py", + "server_args": ["--video-source", "synthetic", "--codec", "mjpeg"], + "client_args": ["--expect-audio", "--require-decoded-video"], + }, + { + "name": "H264 default + audio", + "service_name": "python rtsp multitrack h264 test", + "port": 8555, + "server_script": "rtsp_server_multitrack.py", + "client_script": "rtsp_client_multitrack.py", + "server_args": ["--video-source", "synthetic", "--codec", "h264"], + "client_args": ["--expect-audio"], + }, +] + + +def read_process_output(process): + output = "" + if process.stdout is not None: + output = process.stdout.read() + return output + + +def stop_process(process, timeout=5.0): + if process.poll() is not None: + return + process.terminate() + try: + process.wait(timeout=timeout) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=timeout) + + +def run_case(case): + service_name = f"{case['service_name']} {uuid.uuid4().hex[:8]}" + common_server_args = [ + sys.executable, + case.get("server_script", "rtsp_server_multitrack.py"), + "--duration", "8", + "--service-name", service_name, + "--port", str(case["port"]), + ] + server = subprocess.Popen( + common_server_args + case["server_args"], + cwd=".", + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + ) + + try: + time.sleep(1.0) + client = subprocess.run( + [ + sys.executable, + case.get("client_script", "rtsp_client_multitrack.py"), + "--service-name", service_name, + "--duration", "4", + "--headless", + "--min-video-frames", "5", + "--min-audio-frames", "10", + "--min-audio-peak", "1000", + *case["client_args"], + ], + cwd=".", + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + timeout=15, + ) + server.wait(timeout=10) + server_output = read_process_output(server) + return client.returncode == 0, client.stdout, server_output + finally: + stop_process(server) + + +def main(): + overall_success = True + for case in TEST_CASES: + print(f"=== Running {case['name']} ===") + success, client_output, server_output = run_case(case) + print("--- Client output ---") + print(client_output.rstrip()) + print("--- Server output ---") + print(server_output.rstrip()) + print(f"=== {'PASS' if success else 'FAIL'}: {case['name']} ===\n") + overall_success = overall_success and success + + sys.exit(0 if overall_success else 1) + + +if __name__ == "__main__": + main() diff --git a/python/rtsp_server.py b/python/rtsp_server.py index 171010f06..3b868d3e2 100644 --- a/python/rtsp_server.py +++ b/python/rtsp_server.py @@ -1,165 +1,34 @@ -import sys -import time - -import asyncio - -import argparse - -import mss -import cv2 -import numpy as np - -from zeroconf import IPVersion, ServiceInfo, Zeroconf - -import socket - -from simplejpeg import encode_jpeg - -from support_loader import espp - -# This script sets up an RTSP server that captures frames from the specified -# camera (default is 0) or the display, encodes them as JPEG, and serves them -# over RTSP. - -cap = None -use_display = False - -# initialize the RTSP server -async def init_server(server_ip, server_port=8554, hostname='localhost', media_path='mjpeg/1'): - print(f"Starting RTSP Server IP: {server_ip}, Port: {server_port}, Path: {media_path}") - rtsp_server = espp.RtspServer(espp.RtspServer.Config( - server_address = server_ip, - port = server_port, - path = media_path, - log_level = espp.Logger.Verbosity.warn - )) - rtsp_server.start() - - info = ServiceInfo( - "_rtsp._tcp.local.", - f"python rtsp server._rtsp._tcp.local.", - addresses=[socket.inet_aton(server_ip)], - port=server_port, - properties={"path": media_path}, - server=hostname + '.local.' - ) +"""Audio-enabled MJPEG RTSP server convenience wrapper. - ip_version = IPVersion.V4Only +This preserves the original `rtsp_server.py` camera / `--use-display` workflow +while routing it through the multitrack RTSP server so the stream includes audio +by default. The camera path uses live microphone capture; display capture keeps +the synthetic tone unless `--audio-source microphone` is selected. +""" - # use zeroconf/mDNS to advertise the rtsp server as '_rtsp._tcp.local.' - zeroconf = Zeroconf(ip_version=ip_version) - await zeroconf.async_register_service(info) - print(f"Registered RTSP service: {info}") - - return rtsp_server, zeroconf, info - -# function to capture a frame, resize it, and encode it as JPEG -def get_frame(image_size=(320, 240), jpeg_quality=10): - global cap - global use_display - if cap is None: - print("Video capture is not initialized") - return None - if use_display: - frame = np.array(cap.grab(cap.monitors[1])) # capture the primary monitor - if frame is None: - print("Failed to capture frame from display") - return None - # convert from rgba to bgr - frame = cv2.cvtColor(frame, cv2.COLOR_RGBA2RGB) - else: - # capture a frame from the display or a video source - # for example, using OpenCV or PIL - ret, frame = cap.read() - if not ret: - print("Failed to capture frame") - return None - # resize the frame to a smaller size (320x240) - resized_frame = cv2.resize(frame, image_size) - - # encode the frame as JPEG - image_bytes = encode_jpeg(resized_frame, quality=jpeg_quality, colorspace='BGR') - - return image_bytes - -# make a task to grab a video or the display, resize it, and call -# `rtsp_server.send_frame(frame)` periodically -def task_func(rtsp_server, jpeg_quality=10, image_size=(320, 240)): - image_bytes = get_frame(image_size=image_size, jpeg_quality=jpeg_quality) - if image_bytes is None: - return True # stop the task if we failed to get a frame - # create a JpegFrame object with the image data and bytes - frame = espp.JpegFrame(image_bytes) - # send the frame to the RTSP server - rtsp_server.send_frame(frame) - time.sleep(.01) - return False # we don't want to stop the task - -async def main(): - global cap, use_display - - parser = argparse.ArgumentParser(description='RTSP Server for MJPEG streaming') - parser.add_argument('--port', type=int, default=8554, help='Port number for RTSP server (default: 8554)') - parser.add_argument('--use-display', action='store_true', help='Use display as video source instead of webcam') - parser.add_argument('--camera', type=int, default=0, help='Camera index to use (default: 0)') - parser.add_argument('--jpeg-quality', type=int, default=50, help='JPEG quality (1-100, default: 50)') - parser.add_argument('--image-width', type=int, default=320, help='Width of the image (default: 320)') - parser.add_argument('--image-height', type=int, default=240, help='Height of the image (default: 240)') - args = parser.parse_args() - - # if use_display is set, we will use the display as the video source - use_display = args.use_display - camera_index = args.camera - if use_display: - print("Using display as video source") - else: - print(f"Using camera index {camera_index} as video source") - jpeg_quality = args.jpeg_quality - image_size = (args.image_width, args.image_height) - - # get the address of this machine using python's socket library - fqdn = socket.gethostname() - hostname = fqdn.split('.')[0] # Get the hostname without domain - ip_addr = socket.gethostbyname(hostname) - port = args.port - - rtsp_server, zeroconf, info = await init_server(server_ip=ip_addr, server_port=port, hostname=hostname) +import asyncio +import sys - # initialize the video capture - if use_display: - cap = mss.mss() # use mss to capture the display - else: - # capture a frame from the default camera - cap = cv2.VideoCapture(camera_index) +from rtsp_server_multitrack import main as multitrack_main - task = espp.Task( - lambda: task_func(rtsp_server, jpeg_quality=jpeg_quality, image_size=image_size), - espp.Task.BaseConfig("rtsp server task") - ) - task.start() - print("RTSP Server started, press Ctrl+C to stop...") - try: - while True: - await asyncio.sleep(1) - except asyncio.CancelledError: - print("\nKeyboard interrupt received, stopping...") - except KeyboardInterrupt: - print("\nKeyboard interrupt received, stopping...") +def has_option(argv, option): + return any(arg == option or arg.startswith(f"{option}=") for arg in argv) - print("Releasing resources...") - if use_display: - cap.close() # Close mss display capture - else: - cap.release() - print("Unregistering service...") - await zeroconf.async_unregister_service(info) +def build_argv(argv): + forwarded = list(argv) - print("Stopping zeroconf...") - zeroconf.close() + if not has_option(forwarded, "--codec"): + forwarded.extend(["--codec", "mjpeg"]) + if not has_option(forwarded, "--path"): + forwarded.extend(["--path", "/mjpeg/1"]) + if not has_option(forwarded, "--service-name"): + forwarded.extend(["--service-name", "python rtsp server"]) + if not has_option(forwarded, "--audio") and not has_option(forwarded, "--no-audio"): + forwarded.append("--audio") + return forwarded - sys.exit(0) if __name__ == "__main__": - asyncio.run(main()) + sys.exit(asyncio.run(multitrack_main(build_argv(sys.argv[1:])))) diff --git a/python/rtsp_server_multitrack.py b/python/rtsp_server_multitrack.py new file mode 100755 index 000000000..dc7dbcd6b --- /dev/null +++ b/python/rtsp_server_multitrack.py @@ -0,0 +1,540 @@ +"""Multi-track RTSP server demonstrating the new multi-codec APIs. + +Supports streaming video (MJPEG or H264) plus audio by default. The camera path +captures live microphone audio, while the display and synthetic paths use a +simulated tone unless `--audio-source microphone` is selected explicitly. +Uses the new add_track() / send_frame(track_id, data) APIs. + +Usage: + # MJPEG camera video + live microphone audio (default) + python rtsp_server_multitrack.py + + # MJPEG video only + python rtsp_server_multitrack.py --no-audio + + # H264 video (simulated, uses packetizer only — no real encoder) + python rtsp_server_multitrack.py --codec h264 --no-audio + + # H264 video + audio + python rtsp_server_multitrack.py --codec h264 +""" + +import sys +import time +import queue +import multiprocessing as mp +import struct +import math +import asyncio +import argparse +import socket + +import mss +import cv2 +import numpy as np + +from zeroconf import IPVersion, ServiceInfo, Zeroconf +from simplejpeg import encode_jpeg + +from support_loader import espp + +try: + import sounddevice as sd +except ImportError: + sd = None + +cap = None +video_source = "camera" + + +# --- Frame capture (same as rtsp_server.py) --- +def get_frame(image_size=(320, 240), jpeg_quality=50): + global cap, video_source + if cap is None: + return None + if video_source == "display": + frame = np.array(cap.grab(cap.monitors[1])) + if frame is None: + return None + frame = cv2.cvtColor(frame, cv2.COLOR_RGBA2RGB) + else: + ret, frame = cap.read() + if not ret: + return None + resized_frame = cv2.resize(frame, image_size) + return encode_jpeg(resized_frame, quality=jpeg_quality, colorspace='BGR') + + +def get_synthetic_frame(frame_seq, image_size=(320, 240), jpeg_quality=50): + width, height = image_size + x = np.linspace(0, 255, width, dtype=np.uint16) + y = np.linspace(0, 255, height, dtype=np.uint16) + x_grid = np.tile(x, (height, 1)) + y_grid = np.tile(y[:, None], (1, width)) + frame = np.zeros((height, width, 3), dtype=np.uint16) + frame[..., 0] = (x_grid + frame_seq * 5) % 256 + frame[..., 1] = (y_grid + frame_seq * 3) % 256 + frame[..., 2] = ((x_grid // 2) + (y_grid // 2) + frame_seq * 7) % 256 + frame = frame.astype(np.uint8) + cv2.putText(frame, f"seq:{frame_seq}", (10, height // 2), cv2.FONT_HERSHEY_SIMPLEX, + 0.6, (255, 255, 255), 2, cv2.LINE_AA) + return encode_jpeg(frame, quality=jpeg_quality, colorspace='BGR') + + +# --- Simulated audio: generate a 160-sample (20 ms @ 8 kHz) sine-tone buffer --- +audio_phase = 0.0 + + +def get_audio_frame(frequency=440.0, sample_rate=8000, frame_samples=160): + """Generate a PCM 16-bit mono sine wave frame.""" + global audio_phase + buf = bytearray(frame_samples * 2) + for i in range(frame_samples): + sample = int(16000 * math.sin(audio_phase)) + struct.pack_into(' 0 and (now - start_time) >= args.duration: + break + + sent_frame = False + if now >= next_video_time: + if args.codec == 'mjpeg': + if video_source == "synthetic": + jpeg_bytes = get_synthetic_frame( + frame_seq, image_size=image_size, jpeg_quality=args.jpeg_quality + ) + else: + jpeg_bytes = get_frame(image_size=image_size, jpeg_quality=args.jpeg_quality) + if jpeg_bytes is not None: + rtsp_server.send_frame(video_track_id, jpeg_bytes) + video_ready = True + else: + fake_h264 = get_fake_h264_frame(frame_seq) + rtsp_server.send_frame(video_track_id, fake_h264) + video_ready = True + frame_seq += 1 + next_video_time = max(next_video_time + 1.0 / 30.0, time.monotonic()) + sent_frame = True + + now = time.monotonic() + if args.audio: + if resolved_audio_source == 'microphone': + if now >= next_audio_time: + audio_data = live_audio_source.get_latest_frame() + if audio_data is not None: + rtsp_server.send_frame(audio_track_id, audio_data) + audio_frame_count += 1 + audio_ready = True + next_audio_time = max(next_audio_time + audio_period, time.monotonic()) + sent_frame = True + else: + if now >= next_audio_time: + audio_data = get_audio_frame() + rtsp_server.send_frame(audio_track_id, audio_data) + audio_frame_count += 1 + audio_ready = True + next_audio_time = max(next_audio_time + audio_period, time.monotonic()) + sent_frame = True + + if not service_registered and video_ready and audio_ready: + await zeroconf.async_register_service(info) + print(f"Registered RTSP service: {info}") + service_registered = True + + if sent_frame: + await asyncio.sleep(0) + else: + await asyncio.sleep(0.005) + + except (asyncio.CancelledError, KeyboardInterrupt): + print("\nStopping...") + + print(f"Sent {frame_seq} video frames and {audio_frame_count} audio frames") + + if video_source == "display" and cap is not None: + cap.close() + elif video_source == "camera" and cap is not None: + cap.release() + if live_audio_source is not None: + live_audio_source.stop() + print(f"Microphone capture overflows: {live_audio_source.overflow_count}, " + f"dropped frames: {live_audio_source.dropped_frames}") + + if service_registered: + await zeroconf.async_unregister_service(info) + zeroconf.close() + return 0 + + +if __name__ == "__main__": + sys.exit(asyncio.run(main()))