feat(rtsp/socket): Improve network comms and make RTSP component more stream agnostic, to support audio as well and not just MJPEG

components/rtsp/README.md

@@ -2,94 +2,109 @@
 
 [![Badge](https://components.espressif.com/components/espp/rtsp/badge.svg)](https://components.espressif.com/components/espp/rtsp)
 
-The `rtsp` component provides various classes for implementing both sides of an
-RTSP stream for transmitting MJPEG video data.
+The `rtsp` component provides a flexible, multi-codec RTSP streaming framework
+for ESP32 devices. It supports MJPEG, H.264, and generic audio codecs through
+an extensible packetizer/depacketizer architecture. The component handles only
+RTP packet splitting and reassembly — encoding and decoding of media data is
+performed externally.
 
 <!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
 **Table of Contents**
 
 - [RTSP (Real-Time Streaming Protocol) Component](#rtsp-real-time-streaming-protocol-component)
   - [RTSP Client](#rtsp-client)
   - [RTSP Server](#rtsp-server)
+  - [Packetizers and Depacketizers](#packetizers-and-depacketizers)
   - [Testing and Utilities](#testing-and-utilities)
   - [Example](#example)
 
 <!-- markdown-toc end -->
 
 ## RTSP Client
 
-The `RtspClient` class provides an interface to an RTSP server. It is used to
-send RTSP requests and receive RTSP responses. It also provides an interface
-to the RTP and RTCP sessions that are created as a result of the RTSP
-interactions.
+The `RtspClient` class connects to an RTSP server and receives media streams
+over RTP/UDP. It dispatches incoming RTP packets to codec-specific
+depacketizers based on payload type.
 
-The `RtspClient` currently only supports MJPEG streams, since the ESP32 does
-not have a hardware decoder for H.264 or H.265.
+For **backward compatibility**, setting the `on_jpeg_frame` callback
+automatically creates an `MjpegDepacketizer` for MJPEG streams (payload type
+26). For generic multi-track use, applications can use the `on_frame`
+callback and inspect parsed SDP metadata through `tracks()`.
 
-Additionally the client currently only supports UDP transport for RTP and RTCP
-packets. TCP transport is not supported.
+The client supports:
 
-The user can register a callback function to be notified when new, complete JPEG
-frames are received. The callback function is called with a pointer to the JPEG
-frame.
+* generic `on_frame(track_id, data)` callbacks for multi-track sessions
+* parsed SDP track metadata including media type, payload type, codec name,
+  sample rate, channel count, and resolved control path
+* automatic depacketizer selection for MJPEG, H.264, and generic payloads
+  discovered during `DESCRIBE`
+* an `on_connection_lost` callback for reconnect / rediscovery workflows when
+  the RTSP control socket or RTP stream disappears after playback starts
+* custom depacketizers via `add_depacketizer()` for additional payload types
 
 ## RTSP Server
 
-The `RtspServer` class provides an implementation of an RTSP server. It is used
-to receive RTSP requests and send RTSP responses. It is designed to allow the
-user to send JPEG frames to the server, which will then send them to the client
-over RTP/UDP.
+The `RtspServer` class accepts RTSP connections and streams media over RTP/UDP.
+It supports multiple media tracks, each with its own codec-specific packetizer,
+SSRC, and sequence numbering.
 
-The server currently only supports MJPEG streams, since the ESP32 does not have
-a hardware encoder for H.264 or H.265.
+For **backward compatibility**, calling `send_frame(const JpegFrame&)` lazily
+creates a default MJPEG track. For other codecs, register tracks via
+`add_track()` and send frames with `send_frame(track_id, data)`.
 
-Additionally, the server currently only supports UDP transport for RTP and RTCP
-packets. TCP transport is not supported.
+The server also exposes helpers that are useful for embedded capture loops:
+
+* configurable accept, session-dispatch, and per-session control task stack
+  sizes
+* `has_active_sessions()` to avoid capturing when no client is actively playing
+* `get_capture_cooldown()` and `get_recommended_capture_period()` so an
+  application can slow capture when RTP backpressure is observed
+* a legacy MJPEG `send_frame(std::span<const uint8_t>)` path that preserves the
+  older wire format for existing MJPEG-only users
+
+## Packetizers and Depacketizers
+
+The packetizer/depacketizer abstraction allows the server and client to support
+multiple media codecs without changing the RTSP core:
+
+- **MJPEG** (`MjpegPacketizer` / `MjpegDepacketizer`) — RFC 2435 JPEG over RTP
+- **H.264** (`H264Packetizer` / `H264Depacketizer`) — RFC 6184 with FU-A fragmentation
+- **Generic** (`GenericPacketizer` / `GenericDepacketizer`) — MTU chunking for
+  audio or other pre-encoded payloads, with frame reconstruction based on RTP
+  marker / timestamp boundaries
+
+Custom packetizers can be created by subclassing `RtpPacketizer` or
+`RtpDepacketizer`.
 
 ## Testing and Utilities
 
-We have a few ways for testing the RTSP code:
-- **ESPP Python Library**:
-  [`espp/lib`](https://github.com/esp-cpp/espp/tree/main/lib) contains the code
-  and scripts needed to build the ESPP Python library, which can be used to test
-  the RTSP server and client using Python scripts. There are existing RTSP
-  python tests within
-  [`espp/python`](https://github.com/esp-cpp/espp/tree/main/python) which use
-  this lib and provide various RTSP client / server functionality.
-- **Pure Python**: The [`rtsp/python`](./python) folder contains pure Python
-  scripts that can be used to connect to and display the RTSP stream from an
-  RTSP server, using both mDNS and directly by IP address.
-
-### ESPP Python Library
-
-The [`espp/lib`](https://github.com/esp-cpp/espp/tree/main/lib) folder contains
-the code and scripts needed to build the ESPP Python library, which can be used
-to test the RTSP server and client functionality. This library provides a Python
-interface to the RTSP server and client classes, allowing you to easily send and
-receive RTSP requests and responses, as well as send and receive JPEG frames
-over RTP/UDP.
-
-The [`espp/python`](https://github.com/esp-cpp/espp/tree/main/python) folder
-contains the actual python scripts which run espp c++ code (via `pybind11`). See
-the `README` in that directory for more information on how to use those scripts.
-
-### Pure Python
-
-To facilitate testing and debugging of the RTSP code, there are some
-[python](./python) scripts in this folder which provide various mechanisms for
-connecting to and displaying the RTSP stream from an RTSP server, using both
-mDNS and directly by IP address. These scripts can be used to test the RTSP
-server and client implementations.
-
-See [python/README.md](./python/README.md) for more information on how to use
-these scripts.
+There are several ways to exercise the RTSP stack:
+
+* **ESPP Python library**: build the host library from
+  [`lib`](https://github.com/esp-cpp/espp/tree/main/lib) with `./build.sh` to
+  expose the RTSP client/server classes to Python.
+* **Python harness scripts**:
+  [`python`](https://github.com/esp-cpp/espp/tree/main/python) contains wrapper
+  and multi-track scripts for exercising legacy MJPEG flows, generic multi-track
+  flows, live microphone audio, and end-to-end host validation.
+* **Embedded examples and downstream apps**: the component example plus
+  [`camera-streamer`](https://github.com/esp-cpp/camera-streamer) and
+  [`camera-display`](https://github.com/esp-cpp/camera-display) cover practical
+  server/client integrations.
+
+See [`python/README.md`](../../python/README.md) for more information on the
+host-side scripts.
 
 ## Example
 
-The [example](./example) shows the use of the `espp::RtspServer` and
-`espp::RtspClient` classes provided by the `rtsp` component for performing
-streaming of JPEG images (`espp::JpegFrame`) using the `MJPEG` format over the
-Real Time Streaming Protocol (RTSP) / Real Time Protocol (RTP) packets.
+The [example](./example) demonstrates several RTSP usage patterns selected via
+menuconfig, including:
+
+* legacy MJPEG server + client behavior on the same device
+* server-only MJPEG streaming
+* client-only MJPEG reception from a remote RTSP server
+* startup API tests for packetizers, depacketizers, and server/client setup
+* multi-track streaming with MJPEG video plus generic audio
 
 For more complete example use, see the
 [camera-streamer](https://github.com/esp-cpp/camera-streamer) and

components/rtsp/example/README.md

@@ -1,9 +1,9 @@
 # RTSP Example
 
-This example shows the use of the `espp::RtspServer` and `espp::RtspClient`
-classes provided by the `rtsp` component for performing streaming of JPEG images
-(`espp::JpegFrame`) using the `MJPEG` format over the Real Time Streaming
-Protocol (RTSP) / Real Time Protocol (RTP) packets.
+This example demonstrates several `rtsp` component workflows, ranging from the
+legacy MJPEG API to the newer multi-track server API. The example can run as a
+server, client, or combined self-check depending on the selected menuconfig
+mode.
 
 For more complete example use, see the
 [camera-streamer](https://github.com/esp-cpp/camera-streamer) and
@@ -13,46 +13,57 @@ For more complete example use, see the
 
 ### Hardware Required
 
-This example is designed to be run on an M5Stack ESP32 Timer Cam module (server) or a
-ESP32-S3-Box (client).
+Any supported ESP32 board with Wi-Fi can run this example. The bundled test
+content uses an embedded JPEG image and synthetic audio data, so no camera,
+microphone, or display hardware is required.
 
 ### Configure the project
 
-```
+```sh
 idf.py menuconfig
 ```
 
-You need to configure the `WiFi` network in the `RTSP Example Configuration`
-menuconfig and you can optionally configure the `RTSP Server Port` (default
-8554).
+The `RTSP Example Configuration` menu includes:
+
+* `Example Mode`
+  * `Legacy MJPEG (server + client)` - runs the original MJPEG server and
+    client on the same device
+  * `MJPEG Server only` - serves the embedded JPEG on `/mjpeg/1`
+  * `MJPEG Client only` - connects to a remote MJPEG RTSP server
+  * `Multi-track server (MJPEG + audio)` - serves MJPEG video on track 0 and
+    generic `L16/16000` mono audio on track 1 at `/stream`
+* `Run API tests at startup` - exercises packetizers, depacketizers, and basic
+  client/server construction before networking starts
+* Wi-Fi credentials and retry count
+* `RTSP Server Port`
+* `Remote RTSP server address` when client-only mode is selected
 
 ### Build and Flash
 
-NOTE: this example is designed to be modified into client only or server-only operation depending on the hardware it is deployed to.
-
-Build the project and flash it to the board, then run monitor tool to view serial output:
+Build the project and flash it to the board, then run monitor tool to view
+serial output:
 
-```
+```sh
 idf.py -p PORT flash monitor
 ```
 
 (Replace PORT with the name of the serial port to use.)
 
 (To exit the serial monitor, type ``Ctrl-]``.)
 
-See the Getting Started Guide for full steps to configure and use ESP-IDF to build projects.
+See the Getting Started Guide for full steps to configure and use ESP-IDF to
+build projects.
 
 ## Example Output
 
-Example showing the server (camera), the client (handheld display), and the python client, with both clients connected simultaneously:
-
-https://user-images.githubusercontent.com/213467/236601258-c334e1ba-5e18-4452-b48d-e792ec2ed4fb.mp4
-
-Screenshot showing the server running for a very long time (all day, see timestamp in log > 33,000 seconds):
-![all_day_test](https://user-images.githubusercontent.com/213467/236601320-0d9139d7-0333-4c63-b26f-da4078e141b7.png)
+The serial log reports the selected mode, the local IP address after Wi-Fi
+connects, and the RTSP URI for server modes. Typical output includes:
 
-Screenshot showing the received framerate on the camera-display:
-<img width="638" alt="CleanShot 2023-05-06 at 10 27 21@2x" src="https://user-images.githubusercontent.com/213467/236633241-a2aba704-e10f-4855-b07b-766a8f8d8658.png">
+* `All API tests passed!` when startup API tests are enabled
+* `RTSP URI: rtsp://<ip>:8554/mjpeg/1` in legacy and server-only modes
+* `RTSP URI: rtsp://<ip>:8554/stream` in multi-track mode
+* `Got JPEG frame: <width>x<height>` in client-only mode
+* `Streaming MJPEG video (track 0) + audio (track 1)...` in multi-track mode
 
-Screenshot of main code for camera-streamer output:
-<img width="799" alt="CleanShot 2023-05-06 at 10 29 31@2x" src="https://user-images.githubusercontent.com/213467/236633321-abdd2551-0a53-4be2-b90e-a693dfa89c12.png">
+For host-side end-to-end testing, build the host library in `lib/` with
+`./build.sh` and use the scripts in the repository-root `python/` directory.

components/rtsp/example/main/Kconfig.projbuild

@@ -6,6 +6,55 @@ menu "RTSP Example Configuration"
         help
             The port number of the RTSP server.
 
+    choice RTSP_EXAMPLE_MODE
+        prompt "Example Mode"
+        default RTSP_EXAMPLE_MODE_LEGACY
+        help
+            Select which RTSP example mode to run.
+
+        config RTSP_EXAMPLE_MODE_LEGACY
+            bool "Legacy MJPEG (server + client)"
+            help
+                Runs the original MJPEG server and client on the same device.
+                The server streams a JPEG image and the client receives it.
+                Uses the backward-compatible JpegFrame API.
+
+        config RTSP_EXAMPLE_MODE_SERVER_ONLY
+            bool "MJPEG Server only"
+            help
+                Runs only the RTSP server streaming MJPEG frames.
+                Connect an external RTSP client (e.g. VLC) to view the stream.
+
+        config RTSP_EXAMPLE_MODE_CLIENT_ONLY
+            bool "MJPEG Client only"
+            help
+                Runs only the RTSP client connecting to a remote RTSP server.
+                Requires an external RTSP server to be running.
+
+        config RTSP_EXAMPLE_MODE_MULTITRACK
+            bool "Multi-track server (MJPEG + audio)"
+            help
+                Runs a multi-track RTSP server with an MJPEG video track and
+                a generic audio track, demonstrating the add_track() API
+                and generic send_frame(track_id, data) API.
+
+    endchoice
+
+    config RTSP_EXAMPLE_RUN_API_TESTS
+        bool "Run API tests at startup"
+        default y
+        help
+            Run offline API tests for packetizers, depacketizers, and
+            server/client construction before starting network operations.
+            This exercises the component APIs to catch breaking changes.
+
+    config RTSP_CLIENT_SERVER_ADDRESS
+        string "Remote RTSP server address (client-only mode)"
+        default "192.168.1.100"
+        depends on RTSP_EXAMPLE_MODE_CLIENT_ONLY
+        help
+            IP address of the remote RTSP server to connect to in client-only mode.
+
     config ESP_WIFI_SSID
         string "WiFi SSID"
         default "myssid"