Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,12 @@ EXPORT_TTL_SECONDS=86400
# camera won't decode on-device (e.g. H.265 all the way down).
# Built lazily, only while something is watching.
# MOBILE_STREAM_WIDTH=640 # transcode width in px (floored at 160)
# MAIN_REPAIR_TRANSCODE_ENABLED=false # opt-in, per-camera FULL-RES H.265->H.264 transcode of a main
# whose SDP lacks fmtp (Android/Media3 rejects it with "missing
# attribute fmtp", e.g. some Uniview LPR cams). A cheap copy-remux
# does NOT work here (go2rtc then emits an HEVC aggregation packet
# Media3 can't read), so this is a real re-encode: leave off and
# Android just uses the H.264 sub (SD); on = HD at recorder CPU cost.
# SEGMENT_LOW_CACHE_MAX_BYTES=2147483648 # 2 GiB budget for the low-res playback segment cache

# --- Scrub-preview cache (optional; five of these are also editable in the console, which wins) ---
Expand Down
15 changes: 15 additions & 0 deletions apps/android/app/src/main/java/video/crumb/app/data/Models.kt
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,21 @@ data class LiveStreamsResponse(
@SerialName("webrtc_main_url") val webrtcMainUrl: String? = null,
@SerialName("webrtc_sub_url") val webrtcSubUrl: String? = null,
@SerialName("rtsp_main_url") val rtspMainUrl: String,
/**
* Repaired full-resolution main (`<name>_mainv`): a per-camera H.265->H.264
* **transcode** the server registers only for a main it detected publishes
* video with no `a=fmtp`, and only when the operator opts the repair in (it
* costs recorder CPU). Media3's RTSP client rejects a fmtp-less SDP with
* `IllegalArgumentException: missing attribute fmtp`; a copy-remux would fix
* that but go2rtc then bundles the H.265 parameter sets into an RTP Aggregation
* Packet Media3 cannot depacketize, so the repair has to be a transcode. Full
* resolution (no SD badge), tried right after [rtspMainUrl].
*
* Normally `null`: the camera's main is fine, the operator has not enabled the
* repair, the camera is unmanaged (Frigate-served / legacy), or the server
* predates the field — in every case the client just skips this rung.
*/
@SerialName("rtsp_mainv_url") val rtspMainvUrl: String? = null,
@SerialName("rtsp_sub_url") val rtspSubUrl: String? = null,
/**
* Video-only sub restream (`<name>_subv`): the raw sub run through an ffmpeg
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,18 @@ enum class StreamTier {
/** Full-resolution main stream (`rtsp_main_url`). */
MAIN,

/**
* The server's repaired full-resolution main (`rtsp_mainv_url`, `<name>_mainv`):
* a per-camera H.265->H.264 **transcode** the server registers only for a main
* it has detected publishes video with no `a=fmtp`, and only when the operator
* has opted the repair in (it costs recorder CPU). Unlike [SUBV] this is a
* re-encode, not a copy: a copy-remux of an H.265 main fixes the SDP but go2rtc
* then bundles the parameter sets into an RTP Aggregation Packet Media3 cannot
* depacketize, so only a transcode yields a main this device can actually play.
* Full resolution, so it carries no SD badge. Normally `null`.
*/
MAINV,

/**
* The server's video-only sub restream (`rtsp_subv_url`, `<name>_subv`): the
* raw sub run through an ffmpeg **copy** so go2rtc republishes a proper
Expand All @@ -55,6 +67,7 @@ enum class StreamTier {
/** The URL for [tier], or `null` when this camera does not expose that rung. */
fun LiveStreamsResponse.urlForTier(tier: StreamTier): String? = when (tier) {
StreamTier.MAIN -> rtspMainUrl
StreamTier.MAINV -> rtspMainvUrl
StreamTier.SUBV -> rtspSubvUrl
StreamTier.SUB -> rtspSubUrl
StreamTier.MOBILE -> rtspMobileUrl
Expand All @@ -76,29 +89,32 @@ private fun LiveStreamsResponse.chainOf(vararg order: StreamTier): List<StreamTi
* The wall is low-res by design (N tiles, N decoders), so a camera with a sub
* starts on it and never escalates to the main's bitrate: `subv → sub → mobile`.
* A camera with no sub at all keeps today's behaviour of playing the main, with
* the transcode as its one fallback: `main → mobile`.
* the repaired main (when the server publishes one) and the transcode as its
* fallbacks: `main → mainv → mobile`.
*/
fun LiveStreamsResponse.wallStreamChain(): List<StreamTier> =
if (subStreamUrl() != null) {
chainOf(StreamTier.SUBV, StreamTier.SUB, StreamTier.MOBILE)
} else {
chainOf(StreamTier.MAIN, StreamTier.MOBILE)
chainOf(StreamTier.MAIN, StreamTier.MAINV, StreamTier.MOBILE)
}

/**
* Fallback ladder for **fullscreen live**.
*
* Off a metered link fullscreen is the one place HD is worth it, so it starts on
* the main and walks down: `main → subv → sub → mobile`. On a metered link the
* data-saver order applies (unchanged from before: low-res first, the transcode
* when the camera has no sub), with the main kept as the last resort so a broken
* sub still leaves something to watch rather than a spinner.
* the main and walks down: `main → mainv → subv → sub → mobile`. The repaired main
* (`mainv`) sits right after the raw main so a camera whose main is unplayable for
* a fmtp/packetization reason gets HD from the server-side repair before dropping
* to an SD rung. On a metered link the data-saver order applies (low-res first, the
* transcode when the camera has no sub), with both HD mains kept as the last
* resort so a broken sub still leaves something to watch rather than a spinner.
*/
fun LiveStreamsResponse.fullscreenStreamChain(metered: Boolean): List<StreamTier> =
if (metered) {
chainOf(StreamTier.SUBV, StreamTier.SUB, StreamTier.MOBILE, StreamTier.MAIN)
chainOf(StreamTier.SUBV, StreamTier.SUB, StreamTier.MOBILE, StreamTier.MAIN, StreamTier.MAINV)
} else {
chainOf(StreamTier.MAIN, StreamTier.SUBV, StreamTier.SUB, StreamTier.MOBILE)
chainOf(StreamTier.MAIN, StreamTier.MAINV, StreamTier.SUBV, StreamTier.SUB, StreamTier.MOBILE)
}

/**
Expand Down Expand Up @@ -174,19 +190,32 @@ fun isCodecAgnosticError(errorCode: Int): Boolean =
* Substrings that identify a failure Media3 will hit **every single time** on this
* stream, from the exception chain rather than the error code.
*
* The one that matters today is `RtpH265Reader`'s
* `UnsupportedOperationException("need to implement processAggregationPacket")`,
* verified present in the pinned `media3-exoplayer-rtsp:1.4.1`. RFC 7798 §4.4.2
* Aggregation Packets pack several SMALL NALs into one RTP packet, and Media3 has
* never implemented that path (androidx/media#1008). It is thrown from inside the
* loader, so it reaches the app wrapped in an `IOException` and arrives with an
* IO error code that looks exactly like a network problem — which is precisely
* why the code alone is not enough to classify it.
* Two matter today, both thrown from inside the RTSP loader, so both reach the app
* wrapped in an `IOException` and arrive with an IO error code that looks exactly
* like a network problem — which is precisely why the code alone is not enough to
* classify either:
*
* - `processAggregationPacket`: `RtpH265Reader`'s
* `UnsupportedOperationException("need to implement processAggregationPacket")`,
* verified present in the pinned `media3-exoplayer-rtsp:1.4.1`. RFC 7798 §4.4.2
* Aggregation Packets pack several SMALL NALs into one RTP packet, and Media3 has
* never implemented that path (androidx/media#1008).
* - `missing attribute fmtp`: the RTSP client rejects an SDP whose media track has
* no `a=fmtp` line (an H264/H265 track with no out-of-band parameter sets), the
* `IllegalArgumentException` #483 is about. Depending on where it surfaces this
* can arrive as a `1004 FAILED_RUNTIME_CHECK` (already an unplayable format code)
* OR — as seen on an LPR camera's H.265 MAIN over RTSP — wrapped through
* `RtspPlaybackException` into a `2000` IO code, which the graduated threshold
* would otherwise retry ~30 s before giving up. The SDP does not change between
* attempts, so it is deterministic; the signature makes the step-down immediate
* regardless of the code it happens to wear. NB the server-side `_subv`/`_mainv`
* fmtp repair fixes only cameras it manages and detects — this is the client's
* backstop for the rest.
*
* This is a *narrow* list on purpose: a signature here means "step down now, do
* not retry", so only failures that are structurally deterministic belong in it.
*/
private val UNPLAYABLE_FAILURE_SIGNATURES = listOf("processAggregationPacket")
private val UNPLAYABLE_FAILURE_SIGNATURES = listOf("processAggregationPacket", "missing attribute fmtp")

/**
* True when a failure's [detail] (see [failureDetail]) names a deterministic,
Expand Down Expand Up @@ -369,7 +398,8 @@ fun attachLogLine(
* fact the device is watching a server-side re-encode (#524).
*/
fun sdBadgeLabel(tier: StreamTier?): String? = when (tier) {
null, StreamTier.MAIN -> null
// MAINV is a full-resolution repaired main — HD, so no badge, like MAIN.
null, StreamTier.MAIN, StreamTier.MAINV -> null
StreamTier.SUBV, StreamTier.SUB -> "SD · tap for HD"
StreamTier.MOBILE -> "SD transcode · tap for HD"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,14 @@ class LiveStreamFallbackTest {

private fun streams(
main: String? = "rtsp://u:p@host:18554/drive",
mainv: String? = null,
sub: String? = "rtsp://u:p@host:18554/drive_sub",
subv: String? = null,
mobile: String? = "rtsp://u:p@host:18554/drive_mobile",
) = LiveStreamsResponse(
cameraId = "8f14e45f-ceea-467a-9f3a-8f14e45fceea",
rtspMainUrl = main ?: "rtsp://u:p@host:18554/drive",
rtspMainvUrl = mainv,
rtspSubUrl = sub,
rtspSubvUrl = subv,
rtspMobileUrl = mobile,
Expand Down Expand Up @@ -125,6 +127,48 @@ class LiveStreamFallbackTest {
assertEquals(listOf(StreamTier.SUBV, StreamTier.MOBILE), chain)
}

@Test
fun `fullscreen chain puts the repaired main right after the raw main`() {
val chain = streams(
mainv = "rtsp://u:p@host:18554/drive_mainv",
subv = "rtsp://u:p@host:18554/drive_subv",
).fullscreenStreamChain(metered = false)
assertEquals(
listOf(
StreamTier.MAIN, StreamTier.MAINV,
StreamTier.SUBV, StreamTier.SUB, StreamTier.MOBILE,
),
chain,
)
// The repaired main is HD, so it must carry no SD badge.
assertNull(sdBadgeLabel(StreamTier.MAINV))
}

@Test
fun `metered fullscreen keeps the repaired main a last resort with the raw main`() {
val chain = streams(mainv = "rtsp://u:p@host:18554/drive_mainv")
.fullscreenStreamChain(metered = true)
assertEquals(
listOf(StreamTier.SUB, StreamTier.MOBILE, StreamTier.MAIN, StreamTier.MAINV),
chain,
)
}

@Test
fun `a no-sub camera puts the repaired main before the transcode on the wall`() {
val chain = streams(mainv = "rtsp://u:p@host:18554/drive_mainv", sub = null)
.wallStreamChain()
assertEquals(listOf(StreamTier.MAIN, StreamTier.MAINV, StreamTier.MOBILE), chain)
}

@Test
fun `the repaired main rung is absent when the server does not publish one`() {
// The common case: mainv is null, so the rung is simply not in the ladder
// and every existing chain is unchanged.
assertFalse(StreamTier.MAINV in streams().fullscreenStreamChain(metered = false))
assertFalse(StreamTier.MAINV in streams().wallStreamChain())
}

// ── walking the ladder ────────────────────────────────────────────────────

@Test
Expand Down Expand Up @@ -270,6 +314,47 @@ class LiveStreamFallbackTest {
assertFalse(isUnplayableFailureDetail(null))
}

@Test
fun `the missing-fmtp failure steps down at once instead of waiting out the IO backoff`() {
// An LPR camera's H.265 MAIN advertises an SDP with no `a=fmtp`, so Media3's
// RTSP client throws IllegalArgumentException("missing attribute fmtp"). On
// the MAIN over RTSP it surfaces wrapped through RtspPlaybackException into a
// 2000 IO code (captured on device, #561), which the graduated threshold
// would otherwise retry ~30 s before escaping. The SDP is identical every
// attempt, so the signature must step it down on the FIRST failure.
val detail = failureDetail(
java.io.IOException(
"Source error",
IllegalArgumentException("missing attribute fmtp"),
),
)
assertTrue(detail!!, isUnplayableFailureDetail(detail))
assertTrue(
shouldFallBackToNextTier(
hasNextTier = true,
everReady = false,
errorCode = 2000,
failureDetail = detail,
preFirstFrameFailures = 1,
),
)
// Contrast: a generic 2000 IO failure with no such signature still has to
// repeat across the whole backoff curve before the rung is abandoned — the
// fmtp match must not have widened the codec-agnostic retry path (#560).
(1 until CODEC_AGNOSTIC_FAILURES_BEFORE_FALLBACK).forEach { n ->
assertFalse(
"generic IO failure $n must still retry",
shouldFallBackToNextTier(
hasNextTier = true,
everReady = false,
errorCode = 2000,
failureDetail = failureDetail(java.io.IOException("Source error")),
preFirstFrameFailures = n,
),
)
}
}

@Test
fun `a failure detail never carries stream credentials into logcat`() {
// RTSP URLs here embed the stream user and password; Media3 puts the URL it
Expand Down
13 changes: 7 additions & 6 deletions data/camera-compatibility.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
"streams": {
"main": {
"codec": "H265",
"notes": "Full-range (yuvj420p), RTP aggregation-packet packetization, and a 'PPS id out of range' bitstream quirk. Benign to ffmpeg (server ingest, recording, desktop, and web are unaffected); rejected by strict RTSP HEVC readers."
"notes": "The RTSP SDP advertises the H265 video track with NO 'a=fmtp' (no sprop-vps/sps/pps) — the parameter sets ship only in-band. Android's Media3 RTSP client requires fmtp and rejects the DESCRIBE with 'IllegalArgumentException: missing attribute fmtp' (confirmed on-device 2026-08-08). Also full-range (yuvj420p) with a 'PPS id out of range' bitstream quirk. All benign to ffmpeg, so server ingest, recording, desktop, and web are unaffected."
},
"sub": {
"codec": "H264",
Expand All @@ -44,18 +44,19 @@
{
"summary": "Android fullscreen HD live falls back to SD",
"affects": ["android"],
"detail": "The H265 main uses RTP aggregation packets plus a 'PPS id out of range' bitstream quirk that Android's RTSP HEVC path (Media3/ExoPlayer) cannot handle, so fullscreen live silently drops to the H264 sub and shows an 'SD' badge. It is specific to this camera's stream, not to H265 in general: clean 4K H265 mains from other cameras play full HD on the same phone. Server ingest, recording, desktop, and web are all unaffected because ffmpeg tolerates the quirk. iOS is untested."
"detail": "The confirmed blocker is the missing 'a=fmtp' in the H265 main's SDP: Media3's RTSP client throws 'missing attribute fmtp' at DESCRIBE, before any video decodes, and the tile drops to the H264 sub with an 'SD' badge. It is specific to this camera's stream, not to H265 in general: clean 4K H265 mains from other cameras play full HD on the same phone. Note the H264 '_subv' fmtp repair (an ffmpeg copy that recovers the parameter sets) does NOT translate to this H265 main: a copy-remux restores fmtp but go2rtc then bundles the VPS/SPS/PPS/SEI into an RTP Aggregation Packet at each keyframe, which Media3's RtpH265Reader cannot depacketize (androidx/media#1008) — so the copy only trades the fmtp rejection for a 'processAggregationPacket' crash (verified by packet capture 2026-08-08). Only a full-res H265->H264 transcode ('_mainv', opt-in via MAIN_REPAIR_TRANSCODE_ENABLED) yields an HD main this device can play. Server ingest, recording, desktop, and web are all unaffected."
}
],
"recommended_settings": [
"For phone HD live: set the Main stream codec to H264 in the camera web UI (Setup > Video > Encoding).",
"Leave the Sub stream on H264 (the default).",
"For phone HD live: set the Main stream codec to H264 in the camera web UI (Setup > Video > Encoding). This is the cheapest fix and avoids any server transcode.",
"Alternatively, enable MAIN_REPAIR_TRANSCODE_ENABLED on the server for an opt-in, per-camera H265->H264 transcode of the main (costs recorder CPU while an Android viewer is fullscreen).",
"Leave the Sub stream on H264 (the default); Android steps down to it (SD) automatically and now does so instantly on the deterministic fmtp error.",
"Keep the keyframe (I-frame) interval near 1 second (the default) so the stream starts quickly."
],
"tested": {
"by": "badbread",
"date": "2026-07-10",
"method": "ffprobe on the go2rtc restream plus on-device Android playback; make/model via ONVIF GetDeviceInformation"
"date": "2026-08-08",
"method": "ffprobe + RTP packet capture (tshark) on the go2rtc restream and copy-remux, on-device Android playback (Media3 1.4.1), and CrumbLiveFallback logging (#561); make/model via ONVIF GetDeviceInformation"
},
"references": [
"https://github.com/androidx/media/issues/1008",
Expand Down
1 change: 1 addition & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,7 @@ services:
# value set in .env actually reaches the api (compose only passes listed keys).
MOBILE_STREAM_ENABLED: ${MOBILE_STREAM_ENABLED:-}
MOBILE_STREAM_WIDTH: ${MOBILE_STREAM_WIDTH:-}
MAIN_REPAIR_TRANSCODE_ENABLED: ${MAIN_REPAIR_TRANSCODE_ENABLED:-}
SEGMENT_LOW_CACHE_MAX_BYTES: ${SEGMENT_LOW_CACHE_MAX_BYTES:-}
EXPORT_CACHE_MAX_BYTES: ${EXPORT_CACHE_MAX_BYTES:-}
# Behind the bundled Caddy (or any reverse proxy), set TRUST_PROXY=1 so the
Expand Down
1 change: 1 addition & 0 deletions docs-site/docs/configuration/environment-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ are sized for a phone on a slow link.
|---|---|---|
| `MOBILE_STREAM_ENABLED` | `true` | the on-demand H.264 transcode that mobile clients fall back to when a camera's own streams won't decode on the device, notably a camera that is H.265 all the way down. Turning it off saves server CPU and costs those cameras live view on Android |
| `MOBILE_STREAM_WIDTH` | `640` | transcode width in pixels, floored at 160 |
| `MAIN_REPAIR_TRANSCODE_ENABLED` | `false` | opt-in, per-camera full-resolution H.265 to H.264 transcode of a main stream whose SDP has no `fmtp` attribute. Android's video player rejects such a main ("missing attribute fmtp", seen on some Uniview LPR cameras) and otherwise steps down to the H.264 sub in SD. Leave it off and those cameras play in SD on Android; turn it on to get HD, at the cost of recorder CPU while an Android viewer is watching that camera fullscreen. A cheaper copy-only repair does not work for this case, which is why it is a real re-encode and off by default. The cheapest fix of all, when the camera allows it, is to set the camera's main stream to H.264 in its own web UI |
| `SEGMENT_LOW_CACHE_MAX_BYTES` | `2147483648` (2 GiB) | size budget for the cache of low-resolution playback segments |

## Database backup
Expand Down
Loading
Loading