Live video+audio publishing stutters end to end (not reproducible in MediaPublish alone)

[rickstaa]

Summary

Publishing video + audio through one MediaPublish delivers the stream in far larger bursts than video alone, and drops a large fraction of video frames. It is not a capacity problem: the client uses less CPU in the stuttering case than in the smooth one, so the encoder thread is blocking, not computing.

Practical effect: with an audio track present, live playback freezes roughly every 2 seconds while audio stays continuous. Recorded output is complete and in sync, so it only shows up on a live path.

Measurements

Same webcam source, same runner, same SDK (main @ a0b9129), only the audio track differs.

Frame arrival, measured on both sides of the publish boundary:

	client hands to write_frame	received by the peer
video only	max gap 0.046 - 0.071 s	max gap 0.046 - 0.071 s
video + audio	max gap 0.061 - 0.081 s	max gap 0.354 - 0.541 s

The client emits smoothly in both cases and write_frame never blocks (0.000 s). The receiving side adds nothing of its own: its outbound gap tracks its inbound gap to within 0.003 s. So the unevenness is introduced between write_frame returning and bytes reaching the wire, which is entirely inside MediaPublish.

Video frame loss, client publisher stats (14 s runs, 387 video frames in):

video queue_size=8 (default):  ovf=76  debt=15   -> 23% of video lost
video queue_size=32:           ovf=0   debt=38   -> audio then overflowed instead (ovf=99..144)
video queue_size=64:           ovf=0   debt=14..32  -> audio ovf 0..182, run to run
audio (any of the above):      ovf=0 or 0..182 depending on which queue is deeper

Which track sheds frames, and how many, varies enormously between identical runs. Raising one queue moves the loss to the other.

CPU, whole client process (/usr/bin/time -v):

video + audio (stuttering):  21% CPU, 17.45 s wall
video only   (smooth):       64% CPU, 17.22 s wall

The failing case is idle-waiting. Runner container CPU was 31% on a 32-core host. No GPU involved (software libx264 both directions).

Ruled out

Each of these was implemented and measured, and none changed the result:

• keyframe_interval_s at 0.5, 0.25, and 10.0 (at 10.0 only two segment rotations occurred in a 14 s run and the gaps persisted, so it is not segment rotation)
• Sharing one idle-wait budget across tracks instead of timeout=0.05 per track in _next_encoder_item
• Selecting the track with the earliest head media time instead of round-robin
• Moving the app's frame transform off the event loop with asyncio.to_thread
• Encoder settings: already preset=superfast, tune=zerolatency, bf=0
• frames_dropped_non_monotonic_pts = 0, segments_failed = 0, encoder_errors = 0, segment_writer_put_timeouts = 0

Also worth noting: two video tracks are fine, and measurably smoother than one (max gap 0.22 s, zero gaps over 300 ms). So it is not track count or interleaving in general, it is specific to having an audio track.

Two design points that make it worse

1. queue_size is in frames, so its meaning changes with frame rate. VideoOutputConfig.queue_size = 8 is 0.27 s at 30 fps, while AudioOutputConfig.queue_size = 32 is about 0.68 s at 48 kHz / 1024. Any shared stall therefore overflows video roughly 2.5x sooner than audio, for reasons nobody chose. Specifying the depth in seconds and deriving maxsize per track would remove the asymmetry.

2. debt_skip=True is set on video tracks only (media_publish.py:333). Under pressure the SDK always sheds the stream the user can see and never the one they can hear, so any stall in the shared encoder surfaces as video stutter with clean audio.

Where I would look next

Since the thread is blocking rather than computing, timing inside _run_encoder should name the call: wrap _encode_track_frame, the container.mux() call, and the segment handoff, and log anything over 50 ms. Candidates are backpressure from the trickle segment POST (segments_started=7, completed=6, so one segment is always in flight and a two-track segment is larger), GIL contention between the encoder thread and the event loop, or a lock in the segment writer.

Reproduction

# runner: any app that publishes video plus an audio track
ffmpeg -f v4l2 -input_format mjpeg -video_size 1280x720 -framerate 30 -i /dev/video0 \
       -f alsa -i plughw:1,0 -filter_complex "[1:a]aresample=async=1:first_pts=0[a]" \
       -map 0:v -map "[a]" -fps_mode cfr \
       -c:v libx264 -preset ultrafast -tune zerolatency -pix_fmt yuv420p -g 30 \
       -c:a aac -ar 48000 -f mpegts - \
| <client that publishes both tracks> | ffplay -fflags nobuffer -flags low_delay -framedrop -i -

Compare against the same command with the audio input and -map removed. Read publisher.get_stats().track_queue_stats on both sides.

Found while adding an audio transform to the echo example in livepeer/runner-app-examples#58. Possibly related to #21, which covers the audio-only segment_time path.

[linear-code]

ENG-232

[rickstaa]

Need to test this with a more minimal example.

[rickstaa]

Correction

The original report claimed the bug is inside MediaPublish. A minimal repro disproves that. MediaPublish handles two tracks cleanly in isolation.

Runnable repro on branch rs/repro-two-track-stall, scripts/repro_two_track_stall.py. It publishes synthetic video (optionally plus audio) into a local sink implementing the three trickle endpoints, with no orchestrator, no network and no example app:

python scripts/repro_two_track_stall.py            # video only
python scripts/repro_two_track_stall.py --audio    # video + audio

Result:

video only:   emit 420  sink 420  max_gap 0.038s  ovf=0 debt=0
video+audio:  emit 420  sink 838  max_gap 0.045s  ovf=0 debt=0

Zero drops, zero debt, no bursts. Adding audio costs 7 ms of extra gap, not the 400 ms seen end to end. So the earlier measurements were correct about the symptom but wrong about the location.

What the repro strips away, and what that leaves

The real path has four things this script does not:

1. A blocking PyAV decode() generator running on the client's event loop, which with audio decodes both streams and so does more work per second
2. MediaOutput reading the echoed stream back concurrently
3. Real HTTPS/TLS to the orchestrator rather than plain local HTTP
4. The orchestrator proxying, plus the runner's own decode, transform and publish

Partial result on (1)

Moving the client's decode into a worker thread and feeding frames to the loop through a queue improved the end-to-end gap from 0.81 - 0.96 s to 0.62 s, with more chunks delivered and lower first-byte latency. So event-loop starvation by the synchronous decode is a real contributor, but it does not account for all of it.

Status

The symptom measurements in the original report still stand and are reproducible. The attribution to MediaPublish does not. Remaining suspects are (2), (3), (4) above, and whatever else accounts for the residual gap after fixing (1).

Anyone continuing should start from the repro branch and add one of the four stripped-away pieces at a time until the stall appears.

[rickstaa]

Conclusion for now

Closing this. The attribution in the title was disproven by the minimal repro, and the remaining effect is small enough that it is not worth holding an open issue on without someone actively working it.

What is settled:

• MediaPublish handles video + audio cleanly in isolation: 0.045 s max gap, zero drops, zero debt. Repro on branch rs/repro-two-track-stall, scripts/repro_two_track_stall.py.
• The end-to-end stutter was real, and roughly half of it was the consuming app decoding on its event loop. Fixed app-side in perf(echo): decode off the event loop so uploads are not starved runner-app-examples#68: max gap 1.0 s to 0.62 s, startup 0.45 s faster.
• The residual gap (0.62 s with audio versus 0.42 s video-only) is unexplained. It is not CPU (the stuttering case used less), not frame count (batching audio 47/s to 16/s changed nothing), not segment size, not queue depth, not encoder settings.

Two observations worth keeping regardless of the stutter:

1. queue_size is specified in frames, so its meaning changes with rate. Video's default 8 is 0.27 s at 30 fps while audio's 32 is about 0.68 s, so a shared stall overflows video roughly 2.5x sooner for reasons nobody chose. Specifying it in seconds would remove the asymmetry.
2. debt_skip=True is set on video tracks only, so under pressure the SDK always sheds the stream the user can see and never the one they can hear.

Those two are worth their own issue if anyone wants to act on them. Reopen this if the residual gap becomes a priority; the repro branch is the place to start.