echo: live playback stutters with --mode robot (upstream SDK bug)

[rickstaa]

echo --mode robot publishes a video and an audio track together. On the live path (- in, - out, piped to a player) the picture freezes for a moment roughly every 2 seconds while the audio stays continuous. The same clip recorded to a file is complete and in sync, so this only affects live playback.

The cause is in the SDK, not in this repo: livepeer/livepeer-python-gateway#60.

Frames leave the client's write_frame smoothly (0.061 - 0.081 s gaps, never blocking) and arrive at the runner in 0.354 - 0.541 s bursts; video only, through the identical path, stays at 0.046 - 0.071 s. The runner adds nothing of its own. Nothing in echo/ participates in that span.

Seven mitigations were tried here and upstream, and none changed the result: keyframe_interval_s at 0.5 / 0.25 / 10.0, video queue_size at 32 and 64 (this just moves the loss onto audio), moving the transform off the event loop with asyncio.to_thread, a shared idle-wait budget across tracks, and timestamp-ordered track selection.

Until it is fixed upstream

echo/README.md documents recording to a file and playing it back, which is unaffected. The live variant is mentioned there too and should be treated as the lossy option when audio is involved.

Nothing to do in this repo beyond closing this issue when #60 lands and re-checking the live path.

[linear-code]

ENG-234

[rickstaa]

Correction

I attributed this to MediaPublish in the SDK. That attribution was wrong, and a minimal repro proves it: MediaPublish publishing video + audio in isolation is clean (0.045 s max gap, zero drops, zero debt). See livepeer/livepeer-python-gateway#60 and the runnable script on branch rs/repro-two-track-stall.

So part of this may well be on the examples side. Moving the client's blocking PyAV decode() off the event loop into a worker thread improved the end-to-end gap from 0.81 - 0.96 s to 0.62 s. That is echo/client.py code, not SDK code. It does not close the gap entirely, so something else remains.

The symptom is unchanged and still reproducible: with --mode robot the live path stutters roughly every 2 s while audio stays continuous, and the recorded file is perfect.

Next step for whoever picks this up: start from the SDK repro branch and add back one piece at a time (client decode on the loop, MediaOutput reading concurrently, TLS to the orchestrator, orchestrator proxying) until the stall appears.

[rickstaa]

Conclusion for now

Closing this. Roughly half the stutter was ours and is fixed in #68: the client decoded on its event loop, which stalled the async segment uploads. Moving decode to a worker thread took the max gap from 1.0 s to 0.62 s and cut startup latency by 0.45 s, with no regression for video-only modes.

The residual gap (0.62 s with audio versus 0.42 s video-only) is unexplained and not worth holding an issue open on. It is not CPU, not audio frame count, not segment size, not queue depth, not encoder settings, and MediaPublish handles two tracks cleanly in isolation (repro on the SDK branch rs/repro-two-track-stall).

Practical guidance stays as documented in echo/README.md: record to a file and play it back for a clean demo of --mode robot; the live pipe works but is choppier with audio than without.

See livepeer/livepeer-python-gateway#60 for the full measurements and the two SDK design observations that came out of it.