POC: in-browser video streaming from .mcap files - #4076
Draft
Williangalvani wants to merge 15 commits into
Draft
Conversation
…e them The frontend extracts video from the recording itself now, so the service no longer extracts MP4s and its repair loop is down to giving unindexed recordings an index, which is what random access needs. The static binary that used to do the extraction goes with it. Recordings themselves are read straight from nginx, which already serves the recorder directory under /userdata. Copying those bytes in Python cost the vehicle around eight times the CPU nginx needs for the same read, and made opening a recording ten times slower, since the index is read as a handful of tiny ranges where per request overhead is all there is. Co-authored-by: Cursor <cursoragent@cursor.com>
Reads the MCAP index over HTTP range requests, decompresses only the chunks covering what is being watched, and muxes the H.264/H.265 frames into fragmented MP4 for Media Source Extensions. MSE is used instead of WebCodecs because WebCodecs requires a secure context and BlueOS is served over HTTP. Keyframes are located from the message indexes, so seeking costs a few kilobytes instead of downloading the surrounding chunks. The chunk index is read in 256 kB windows as playback and seeking reach them, so opening a recording costs about 40 kB rather than the 1.7 MB index a 1.3 GB file carries, and the record walker reports how far it got so a window can resume where the previous one stopped. Seeking has to leave the playhead alone. An append that was in flight when a seek arrived would move the playhead to its own start time, leaving the element waiting forever for media before the seek target that is never read, since reading only goes forward. Reading also holds off while a seek points the stream at its new time, because a stall or time update arriving in that window started reading from the old position again and quietly pulled hundreds of megabytes. The harness exercises playback and a mid-file seek for every video track of a recording, which is how the window sizes and the keyframe heuristic were measured against real recordings. Co-authored-by: Cursor <cursoragent@cursor.com>
Recordings are listed as MCAP and streamed straight from the vehicle, with duration and stream names read from the recording summary alone. Co-authored-by: Cursor <cursoragent@cursor.com>
A chunk holds every channel recorded in its time span, so the second camera of a recording was already being downloaded and thrown away while the first one played. Chunks are now decompressed once and shared, with downloads in flight joined rather than repeated, which makes the extra streams cost decode time and nothing on the link: both streams of a 4K plus 720p recording read 13.2 MB for four seconds, the same as the 4K stream alone. Recordings therefore show all their streams side by side instead of asking which one to watch. Fragments are timestamped from the recording clock, so the streams share a timeline and the largest one carries the controls, with the rest following its position and pulled back when they drift. Co-authored-by: Cursor <cursoragent@cursor.com>
Recordings hold H264 or H265 frames, so writing a video file out of one is a matter of wrapping those frames in a container: nothing is decoded or encoded, and the picture is exactly what the camera sent. Each stream of a recording gets a save button that reports progress and can be stopped, and the file is an ordinary MP4 rather than a fragmented one so players and editors that expect sample tables read it too. Saving reads through the same shared chunks as playback, so saving every stream of a recording costs one download: exporting both streams of a two camera recording together reads 42.6 MB, the same as either one alone, and produces files byte identical to exporting them one after the other. Co-authored-by: Cursor <cursoragent@cursor.com>
A recording that plays badly can be badly recorded or badly decoded, and there was no way to tell which. A button in the player footer puts a panel over every stream with what the pipeline knows: frames read out of the recording, keyframes, frames the recorder never wrote, frames skipped while waiting for a keyframe, frames holding no video data at all, what the browser decoder decoded and dropped, decode errors, and the measured frame rate, bitrate and buffer. Lost frames come from the per channel sequence numbers that MCAP messages carry and the reader used to skip over, so a gap is a frame that never reached the file rather than a guess. Seeking moves those numbers on purpose and is not counted as loss. Reading every message of four recordings found no gaps, and reading one with every second up to every fiftieth message removed reported exactly the number missing in each case. Streams that fail now report what they read before giving up, so a recording that cannot be decoded still shows what it cost to find that out, and the panel stays out of the way while a stream is showing an error. Co-authored-by: Cursor <cursoragent@cursor.com>
Saving a whole stream downloads practically the whole recording, and not because of the MP4 side: frames sit in zstd compressed MCAP chunks that also carry every other topic of their time span, so a chunk can only be read whole. Saving the 0.16 MB stream of a 42 MB recording read 42.57 MB of it, and there is nothing a different container or a streamed write could do about that. Choosing a part to save cuts the read down to the chunks the bounds fall in: a 30 second clip of a 397 MB recording reads 66 MB instead of 395 MB. The choice is a range slider over the whole recording, opened by the scissors in the footer, so the part being saved is visible and can be dragged, with the playback position drawn over it and buttons to place a bound exactly where playback sits. Dragging a bound shows the frame it lands on, and a selection covering everything reads as the whole recording, so the file keeps its plain name. Saving starts at the keyframe at or before the start, since nothing decodes before one, so a clip can open a moment earlier than asked for. The two downloads a recording offers now say what they hand over, since the player showed two bare download icons stacked above each other with nothing to tell the stream apart from the original file. The original moves up beside the dialog title, which also drops a whole row of controls from under the player. Co-authored-by: Cursor <cursoragent@cursor.com>
bun.lockb was written by a bun old enough that 1.2.21 refuses to parse it, so every build printed "Ignoring lockfile" and resolved the whole tree from scratch. That is how the deployed image ended up running dependency versions nobody had ever built locally. Replace it with a text bun.lock generated by the same bun the image uses, and install with --frozen-lockfile in both the image and CI so a stale lockfile fails the build instead of being silently ignored. CI now runs the same bun as the image, so its lint and build cover what we ship. Which versions get hoisted mattered here. vue-tooltip-directive requires tslib without declaring it, and when the hoisted copy was tslib 1.x, its UMD exporter published the legacy two argument __classPrivateFieldGet onto the global object, so packages that fall back to it (@foxglove/rosmsg2-serialization does) called .get() on the WeakSet used to brand private methods and broke MCAP video playback on the deployed image with "Le.get is not a function". Declaring tslib directly keeps a v2 copy at the top level regardless of hoisting order, leaving tslib 1.x only under tsutils where it is never bundled. Co-authored-by: Cursor <cursoragent@cursor.com>
It was disabled in 8ea5412 because its automatic actions caused instability, and those actions were the extraction loop that transcoded every recording it found, which is gone: the browser reads video out of the recording now and nginx serves the bytes. What the service still does is list recordings, give unindexed ones an index, make thumbnails and delete files, and the recordings page cannot find a recording without it, so leaving it off means a page that polls into 502s every five seconds. Co-authored-by: Cursor <cursoragent@cursor.com>
Repairing rewrites the whole recording, so doing it from a loop meant the vehicle could rewrite a file that something else was still writing, and it spent disk bandwidth on files nobody was looking at. The service now exposes the repair as a request, refuses one on a recording that is still being written, and reports why a repair failed. The records page offers it on the recordings it cannot read, and reads them again once the vehicle is done. Co-authored-by: Cursor <cursoragent@cursor.com>
Recordings are MCAP now, decoded by the frontend, so nothing on the vehicle needs gstreamer to look inside them. This takes out the thumbnail endpoint and its pipelines, and stops listing MP4 files that no longer get produced. Co-authored-by: Cursor <cursoragent@cursor.com>
With MP4 extraction gone from the vehicle, the records page had nothing to show on each card. Decode one mid-recording keyframe over range requests, cache the JPEG in IndexedDB, and paint it under the play button. Co-authored-by: Cursor <cursoragent@cursor.com>
Fetching, previewing and playing recordings can saturate the link the pilot needs for control. Stop every in-flight download the moment the vehicle arms, and keep the page dark until it is safe again. Co-authored-by: Cursor <cursoragent@cursor.com>
The clip range, progress and stream stats were scattered under each video. Pull them into one Export MP4 panel and a compact footer so saving a part of a recording is obvious without hunting for icon buttons. Co-authored-by: Cursor <cursoragent@cursor.com>
Seeking backwards was silently undone once media further ahead had been buffered: the player counted anything buffered past a gap as media ahead of the playhead, skipped restarting its reads, and then moved the playhead forward onto that stale media. Picking an export range seeks the stream, so touching the trim bounds was enough to leave the video stuck. Media is now only considered playable when it covers the playhead itself, the playhead is left alone while a restart is on its way, and restarts happen one at a time so overlapping seeks cannot leave the reader idle. Ending the media stream is also held back until the buffer reaches the end of the recording, since it shortens the stream to whatever is buffered and would otherwise put the rest of the recording out of reach. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This was vibe-coded and still has some unrelated changes and odd decisions in it. but I'm opening the pr for the curious to see.