APP-16408 handle depth cam by DTCurrie · Pull Request #71 · viamrobotics/test-widgets

DTCurrie · 2026-05-19T18:42:59Z

Fixes the camera widget's source dropdown so switching between sources like color and depth actually updates the displayed image. The dropdown was effectively inert for multi-source cameras (e.g., RealSense), and the depth source rendered as a blank feed because the browser cannot decode Viam's custom depth MIME type.

Frontend

pick-image-for-source.ts: new helper that selects the NamedImage matching the chosen source, falling back to the first image. Used as a defensive client-side filter for cameras whose Images() ignores filter_source_names and returns every source.
decode-viam-depth.ts: new decoder for image/vnd.viam.dep. Parses the 24-byte header (8-byte DEPTHMAP magic, two big-endian uint64 dimensions) and the uint16 big-endian depth grid, then emits a colorized Uint8ClampedArray using the same hue sweep (30° warm to 230° cool) that RDK's DepthMap.ToPrettyPicture produces. Zero-depth pixels render as opaque black to match RDK.
live-or-polling-video.svelte: takes a new sourceName prop. The polling-image effect now uses pickImageForSource instead of data.images[0], and routes image/vnd.viam.dep frames through decodeViamDepth to a putImageData call on the existing canvas so they participate in the same capture stream and FPS counter as JPEG frames.
export-screenshot.svelte: takes a sourceName prop so screenshots respect the dropdown.
camera.svelte: passes selectedSource through to LiveOrPollingVideo and ExportScreenshot, and uses pickImageForSource in the Add current image to dataset handler so saved frames also match the selected source.

Why?

Why filter client-side when the proto already supports filter_source_names?

The server-side filter is honored by some camera implementations (e.g., the fake file source) and ignored by others. The RealSense module on the vino-1 left-cam returns every source regardless of the filter, so the existing data.images[0] indexing always rendered the same frame even though the query key changed. Picking the matching sourceName on the client makes the widget work with both kinds of implementations and is a no-op when the server filters.

Why decode image/vnd.viam.dep ourselves?

image/vnd.viam.dep is a Viam-specific binary format (defined in rdk/rimage/depth_map_raw.go). Browsers have no codec for it, so setting img.src to a blob with that MIME type silently fails to load and the canvas never updates, which is why selecting depth looked frozen and the FPS counter stopped advancing. Decoding to an ImageData and writing straight to the existing canvas reuses the same captureStream plumbing as JPEG frames, so the live feed, FPS counter, and picture-in-picture all keep working with no special-casing downstream.

Why match RDK's hue sweep instead of grayscale?

The initial grayscale visualization appeared almost entirely black because depth values in a typical room span a narrow range within the uint16 range, and grayscale uses only one channel. RDK's ToPrettyPicture maps (depth - min) / (max - min) to hue 30° to 230° at full saturation and value, which spreads the same range across all three channels and matches what other Viam tooling shows for depth.

Testing

Ran pnpm check, pnpm lint, and pnpm test. Added two unit test files:

pick-image-for-source.spec.ts: covers matching by source name, fallback to first image, empty input, and missing source name.
decode-viam-depth.spec.ts: covers header parsing, warm/cool hue mapping, opaque-black handling of zero-depth pixels, and rejection of truncated buffers.

Verified visually against vino-1's left-cam and right-cam in both color and depth mode that switching the dropdown updates the feed at the polling rate.

changeset-bot · 2026-05-19T18:43:07Z

🦋 Changeset detected

Latest commit: 5b30904

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@viamrobotics/test-widgets	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

- export-screenshot: decode depth frames to PNG before download so the exported file is viewable; derive extension from MIME type rather than hardcoding .jpeg; move filename construction after source resolution - decode-viam-depth.spec.ts: replace optional chaining on received values with explicit toBeDefined() guards; add all-zero frame edge case covering the Number.isFinite(min) fallback path - live-or-polling-video: surface decodeViamDepth failure to lastError instead of silently freezing the feed Co-authored-by: Devin T. Currie <DTCurrie@users.noreply.github.com>

Co-authored-by: Devin T. Currie <DTCurrie@users.noreply.github.com>

mattmacf98

LGTM with one nit: for containing more of the depth decoding only in decode-viam-depth

Consolidates depth-to-PNG-blob conversion into getBlobForViamDepth in decode-viam-depth.ts. Both live-or-polling-video.svelte and export-screenshot.svelte now use it instead of duplicating the decode → canvas → toBlob pipeline inline. Also adds the DEPTHMAP magic-byte validation to decodeViamDepth, fixing the broken placement introduced in the previous commit. Co-authored-by: Devin T. Currie <DTCurrie@users.noreply.github.com>

…nto app-16408-handle-depth-cam

…obotics/test-widgets into app-16408-handle-depth-cam

handle depth cam

ea7611d

DTCurrie requested review from mattmacf98 and micheal-parks May 19, 2026 18:42

This comment was marked as outdated.

Sign in to view