[feature request] native frame-at-timestamp export (no external ffmpeg dependency for downstream tools)

[kylebrodeur]

Problem

cap export already renders a full video natively, but there's no way to pull
a single still frame at a given timestamp (or read a segment's exact duration)
without shelling out to a separately-installed ffmpeg/ffprobe. Any tool
built on top of Cap that needs still frames — thumbnailing, a step-by-step
doc generator, a diffing/QA tool — ends up carrying that whole external
dependency itself, with all its fragility: PATH lookup, version drift,
platform-specific installs, and (what actually happened to us) a Homebrew
library-version mismatch silently breaking frame extraction with no clear
error.

Concrete case

cap-tools (a Python CLI built on Cap) has a capt guide command that
extracts a screenshot per detected click from a .cap recording's
display.mp4. It shelled out to ffmpeg -ss <t> -i display.mp4 -vframes 1
per click and ffprobe for segment duration. On 2026-07-31, a routine
brew upgrade on a dev machine left the installed ffmpeg linked against a
now-missing libSvtAv1Enc.3.dylib, and every call silently produced zero
frames — the tool's own error handling didn't even catch it as an ffmpeg
problem, since subprocess.run still spawns the process, it just exits
non-zero. The realistic fix for us was vendoring
PyAV (FFmpeg statically compiled into the Python
wheel) to remove the external dependency entirely — but that's real
duplicated effort (and duplicated FFmpeg-linking risk) that every Cap-based
tool has to solve independently, when Cap's own CLI already has a native,
often hardware-accelerated decode path (cap export --force-ffmpeg-decoder
exists specifically as an opt-out of the platform decoder, implying the
default path isn't shelling out to system ffmpeg at all).

Proposal

Add a way to pull one or more still frames — and/or just a duration query —
through the same native pipeline cap export already uses, so no consumer
of Cap recordings needs its own vendored or system video toolchain just to
get a thumbnail.

cap export <project.cap> --frames-at 3.5,7.2,12.0 --out-dir frames/ --json
cap project duration <project.cap> --json   # or fold into `project validate`'s existing output

• Frame output could reuse --format's existing container knowledge (jpg/png)
  and --resolution for downscaling — same flags as cap export already has.
• Duration is probably the cheaper win alone: cap project validate already
  reads the project; adding a durationSeconds field to its JSON output would
  let tools drop ffprobe even without full frame-export support.

Why now

Cap's CLI surface is clearly meant to be the foundation other tools build on
(cap agents install, the MCP server, the whole "designed to be driven by
automation and AI agents" framing in cap --help). Every downstream tool
that wants a still frame currently has to solve the exact fragility we just
hit, independently, on their own.

Status

No PR yet — wanted to check appetite and naming/shape preference first.
Happy to prototype against crates/export if this is a direction you'd take
a contribution on.

[linear]

CAP-771

[tembo]

Good write-up. A chunk of what you're asking for already exists in the CLI today, which is worth knowing before shaping the PR — the real gaps are CLI surface and discoverability, not a missing native capability.

Frame-at-timestamp already works: cap export-preview

There's already a top-level cap export-preview command that renders a single still frame at a given timestamp through the exact same native pipeline as cap export — same ExporterBase builder, same decoders, same FrameRenderer:

• CLI args: ExportPreview in apps/cli/src/export.rs — --frame-time <f64>, --settings-json, --force-ffmpeg-decoder
• Impl: render_preview in crates/export/src/preview.rs — builds ExporterBase, calls segment_media.decoders.get_frames(segment_time, ...), renders via FrameRenderer::render_immediate, encodes to JPEG.

So today a downstream tool can already pull a thumbnail with zero external ffmpeg by running cap export-preview foo.cap --frame-time 3.5 --settings-json '{"fps":60,"resolution_base":{"x":1920,"y":1080},"compression_bpp":0.15}' and base64-decoding the jpeg_base64 field from the JSON result (ExportPreviewResult).

What's missing vs your proposal, and would make a clean PR:

• It returns base64 JPEG on stdout rather than writing image files to --out-dir. Adding a file-output mode (reusing --format's jpg/png knowledge + --resolution) is a thin layer on top of render_preview.
• It's one frame per invocation — no --frames-at a,b,c batch. Note render_preview rebuilds ExporterBase (the heavy part — wgpu device/adapter setup + decoder spawn) per call, so a multi-frame mode should build once and loop render_preview_with_base N times to avoid re-paying setup per timestamp.
• It requires --settings-json instead of the friendly --format/--resolution/--quality flags cap export already exposes — settings_from_flags in export.rs is the pattern to copy.
• It's not in cap guide --json (the agent capability manifest). The guide lists export but omits export-preview, so it's effectively undiscoverable for agents — fixing that is a one-liner in guide.rs and probably the highest-value piece on its own.

Duration is already computed natively (in-process, no ffprobe)

cap_rendering::get_duration (crates/rendering/src/lib.rs:1844) already computes project duration from ProjectRecordingsMeta::duration() + get_source_duration(). The source duration is read via Video::new using ffmpeg::format::input — i.e. the Rust FFmpeg library bindings, not a system ffprobe subprocess. ExportPreviewResult.total_frames is already derived from it (preview.rs:245).

So your "cheaper win alone" instinct is right and very small: cap project validate (project.rs:275) and cap project inspect (project.rs:52) already load RecordingMeta + ProjectConfiguration — they'd just need to build ProjectRecordingsMeta from the studio meta and call get_duration to emit a durationSeconds field in the existing ValidationReport / ProjectInspection JSON. That alone lets every downstream tool drop ffprobe without touching frame export.

One clarification on --force-ffmpeg-decoder

Your read is correct, and worth extending: that flag selects the in-process FFmpeg software decoder vs the platform hardware decoder (AVAssetReader on macOS / MediaFoundation on Windows), with automatic fallback to FFmpeg if the platform decoder fails (decoder/mod.rs:628). Both paths go through cap_video_decode — neither shells out to a separately-installed system ffmpeg/ffprobe binary. Cap links FFmpeg as a Rust library. So the "remove the external dependency" property you want already holds for any consumer that uses Cap's pipeline; the fragility you hit is purely from going around Cap to call system ffmpeg directly, which this feature would let tools stop doing.

Suggested scope split

Given the above, this splits cleanly into two independent PRs:

1. Duration — add durationSeconds to project validate/inspect JSON + document export-preview in cap guide. Tiny, high leverage, unblocks the ffprobe drop immediately.
2. Frame files — add --frames-at/--out-dir/--format to a frame-export mode reusing render_preview with a shared ExporterBase. Bigger, but crates/export is exactly the right place as you suggested.

Happy to help review a prototype against crates/export if you go ahead.

[kylebrodeur]

Thanks, this resolved what I needed.

I integrated cap export-preview directly into capt-tools as the primary path for frame extraction, decoding jpeg_base64 straight to a file. Kept my old PyAV extraction as an automatic fallback for machines without Cap installed. Tested live against a real recording, both with and without cap on PATH, correct resolution either way.

Turns out I didn't need the duration/ffprobe fix myself. PyAV already gives me that with no ffmpeg dependency, so it was never actually blocking me. Still think it'd be worth having durationSeconds built into Cap natively though, so other tools don't have to reach for their own workaround.

The batch/file-output mode is worth doing too, along with the guide manifest listing. I'd consider picking up either of these at some point.