feat: reel mode (9:16), zoom-out blur bg, per-section PIP scale#3
feat: reel mode (9:16), zoom-out blur bg, per-section PIP scale#3amitayks wants to merge 15 commits intotadaspetra:mainfrom
Conversation
… PIP scale
Reel Mode (9:16 Vertical Output)
---------------------------------
- Add `outputMode` project setting ('landscape' | 'reel') with persistence
- Add `normalizeOutputMode()`, `normalizeReelCropX()` to domain model
- Add `resolveOutputSize()` reel branch returning 9:16 dimensions
(e.g. 608x1080 for 1080p source)
- Add `reelCropX` per-keyframe property (-1..+1) controlling horizontal
crop position within the 16:9 source frame
- Add crop overlay in editor preview: semi-transparent dark regions outside
the crop area with dashed white boundary
- Add draggable crop region on canvas for repositioning reelCropX
- Add crop preset buttons (left/center/right) for quick positioning
- Add 16:9 / 9:16 toggle button group in editor controls
- Build animated crop transitions in ffmpeg using `buildNumericExpr()`
for smooth 0.3s interpolation between sections
Zoom-Out with Blur Background (Reel Mode)
------------------------------------------
- Allow `backgroundZoom` range 0.5–3.0 in reel mode (was 1.0–3.0)
- Add `MIN_REEL_BACKGROUND_ZOOM` constant and reel-aware
`normalizeBackgroundZoom(value, outputMode)`
- When zoom < 1.0 in reel mode, render a darkened background
(colorlevels 20% brightness) behind the zoomed-out content
- Static zoom-out: uniform scale + centered overlay on darkened bg
- Animated zoom crossing 1.0 boundary: split pipeline with zoompan
(clamped to max(1, zoom)) + dynamic scale(eval=frame) for the
sub-1.0 portion, overlaid on darkened background
- Clamp zoom values to 1.0 when switching back to landscape mode
Per-Section PIP Scale
---------------------
- Add `pipScale` per-keyframe property (0.15–0.50, default 0.22)
- Add `normalizePipScale()` to domain model and keyframe normalization
- PIP size slider now controls current section's anchor `pipScale`
instead of a global project setting
- Compute PIP pixel size as `round(effectiveCanvasW * pipScale)`
- Re-snap PIP position to nearest corner when scale changes
- Add smooth 0.3s animated PIP size transitions between sections
- Static pipScale: single fixed scale in ffmpeg (no expression overhead)
- Animated pipScale: two-stage ffmpeg pipeline —
1) scale to max pip size (fixed) for format/geq round corners
2) animated scale(eval=frame) after geq for actual size
3) overlay with eval=frame to handle variable-size input
- Add `pipScale` to section operations: split, apply-to-future,
sync anchors, render keyframes, render sections, project snapshot
Bug Fixes
---------
- Fix `resolveOutputSize()` to accept `sourceHeight` parameter
(was ignored, causing incorrect output dimensions)
- Fix hardcoded 1920x1080 camera black fallback in render-service
to use correct dimensions per output mode
- Fix overlay filter missing `eval=frame` flag, which caused PIP
position to freeze at first frame when using animated expressions
- Fix `snapToNearestCorner()` to accept effective canvas dimensions
and pip size parameters instead of using hardcoded globals
Tests
-----
- Add unit tests for `normalizeReelCropX`, `normalizeOutputMode`,
`normalizePipScale`, and keyframe normalization with new properties
- Add unit tests for `resolveOutputSize` in reel and landscape modes
- Add unit tests for `buildScreenFilter` with reel crop (static and
animated), zoom-out blur background (static and animated)
- Add unit tests for `buildFilterComplex` with reel output dimensions,
static pipScale, animated pipScale (two-stage scale), and defaults
- Add unit tests for `normalizeSectionInput` with reelCropX and pipScale
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add openspec specs (reel-mode, pip-overlay, reel-zoom-out) and archived change artifacts documenting the feature design process - Add .DS_Store and .agents/ to gitignore Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add opsx slash commands for managing OpenSpec changes: apply, archive, bulk-archive, continue, explore, ff, new, onboard, sync, verify Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevent the 9:16 crop region from capturing black side bars when screenFitMode is 'fit'. Crop boundaries now use the actual video content width instead of CANVAS_W, in both editor preview and FFmpeg render pipelines. Zoom-out constraints also stack correctly with fit-mode content width. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each keyframe now stores savedLandscape/savedReel slots so zoom, PIP position, pan, and crop settings are independently preserved per mode. First-time mode entry uses defaults. Undo/redo includes outputMode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add heart toggle to save/unsave timeline sections in the sidebar. Saved sections survive deletion (shown grayed out with re-add button), and unreferenced take files are staged to .deleted/ for cleanup. Includes undo/redo support, cleanup on project open/switch/exit, and a Save & Clean button on the home screen. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three regressions from the landscape fit-scaling fix: 1. Reel render crash: baseFilter applied landscape fit scaling in reel mode, shrinking frame to landscapeH. Reel crop (at sourceHeight) then exceeded frame height. Fix: skip landscape scaling in reel mode. 2. Zoom-out blur visible at zoom=1: zoom-out pipeline used landscapeW/H for zoompan output and scale, but screen_base was at raw source dimensions in reel+preprocessed. Fix: introduce baseW/baseH matching actual screen_base dimensions. 3. Per-mode visual state lost: syncSectionAnchorKeyframes dropped savedReel/savedLandscape when rebuilding anchors. Also, normalizeKeyframes and enterEditor clamped zoom-out values to landscape minimum (1.0) before knowing output mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 4-corner snap with 9-point grid (tl, tc, tr, ml, center, mr, bl, bc, br). PIP snap point is saved per-section and per-mode (reel/landscape), so each layout remembers its camera position. Resize no longer re-snaps to a different point — it recalculates position for the current snap point at the new size. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major feature: users can drop external images and videos onto the editor
canvas to overlay them on top of screen recordings at specific time
ranges, positioned and sized freely, with smooth fade transitions.
## Media Overlay Data Model
- Overlay segment structure: id, mediaPath, mediaType (image/video),
startTime/endTime, sourceStart/sourceEnd, per-mode position/size
(landscape + reel), saved flag for heart/archive
- normalizeOverlays() with validation, no-overlap enforcement, sorting
- normalizeOverlayPosition() for {x, y, width, height} objects
- generateOverlayId() with timestamp+counter uniqueness
- Persistence in project.timeline.overlays + savedOverlays
## File Import & Management
- Drag-and-drop import: drop image/video onto canvas, copies to
overlay-media/ folder with unique timestamped name
- Duplicate detection: compares file content against existing files,
reuses path if identical (no duplicate copies)
- Reference counting: media files staged to .deleted/ only when all
overlay segments referencing them are removed
- IPC channels: importOverlayMedia, stageOverlayFile, unstageOverlayFile
- webUtils.getPathForFile for Electron contextIsolation compatibility
## Timeline UI
- New overlay track row above section track (indigo-tinted)
- Overlay segments rendered as colored bands with type icon + filename
- Selection with mutual exclusion (overlay vs section)
- Trim handles on selected overlay (left/right edge drag)
- Split at playhead (same media, divided sourceStart/sourceEnd)
- Drag-to-reposition: ghost band floats during drag, collision
resolution on drop (pushes existing overlays)
- Delete with saved/archive support (heart toggle pattern)
## Canvas Drawing & Interaction
- Overlay drawn between screen recording and PIP camera (z-order)
- Free-placement drag (no snap grid, overflow allowed)
- Corner resize with aspect ratio lock (40px inward hit zones)
- Overflow visualization (out-of-bounds at alpha 0.3)
- Resize cursor on hover (nwse-resize, nesw-resize, move)
- Visual resize handles (14px indigo squares at corners)
- Per-mode positioning: landscape and reel slots set at creation
## Playback & Transitions
- getOverlayStateAtTime() extracted to overlay-utils.js (testable)
- Fade in/out over 0.3s (TRANSITION_DURATION) at segment boundaries
- Position/size interpolation between same-media adjacent segments
(second segment handles transition, avoids double-animation)
- Video overlay sync: single reusable <video> element, synced on
seek and during playback, paused on editor pause
- Image overlay cache: Map<mediaPath, HTMLImageElement>
- Video aspect ratio probed via loadedmetadata for correct proportions
## Render Pipeline (FFmpeg)
- buildOverlayFilter(): composable filter chain fragment for overlays
- Image inputs: -loop 1 -t {duration}, video inputs: trim+setpts
- PTS shift (setpts=PTS+startTime/TB) aligns fade with enable window
- Fade in/out with alpha=1 at correct rendered timeline times
- Time-bounded: enable='between(t,start,end)' on overlay filter
- Z-order fix: screen → overlay media → PIP (not screen → PIP → overlay)
- Reel mode: uses reel position slot, scales to reel output dimensions
## Sidebar Panel
- Tab buttons: "Segments" (section list) and "Overlays" (overlay list)
- Overlay list: sorted by time, type icon, filename, heart toggle
- Heart/save: toggle saved flag, saved overlays survive deletion
(moved to savedOverlays), re-add with [+] button
- savedOverlays persisted through undo/redo, project save/load
## Editor Controls Redesign
- Removed visible buttons (Undo, Redo, Play, Split, Camera, etc.)
— all triggered via keyboard shortcuts
- Photoshop-style scrub controls for Zoom, PIP size, Overlay size
— drag label left/right to adjust, hidden range inputs
- Overlay size control: appears when overlay selected, 5%-500% range,
maintains aspect ratio, centers resize
- Updated keyboard hints in UI
## Keyboard Navigation
- Cmd+Arrow Left/Right: ±5 frames (fast seek)
- Arrow Up: jump to next section/overlay boundary
- Arrow Down: jump to previous section/overlay boundary
- getTimelineBoundaries() collects all section + overlay start/end times
## Playback Improvements
- Auto-loop: playback loops to start when reaching timeline end
- Space bar at end: seeks to 0 before playing (no stuck state)
- Overlay video paused in editorPause()
- Overlay state cleanup in clearEditorState()
## Tests Added (36 new)
- 12 unit tests: normalizeOverlays, normalizeOverlayPosition,
generateOverlayId, project persistence
- 11 unit tests: getOverlayStateAtTime (fade, interpolation, modes)
- 6 unit tests: buildOverlayFilter (image, video, multiple, reel,
same-media fade suppression)
- 2 integration tests: duplicate file detection
- 4 integration tests: overlay file import, staging, unstaging, cleanup
- 1 integration test: render pipeline with overlay
## OpenSpec
- 6 new specs synced to main (canvas, data, files, playback, render,
timeline)
- take-file-cleanup spec updated with overlay staging scenarios
- Change archived: 2026-03-21-media-overlay-track
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nder Record mouse cursor position during screen capture (10Hz, main-process timer with zero renderer IPC overhead), then use that trail data to automatically pan the zoomed viewport to follow the cursor during playback and render. Key changes: - Mouse trail utilities (lookupSmoothedMouseAt, subsampleTrail) with EMA smoothing and binary-search interpolation - IPC: start/stop mouse trail capture in main process, save trail JSON - Data model: autoTrack + autoTrackSmoothing on keyframes, per-mode, persisted via normalizeKeyframes - Editor preview: getStateAtTime overrides focusX/focusY from trail when autoTrack enabled and zoomed - Render pipeline: expandAutoTrackKeyframes generates synthetic keyframes from subsampled trail, feeding animated zoompan expressions - UI: Track toggle + Smooth scrub controls (visible when zoomed + trail exists), manual pan disabled when tracking - Fix: mousePath now saved as relative path in project JSON (consistent with screenPath/cameraPath) - Fix: static zoom (single keyframe) uses crop+scale instead of zoompan to avoid render hangs Tests: 191 passing (mouse-trail unit tests, render integration test for auto-track zoompan expressions, project normalization tests) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The media-stream-cleanup openspec change will be implemented and committed separately from the mouse-auto-track feature. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Release all media resources (ScreenCaptureKit sessions, camera streams, AudioContext, MediaRecorder, WebSocket) on window close and app quit to prevent stale capture session accumulation across dev restarts. Additionally, streams are lazily cleaned after 30s idle when the user leaves the recording view — re-acquired automatically on return. - Add cleanupAllMedia() in renderer with idempotent, null-safe guards - Wire beforeunload handler to release all streams on window close - Add before-quit handler in main process for mouse trail timer cleanup - Add 30s idle timer in setWorkspaceView for lazy stream release - Add 22 unit tests for cleanup functions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generalize the overlay system from a single no-overlap track to N tracks (starting with 2) using a trackIndex field on overlay segments. Overlays on different tracks can overlap in time, enabling two media items visible simultaneously with z-order: screen → track 0 → track 1 → PIP. - Add trackIndex field with per-track no-overlap enforcement in normalizeOverlays - Two overlay track rows in timeline UI with cross-track drag-to-move - Per-track canvas drawing, hit-testing (higher track wins), and video sync - Two reusable <video> elements for simultaneous video overlay playback - Track-ordered ffmpeg compositing in render pipeline - Fix pre-existing render position interpolation timing bug (use absolute times) - Skip fade-in/fade-out at video start/end boundaries - Backward compatible: missing trackIndex defaults to 0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Media streams (screen, camera, audio) are no longer acquired when opening a project to the timeline — only when the recording view is entered. Adds defensive cleanup on both renderer and main process startup to release any lingering resources from a previous forced close. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Hey, sorry just saw this! I appreciate this PR but unfortunately right now I am fully focused on stability for longer recording sessions. I don't want to add any more features until this is solved. However, I do think most of the features you have here will eventually make it into the project. If you need them more urgently I recommend you fork the repo. I will close this PR because it has a ton of merge conflicts from the stability changes I made, but I will definitely keep the issues open for whenever it's a good time to add them! |
|
Totally understand, stability first makes sense! No worries on closing this. On that note — we actually ran into the same long-recording playback issue (lag + choppy scrubbing for anything over ~1 min). We shipped a fix on our fork that generates lightweight proxy MP4s per take in the background (H264, half-res, frequent keyframes). The editor uses the proxy for display while exports still use the original full-quality source. If you're curious: amitayks/loop@ |
|
@amitayks Im curious to hear about how you worked about this. Do you know a lot about this topic (video rendering/playback etc)? What is your workflow, because I see specs defined? I am learning as I go so curious to hear how you are doing it |
|
Honestly not an expert either — I used to work with video editing so I have some intuition for what feels smooth vs laggy from a user perspective, but the technical side is learning-as-I-go too. My workflow is basically: research the problem space first, then spec out options before writing code. The specs you see are from an OpenSpec-style workflow — I write a proposal (why), design doc (how, with alternatives considered), and task breakdown before implementation. Helps me not go in circles. For this specific problem we explored three options:
We went with option 1 because you're constantly scrubbing/editing, and waiting for a re-render after each edit (option 2) would kill the flow. The proxy approach keeps things smooth at all times with zero UX friction. The core insight was that VP9 WebM has sparse keyframes (every 2-5s) which makes seeking brutal for long recordings — the browser has to decode up to 90 frames per seek. H264 MP4 with keyframes every 0.5s makes seeks nearly instant. Happy to chat more about any of this! |
|
@amitayks That makes sense. I sent the link of the commit you sent to Claude yesterday, and it seemed like it implemented everything nicely into the architecture. I made a lot of changes for long recordings (rendering and what you mentioned) in the last few days. If you have any more improvements related to performance would love a PR related to that :), or even some insights from your own building. |
|
Thanks! Glad it resonated. Here are the performance/stability improvements I've integrated:
I'd gladly open a PR with those, but they're currently interleaved with some feature work I built along the way — things like reel mode (9:16 output), multi-track media overlays, mouse auto-tracking for zoom, and per-section PIP scaling. I think you'd like those too, but understand if you want to keep the scope tight for now. I'll open a PR that includes all the latest improvements with docs about each one. |
|
Maybe you know, but whats the best approach for this in open source? Because I've also added a few features now too.
Those things you added sound nice, but it seems like we diverged a bit, and it would be a complicated merge |
|
Yeah merging diverged branches is always tricky in open source. Honestly I think the best approach here is: I'll convert my fork to TypeScript to match yours, and open a fresh PR from there. That way you get the TS foundation you set up plus the features I've been building. A few things I went deep on that might be worth looking at: Media overlays — two dedicated overlay tracks that support both images and video, with drag-to-resize on the canvas. Feels more flexible than per-section image replacement since overlays can span across sections and stack independently Take file cleanup — Reel mode (9:16) — full portrait output with draggable crop overlay and per-mode state preservation Mouse auto-tracking — captures cursor position during recording and auto-pans the zoomed viewport to follow it Export polish — easeInOut transitions that match the editor preview exactly, plus frequent keyframes so text stays crisp I've been using this daily for my own recordings so these have been tested pretty thoroughly. Happy to do the TS conversion work and put together a clean PR — would that work for you? |
|
Yeah I think that would work. Would you mind doing a PR for each feature separately? Or at least separate commits, so I can test them as well before merging them in? Appreciate you working through this with me :) |
|
Hey! Really enjoying working on this project — your architecture is clean and fun to build on. I've fully converted everything to strict TypeScript, including Along the way I found and fixed a couple of real bugs in the rendering pipeline:
Being honest — this can't be a simple merge. My TS conversion includes all the features I've been building (overlays, reel mode, auto-track, etc.) interleaved throughout, so I can't split them into separate PRs at this point. I opened #8 with everything pushed. I think what would benefit you most is the render/proxy pipeline fixes and the It would essentially replace the whole codebase, so totally understand if that's too big a jump. Either way, love what you're building and happy to keep collaborating! |
2 participants
Summary
Closes #2
reelCropXper-keyframe crop positioning, draggable crop overlay in editor, animated crop transitions in ffmpeg, crop preset buttonspipScaleper-keyframe (0.15–0.50), animated PIP size transitions via two-stage ffmpeg scale pipeline, slider controls current sectionresolveOutputSizesourceHeight, camera black fallback dimensions, overlayeval=framefor animated PIP,snapToNearestCornerparameterized10 files changed, 1031 insertions, 66 deletions. Comprehensive unit tests included.
Test plan
npm run checkpasses (lint, typecheck, unit tests)🤖 Generated with Claude Code
Closes #4
Closes #5
Closes #6
Closes #7