feat: strict TypeScript conversion + render/proxy pipeline fixes#8
Open
amitayks wants to merge 18 commits intotadaspetra:mainfrom
Open
feat: strict TypeScript conversion + render/proxy pipeline fixes#8amitayks wants to merge 18 commits intotadaspetra:mainfrom
amitayks wants to merge 18 commits intotadaspetra:mainfrom
Conversation
… PIP scale
Reel Mode (9:16 Vertical Output)
---------------------------------
- Add `outputMode` project setting ('landscape' | 'reel') with persistence
- Add `normalizeOutputMode()`, `normalizeReelCropX()` to domain model
- Add `resolveOutputSize()` reel branch returning 9:16 dimensions
(e.g. 608x1080 for 1080p source)
- Add `reelCropX` per-keyframe property (-1..+1) controlling horizontal
crop position within the 16:9 source frame
- Add crop overlay in editor preview: semi-transparent dark regions outside
the crop area with dashed white boundary
- Add draggable crop region on canvas for repositioning reelCropX
- Add crop preset buttons (left/center/right) for quick positioning
- Add 16:9 / 9:16 toggle button group in editor controls
- Build animated crop transitions in ffmpeg using `buildNumericExpr()`
for smooth 0.3s interpolation between sections
Zoom-Out with Blur Background (Reel Mode)
------------------------------------------
- Allow `backgroundZoom` range 0.5–3.0 in reel mode (was 1.0–3.0)
- Add `MIN_REEL_BACKGROUND_ZOOM` constant and reel-aware
`normalizeBackgroundZoom(value, outputMode)`
- When zoom < 1.0 in reel mode, render a darkened background
(colorlevels 20% brightness) behind the zoomed-out content
- Static zoom-out: uniform scale + centered overlay on darkened bg
- Animated zoom crossing 1.0 boundary: split pipeline with zoompan
(clamped to max(1, zoom)) + dynamic scale(eval=frame) for the
sub-1.0 portion, overlaid on darkened background
- Clamp zoom values to 1.0 when switching back to landscape mode
Per-Section PIP Scale
---------------------
- Add `pipScale` per-keyframe property (0.15–0.50, default 0.22)
- Add `normalizePipScale()` to domain model and keyframe normalization
- PIP size slider now controls current section's anchor `pipScale`
instead of a global project setting
- Compute PIP pixel size as `round(effectiveCanvasW * pipScale)`
- Re-snap PIP position to nearest corner when scale changes
- Add smooth 0.3s animated PIP size transitions between sections
- Static pipScale: single fixed scale in ffmpeg (no expression overhead)
- Animated pipScale: two-stage ffmpeg pipeline —
1) scale to max pip size (fixed) for format/geq round corners
2) animated scale(eval=frame) after geq for actual size
3) overlay with eval=frame to handle variable-size input
- Add `pipScale` to section operations: split, apply-to-future,
sync anchors, render keyframes, render sections, project snapshot
Bug Fixes
---------
- Fix `resolveOutputSize()` to accept `sourceHeight` parameter
(was ignored, causing incorrect output dimensions)
- Fix hardcoded 1920x1080 camera black fallback in render-service
to use correct dimensions per output mode
- Fix overlay filter missing `eval=frame` flag, which caused PIP
position to freeze at first frame when using animated expressions
- Fix `snapToNearestCorner()` to accept effective canvas dimensions
and pip size parameters instead of using hardcoded globals
Tests
-----
- Add unit tests for `normalizeReelCropX`, `normalizeOutputMode`,
`normalizePipScale`, and keyframe normalization with new properties
- Add unit tests for `resolveOutputSize` in reel and landscape modes
- Add unit tests for `buildScreenFilter` with reel crop (static and
animated), zoom-out blur background (static and animated)
- Add unit tests for `buildFilterComplex` with reel output dimensions,
static pipScale, animated pipScale (two-stage scale), and defaults
- Add unit tests for `normalizeSectionInput` with reelCropX and pipScale
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add openspec specs (reel-mode, pip-overlay, reel-zoom-out) and archived change artifacts documenting the feature design process - Add .DS_Store and .agents/ to gitignore Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add opsx slash commands for managing OpenSpec changes: apply, archive, bulk-archive, continue, explore, ff, new, onboard, sync, verify Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevent the 9:16 crop region from capturing black side bars when screenFitMode is 'fit'. Crop boundaries now use the actual video content width instead of CANVAS_W, in both editor preview and FFmpeg render pipelines. Zoom-out constraints also stack correctly with fit-mode content width. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each keyframe now stores savedLandscape/savedReel slots so zoom, PIP position, pan, and crop settings are independently preserved per mode. First-time mode entry uses defaults. Undo/redo includes outputMode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add heart toggle to save/unsave timeline sections in the sidebar. Saved sections survive deletion (shown grayed out with re-add button), and unreferenced take files are staged to .deleted/ for cleanup. Includes undo/redo support, cleanup on project open/switch/exit, and a Save & Clean button on the home screen. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three regressions from the landscape fit-scaling fix: 1. Reel render crash: baseFilter applied landscape fit scaling in reel mode, shrinking frame to landscapeH. Reel crop (at sourceHeight) then exceeded frame height. Fix: skip landscape scaling in reel mode. 2. Zoom-out blur visible at zoom=1: zoom-out pipeline used landscapeW/H for zoompan output and scale, but screen_base was at raw source dimensions in reel+preprocessed. Fix: introduce baseW/baseH matching actual screen_base dimensions. 3. Per-mode visual state lost: syncSectionAnchorKeyframes dropped savedReel/savedLandscape when rebuilding anchors. Also, normalizeKeyframes and enterEditor clamped zoom-out values to landscape minimum (1.0) before knowing output mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 4-corner snap with 9-point grid (tl, tc, tr, ml, center, mr, bl, bc, br). PIP snap point is saved per-section and per-mode (reel/landscape), so each layout remembers its camera position. Resize no longer re-snaps to a different point — it recalculates position for the current snap point at the new size. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major feature: users can drop external images and videos onto the editor
canvas to overlay them on top of screen recordings at specific time
ranges, positioned and sized freely, with smooth fade transitions.
## Media Overlay Data Model
- Overlay segment structure: id, mediaPath, mediaType (image/video),
startTime/endTime, sourceStart/sourceEnd, per-mode position/size
(landscape + reel), saved flag for heart/archive
- normalizeOverlays() with validation, no-overlap enforcement, sorting
- normalizeOverlayPosition() for {x, y, width, height} objects
- generateOverlayId() with timestamp+counter uniqueness
- Persistence in project.timeline.overlays + savedOverlays
## File Import & Management
- Drag-and-drop import: drop image/video onto canvas, copies to
overlay-media/ folder with unique timestamped name
- Duplicate detection: compares file content against existing files,
reuses path if identical (no duplicate copies)
- Reference counting: media files staged to .deleted/ only when all
overlay segments referencing them are removed
- IPC channels: importOverlayMedia, stageOverlayFile, unstageOverlayFile
- webUtils.getPathForFile for Electron contextIsolation compatibility
## Timeline UI
- New overlay track row above section track (indigo-tinted)
- Overlay segments rendered as colored bands with type icon + filename
- Selection with mutual exclusion (overlay vs section)
- Trim handles on selected overlay (left/right edge drag)
- Split at playhead (same media, divided sourceStart/sourceEnd)
- Drag-to-reposition: ghost band floats during drag, collision
resolution on drop (pushes existing overlays)
- Delete with saved/archive support (heart toggle pattern)
## Canvas Drawing & Interaction
- Overlay drawn between screen recording and PIP camera (z-order)
- Free-placement drag (no snap grid, overflow allowed)
- Corner resize with aspect ratio lock (40px inward hit zones)
- Overflow visualization (out-of-bounds at alpha 0.3)
- Resize cursor on hover (nwse-resize, nesw-resize, move)
- Visual resize handles (14px indigo squares at corners)
- Per-mode positioning: landscape and reel slots set at creation
## Playback & Transitions
- getOverlayStateAtTime() extracted to overlay-utils.js (testable)
- Fade in/out over 0.3s (TRANSITION_DURATION) at segment boundaries
- Position/size interpolation between same-media adjacent segments
(second segment handles transition, avoids double-animation)
- Video overlay sync: single reusable <video> element, synced on
seek and during playback, paused on editor pause
- Image overlay cache: Map<mediaPath, HTMLImageElement>
- Video aspect ratio probed via loadedmetadata for correct proportions
## Render Pipeline (FFmpeg)
- buildOverlayFilter(): composable filter chain fragment for overlays
- Image inputs: -loop 1 -t {duration}, video inputs: trim+setpts
- PTS shift (setpts=PTS+startTime/TB) aligns fade with enable window
- Fade in/out with alpha=1 at correct rendered timeline times
- Time-bounded: enable='between(t,start,end)' on overlay filter
- Z-order fix: screen → overlay media → PIP (not screen → PIP → overlay)
- Reel mode: uses reel position slot, scales to reel output dimensions
## Sidebar Panel
- Tab buttons: "Segments" (section list) and "Overlays" (overlay list)
- Overlay list: sorted by time, type icon, filename, heart toggle
- Heart/save: toggle saved flag, saved overlays survive deletion
(moved to savedOverlays), re-add with [+] button
- savedOverlays persisted through undo/redo, project save/load
## Editor Controls Redesign
- Removed visible buttons (Undo, Redo, Play, Split, Camera, etc.)
— all triggered via keyboard shortcuts
- Photoshop-style scrub controls for Zoom, PIP size, Overlay size
— drag label left/right to adjust, hidden range inputs
- Overlay size control: appears when overlay selected, 5%-500% range,
maintains aspect ratio, centers resize
- Updated keyboard hints in UI
## Keyboard Navigation
- Cmd+Arrow Left/Right: ±5 frames (fast seek)
- Arrow Up: jump to next section/overlay boundary
- Arrow Down: jump to previous section/overlay boundary
- getTimelineBoundaries() collects all section + overlay start/end times
## Playback Improvements
- Auto-loop: playback loops to start when reaching timeline end
- Space bar at end: seeks to 0 before playing (no stuck state)
- Overlay video paused in editorPause()
- Overlay state cleanup in clearEditorState()
## Tests Added (36 new)
- 12 unit tests: normalizeOverlays, normalizeOverlayPosition,
generateOverlayId, project persistence
- 11 unit tests: getOverlayStateAtTime (fade, interpolation, modes)
- 6 unit tests: buildOverlayFilter (image, video, multiple, reel,
same-media fade suppression)
- 2 integration tests: duplicate file detection
- 4 integration tests: overlay file import, staging, unstaging, cleanup
- 1 integration test: render pipeline with overlay
## OpenSpec
- 6 new specs synced to main (canvas, data, files, playback, render,
timeline)
- take-file-cleanup spec updated with overlay staging scenarios
- Change archived: 2026-03-21-media-overlay-track
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nder Record mouse cursor position during screen capture (10Hz, main-process timer with zero renderer IPC overhead), then use that trail data to automatically pan the zoomed viewport to follow the cursor during playback and render. Key changes: - Mouse trail utilities (lookupSmoothedMouseAt, subsampleTrail) with EMA smoothing and binary-search interpolation - IPC: start/stop mouse trail capture in main process, save trail JSON - Data model: autoTrack + autoTrackSmoothing on keyframes, per-mode, persisted via normalizeKeyframes - Editor preview: getStateAtTime overrides focusX/focusY from trail when autoTrack enabled and zoomed - Render pipeline: expandAutoTrackKeyframes generates synthetic keyframes from subsampled trail, feeding animated zoompan expressions - UI: Track toggle + Smooth scrub controls (visible when zoomed + trail exists), manual pan disabled when tracking - Fix: mousePath now saved as relative path in project JSON (consistent with screenPath/cameraPath) - Fix: static zoom (single keyframe) uses crop+scale instead of zoompan to avoid render hangs Tests: 191 passing (mouse-trail unit tests, render integration test for auto-track zoompan expressions, project normalization tests) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The media-stream-cleanup openspec change will be implemented and committed separately from the mouse-auto-track feature. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Release all media resources (ScreenCaptureKit sessions, camera streams, AudioContext, MediaRecorder, WebSocket) on window close and app quit to prevent stale capture session accumulation across dev restarts. Additionally, streams are lazily cleaned after 30s idle when the user leaves the recording view — re-acquired automatically on return. - Add cleanupAllMedia() in renderer with idempotent, null-safe guards - Wire beforeunload handler to release all streams on window close - Add before-quit handler in main process for mouse trail timer cleanup - Add 30s idle timer in setWorkspaceView for lazy stream release - Add 22 unit tests for cleanup functions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generalize the overlay system from a single no-overlap track to N tracks (starting with 2) using a trackIndex field on overlay segments. Overlays on different tracks can overlap in time, enabling two media items visible simultaneously with z-order: screen → track 0 → track 1 → PIP. - Add trackIndex field with per-track no-overlap enforcement in normalizeOverlays - Two overlay track rows in timeline UI with cross-track drag-to-move - Per-track canvas drawing, hit-testing (higher track wins), and video sync - Two reusable <video> elements for simultaneous video overlay playback - Track-ordered ffmpeg compositing in render pipeline - Fix pre-existing render position interpolation timing bug (use absolute times) - Skip fade-in/fade-out at video start/end boundaries - Backward compatible: missing trackIndex defaults to 0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Media streams (screen, camera, audio) are no longer acquired when opening a project to the timeline — only when the recording view is entered. Adds defensive cleanup on both renderer and main process startup to release any lingering resources from a previous forced close. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Recordings over ~1 minute caused noticeable lag during editor playback and scrubbing because the editor played back raw VP9/VP8 .webm files directly. VP9 has sparse keyframes (every 2-5s), making every seek decode potentially dozens of frames, and software-decoding 1920×1080 VP9 while running canvas compositing saturated the renderer thread. This change introduces per-take proxy MP4 generation — a lightweight H264 transcode of each screen recording that the editor uses for display, while exports continue to use the original full-quality source. Proxy generation: - New proxy-service.js: builds and runs ffmpeg to transcode .webm → .mp4 (960×540, H264, CRF 23, ultrafast preset, 2 threads, keyframe every 0.5s, AAC 64k, +faststart for instant browser load) - Concurrency queue (max 2 simultaneous ffmpeg jobs) prevents CPU overload - Writes to .tmp path first, renames on success, deletes .tmp on failure - Triggered automatically after each recording stops (fire-and-forget) - On project open, queues generation for any takes missing a proxy (backward-compatible with existing projects) IPC and data model: - New proxy:generate IPC handler with progress events (started/progress/done/error) - proxyPath field added to take data model (persisted in project.json as relative path, resolved to absolute on load — same pattern as screenPath) - proxyPath included in file staging/unstaging for take cleanup lifecycle Editor integration: - getOrCreateTakeVideos() uses proxy when available, falls back to original - Hot-swap: when proxy finishes, the cached video element's src is updated in-place, preserving playback state and restarting the draw loop - Source resolution probe: a throwaway video element reads dimensions from the original .webm (not the proxy) so exports render at full resolution - Timeline progress indicator: animated amber bar on section bands shows real-time proxy generation progress (percent-width, CSS transition) - Draw loop safety: 200ms fallback timer prevents requestVideoFrameCallback stalls during src swaps or section transitions Export is unaffected — render-service.js always reads from take.screenPath (the original .webm), never from proxyPath. Tests: 243 pass, proxy-service.js at 100% statement coverage. New tests cover proxy generation (happy path, failure cleanup, concurrency limit), IPC handler (success/error/destroyed-sender), proxyPath persistence round-trip, and file staging with proxy files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…xy change Export quality improvements: - Add -g (keyframe interval) to export output args, set to targetFps * 2 (keyframe every 2s). Keeps text crisp throughout the video instead of periodic blur from sparse default keyframes (~8s apart). - Replace linear interpolation with easeInOut curve in all ffmpeg filter transition expressions (buildNumericExpr, buildPosExpr, buildAlphaExpr, buildCamFullAlphaExpr). Export PIP movement, zoom, fade, and fullscreen transitions now match the editor preview exactly. OpenSpec maintenance: - Archive editor-proxy-generation change (all 21 tasks complete) - Sync delta specs to main: new take-proxy-files spec, updated take-file-cleanup spec with proxyPath staging/unstaging scenarios Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Convert all 24 production files and 22 test files from JavaScript to TypeScript with strict: true, noUncheckedIndexedAccess, and project references enforcing module boundaries. - Add tsc --build pipeline compiling src/ to dist/ - Create 4 tsconfig project refs (shared, main, preload, renderer) - Define 15+ domain interfaces, 22 service types, typed IPC contract - All 243 tests pass, lint clean, e2e and packaging smoke verified - Zero .js source files remain in src/ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full strict TypeScript conversion of all 24 production files and 22 test files, plus render pipeline bug fixes discovered during testing.
TypeScript Conversion
noUncheckedIndexedAccess— noanyescape hatches, includingrenderer/app.tsshared,main,preload,renderer) enforcing module boundaries at compile timeElectronAPIIPC interface, domain types (Project,Section,Keyframe,Overlay), service interfaces.js/.mjsto unified.test.tsRender Pipeline Fixes
-r 30on input → proper 30fps, ~113MBfpsfilter reprocessed the already-30fps composited stream after the overlay step, dropping camera frames. Removed the double filtering → camera smooth againSmall Improvements
Note
This branch includes all prior feature work (overlays, reel mode, auto-track, etc.) interleaved with the TS conversion — can't be separated into individual PRs at this point. See comment on #3 for context.
Test plan
npm run checkpasses (lint, typecheck, 243 unit tests, e2e, packaging)🤖 Generated with Claude Code