feat: AI-friendly editor API + effects layer#9
Open
ziqiangai wants to merge 12 commits into
Open
Conversation
Adds a second, deterministic-by-default surface on `EditorApi` for
callers without UI state (AI tool-loops, scripts, tests). Existing
positional methods are untouched — legacy code compiles.
Core (@aicut/core):
- New `EditResult<T>` discriminated union — every new mutator returns
`{ ok: true, data }` or `{ ok: false, reason, hint? }`. Reasons are
a typed union (`clip-not-found` | `time-outside-clip` | `overlap` |
`invalid-time` | `invalid-range` | `source-not-found` |
`source-load-failed` | `internal-error`), so AI callers can branch
without guessing.
- `Editor.batch(label, fn)` — wraps `beginInteraction / endInteraction`
with try/finally + async support. N mutations become one undo
entry. `label` reserved for future labeled-history UI.
- `Editor.splitClip({ clipId, timeMs })` — split a specific clip, no
more first-match-wins across other tracks. Fails cleanly outside
the clip's [start, end).
- `Editor.moveClipTo({ clipId, toTrackId?, startMs?, onOverlap })` —
explicit destination. `onOverlap: "error"` (default) refuses when
taken; `"auto"` opts into today's smart-routing.
- `Editor.trimClip({ clipId, edge, timeMs })` — move one edge without
the ripple-close side effects the UI-facing `trimLeft` applies.
- `Editor.deleteClip({ clipId })` — typed removal.
- `Editor.addClip({ sourceUrl | sourceId, trackId, startMs, inMs?,
outMs?, onOverlap?, meta? })` — one-shot: probe the URL for
duration + dimensions (via new `probeMediaSource` helper), register
the source, insert the clip. Async. `meta` accepted but silently
dropped in MVP (Clip.meta field lands in a future release; passing
it now is forward-compatible).
- `Editor.captureFrame({ timeMs, source: 'composite' | 'raw', clipId?,
format?, quality?, maxWidth? })` — grab a still frame as JPEG/PNG
blob. Composite mode reads the compositor canvas after seek + rAF
(requires CanvasCompositorEngine or WebCodecsEngine). Raw mode
spawns a hidden `<video>` seeked to the source-local equivalent
time — engine-independent. Purpose: feed AI vision models without
requiring hosts to plumb the compositor themselves.
- Pure query helpers on Project (in `model.ts`, re-exported from
core index):
- `findClipAt(project, timeMs, trackIndex?)` — first matching clip.
- `getClipsInRange(project, startMs, endMs, trackIndex?)` — all
overlapping clips.
- `getClipsOnTrack(project, trackIndex)` — clips on one track.
- `timelineToSourceMs(clip, timelineMs)` / `sourceToTimelineMs(...)`
— the coordinate math AI would otherwise re-derive per tool call.
- `probeMediaSource(url)` — the metadata probe `addClip` uses
internally, exposed for hosts that want to pre-fetch before
deciding whether to insert.
React (@aicut/react):
- Re-exports the model helpers + probe + `EditResult` / `EditErrorReason`
types so hosts stay on one import.
Demo:
- New `#/api` route: **API playground** page. Card layout — one card
per new method with form inputs bound to its params, Run button,
pretty-printed EditResult log. Runs against a live editor with the
compositor canvas engine (so captureFrame composite mode works).
Purpose: dogfood the ergonomics before wiring AI on top. If a form
feels awkward, a tool spec built from the same shape will feel
worse.
- Router gains "API playground 🧪" tab.
Tests: 125/125 pass across 9 files. New coverage:
editor.test.ts — batch (sync/async/throwing/coalescing) +
splitClip/moveClipTo/trimClip/deleteClip/addClip variants.
model-query.test.ts — findClipAt / getClipsInRange /
getClipsOnTrack / timelineToSourceMs / sourceToTimelineMs.
Not in this commit (post-MVP):
- `validateProject` helper (#169)
- `fillGap` composite (#171)
- `findGapsOnTrack` helper (#172)
- AiCopilot LLM demo (deferred until MVP APIs prove ergonomic in
the playground)
Core: OperationEvent type + 'operation' event on the bus. Every AI-facing mutator (splitClip / moveClipTo / trimClip / deleteClip / addClip) fires exactly one event with kind + args + result + before/after project snapshots + batchId when inside batch(). Nested batch() calls reuse the outermost id. 9 new unit tests (134/134 pass). @aicut/effects (new package, WIP): package skeleton + types + geometry helpers. AiCutEffects component + character SVG + default splitEffect / moveEffect handlers not landed yet — pausing here to hotfix a useEditorState infinite-loop on main.
Ships the effects overlay on top of the operation event bus. Every
`splitClip` / `moveClipTo` on the timeline gets a stick figure
animation that reads as "the little worker did that", verifying the
architecture works end-to-end before the character art gets replaced
with real designs.
@aicut/effects (new package):
- `<AiCutEffects>` — the overlay component. Mount once inside
`<EditorProvider>`. Subscribes to `editor.on("operation", ...)`,
builds an `EffectContext` (timeline / preview rects + coordinate
helpers) at emit time, dispatches to a registered handler for that
op kind, renders whatever JSX the handler returns into a fixed-
position pointer-events:none overlay. Holds one active effect per
kind — a second op of the same kind replaces the first via React
key change. Different kinds run concurrently.
Fully opt-in:
- `enabled={false}` disables the whole layer (no subscription,
no render).
- `effects={{ splitClip: myFn }}` swaps a single handler.
- `effects={{ splitClip: false }}` disables one kind without a
replacement.
- `zIndex` prop controls stacking.
- `useOperationStream(handler)` — low-level hook for hosts who want
Canvas / non-React overlays / telemetry loggers without going
through the JSX handler shape.
- `<StickFigure pose facing scale color />` — minimum-viable SVG
character with 7 poses (idle / walking / cutting / lifting /
carrying / dropping / waving). Intentionally crude so it reads as
"intent placeholder" — hosts will swap in Lottie / Rive / hand-
drawn art. Pose type is a widened string union so custom
character components can implement any subset.
- `defaultSplitEffect` — worker walks in from the right, parks at
the cut point, sawing pose + flash line for a beat, walks off the
left. Total ~1.7s.
- `defaultMoveEffect` — worker enters near the source clip, lifts a
ghost rectangle overhead, walks along the timeline to the
destination (source rect read from `op.beforeProject`, dest from
`op.afterProject`), drops the ghost onto the new position, exits.
Total ~2.1s.
- `buildEffectContext(editor)` — geometry helper. Reads DOM live:
timeline root rect, preview host rect, `timelineToScreenX(ms)` +
`clipToScreenRect(clipId)` that account for header width + scroll
offset + CSS-var track height. Exported so custom effects can reuse
it.
- Trim / delete / add ship without a default (`false` in the built-
in map) — first pass keeps the shipped surface honest with 2 real
animations, more follow when the shape is proven.
Demo:
- `#/api` playground now mounts `<AiCutEffects>` outside the shell
(so its fixed overlay stacks correctly) and gains a
`🎭 Effects: on/off` chip in the header for A/B toggling.
- Fire splitClip → stick figure walks in, saws over the cut column,
walks off. Fire moveClipTo → figure lifts a ghost clip, carries
it to the destination row, drops. Toggle chip disables both.
Everything is decoupled:
- `@aicut/core` never learns about characters or SVG — it just
emits `operation` events with data.
- `@aicut/react` doesn't depend on `@aicut/effects`. A host that
doesn't import `@aicut/effects` gets zero animation code.
- Effect handlers are pure functions of `(op, ctx, onComplete)`
→ JSX — hosts swap any subset without touching the library.
- Character components take a `pose` prop only — swap
`<StickFigure>` for anything with the same prop shape.
Not in this pass:
- trim / delete / add effects (schema in place, just no defaults)
- audio / sound effects
- character art polish (this is a "prove the pipes work" figure)
- AiCopilot demo (planned as follow-up now that operation events
are the natural feedback surface for it)
splitClip: rebuilt as drop → strike → recoil (~800ms). Strike + yellow flash fires ~200ms after mount so the visual peak lands close to the data commit instead of trailing 600ms behind. Cubic-bezier easing per phase for anticipation / snap / follow-through. moveClipTo: rebuilt as spawn → lift → carry → drop → exit (~1.45s). Carry phase follows a parabolic arc scaled by horizontal distance (short moves stay flat, long moves arc high). Walk-cycle pose swap every 140ms plus a small vertical bob reads as steps instead of a slide. Both stay within the pure overlay layer — no core / timeline coupling.
Playwright coverage for @aicut/effects wired into the /#/api playground. Asserts: - overlay mounts at idle (parent listener container present) - splitClip Run → [data-effect-kind="splitClip"] node appears + unmounts - moveClipTo Run → [data-effect-kind="moveClipTo"] node appears + unmounts - Effects toggle off → overlay detaches; further ops render nothing Timing assertions are generous (500ms mount, 1.5s split unmount, 2.5s move unmount) to accommodate RAF-scheduled state transitions without being flaky. Animation quality itself is not asserted here — that's a visual review, not a spec.
The previous StickFigure-based defaults felt stiff for two structural
reasons: (1) pose swaps between rendered SVG frames read as stop-motion
because there's no interpolation between poses, and (2) the causal
timing was inverted — the timeline data mutated at t=0 but the visual
"cut" landed 200-600ms later, so the split appeared before the action.
This rewrite:
- Drops StickFigure entirely.
- Adds a <Bear> character using two AI-generated WebP sticker assets
(chop pose = arms raised, carry pose = arms outstretched). Assets
are inlined as base64 data URLs from bears.ts so consumers get a
zero-config import.
- Every effect uses ONE bear image throughout — motion comes from
CSS transforms + keyframes, never pose swapping. GPU-only path,
stays 60fps.
- defaultSplitEffect: at t=0 a bright light beam appears at the cut
point (causal anchor — the beam IS the split), while the bear pops
in above with arms cocked, swings down through the beam ~250ms in,
and floats away. ~700ms total.
- defaultMoveEffect: source-position ring pulse signals grab, ghost
clip lifts into the bear's outstretched arms, both slide along a
flat ease-in-out curve to the destination with intrinsic wobble,
destination ring pulse signals drop, then fade. ~900ms total.
Design references (raw candidate JPEGs from the seedream generation
run) live under packages/effects/design/bears/ for future re-picks —
not shipped to npm.
…the real clip
Two changes, both from user feedback while iterating on the bear-based
choreography:
1. Slower durations for the testing phase so the choreography reads
clearly frame by frame:
splitClip: 700ms → 1400ms
moveClipTo: 900ms → 1800ms
Halve these back once the shape is locked in.
2. moveClipTo ghost redesigned so it looks like the real timeline clip
being lifted, not a dashed drag-preview box. Previously it was a
thin (55%-height) purple bar with a dashed border, which reads as
"here's a placeholder while the bear walks" — but the user's mental
model is "the bear lifts the actual video strip up and carries it".
Now:
• Ghost matches source-clip footprint exactly (full width AND
height).
• Solid purple gradient fill styled like real clip bars, with a
small `clipId` label inside so it's recognisable as *this*
specific clip.
• Squash-on-arrival lands with a slight vertical squeeze then
relaxes — the "put it down" cue.
• New fading outline stays briefly at the source position (25%
opacity by 25% through the effect, gone by 40%) to soften the
fact that the underlying timeline data has already teleported.
Bridges the causal gap between data mutation and visual arrival.
Fundamental rewrite of the AI-op feedback pattern. The previous overlay
approach was structurally capped: the DOM overlay could only paint a
ghost of the real clip, never the clip itself, so moves read as
"data teleported, mascot chased it" and no amount of tweaking made
that feel professional. This lands the animation INSIDE the canvas
renderer where the actual clip bars live.
Two low-level Timeline APIs on `@aicut/core`:
animateClip(clipId, {
fromStartMs, fromTrackIndex,
toStartMs, toTrackIndex,
durationMs?,
})
Registers a per-clip render-time position tween. While active,
`buildDrawState` interpolates (start, trackIndex) with an ease-out
cubic and `drawTrackRow` picks up the overrides — the clip's data
model is unchanged, only its painted position moves. TrackIndex is
a float during transit so the clip physically slides between rows.
The raf loop stays alive while any tween is in flight; finished
entries auto-clear inside buildDrawState.
flashCut(trackIndex, atMs, durationMs = 220)
Registers a bright vertical flash on top of the clip body at
(trackIndex, atMs) — the canonical "a cut just happened here" cue.
Sharp 12% attack, long decay with a soft outward glow, alpha
pre-computed per frame so drawing is stateless.
`DrawState` gained two matching read-only fields (clipPositionOverrides,
flashes) so the pure `draw*` functions stay stateless. A new
`drawFlashes` pass runs inside the tracks clip region so flashes
inherit the same scroll transform as the clips they mark.
@aicut/react `<Timeline>` primitive now subscribes to `editor.on
("operation", …)` and translates events into those two APIs:
moveClipTo → animateClip from beforeProject rect → afterProject rect
splitClip → flashCut at (cut track, cut time)
Durations are temporarily generous (moveClipTo 560ms, splitClip 420ms)
for the current testing phase — halve back to ~280ms / ~180ms once the
shape is locked in.
…fault
Now that <Timeline> handles the primary AI-op feedback in-canvas via
animateClip / flashCut, running the bear overlay on top would be
double-drawing the same event. Change the default:
- <AiCutEffects> DEFAULT_EFFECTS all `false`. Hosts get no overlay
handlers out of the box; the bear is opt-in via an explicit
`effects={{ splitClip: defaultSplitEffect, moveClipTo: defaultMoveEffect }}`
prop.
- defaultSplitEffect / defaultMoveEffect stay exported so hosts who
want the playful vibe can wire them without pulling from internals.
- API playground chip renames to "🧸 Bear overlay: on/off" and starts
off — canvas animation is what visitors see first, matching the
real product surface. Chip flips the effects prop between the
opt-in map and undefined.
- e2e spec rewritten for the new semantics: the overlay is empty by
default; enabling the chip is what makes the bear effects mount.
… relax Split now shows the clip being cleaved, not just a flash line landing. The two new halves briefly pull apart along the cut edge in pixel space, then gently close back into contact with the flash decaying alongside. Reads as a physical cut rather than a lighting cue. Core additions: DrawState.clipPositionOverrides value gained a `startPxOffset?: number` field — a pixel-space nudge applied to a clip's rendered startX after pxPerSec conversion. Effects author in screen pixels regardless of the current zoom, which is exactly what a "constant-width gap" wants. Timeline.animateSplit(leftId, rightId, cutAtMs, trackIndex, opts?) registers a physical-cleave animation for both halves at once. The gap follows an asymmetric envelope: fast open (0 → 35% ease-out) to a peak, then long relax (35% → 100% ease-out cubic) back to zero. Internally also fires flashCut over ~70% of the same window, so the cut line is bright while the gap is open and fades before the halves fully close. drawClipAt gained a `startPxOffset` parameter (defaults to 0) that drawTrackRow now threads through from the override map. React primitive: splitClip operations now dispatch to animateSplit using the two newClipIds from result.data (falling back to flashCut only when those aren't present — defensive, since core always returns them on ok=true). Testing-phase parameters: durationMs=520, peakGapPx=14 for slow visual review. Ship at durationMs ~260 / peakGapPx ~7.
Bumped the testing-phase durations so the choreography reads clearly during walkthroughs. Ship targets stay the same (~260-280ms). moveClipTo animateClip: 560ms → 820ms splitClip animateSplit: 520ms → 780ms, peakGapPx 14 → 16
…media
Domain-neutral primitive: a clip can carry a `placeholder` field with
optional label / badge / progress. When set, the canvas paints a
distinct "not solid content" look (diagonal white stripes scrolling
at 32 px/sec over a deep-purple gradient, dashed border, centered
label with animated dots, top-right badge, optional bottom progress
bar) instead of thumbnails.
Motivation: AI-generated video is going to be a common source, so
you often want to reserve a slot on the timeline while the model is
still generating, then swap in real media on completion. Core stays
out of the AI semantics — it just knows "no playable media yet, show
the placeholder look". The host owns the meaning (AI generation, an
upload in flight, a queued render task, etc.) and drives label /
badge / progress via updatePlaceholder.
New Editor API:
- addPlaceholderClip({ trackId, startMs, durationMs, label?, badge?,
progress?, onOverlap? }) → EditResult<{ clipId, sourceId }>
Reserves a slot and creates a synthetic MediaSource with
duration=slot so downstream time-slot math stays uniform.
- updatePlaceholder(clipId, { label?, badge?, progress? })
Merges into the existing placeholder object; explicit undefined
clears a field (e.g. progress removed → bar disappears).
- replaceClipSource({ clipId, sourceUrl })
Async — probes the URL, creates a real MediaSource, points the
clip at it, clears placeholder. Preserves start + track index.
New error reason `not-media`: `splitClip` now rejects a placeholder
clip with this reason (nothing meaningful to split). Move / trim /
delete all work as normal since they're time-slot ops.
New OperationEvent kinds: `addPlaceholderClip`, `updatePlaceholder`,
`replaceClipSource` — subscribe via editor.on("operation", …) to
react to the placeholder lifecycle.
Canvas render (packages/core/src/timeline/draw.ts):
- drawTrackRow branches to drawPlaceholderClipAt when clip.placeholder
is present.
- Stripe phase advances from performance.now(); Timeline.scheduleRender's
heartbeat now keeps the raf loop alive while any placeholder is on
screen, so the pattern crawls smoothly even without other animation.
- fitLabel utility truncates the center text with an ellipsis via
binary search so the label always fits the clip width.
New playground card `addPlaceholderClip` in examples/react-demo:
adds a placeholder, ticks progress every 220ms (simulated polling),
click "complete" to replaceClipSource with sample.mp4 or "cancel" to
deleteClip. Demonstrates the whole AI-generation lifecycle end-to-end.
Also fix a gitignore gap: scripts/.publish-creds.* was ignored on
main (added there in the CodeArtifact rename) but not on this branch.
Adding the rule so local publish tokens stay out of commits here too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EditorApion@aicut/core: structuredEditResult<T>returns, explicit targets, atomicbatch(), and an operation event bus. Old-shape mutators still ship; the new ones sit alongside.@aicut/effectspackage (unpublished): decoupled overlay that renders animated feedback per edit operation. Ships with defaultsplitClipandmoveClipTostick-figure effects; every effect is swappable or disable-able./#/apiplayground route inexamples/react-demo— 9 cards, one per new API, showing structured return values live-editing a shared timeline.Changes
@aicut/coreEditResult<T>+EditErrorReasonunionbatch(label, fn): coalesces N mutations into one undo entry (sync + async)splitClip,moveClipTo,trimClip,deleteClip,addClip— allEditResult-returning, all emitOperationEventsfindClipAt,getClipsInRange,getClipsOnTrack,timelineToSourceMs,sourceToTimelineMscaptureFrame: composite-or-raw canvas capture →BlobprobeMediaSource: URL → duration/dimensions via headless<video>on(\"operation\", …): event stream with before/after project snapshots andbatchId@aicut/effects(new)<AiCutEffects>overlay component +useOperationStreamlow-level hookdefaultSplitEffect(drop → strike → recoil, ~800ms) with cubic-bezier easing per phase; strike + flash line lands ~200ms after mount so the visual peak sits near the data commit instead of trailing itdefaultMoveEffect(spawn → lift → parabolic arc + walk cycle → drop → exit, ~1.45s); arc height scales with move distance<StickFigure>character with 7 poses; thecharacterprop lets you replace it (Lottie / Rive / whatever)Demo
examples/react-demo/src/ApiPlayground.tsxmounted at/#/apiin the routerTests
@aicut/core—editor.test.tscovers batch semantics,EditResultfailure paths, and operation events;model-query.test.tscovers pure helpers./#/apiplayground.Publish plan
@aicut/effectsis a new package — first publish needs--access public.@aicut/core/@aicut/react/@aicut/vuebumps go out viascripts/publish.shafter merge, per the existing OTP workflow.Test plan
/#/api, exercise each card, verifyEditResultJSONpnpm --filter @aicut/core test— 134/134 passpnpm --filter react-demo typecheckclean