design: kernel + plugin architecture for FrameScript (Vue support as one plugin)

[ubugeeei]

Goal

This fork (FrameScript.vue) currently lives as a hard fork of upstream FrameScript. The intent of this issue is to redesign FrameScript around a tiny framework-agnostic kernel and a plugin substrate that everything else (including the React renderer) is built on. Vue support is then one plugin among many — same status as the React renderer, the media primitives, the animation system, and the studio chrome.

This is a single design-doc tracker. Once the design is locked we file individual PRs per phase against ubugeeei/FrameScript upstream and close this with a link to the merged series.

What the kernel actually owns

Cut the kernel down to four things:

@frame-script/core
├─ ProjectSettings                     // { name, width, height, fps }
├─ FrameStore                          // current frame; subscribe / set / get
├─ TimelineStore                       // clip registry + visibility
├─ AudioPlanStore                      // audio-segment registry
└─ PluginHost                          // plugin registration + lifecycle + bridge namespace

That's it. No <Project>. No <Clip>. No <Video>. No React. No Vue. No useAnimation. Those are all plugins.

The kernel is framework-agnostic, ~600 lines of TS, no DOM dependencies (only DOM access lives in the bridge for the render harness).

Plugin contract

interface FrameScriptPlugin<Options = void> {
  readonly id: string                                  // "@frame-script/react", "@frame-script/vue", ...
  readonly version: string

  // Plugins this plugin needs at host setup time.
  readonly requires?: readonly string[]

  // Called once when the plugin is registered with the host.
  setup(host: PluginHost, options?: Options): void | (() => void)
}

interface PluginHost {
  // Core stores (framework-agnostic).
  readonly frame: FrameStore
  readonly timeline: TimelineStore
  readonly audioPlan: AudioPlanStore
  readonly settings: ProjectSettings

  // Bridge namespace, scoped per plugin. No more flat window.__frameScript.
  // Renders out as window.__frameScript = { core: {...}, animation: {...}, vue: {...} }.
  registerBridge<T>(namespace: string, api: T): () => void

  // Inter-plugin ready contract: any plugin can publish a signal, any plugin
  // (notably the render harness) can wait on them.
  registerReady(name: string, contract: ReadyContract): () => void
  awaitAllReady(): Promise<void>

  // Lifecycle the host emits to plugins that opt in.
  on(event: "mount" | "unmount" | "frame", cb: (payload: any) => void): () => void
}

interface ReadyContract {
  isReady(): boolean
  whenReady(): Promise<void>
  // optional, used by the render harness when stepping frame-by-frame
  whenReadyAt?(frame: number): Promise<void>
}

Nothing in this contract mentions Vue, React, video, images, animation, or the studio. Those are plugin concerns.

Standard plugins (decomposition of today's monolith)

Plugin	Owns	Bridge namespace	Ready contracts
@frame-script/react	Renderer. Exports <Project>, <TimeLine>, <Clip>, <FillFrame> as React components that read core stores and call timeline.registerClip / audioPlan.register.	react	—
@frame-script/vue	Renderer. Same primitive surface as Vue components / composables. Wires Vue Suspense / async setup into vue.whenSuspenseReady.	vue	vue
@frame-script/media	<Video>, <Audio>, <Image>, <Sound>. Contributes audio segments via audioPlan.register. Fetches metadata via the backend client.	media	images, mediaMetadata
@frame-script/animation	useAnimation for React and the Vue equivalent. Frame-driven derivations.	animation	animations
@frame-script/audio-waveform	Waveform rendering for sound clips.	audioWaveform	audioWaveform
@frame-script/psd	<PsdCharacter> and friends, currently src/lib/character/*.	psd	psd, psdAtFrame
@frame-script/webgl	<Three> wrapper, src/lib/webgl/*.	webgl	webgl, webglAtFrame
@frame-script/draw-text	src/lib/animation/effect/draw-text.tsx + MathJax / opentype subtree.	drawText	drawText
@frame-script/studio-ui	Full editor chrome: timeline scrubber, clip panel, render dialogs. Loaded only in dev mode, never in render.	studio	—
@frame-script/backend-client	Centralised backend URL / token resolver. Used by media and others. No primitives.	backend	—

Counting roughly: ten plugins replace the current monolith. The current src/lib/ directory decomposes into these without renames. The dependency graph is shallow — media and animation depend on core; vue and react depend on core; studio-ui depends on react. No deep tree.

What the project file becomes

defineProject(config) is the new entry point. Users assemble exactly the plugins they need, Vite-style:

// project/project.ts  (renderer-agnostic header)
import { defineProject } from "@frame-script/core"
import vue from "@frame-script/vue"
import media from "@frame-script/media"
import animation from "@frame-script/animation"
import drawText from "@frame-script/draw-text"
import ProjectComponent from "./project.vue"

export default defineProject({
  settings: { name: "demo", width: 1920, height: 1080, fps: 60 },
  plugins: [
    vue(),
    media(),
    animation(),
    drawText(),
  ],
  root: ProjectComponent,
})

Render mode is the same minus @frame-script/studio-ui. The Rust render binary doesn't care — it still drives via window.__frameScript.core.setFrame() and waits on the namespaced ready contracts.

For existing React users we ship a convenience preset:

import { defineProject, presetReact } from "@frame-script/core"
import Project from "./project.tsx"

export default defineProject({
  settings: { ... },
  plugins: presetReact(),  // = [react(), media(), animation(), drawText(), audioWaveform(), studioUi()]
  root: Project,
})

Same shape, same API for everyone. Vue users swap presetReact() for presetVue() (or compose manually). Future Svelte / Solid / Lit support is just another renderer plugin and the kernel never learns about it.

Bridge namespacing concretely

Before:

window.__frameScript = {
  setFrame, getFrame,
  waitAnimationsReady, getAnimationsPending,
  waitImagesReady,     getImagesPending,
  waitAudioWaveformsReady, getAudioWaveformsPending,
  waitPsdReady, waitPsdFrame, getPsdPending,
  waitWebGLReady, waitWebGLFrame, getWebGLPending,
  waitDrawTextReady, getDrawTextPending,
  waitMediaMetadataReady, getMediaMetadataPending,
  waitCanvasFrame,
}

After:

window.__frameScript = {
  core: { setFrame, getFrame },
  animation: { whenReady, pending },
  media:     { whenImagesReady, whenMetadataReady, pending },
  audioWaveform: { whenReady, pending },
  psd:       { whenReady, whenReadyAt, pending },
  webgl:     { whenReady, whenReadyAt, pending },
  drawText:  { whenReady, pending },
  canvas:    { whenFrame },
  vue:       { whenSuspenseReady },  // present only if @frame-script/vue is installed
}

The render binary calls awaitAllReady() which iterates every registered namespace's whenReady — no hardcoded list. New plugins (Svelte, future media types) just appear with their own namespace and the harness picks them up automatically.

Current fork delta — what already exists on this branch

For reviewers sizing the migration cost. Diff against main: 65 files, +4974 / −1087.

A. Pure additions (no behaviour change upstream-side)

File	Lines	Migrates into
src/lib/vue.tsx	1467	@frame-script/vue
src/lib/audio-plan.tsx	186	@frame-script/core (AudioPlanStore)
src/lib/backend.ts	47	@frame-script/backend-client
src/lib/frame-script-bridge.ts	56	@frame-script/core (PluginHost.registerBridge)
src/lib/media-metadata.ts	51	@frame-script/media
src/lib/video/canvas-frame-registry.ts	26	@frame-script/media (canvas namespace)
project/project.vue	165	Template for @frame-script/vue
docs/docs/vue-sfc.md (+ i18n)	79	Docs for the Vue plugin

B. Upstream-leaking refactors (already required, regardless of Vue)

File	What changed	Status
src/lib/project.tsx	<Project> wraps in <TimelineStoreProvider> + <AudioPlanProvider>	Becomes the React plugin's mount logic.
src/lib/timeline.tsx	Replaced singletons (globalClips, globalHidden) with store + context	Stores move into core; React adapter stays in the React plugin.
src/lib/clip.tsx	Clip end-clamp removed; Serial inclusive-span arithmetic	Lands upstream regardless of plugin work.
src/lib/frame.tsx	Exported createFrameStore; bridge registration via registerFrameScriptApi	Store moves into core; React adapter stays in the React plugin.
src/lib/audio.ts, sound/sound.tsx, video/video.tsx, audio-waveform.ts, image.tsx, webgl/index.ts, video/video-render.tsx	Migrated to backendFetch + buildBackendUrl; sync XHR removed	Lands as part of the media / backend-client plugins.
vite.studio.config.ts, vite.render.config.ts	Added @vitejs/plugin-vue	Vue plugin will ship its own @frame-script/vue/vite.

C. Out of scope here

backend/ security hardening, render IPC validation, CI, prod-readiness work (#74–#79 and surrounding PRs). Independent of the plugin substrate.

Phased plan

Phase 0 — Carve out the kernel and the plugin host

• K.1 Move ProjectSettings, FrameStore, TimelineStore, AudioPlanStore into @frame-script/core. No React, no Vue, no DOM beyond window.__frameScript.
• K.2 Implement PluginHost with registerBridge, registerReady, awaitAllReady, lifecycle events.
• K.3 Implement defineProject(config) and the presetReact() / presetVue() presets.
• K.4 Replace flat window.__frameScript with the namespaced shape. Keep a temporary back-compat shim that mirrors the legacy keys for one release.

Outcome: upstream still ships exactly the same studio for React users (because presetReact() is the default), but underneath the kernel knows nothing about React anymore.

Phase 1 — Extract the standard plugins

Each row in the plugin table above becomes its own PR. Each PR is a move + rewire, not a feature change.

• P.1 Extract @frame-script/react (the existing JSX primitives + hooks).
• P.2 Extract @frame-script/media (Video / Audio / Image / Sound + metadata tracker).
• P.3 Extract @frame-script/animation.
• P.4 Extract @frame-script/audio-waveform.
• P.5 Extract @frame-script/psd, @frame-script/webgl, @frame-script/draw-text.
• P.6 Extract @frame-script/studio-ui (React-only for now).
• P.7 Extract @frame-script/backend-client.

Outcome: src/lib/* is empty in upstream. Everything lives in packages/<plugin>/ consumed by the kernel.

Phase 2 — Ship @frame-script/vue as one more plugin

• V.1 Port src/lib/vue.tsx into the plugin package. It depends only on @frame-script/core and (where it needs to talk to media / animation / etc.) the plugin contracts of those plugins.
• V.2 Vue-native composables for animation, media (<Video> / <Audio> / <Image> / <Sound> as Vue components).
• V.3 Wire Vue Suspense / async setup into vue.whenSuspenseReady.
• V.4 Ship @frame-script/vue/vite for the Vite plugin (@vitejs/plugin-vue + auto-configuration).

Outcome: Vue users pnpm add @frame-script/vue and use presetVue(). Zero upstream code knows the word "Vue".

Phase 3 — Open the door

• F.1 Cookbook for writing a renderer plugin (Svelte / Solid / Lit reference).
• F.2 Framework-agnostic studio-ui (move chrome to web components or framework-detected).
• F.3 Plugin marketplace doc / registry convention.

Why this shape (not just "Vue addon hooks")

A "Vue addon" plan fixes the symptom (vue.tsx needs upstream-leaking refactors) but not the cause — upstream knows about each subsystem by name. As soon as upstream wants to ship Svelte support, somebody wants to maintain a Solid renderer, or the studio chrome gets redesigned, we are back here filing another design issue. The kernel + plugin model pays the refactor cost once and stops paying it.

It also lets the fork stop being a fork: @frame-script/vue is just a sibling of @frame-script/react in the same repo (or in a separate one), with the kernel as the only shared contract.

Open design questions

1. Plugin loading: synchronous registration via defineProject({ plugins: [...] }), or async with await host.load(...)? Async lets us code-split renderer + studio-ui; sync is simpler. Recommend sync registration with lazy module imports inside each plugin.
2. Cross-renderer projects: do we want to support a project that mixes Vue components and React components in the same timeline? Technically possible (both renderers reading the same TimelineStore), UX is weird. Recommend: forbid in v1, leave the door open.
3. Plugin options shape: vue({ ssr: true }) vs vue.with({ ssr: true }) vs vueOptions({ ssr: true }). Recommend factory functions returning a plugin instance, like Vite (vue({ ... })).
4. Bridge namespace ownership: first-come-first-served, or reserved namespace list maintained in core? Recommend a reserved list for the standard plugins (core, animation, media, psd, webgl, drawText, canvas, audioWaveform, vue, react, studio). Anything else any plugin can register.
5. Back-compat shim duration: how long do we keep the flat window.__frameScript.setFrame etc. mirror? Recommend one minor release with @deprecated, removed in the next major.

Asks

1. Sanity-check the kernel surface (the four stores + PluginHost) — anything missing or extra?
2. Confirm the plugin table and namespace assignments.
3. Confirm defineProject + presetReact / presetVue as the project file entry point.
4. Pick a position on each of the five open design questions.
5. Sign off on K.1 + K.2 being the first upstream PR (kernel + PluginHost, no plugins migrated yet, no behaviour change visible to users beyond the namespaced bridge).