diff --git a/.changeset/calm-masks-protect.md b/.changeset/calm-masks-protect.md new file mode 100644 index 0000000..760af51 --- /dev/null +++ b/.changeset/calm-masks-protect.md @@ -0,0 +1,11 @@ +--- +"@seqvio/browser-recorder": minor +"@seqvio/capture": patch +"@seqvio/terminal-narrator": patch +--- + +Add fail-closed selector and rectangle privacy masks for browser capture plans, +retain non-sensitive mask evidence in recording manifests, and keep the `serve` +CLI process alive until it is terminated. Promote the versioned Terminal and +Browser CLI/artifact contract to stable after the Windows/Linux/macOS runtime +matrix passed. diff --git a/.env.example b/.env.example index 35cb49e..3f260ff 100644 --- a/.env.example +++ b/.env.example @@ -25,3 +25,8 @@ OPENAI_BASE_URL=https://api.openai.com/v1 # MiniMax # `mmx auth login` is still required; this only controls the default voice. # MINIMAX_TTS_VOICE= + +# Optional external Python Manim adapter +# seqvio-doctor discovers .venv-manim, then falls back to `python`. +# The adapter API uses `python` unless pythonCommand is passed explicitly. +# SEQVIO_MANIM_PYTHON=/absolute/path/to/venv/bin/python diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6556239..8d4c3a9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,7 +2,7 @@ name: CI on: push: - branches: [main] + branches: [main, 'perf/**'] pull_request: branches: [main] diff --git a/.gitignore b/.gitignore index 3f5a1a0..b7c66fa 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ config.local.json # Dependencies node_modules/ .tools/ +.venv-manim/ # Build outputs dist/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 5c7b0ad..45174fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,9 @@ This project follows a simple keep-a-changelog style: ### Added +- Added complete documentation for the experimental Python Manim adapter, + including cross-platform setup, deterministic external rendering, manifests, + cache behavior, `ManimClip`, ExplanationBeat markers, and troubleshooting. - Added the CompositionDocument `ExplanationBeat` contract for jointly authored narration cues, exact phrase anchors, visual actions, and capture evidence. - Added post-TTS phrase resolution and semantic scene-local time maps. Fine TTS diff --git a/README.md b/README.md index 405cb68..76d8705 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,3 @@ - # Seqvio [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE) @@ -11,7 +10,7 @@ English | [简体中文](./README.zh-CN.md) Seqvio gives coding agents a capture-to-explanation path for turning real terminal/browser activity and technical ideas into clear narrated videos. Human-readable `EDITORIAL.md` and `VISUAL-DESIGN.md` artifacts make content choices and visual direction reviewable before the executable `ExplainerDocument` IR binds spoken cues to visual actions with `ExplanationBeat`s. -> **Current status:** The repository supports explicit React/TSX compositions and `ExplainerDocument` with complete `whiteboard`, `code`, `diagram`, `terminal`, and `browser` compiler paths. Phrase-anchored ExplanationBeats drive logical visual timing, post-TTS semantic time maps, speech/highlight QA, and deterministic local rendering. Terminal and browser pipelines compile real recorded steps through the same IR. Capture CLI contract `2.0` names the canonical IR artifact `explainer.json`. +> **Current status:** The repository supports explicit React/TSX compositions and `ExplainerDocument` with public `whiteboard`, `code`, `diagram`, `terminal`, and `browser` compiler paths, plus experimental `infographic` and externally rendered Python `manim` paths. Phrase-anchored ExplanationBeats drive logical visual timing, post-TTS semantic time maps, speech/highlight QA, and deterministic local rendering. Terminal and browser pipelines compile real recorded steps through the same IR. Capture CLI contract `2.0` names the canonical IR artifact `explainer.json`. ## Demo @@ -31,10 +30,10 @@ primary product demo. Seqvio has two separate pieces: -| Piece | What it is | Install with | -| --- | --- | --- | -| **Agent skill** | Teaches Cursor and other agents how to author TSX compositions and run the render workflow | `npx skills add ...` | -| **Renderer CLI** | Runs `seqvio-render`, `seqvio-audio`, and `seqvio-qa` | `npm install @seqvio/renderer` or a local repo build | +| Piece | What it is | Install with | +| ---------------- | ------------------------------------------------------------------------------------------ | ---------------------------------------------------- | +| **Agent skill** | Teaches Cursor and other agents how to author TSX compositions and run the render workflow | `npx skills add ...` | +| **Renderer CLI** | Runs `seqvio-render`, `seqvio-audio`, and `seqvio-qa` | `npm install @seqvio/renderer` or a local repo build | Installing the skill alone is **not** enough to render videos. You also need the CLI (or a local checkout of this repository). @@ -67,6 +66,8 @@ Public packages: `@seqvio/core`, `@seqvio/whiteboard`, `@seqvio/scatterbrain`, `@seqvio/product-demo`, `@seqvio/technical`, and `@seqvio/renderer`. Experimental capture packages: `@seqvio/capture`, `@seqvio/browser-recorder`, and `@seqvio/terminal-narrator`. +The optional experimental `@seqvio/manim-adapter` workspace invokes the Python +package `manim` for externally rendered mathematical animation. Install optional style/component packages when a composition imports them directly: @@ -132,7 +133,7 @@ node packages/browser-recorder/dist/cli.js serve --port 4175 node packages/browser-recorder/dist/cli.js record --plan plan.json --jobId demo --json ``` -Open `http://127.0.0.1:4175`. The built-in sample runs without an AI provider; configure a planner webhook only when AI-generated action plans are required. See the [browser recorder README](./packages/browser-recorder/README.md) for the plan contract and pre-stable CLI boundaries. +Open `http://127.0.0.1:4175`. The built-in sample runs without an AI provider; configure a planner webhook only when AI-generated action plans are required. See the [browser recorder README](./packages/browser-recorder/README.md) for the plan contract and stable adapter boundaries. **Requirements:** Node.js `>=18`, Chromium (via Puppeteer), FFmpeg (bundled in `@seqvio/renderer`). Local repo development uses npm workspaces and `package-lock.json`. Verify the complete local toolchain with `seqvio-doctor` or `npm run doctor` in a repository checkout. @@ -140,6 +141,19 @@ Open `http://127.0.0.1:4175`. The built-in sample runs without an AI provider; c [`@seqvio/terminal-narrator`](./packages/terminal-narrator) uses `node-pty` and xterm-backed snapshots to preserve terminal state and recorded step timing. It compiles each observed step into a Terminal scene plus jointly-authored narration cues and capture-backed ExplanationBeats. `--withAudio` synthesizes and muxes narration; hard captions are added only when `--burnCaptions` is also explicit. +### Optional Python Manim adapter + +[`@seqvio/manim-adapter`](./packages/manim-adapter) is a TypeScript/Node.js +adapter for the external Python package `manim`; it is not a JavaScript Manim +implementation. It renders equations, graphs, and geometric constructions to +validated media and a content-addressed manifest. `ManimClip` from +`@seqvio/technical` then adds that seekable media to the Seqvio timeline, where +named markers can align with phrase-anchored ExplanationBeats. + +Python Manim is optional and only required for generating this external media. +See the [Manim integration guide](./docs/MANIM-INTEGRATION.md) for setup on +Windows, macOS, and Linux, adapter commands, cache behavior, and IR/TSX usage. + ## What You Can Build - Product and framework intro videos @@ -150,18 +164,19 @@ Open `http://127.0.0.1:4175`. The built-in sample runs without an AI provider; c Start from examples: -| Example | Description | -| --- | --- | -| [`seqvio-overview-en.tsx`](./examples/compositions/seqvio-overview-en.tsx) | Narrated English product overview | -| [`seqvio-overview-zh.tsx`](./examples/compositions/seqvio-overview-zh.tsx) | Narrated Chinese product overview | -| [`seqvio-audio-demo.tsx`](./examples/compositions/seqvio-audio-demo.tsx) | Audio and caption metadata | -| [`seqvio-style-manifest-demo.tsx`](./examples/compositions/seqvio-style-manifest-demo.tsx) | Whiteboard style preset manifest demo | -| [`seqvio-product-demo-preview.tsx`](./examples/compositions/seqvio-product-demo-preview.tsx) | Product walkthrough components demo | -| [`seqvio-scatterbrain.tsx`](./examples/compositions/seqvio-scatterbrain.tsx) | Sticky-note / workshop style demo | -| [`loop-engineering-explainer.tsx`](./examples/compositions/loop-engineering-explainer.tsx) | Long-form narrated explainer composition | -| [`technical-explainer.tsx`](./examples/compositions/technical-explainer.tsx) | Technical explainer with code walkthrough and architecture diagram | -| [`technical-demo.tsx`](./examples/compositions/technical-demo.tsx) | Terminal demo and ANSI rendering showcase | -| [`packages/whiteboard/examples/`](./packages/whiteboard/examples/) | Single-scene whiteboard samples | +| Example | Description | +| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| [`seqvio-overview-en.tsx`](./examples/compositions/seqvio-overview-en.tsx) | Narrated English product overview | +| [`seqvio-overview-zh.tsx`](./examples/compositions/seqvio-overview-zh.tsx) | Narrated Chinese product overview | +| [`seqvio-audio-demo.tsx`](./examples/compositions/seqvio-audio-demo.tsx) | Audio and caption metadata | +| [`seqvio-style-manifest-demo.tsx`](./examples/compositions/seqvio-style-manifest-demo.tsx) | Whiteboard style preset manifest demo | +| [`seqvio-product-demo-preview.tsx`](./examples/compositions/seqvio-product-demo-preview.tsx) | Product walkthrough components demo | +| [`seqvio-scatterbrain.tsx`](./examples/compositions/seqvio-scatterbrain.tsx) | Sticky-note / workshop style demo | +| [`loop-engineering-explainer.tsx`](./examples/compositions/loop-engineering-explainer.tsx) | Long-form narrated explainer composition | +| [`technical-explainer.tsx`](./examples/compositions/technical-explainer.tsx) | Technical explainer with code walkthrough and architecture diagram | +| [`technical-demo.tsx`](./examples/compositions/technical-demo.tsx) | Terminal demo and ANSI rendering showcase | +| [`manim-end-to-end-validation.tsx`](./examples/compositions/manim-end-to-end-validation.tsx) | Narrated playback of externally rendered graph and proof animation | +| [`packages/whiteboard/examples/`](./packages/whiteboard/examples/) | Single-scene whiteboard samples | ## How It Works @@ -192,13 +207,13 @@ See [`docs/COMPOSITION-AUTHORING.md`](./docs/COMPOSITION-AUTHORING.md) for the a The skill lives in [`skills/seqvio/SKILL.md`](./skills/seqvio/SKILL.md) with supporting references: -| Reference | Purpose | -| --- | --- | -| [`authoring-patterns.md`](./skills/seqvio/references/authoring-patterns.md) | TSX composition patterns and timing rules | -| [`audio-workflow.md`](./skills/seqvio/references/audio-workflow.md) | Extract, synthesize, and mux narration | -| [`render-workflow.md`](./skills/seqvio/references/render-workflow.md) | Build, render, and smoke-test commands | +| Reference | Purpose | +| --------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| [`authoring-patterns.md`](./skills/seqvio/references/authoring-patterns.md) | TSX composition patterns and timing rules | +| [`audio-workflow.md`](./skills/seqvio/references/audio-workflow.md) | Extract, synthesize, and mux narration | +| [`render-workflow.md`](./skills/seqvio/references/render-workflow.md) | Build, render, and smoke-test commands | | [`production-techniques.md`](./skills/seqvio/references/production-techniques.md) | Voice-first timing, reference-style analysis, and visual QA rules | -| [`planning-workflow.md`](./skills/seqvio/references/planning-workflow.md) | Editorial/visual planning and agent handoff | +| [`planning-workflow.md`](./skills/seqvio/references/planning-workflow.md) | Editorial/visual planning and agent handoff | Install the skill (see [Quick Start](#quick-start)): @@ -228,6 +243,8 @@ Seqvio is the visual language for coding agents that need to explain, not merely - `@seqvio/scatterbrain` sticky-note / cork-board components - `@seqvio/product-demo` components: `ProductDemoScene`, `BrowserFrame`, `ScreenshotPlaceholder`, `CursorPath`, `Callout`, `ProductTitle` - `@seqvio/technical` components: `TechnicalScene`, `AnnotationTarget`, `CodeWalkthrough`, `ArchitectureDiagram`, `TerminalDemo`, plus ANSI/grid utilities and bundled code fonts +- `@seqvio/technical` `ManimClip` for deterministic seeking and narration-aligned markers in externally rendered mathematical animation +- Experimental `@seqvio/manim-adapter` for Python/Manim preflight, deterministic execution, media probing, content-addressed manifests, and cache reuse - Terminal scene support in the composition-document IR (`events` / `steps` / `commands`) with validation and TSX compilation - Browser scene support with recorded video, cursor/focus/click metadata, exact action clocks, and time-mapped media seeking - `ExplanationBeat` cues, exact phrase anchors, visual actions, capture evidence, post-TTS `outputFrame`s, and semantic `sceneTimings[].timeMap` @@ -295,17 +312,18 @@ Voiceover is muxed automatically from the manifest. **Do not** add `--burnCaptio ## Packages -| Package | Description | -| --- | --- | -| [`@seqvio/whiteboard`](./packages/whiteboard) | Whiteboard drawing components and timing helpers | -| [`@seqvio/core`](./packages/core) | Composition container, scenes, transitions, and timeline runtime | -| [`@seqvio/scatterbrain`](./packages/scatterbrain) | Sticky-note / cork-board style components | -| [`@seqvio/product-demo`](./packages/product-demo) | Browser frames, cursor paths, screenshot placeholders, callouts, and product walkthrough components | -| [`@seqvio/technical`](./packages/technical) | Technical explainer runtime: code walkthroughs, architecture diagrams, terminal demos, annotations, and bundled fonts | -| [`@seqvio/terminal-narrator`](./packages/terminal-narrator) | Pre-stable node-pty/xterm capture → IR/ExplanationBeat → optional narrated MP4 | -| [`@seqvio/browser-recorder`](./packages/browser-recorder) | Pre-stable Chromium action capture with exact action timing → IR/ExplanationBeat | -| [`@seqvio/capture`](./packages/capture) | Shared experimental capture session and artifact contracts | -| [`@seqvio/renderer`](./packages/renderer) | TSX bundler plus `seqvio-render` and `seqvio-audio` CLIs | +| Package | Description | +| ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| [`@seqvio/whiteboard`](./packages/whiteboard) | Whiteboard drawing components and timing helpers | +| [`@seqvio/core`](./packages/core) | Composition container, scenes, transitions, and timeline runtime | +| [`@seqvio/scatterbrain`](./packages/scatterbrain) | Sticky-note / cork-board style components | +| [`@seqvio/product-demo`](./packages/product-demo) | Browser frames, cursor paths, screenshot placeholders, callouts, and product walkthrough components | +| [`@seqvio/technical`](./packages/technical) | Technical explainer runtime: code walkthroughs, architecture diagrams, terminal demos, annotations, and bundled fonts | +| [`@seqvio/terminal-narrator`](./packages/terminal-narrator) | Stable node-pty/xterm capture contract → IR/ExplanationBeat → optional narrated MP4 | +| [`@seqvio/browser-recorder`](./packages/browser-recorder) | Stable Chromium action capture with exact action timing → IR/ExplanationBeat | +| [`@seqvio/capture`](./packages/capture) | Shared experimental capture session and artifact contracts | +| [`@seqvio/manim-adapter`](./packages/manim-adapter) | Experimental adapter that invokes Python Manim and validates/caches the rendered media manifest | +| [`@seqvio/renderer`](./packages/renderer) | TSX bundler plus `seqvio-render` and `seqvio-audio` CLIs | ## Documentation @@ -316,6 +334,7 @@ Recommended reading: - [`docs/COMPOSITION-AUTHORING.md`](./docs/COMPOSITION-AUTHORING.md) — authoring contract and API rules - [`docs/EXPLANATION-BEAT-TIMING.md`](./docs/EXPLANATION-BEAT-TIMING.md) — joint narration/visual timing and post-TTS alignment - [`docs/CAPTURE-CLI-CONTRACT.md`](./docs/CAPTURE-CLI-CONTRACT.md) — capture commands, JSON output, exit codes, and artifacts +- [`docs/MANIM-INTEGRATION.md`](./docs/MANIM-INTEGRATION.md) — optional Python Manim setup, adapter rendering, manifests, and timeline integration - [`docs/TROUBLESHOOTING.md`](./docs/TROUBLESHOOTING.md) — renderer, audio, and environment issues - [`examples/compositions/README.md`](./examples/compositions/README.md) — example catalog and conventions - [`skills/seqvio/SKILL.md`](./skills/seqvio/SKILL.md) — agent production loop @@ -331,8 +350,8 @@ Full phase ordering and the reasoning behind it: [`docs/ROADMAP.md`](./docs/ROAD 1. **Singular capture/IR path** - shared dispatcher routing and legacy writer removal are complete; stabilize adapter CLIs around `CaptureSession -> ExplainerDocument`. 2. **ExplanationBeat timing** - now implemented across all stable scenes, including capture evidence and post-TTS phrase alignment. -3. **Release QA** - baseline/capture profiles now cover visual, pacing, audio, media, and semantic Beat failures; screenshot privacy masking remains deferred. -4. **Packaging and promotion** - CLI/artifact contract `1.0` is implemented; verify supported npm/runtime hosts before lifecycle promotion. Screenshot privacy remains deferred. +3. **Release QA** - baseline/capture profiles now cover visual, pacing, audio, media, semantic Beat failures, and deterministic browser privacy masks; OCR is not a security boundary. +4. **Packaging and promotion** - CLI/artifact contract `2.0` and supported-host lifecycle promotion are complete; npm release publication remains an external release action. Product positioning and scope: diff --git a/README.zh-CN.md b/README.zh-CN.md index 41eaf73..573bc26 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -10,7 +10,7 @@ Seqvio 为 coding agent 提供从真实系统捕获到讲解视频的完整路径。人类可读的 `EDITORIAL.md` 与 `VISUAL-DESIGN.md` 先让内容取舍和视觉方向可审阅，再由正式 `ExplainerDocument` IR 通过 `ExplanationBeat` 绑定旁白短语与视觉动作。 -> **当前状态：** 仓库支持显式 React/TSX composition，以及拥有完整 `whiteboard`、`code`、`diagram`、`terminal`、`browser` 编译路径的 `ExplainerDocument`。短语锚定的 ExplanationBeat 驱动逻辑视觉时间、TTS 后语义 timeMap、语速/高亮 QA 和确定性本地渲染。Capture CLI contract `2.0` 将正式 IR 产物统一命名为 `explainer.json`。 +> **当前状态：** 仓库支持显式 React/TSX composition，以及拥有 public `whiteboard`、`code`、`diagram`、`terminal`、`browser` 编译路径和实验性 `infographic`、外部 Python `manim` 编译路径的 `ExplainerDocument`。短语锚定的 ExplanationBeat 驱动逻辑视觉时间、TTS 后语义 timeMap、语速/高亮 QA 和确定性本地渲染。Capture CLI contract `2.0` 将正式 IR 产物统一命名为 `explainer.json`。 ## Demo @@ -28,10 +28,10 @@ Seqvio 为 coding agent 提供从真实系统捕获到讲解视频的完整路 Seqvio 分两部分，需要分别安装： -| 组件 | 作用 | 安装方式 | -| --- | --- | --- | -| **Agent skill** | 教 Cursor 等 agent 如何编写 TSX composition 并走渲染流程 | `npx skills add ...` | -| **Renderer CLI** | 执行 `seqvio-render`、`seqvio-audio`、`seqvio-qa` | `npm install @seqvio/renderer` 或本地仓库 build | +| 组件 | 作用 | 安装方式 | +| ---------------- | -------------------------------------------------------- | ----------------------------------------------- | +| **Agent skill** | 教 Cursor 等 agent 如何编写 TSX composition 并走渲染流程 | `npx skills add ...` | +| **Renderer CLI** | 执行 `seqvio-render`、`seqvio-audio`、`seqvio-qa` | `npm install @seqvio/renderer` 或本地仓库 build | **只执行 `npx skills add` 不够**，还需要安装 CLI（或 clone 本仓库并 build）才能真正渲染视频。 @@ -63,6 +63,8 @@ seqvio-render --help Public 包：`@seqvio/core`、`@seqvio/whiteboard`、`@seqvio/scatterbrain`、 `@seqvio/product-demo`、`@seqvio/technical`、`@seqvio/renderer`。实验性捕获包： `@seqvio/capture`、`@seqvio/browser-recorder`、`@seqvio/terminal-narrator`。 +可选的实验性 `@seqvio/manim-adapter` workspace 会调用 Python 包 `manim`， +用于在外部渲染数学动画。 当 composition 直接 import 可选视觉包时，可额外安装： @@ -127,12 +129,24 @@ node packages/browser-recorder/dist/cli.js serve --port 4175 node packages/browser-recorder/dist/cli.js record --plan plan.json --jobId demo --json ``` -打开 `http://127.0.0.1:4175`。内置示例无需 AI provider；只有需要 AI 生成 action plan 时才配置 planner webhook。plan 契约和 pre-stable CLI 边界见 [browser recorder README](./packages/browser-recorder/README.md)。 +打开 `http://127.0.0.1:4175`。内置示例无需 AI provider；只有需要 AI 生成 action plan 时才配置 planner webhook。plan 契约和 stable adapter 边界见 [browser recorder README](./packages/browser-recorder/README.md)。 ### Terminal 捕获适配器 [`@seqvio/terminal-narrator`](./packages/terminal-narrator) 使用 `node-pty` 与 xterm 快照保留终端状态和真实步骤时间，并把每个观察步骤编译成 Terminal 场景、旁白 cue 与基于捕获证据的 ExplanationBeat。`--withAudio` 只负责合成并混入旁白；仅当同时显式添加 `--burnCaptions` 时才烧录字幕。 +### 可选的 Python Manim 适配器 + +[`@seqvio/manim-adapter`](./packages/manim-adapter) 是外部 Python 包 `manim` +的 TypeScript/Node.js 适配层，并不是 JavaScript 版 Manim。它把公式、图表和 +几何构造渲染成经过校验的媒体与内容寻址 manifest；随后由 +`@seqvio/technical` 的 `ManimClip` 将媒体作为可 seek 内容接入 Seqvio +时间线，并让命名 marker 与短语锚定的 ExplanationBeat 对齐。 + +只有生成这些外部媒体时才需要安装 Python Manim。Windows、macOS、Linux +安装方法、adapter 命令、缓存规则和 IR/TSX 接入方式见 +[Manim 集成指南](./docs/MANIM-INTEGRATION.md)。 + **环境要求：** Node.js `>=18`、Chromium（Puppeteer）、FFmpeg（`@seqvio/renderer` 已内置）。本地仓库开发使用 npm workspaces 和 `package-lock.json`。可运行 `seqvio-doctor`，或在仓库中运行 `npm run doctor`，一次校验完整本地工具链。 ## 可以做什么 @@ -145,18 +159,19 @@ node packages/browser-recorder/dist/cli.js record --plan plan.json --jobId demo 示例入口： -| 示例 | 说明 | -| --- | --- | -| [`seqvio-overview-zh.tsx`](./examples/compositions/seqvio-overview-zh.tsx) | 中文旁白产品介绍 | -| [`seqvio-overview-en.tsx`](./examples/compositions/seqvio-overview-en.tsx) | 英文旁白产品介绍 | -| [`seqvio-audio-demo.tsx`](./examples/compositions/seqvio-audio-demo.tsx) | 音频和字幕元数据 | -| [`seqvio-style-manifest-demo.tsx`](./examples/compositions/seqvio-style-manifest-demo.tsx) | 白板 style preset manifest 示例 | -| [`seqvio-product-demo-preview.tsx`](./examples/compositions/seqvio-product-demo-preview.tsx) | 产品 walkthrough 组件示例 | -| [`seqvio-scatterbrain.tsx`](./examples/compositions/seqvio-scatterbrain.tsx) | 便签 / workshop 风格示例 | -| [`loop-engineering-explainer.tsx`](./examples/compositions/loop-engineering-explainer.tsx) | 长篇旁白讲解 composition | -| [`technical-explainer.tsx`](./examples/compositions/technical-explainer.tsx) | 技术讲解：代码走读与架构图 | -| [`technical-demo.tsx`](./examples/compositions/technical-demo.tsx) | 终端演示与 ANSI 渲染展示 | -| [`packages/whiteboard/examples/`](./packages/whiteboard/examples/) | 单场景白板示例 | +| 示例 | 说明 | +| -------------------------------------------------------------------------------------------- | -------------------------------- | +| [`seqvio-overview-zh.tsx`](./examples/compositions/seqvio-overview-zh.tsx) | 中文旁白产品介绍 | +| [`seqvio-overview-en.tsx`](./examples/compositions/seqvio-overview-en.tsx) | 英文旁白产品介绍 | +| [`seqvio-audio-demo.tsx`](./examples/compositions/seqvio-audio-demo.tsx) | 音频和字幕元数据 | +| [`seqvio-style-manifest-demo.tsx`](./examples/compositions/seqvio-style-manifest-demo.tsx) | 白板 style preset manifest 示例 | +| [`seqvio-product-demo-preview.tsx`](./examples/compositions/seqvio-product-demo-preview.tsx) | 产品 walkthrough 组件示例 | +| [`seqvio-scatterbrain.tsx`](./examples/compositions/seqvio-scatterbrain.tsx) | 便签 / workshop 风格示例 | +| [`loop-engineering-explainer.tsx`](./examples/compositions/loop-engineering-explainer.tsx) | 长篇旁白讲解 composition | +| [`technical-explainer.tsx`](./examples/compositions/technical-explainer.tsx) | 技术讲解：代码走读与架构图 | +| [`technical-demo.tsx`](./examples/compositions/technical-demo.tsx) | 终端演示与 ANSI 渲染展示 | +| [`manim-end-to-end-validation.tsx`](./examples/compositions/manim-end-to-end-validation.tsx) | 外部渲染图表和证明动画的旁白播放 | +| [`packages/whiteboard/examples/`](./packages/whiteboard/examples/) | 单场景白板示例 | ## 工作原理 @@ -187,13 +202,13 @@ node packages/browser-recorder/dist/cli.js record --plan plan.json --jobId demo Skill 主文件：[`skills/seqvio/SKILL.md`](./skills/seqvio/SKILL.md)，参考文档： -| 参考 | 用途 | -| --- | --- | -| [`authoring-patterns.md`](./skills/seqvio/references/authoring-patterns.md) | TSX composition 模式与 timing 规则 | -| [`audio-workflow.md`](./skills/seqvio/references/audio-workflow.md) | 提取、合成、混流旁白 | -| [`render-workflow.md`](./skills/seqvio/references/render-workflow.md) | build、render、smoke test 命令 | +| 参考 | 用途 | +| --------------------------------------------------------------------------------- | ---------------------------------------------- | +| [`authoring-patterns.md`](./skills/seqvio/references/authoring-patterns.md) | TSX composition 模式与 timing 规则 | +| [`audio-workflow.md`](./skills/seqvio/references/audio-workflow.md) | 提取、合成、混流旁白 | +| [`render-workflow.md`](./skills/seqvio/references/render-workflow.md) | build、render、smoke test 命令 | | [`production-techniques.md`](./skills/seqvio/references/production-techniques.md) | voice-first timing、参考风格分析和视觉 QA 规则 | -| [`planning-workflow.md`](./skills/seqvio/references/planning-workflow.md) | Editorial/Visual 规划和 agent handoff | +| [`planning-workflow.md`](./skills/seqvio/references/planning-workflow.md) | Editorial/Visual 规划和 agent handoff | 安装 skill（见 [快速开始](#快速开始)）： @@ -223,6 +238,8 @@ Seqvio 是 coding agent 用来解释想法的视觉语言，而不只是一个 - `@seqvio/scatterbrain`：便签 / cork-board 风格组件 - `@seqvio/product-demo`：`ProductDemoScene`、`BrowserFrame`、`ScreenshotPlaceholder`、`CursorPath`、`Callout`、`ProductTitle` - `@seqvio/technical`：`TechnicalScene`、`AnnotationTarget`、`CodeWalkthrough`、`ArchitectureDiagram`、`TerminalDemo`，以及 ANSI/grid 工具函数和内置代码字体 +- `@seqvio/technical` 的 `ManimClip`：对外部渲染的数学动画进行确定性 seek，并让 marker 与旁白对齐 +- 实验性 `@seqvio/manim-adapter`：检查 Python/Manim、确定性执行、媒体探测、内容寻址 manifest 和缓存复用 - 终端场景在 composition-document IR 中的支持（`events` / `steps` / `commands`），含校验和 TSX 编译 - Browser 场景支持录制视频、光标/聚焦/点击元数据、真实动作时钟和 time-mapped 媒体 seek - `ExplanationBeat` cue、精确短语锚点、视觉动作、捕获证据、TTS 后 `outputFrame` 和语义 `sceneTimings[].timeMap` @@ -291,17 +308,18 @@ node packages/renderer/dist/cli.js \ ## Packages -| Package | 说明 | -| --- | --- | -| [`@seqvio/whiteboard`](./packages/whiteboard) | 白板绘制组件和 timing helpers | -| [`@seqvio/core`](./packages/core) | Composition 容器、场景、转场和 timeline runtime | -| [`@seqvio/scatterbrain`](./packages/scatterbrain) | 便签 / cork-board 风格组件 | -| [`@seqvio/product-demo`](./packages/product-demo) | 浏览器框、光标路径、截图占位、callout 和产品 walkthrough 组件 | -| [`@seqvio/technical`](./packages/technical) | 技术讲解 runtime：代码走读、架构图、终端演示、标注和内置字体 | -| [`@seqvio/terminal-narrator`](./packages/terminal-narrator) | Pre-stable node-pty/xterm 捕获 → IR/ExplanationBeat → 可选旁白 MP4 | -| [`@seqvio/browser-recorder`](./packages/browser-recorder) | Pre-stable Chromium action 捕获，保留真实动作时间 → IR/ExplanationBeat | -| [`@seqvio/capture`](./packages/capture) | 实验性的共享 capture session 和 artifact 契约 | -| [`@seqvio/renderer`](./packages/renderer) | TSX bundler，以及 `seqvio-render` / `seqvio-audio` CLI | +| Package | 说明 | +| ----------------------------------------------------------- | ------------------------------------------------------------------ | +| [`@seqvio/whiteboard`](./packages/whiteboard) | 白板绘制组件和 timing helpers | +| [`@seqvio/core`](./packages/core) | Composition 容器、场景、转场和 timeline runtime | +| [`@seqvio/scatterbrain`](./packages/scatterbrain) | 便签 / cork-board 风格组件 | +| [`@seqvio/product-demo`](./packages/product-demo) | 浏览器框、光标路径、截图占位、callout 和产品 walkthrough 组件 | +| [`@seqvio/technical`](./packages/technical) | 技术讲解 runtime：代码走读、架构图、终端演示、标注和内置字体 | +| [`@seqvio/terminal-narrator`](./packages/terminal-narrator) | Stable node-pty/xterm 捕获契约 → IR/ExplanationBeat → 可选旁白 MP4 | +| [`@seqvio/browser-recorder`](./packages/browser-recorder) | Stable Chromium action 捕获，保留真实动作时间 → IR/ExplanationBeat | +| [`@seqvio/capture`](./packages/capture) | 实验性的共享 capture session 和 artifact 契约 | +| [`@seqvio/manim-adapter`](./packages/manim-adapter) | 调用 Python Manim，并校验、缓存渲染媒体 manifest 的实验性适配层 | +| [`@seqvio/renderer`](./packages/renderer) | TSX bundler，以及 `seqvio-render` / `seqvio-audio` CLI | ## 文档 @@ -312,6 +330,7 @@ node packages/renderer/dist/cli.js \ - [`docs/COMPOSITION-AUTHORING.md`](./docs/COMPOSITION-AUTHORING.md) - [`docs/EXPLANATION-BEAT-TIMING.md`](./docs/EXPLANATION-BEAT-TIMING.md) - [`docs/CAPTURE-CLI-CONTRACT.md`](./docs/CAPTURE-CLI-CONTRACT.md) +- [`docs/MANIM-INTEGRATION.md`](./docs/MANIM-INTEGRATION.md) — Python Manim 安装、adapter 渲染、manifest 与时间线接入 - [`docs/TROUBLESHOOTING.md`](./docs/TROUBLESHOOTING.md) - [`examples/compositions/README.md`](./examples/compositions/README.md) - [`skills/seqvio/SKILL.md`](./skills/seqvio/SKILL.md) @@ -327,8 +346,8 @@ node packages/renderer/dist/cli.js \ 1. **统一捕获/IR 路径** -- shared dispatcher 路由与 legacy writer 移除已经完成，下一步围绕 `CaptureSession -> ExplainerDocument` 稳定适配器 CLI。 2. **ExplanationBeat 时间模型** -- 已覆盖所有稳定场景，包括捕获证据和 TTS 后短语对齐。 -3. **发布 QA** -- baseline/capture profile 已覆盖视觉、节奏、音频、媒体和语义 Beat 故障；截图隐私 masking 暂缓。 -4. **打包与晋级** -- CLI/产物 contract `1.0` 已完成；验证支持的 npm/runtime 主机后再晋级生命周期。截图隐私仍暂缓。 +3. **发布 QA** -- baseline/capture profile 已覆盖视觉、节奏、音频、媒体、语义 Beat 故障和确定性的浏览器隐私遮罩；OCR 不作为安全边界。 +4. **打包与晋级** -- CLI/产物 contract `2.0` 已完成；验证支持的 npm/runtime 主机后再晋级生命周期。 产品定位与范围： diff --git a/TERMINAL-NARRATOR-PLAN.md b/TERMINAL-NARRATOR-PLAN.md deleted file mode 100644 index 2f49115..0000000 --- a/TERMINAL-NARRATOR-PLAN.md +++ /dev/null @@ -1,216 +0,0 @@ -# Terminal Narrator 实施计划 - -> Coding Agent CLI 执行录屏 → 旁白讲解视频自动化 -> 包名：`@seqvio/terminal-narrator` -> 范围：Phase 0 / Phase 1 / Phase 3（**不含 MCP 产品化入口**） - ---- - -## 1. 目标 - -为 Seqvio 增加一条与 [`packages/browser-recorder`](packages/browser-recorder) 同构的终端演示管线： - -```text -plan.json → record (node-pty) → recording-manifest.json → compose.tsx → seqvio-render → final.mp4 -``` - -差异化在于 **narration-first**：捕获真实终端 I/O 后，按步骤生成旁白/字幕元数据，驱动 `TerminalDemo` 回放与高亮。 - ---- - -## 2. 包命名（已定） - -| 项 | 名称 | -|----|------| -| npm 包 | `@seqvio/terminal-narrator` | -| CLI | `seqvio-terminal-narrator` / `seqvio-terminal` | -| 输出目录 | `output/terminal-narrator//` | - -内部模块仍用动词命名（`record.ts`、`compose.ts`），与 browser-recorder 一致。 - ---- - -## 3. 核心架构：双层模式 - -```mermaid -flowchart TB - subgraph orchestrator [OrchestratorLayer] - MasterAgent["主控 Agent / CLI 胶水层"] - Recorder["terminal-narrator"] - MasterAgent -->|"生成 plan.json"| Recorder - end - - subgraph subprocess [RecordedSubprocess] - PTY["node-pty 虚拟终端"] - end - - subgraph target [TargetProcess] - ClaudeCLI["Claude Code / shell CLI"] - InnerAI["内部 AI 执行 Skill"] - ClaudeCLI --> InnerAI - end - - Recorder -->|"spawn + write stdin"| PTY - PTY --> ClaudeCLI - Recorder -->|"stdout + timestamps"| Manifest["recording-manifest.json"] - Manifest --> Compose["compose.tsx + meta.audio"] - Compose --> Render["seqvio-render"] - Render --> MP4["final.mp4"] -``` - -**关键约束**：录制引擎必须在 PTY 之外，避免「Agent 录自己」的死循环。 - ---- - -## 4. 编排机制（本阶段采用） - -| 机制 | 说明 | 本计划 | -|------|------|--------| -| 机制 1：外壳 Agent 模拟人类 | 主控向 PTY 写 `/my-skill` 等指令 | **采用（捕获层）** | -| 机制 2：MCP 沙盒重跑 | Claude Code 调用录屏工具 | **本阶段不做** | -| 机制 3：日志回放渲染 | 基于 manifest 加速回放 | **采用（渲染层）** | - ---- - -## 5. 技术路线 - -**Phase 0**：`node-pty` 捕获 + timestamped manifest + **asciinema cast v2 导出** -**Phase 1**：`TerminalDemo`（VHS 风格：渐变舞台 + macOS 窗框 + 打字机）+ Seqvio 渲染 -**Phase 3**：步骤摘要 → narration/captions → 可选 `--withAudio` TTS - -机制 1 产品化入口：`record --sample-claude --skill "/my-skill ..."`（外壳写入 Claude Code PTY）。 -另提供 **真工具链引擎**：`record --sample-vhs --engine asciinema-vhs`（`asciinema rec` + Charm `vhs` `.tape`，经 WSL）。 -`native` 引擎不引入 VHS 二进制；`session.cast` 仍可给外部 asciinema/VHS 复用。 - ---- - -## 6. 分阶段交付 - -### Phase 0 — terminal-narrator 包（捕获） - -**目录**：[`packages/terminal-narrator`](packages/terminal-narrator) - -| 文件 | 职责 | -|------|------| -| `types.ts` | `TerminalNarratorPlan`、`TerminalRecordingManifest` | -| `validate.ts` | plan 校验 | -| `record.ts` | node-pty spawn、按 inputs 写 stdin、捕获 stdout 事件 | -| `sample.ts` | Windows cmd 示例 plan | -| `cli.ts` | `record --plan plan.json` / `--sample` | - -**plan.json 契约（v1.0）**： - -```json -{ - "version": "1.0", - "name": "Skill demo", - "viewport": { "width": 1280, "height": 720 }, - "renderFps": 30, - "shell": { "command": "claude", "args": [], "cwd": ".", "cols": 120, "rows": 36 }, - "inputs": [ - { "id": "run-skill", "label": "运行 /my-skill", "text": "/my-skill", "afterMs": 3000 } - ], - "finalWaitMs": 1200, - "timeoutMs": 120000 -} -``` - -**job 输出**： - -- `plan.json` -- `recording-manifest.json`（events、steps、durationMs） -- `composition.tsx`（Phase 1 生成） -- `final.mp4` - -**安全**：manifest 阶段 secret redaction（env 中 token/key 值替换为 `[REDACTED]`）。 - ---- - -### Phase 1 — TerminalDemo 组件（渲染） - -**目录**：[`packages/technical/src/TerminalDemo.tsx`](packages/technical/src/TerminalDemo.tsx) - -- macOS 窗口框（复用 CodeWalkthrough 视觉语言） -- 按帧回放 `events[]`，支持打字机进度 -- `steps[]` 驱动当前步骤高亮卡片 -- 对接 [`TerminalSceneSpec`](packages/core/src/composition-document/schema.ts)（`events` / `steps` 字段） -- [`compose.ts`](packages/terminal-narrator/src/compose.ts) 生成含 `TerminalDemo` 的 TSX -- core 编译器 [`compile.ts`](packages/core/src/composition-document/compile.ts) 增加 `terminal` 分支 - ---- - -### Phase 3 — 旁白与字幕同步 - -**不依赖 MCP**；在 compose 阶段从 manifest 生成 narration/captions： - -```text -stdin 写入时间 → stdout echo / 首行有效输出匹配（refineStepTimings） - → steps[].label + 该步骤窗口内 stdout 摘要 - → meta.audio.narration（startMs/endMs 对齐 refined step 边界） - → meta.audio.captions - → seqvio-render（lockToAudio: true） - → 可选 --withAudio → seqvio-audio synthesize（Edge-TTS / ElevenLabs） -``` - -后续可增强：LLM 改写步骤旁白文案（仍走同一 manifest 时间轴）。 - ---- - -## 7. 实施任务清单 - -| ID | 任务 | 状态 | -|----|------|------| -| P0 | `packages/terminal-narrator`：types / validate / record / cli / sample | 完成 | -| P0 | secret redaction | 完成 | -| P1 | `@seqvio/technical` TerminalDemo 组件 | 完成 | -| P1 | core TerminalSceneSpec + compile 分支 | 完成 | -| P1 | terminal-narrator compose + pipeline + render | 完成 | -| P3 | compose 生成 narration/captions 元数据 | 完成 | -| P3 | `--withAudio` 一键合成旁白并 mux | 完成 | -| P3 | stdout echo 精确 step/caption 时间切分 | 完成 | -| M1 | Claude Code sample + 人类打字 / readyPattern / startupWait | 完成 | -| M1 | asciinema `session.cast` 导出 | 完成 | -| M1 | TerminalDemo VHS 渐变舞台外观 | 完成 | -| P3 | 文档：README + 示例 plan | 完成 | -| — | root `package.json` build 脚本纳入 terminal-narrator | 完成 | -| — | 单元测试（validate / compose / timing / audio / cast） | 完成 | - -**明确不做（本阶段）**： - -- MCP / Skill 对外接口（原 Phase 2） -- 沙盒 git clone + fixture repo 自动化 -- Claude Code 内「@Agent 录演示」一键入口 - ---- - -## 8. 风险与缓解 - -| 风险 | 缓解 | -|------|------| -| 会话耗时长 | 渲染层压缩空白；plan 设 timeoutMs | -| 密钥泄露 | redact + 录制前剥离敏感 env | -| Windows PTY 差异 | sample 用 cmd.exe；shell 可配置 | -| 输出非确定性 | 接受多次 take；fixture 命令做 smoke test | -| node-pty 原生依赖 | `npm install` 时编译；CI 需对应环境 | - ---- - -## 9. 验收标准 - -1. `node packages/terminal-narrator/dist/cli.js record --sample` 在 Windows 产出 `final.mp4` -2. 视频中终端按步骤回放，步骤卡片随时间切换 -3. `composition.tsx` 含 `meta.audio.narration` 与 captions -4. `seqvio-render` 可渲染该 composition（可选 `--audioManifest` _mux 旁白_） -5. 无 MCP 依赖；CLI + plan.json 即可端到端运行 - ---- - -## 10. 与 browser-recorder 对照 - -| | browser-recorder | terminal-narrator | -|--|------------------|-------------------| -| 捕获 | Puppeteer DOM | node-pty TTY | -| manifest | cursor/focus/clicks | events/steps | -| 渲染组件 | RecordedBrowserDemo | TerminalDemo | -| 旁白 | 未内置 | Phase 3 内置 | -| Web UI | 有 | 无（CLI only） | diff --git a/docs/CAPTURE-CLI-CONTRACT.md b/docs/CAPTURE-CLI-CONTRACT.md index fc826cd..6dda06b 100644 --- a/docs/CAPTURE-CLI-CONTRACT.md +++ b/docs/CAPTURE-CLI-CONTRACT.md @@ -1,11 +1,11 @@ # Capture CLI Contract -> Current contract version: `1.0` -> Adapter lifecycle: `pre-stable` +> Current contract version: `2.0` +> Adapter lifecycle: `stable` Terminal and Browser adapters share command, result, exit-code, QA, audio, and -artifact conventions. The contract is versioned while supported-host -verification remains pre-stable. +artifact conventions. The contract is versioned and the supported-host +verification gate is stable for the current release. ## Commands @@ -41,8 +41,8 @@ go to stderr. Success includes: ```json { "ok": true, - "cliContractVersion": "1.0", - "lifecycle": "pre-stable", + "cliContractVersion": "2.0", + "lifecycle": "stable", "adapter": "terminal", "jobId": "demo", "jobDir": ".../demo", @@ -56,32 +56,32 @@ Failures use the same envelope with `ok: false`, a stable `exitCode`, and an ## Exit Codes -| Code | Meaning | -| --- | --- | -| `0` | Success | -| `2` | Invalid command, option, plan, job id, or conflicting input | -| `3` | Capture, compilation, synthesis, server, render, or QA failure | -| `4` | Unexpected CLI-internal failure | +| Code | Meaning | +| ---- | -------------------------------------------------------------- | +| `0` | Success | +| `2` | Invalid command, option, plan, job id, or conflicting input | +| `3` | Capture, compilation, synthesis, server, render, or QA failure | +| `4` | Unexpected CLI-internal failure | ## Job Artifacts Every completed job writes `artifacts.json`. Failed jobs write it after the job directory has been created. Paths inside it are relative to the job directory. -| Artifact | Terminal | Browser | -| --- | --- | --- | -| `plan.json` | yes | yes | -| `recording-manifest.json` | yes | yes | -| `capture-manifest.json` | yes | yes | -| `explainer.json` | yes | yes | -| `composition.tsx` | yes | yes | -| `audio-manifest.json` | yes | yes | +| Artifact | Terminal | Browser | +| ------------------------------ | ---------- | ---------- | +| `plan.json` | yes | yes | +| `recording-manifest.json` | yes | yes | +| `capture-manifest.json` | yes | yes | +| `explainer.json` | yes | yes | +| `composition.tsx` | yes | yes | +| `audio-manifest.json` | yes | yes | | `audio-manifest.resolved.json` | with audio | with audio | -| `session.cast` | yes | no | -| `raw.mp4` | no | yes | -| `final.mp4` | yes | yes | -| `qa-report.json` | yes | yes | -| `artifacts.json` | yes | yes | +| `session.cast` | yes | no | +| `raw.mp4` | no | yes | +| `final.mp4` | yes | yes | +| `qa-report.json` | yes | yes | +| `artifacts.json` | yes | yes | The Browser server exposes the same manifest at `GET /api/jobs//artifacts` after a job completes. @@ -103,9 +103,13 @@ and `message` rather than infer sub-step meaning from percentage alone. ## Current Boundary -Contract `1.0` is implemented and covered by tests. Windows package/CLI host +Contract `2.0` is implemented and covered by tests. Browser plans can declare +selector- or rectangle-based opaque privacy masks that are applied before frame +capture. Required selectors fail closed when they never match; automatic OCR is +not used as a security boundary. Windows package/CLI host verification passes locally; a Windows/Linux/macOS CI matrix verifies clean installation, build, adapter tests, npm package contents, and CLI contract. -Adapters remain pre-stable until that matrix passes on the repository host. -Screenshot OCR/masking is not part of this contract and remains intentionally -deferred; raw Browser capture must be treated as sensitive input. +The configured Windows/Linux/macOS matrix has passed for the current contract; +package lifecycle remains experimental while API/package promotion proceeds. +Raw Browser captures must still be treated as sensitive input and reviewed +before publication. diff --git a/docs/COMPOSITION-AUTHORING.md b/docs/COMPOSITION-AUTHORING.md index 51f346b..84431cd 100644 --- a/docs/COMPOSITION-AUTHORING.md +++ b/docs/COMPOSITION-AUTHORING.md @@ -25,17 +25,13 @@ visual timestamps for the same scene. } ], "explanation": { - "cues": [ - { "id": "voice", "text": "Now the client sends the request." } - ], + "cues": [{ "id": "voice", "text": "Now the client sends the request." }], "beats": [ { "id": "send-request", "cueId": "voice", "anchor": { "text": "sends the request" }, - "visuals": [ - { "targetId": "request-line", "action": "focus" } - ] + "visuals": [{ "targetId": "request-line", "action": "focus" }] } ] } @@ -54,6 +50,12 @@ lower-level contract and does not automatically provide phrase-level visual alignment unless the corresponding ExplanationBeat and scene timing metadata are also present. +Mathematical animation is a two-stage optional path: the experimental +`@seqvio/manim-adapter` invokes the external Python package `manim`, then an +ExplainerDocument `manim` scene or `@seqvio/technical` `ManimClip` consumes the +pre-rendered media. See [`MANIM-INTEGRATION.md`](./MANIM-INTEGRATION.md) for the +runtime boundary, setup, manifest, marker, and cache contracts. + ## Quick start 1. Create a `.tsx` file under `examples/compositions/` or your project. @@ -72,10 +74,10 @@ node dist/cli.js \ ## File contract -| Export | Required | Purpose | -|--------|----------|---------| -| `default` | Yes | Root scene or `VideoComposition` | -| `meta` | Yes | `{ duration: frames, fps: number }` for the renderer | +| Export | Required | Purpose | +| --------- | -------- | ---------------------------------------------------- | +| `default` | Yes | Root scene or `VideoComposition` | +| `meta` | Yes | `{ duration: frames, fps: number }` for the renderer | Imports resolve via esbuild aliases: @@ -127,12 +129,12 @@ Supported transitions: `fade`, `slide`, and `wipe` (defined in `packages/core/sr ## Examples -| Path | Description | -|------|-------------| -| `examples/compositions/seqvio-intro.tsx` | 4-scene product intro | -| `packages/core/examples/multi-scene-demo.tsx` | Scene + fade transition API | -| `packages/whiteboard/examples/01-hello-world.tsx` | Minimal single scene | -| `packages/whiteboard/examples/04-framework-intro.tsx` | Long single-scene intro | +| Path | Description | +| ----------------------------------------------------- | --------------------------- | +| `examples/compositions/seqvio-intro.tsx` | 4-scene product intro | +| `packages/core/examples/multi-scene-demo.tsx` | Scene + fade transition API | +| `packages/whiteboard/examples/01-hello-world.tsx` | Minimal single scene | +| `packages/whiteboard/examples/04-framework-intro.tsx` | Long single-scene intro | ## Whiteboard theme (refined defaults) @@ -165,11 +167,11 @@ With `excalidrawTheme` / `handDrawn`: **Virgil** (Latin) and **Long Cang 龙苍* Props: -| Prop | Values | Notes | -|------|--------|-------| -| `textRender` | `fill` \| `stroke` \| `stroke-wash` | On `DrawText`; default solid fill | -| `type` | `rounded-rectangle` | Rounded corners via `borderRadius` | -| `fillColor` | explicit | Overrides theme wash when set | +| Prop | Values | Notes | +| ------------ | ----------------------------------- | ---------------------------------- | +| `textRender` | `fill` \| `stroke` \| `stroke-wash` | On `DrawText`; default solid fill | +| `type` | `rounded-rectangle` | Rounded corners via `borderRadius` | +| `fillColor` | explicit | Overrides theme wash when set | ## CLI reference diff --git a/docs/EXPLAINER-FRAMEWORK-IMPROVEMENT-PLAN.md b/docs/EXPLAINER-FRAMEWORK-IMPROVEMENT-PLAN.md deleted file mode 100644 index d6aac69..0000000 --- a/docs/EXPLAINER-FRAMEWORK-IMPROVEMENT-PLAN.md +++ /dev/null @@ -1,461 +0,0 @@ -# Seqvio Explainer Framework Improvement Plan - -> **Status:** active implementation plan. -> -> **Purpose:** turn Seqvio from a broad collection of video components into a -> reliable capture-to-explanation framework for coding agents. This document is -> an execution plan, subordinate to [`VISION.md`](./VISION.md) and -> [`ROADMAP.md`](./ROADMAP.md). -> -> **Baseline date:** 2026-08-01. Statements about current behavior were checked -> against the repository at this date; update this document when implementation -> changes invalidate them. - -## 1. Outcome - -Seqvio should provide one dependable path from real system activity to a short, -clear, narrated technical explanation: - -```text -real system activity - -> CaptureSession - -> CaptureManifest - -> ExplainerDocument - -> ExplanationBeat cues + visual actions + capture evidence - -> TTS-resolved semantic time map - -> supported explainer scenes - -> deterministic render - -> executable QA - -> MP4 + machine-readable diagnostics -``` - -The framework succeeds when a coding agent can capture what actually happened, -explain it from captured evidence, render it repeatedly, and detect common video -failures without relying on manual inspection. - -## 2. Current Baseline - -### What is already working - -- `@seqvio/capture` defines a shared `CaptureSession -> CaptureManifest` - contract for terminal and browser sources. -- Terminal and browser production pipelines currently compile through - `ExplainerDocument` before generating TSX. -- `ExplainerDocument` has complete compiler paths for `whiteboard`, `code`, - `diagram`, `terminal`, and `browser` scenes. -- ExplanationBeat authoring joins narration cues, exact phrase anchors, visual - actions, and capture evidence. The compiler creates logical source timing; - post-TTS reflow resolves speech frames and semantic scene time maps. -- Terminal and Browser capture compilers generate ExplanationBeats per recorded - step. Browser recordings now persist exact action start times. -- Baseline/capture QA covers visual layout, pacing, audio health, capture state - and media, unresolved/reversed Beats, and semantic time-map integrity. -- A deterministic 1280x720 release smoke renders and fully decodes both capture - families without network or provider credentials. -- Terminal rendering now has a deterministic xterm-backed state path and - explicit visual controls in IR `renderOptions`. - -### Gaps that block the product promise - -- Terminal and Browser CLI/artifact contract `1.0` is implemented with explicit - experimental package and pre-stable adapter labels. Windows verification - passes locally; Linux/macOS matrix confirmation remains before promotion. -- Terminal and Browser production pipelines now use the shared capture - dispatcher; duplicate `writeComposition` paths and tests have been removed. -- Stable scene types now compile to real components. The former `chat`, `diff`, - and `infographic` types were removed: generic chat playback, generated-code - comparison, and generic panel layouts do not belong in the current capture - contract. A future captured agent-session contract must be designed from real - events rather than reusing the removed `chat` shape. -- Stateful and asynchronous scene components share a framework-wide - prepare/ready/render/dispose contract with per-stage deadlines, stable errors, - browser-stage state, and explicit disposal. Cross-platform semantic goldens - and same-host pixel determinism now run in the three-host CI matrix. -- Screenshot privacy masking/OCR remains intentionally deferred. Text, URL, and - credential-like capture scanning exists but does not make raw screenshots safe. -- Root/stable-train versions, package lifecycle markers, changesets grouping, - local dependency versions, and scene capability documentation are governed by - a checked machine-readable release policy. Release publication remains pending. -- Browser-per-frame rendering is adequate for short videos but lacks a published - performance baseline and regression budget. - -## 3. Product Decisions - -These decisions constrain all phases of this plan. - -1. **Capture-to-explanation is the primary product path.** Hand-authored TSX - remains supported, but new product investment starts from captured evidence. -2. **`ExplainerDocument` is the canonical interchange contract.** Generated - TSX is a compilation target, not a second planning model. -3. **Only complete scene types are public.** A valid document must never silently - produce a placeholder in a release build. -4. **Voice remains the clock.** Scene duration, captions, highlights, and captured - actions derive from one resolved timeline. -5. **Real state outranks reconstructed state.** Capture artifacts retain source - timing and provenance; narration describes captured results rather than the - intended plan. -6. **Determinism is a framework contract.** Every renderable component must be - correct when rendering starts at an arbitrary frame, seeks backward, or - renders the same frame more than once. -7. **The renderer stays replaceable infrastructure.** Work on it must improve - correctness, diagnostics, or necessary local performance, not expand into a - generic effects or studio feature race. -8. **No internal LLM dependency.** A host agent may provide narration or repair - decisions through explicit provider interfaces; core execution remains local - and deterministic. - -## 4. Work Plan - -### Phase 0 - Align the Contract and the Documentation - -**Goal:** make the repository describe one truthful, reachable product path. - -Deliverables: - -- Rewrite the top-level capability map around capture, explanation, render, and - verification; label hand-authored visual styles as supporting capabilities. -- Correct `ROADMAP.md` and `IMPLEMENTATION-PLAN.md` to reflect that terminal and - browser pipelines already compile through IR. -- Define public, experimental, internal, deprecated, and removed lifecycle labels - and apply them consistently to packages and exports. -- Keep terminal/browser composition generation on the shared dispatcher and - reject reintroduction of direct string-writer paths. -- Align root/package versions and add a release check that rejects inconsistent - versions or stale capability markers. -- Keep the implemented canonical Terminal/Browser commands and versioned - `artifacts.json` contract covered by process-level and pipeline tests. - -Implemented as of 2026-08-01: lifecycle/version policy, stable release train, -changesets alignment, five-scene capability registry, agent-authoring filtering, -machine-readable capability snapshot, and CI/release drift verification. - -Exit criteria: - -- Documentation contains no claim that contradicts the current production path. -- A contributor can identify the canonical data flow from `docs/README.md` in one - navigation step. -- Every exported feature has an explicit lifecycle state. -- CI detects version and capability-documentation drift. - -### Phase 1 - Make the IR Honest and Singular - -**Goal:** remove ambiguous authoring paths and ensure every valid IR scene has a -complete, testable result. - -Deliverables: - -- Remove the generic placeholder compiler from production behavior. -- Keep `chat`, `diff`, and `infographic` outside the stable schema. A future - captured-agent-session type requires its own evidence, timing, component, and - QA contract before it can become public. -- Define a scene capability registry containing schema version, compiler support, - required package, lifecycle state, and QA rules. Use it to drive validation and - agent-facing capability descriptions. -- Make capture adapters call the shared - `compileCaptureManifestToExplainerDocument` dispatcher rather than invoking - adapter compilers through parallel orchestration code. -- Persist `CaptureManifest`, `ExplainerDocument`, resolved timeline, generated - TSX, and audio manifest as named artifacts for every pipeline run. -- Treat ExplainerDocument changes as explicitly breaking until the first - stable IR release; do not build migrations for temporary pre-stable documents. -- Keep Storyboard v1 as an isolated whiteboard input while it remains useful. It - is not an ExplainerDocument migration path and creates no compatibility work. - -Exit criteria: - -- Stable validation accepts only scene types that compile without placeholders. -- Terminal and browser use the same dispatcher and artifact layout. -- No production pipeline imports a legacy composition writer. -- Every public schema change has an explicit breaking-release note until the IR - receives a stable compatibility policy. -- Golden IR fixtures compile deterministically to normalized TSX. - -### Phase 2 - Establish Deterministic Playback Contracts - -**Goal:** make stateful terminal, browser, audio, and annotation scenes reliable -under arbitrary frame access. - -Progress as of 2026-08-01: - -- Terminal frame-state tests cover forward, reverse, repeated, and shuffled - requests; renderer readiness now waits for every mounted xterm instance rather - than one global last-writer promise. -- Browser camera, cursor, and click geometry use an exported pure frame-state - resolver with forward/reverse/repeated-order tests. Click resolution is - timestamp-based and independent of manifest array order. -- Code insert/replace line identities are local to each frame calculation rather - than a module-global counter, so repeated and reverse seeks preserve semantic - targets. -- A real Chromium conformance test generates a moving video fixture, renders a - mixed terminal/browser scene in forward, reverse, and repeated frame order, - and enforces PSNR plus significant-pixel thresholds. Missing seekable media now - fails with an explicit browser-runtime diagnostic. -- The core adapter contract now supports `prepare`, `ready`, deterministic - `render({ frame, fps, timeSeconds })`, and `dispose`. Renderer initialization - awaits preparation/readiness, and each requested frame awaits asynchronous - rendering before capture. Legacy `seek()` adapters remain compatible. -- Lifecycle tests cover forward, backward, repeated, and shuffled frame access, - adapter failure wrapping, stage timeouts, and disposal timeouts. Core and - browser runtime deadlines plus diagnostic fields are documented in - `RENDER-LIFECYCLE.md`. -- Missing, corrupt-header, and truncated mid-stream browser media fixtures are - covered. A committed five-frame semantic golden fixes Terminal and Browser - interaction state across operating systems; each host also enforces repeated - and shuffled-frame pixel equivalence and uploads environment-tagged evidence. -- A second Chromium fixture covers Whiteboard, Code, and Diagram scene starts, - narration-expanded local clocks, both sides of transitions, burned captions, - annotations, reverse seeks, and repeated-frame pixels. Annotation measurement - now participates in the readiness barrier and cannot reuse previous-frame DOM - bounds. - -Deliverables: - -- Specify a render lifecycle with `prepare`, `ready`, `render(frame)`, and - `dispose` semantics, including timeout and diagnostic behavior. -- Require scene state to be derived from immutable input plus the requested frame; - caches may improve speed but may not affect output. -- Add a standard readiness barrier for fonts, images, syntax highlighting, - browser video metadata, xterm initialization, and other asynchronous resources. -- Add conformance tests that render frames in sequential, reverse, repeated, and - randomized order and compare pixel hashes or semantic snapshots. -- Define terminal fidelity fixtures for ANSI colors, cursor shape, wrapping, - resizing, carriage return, backspace, alternate-screen behavior, Unicode, and - long-output scrolling. Unsupported control sequences must emit diagnostics. -- Define browser fidelity fixtures for cursor interpolation, clicks, focus zoom, - source-video seeking, viewport scaling, and missing/corrupt media. -- Make captions, narration cues, annotations, and scene actions consume the same - resolved timeline representation. - -Exit criteria: - -- Supported scenes produce equivalent output regardless of frame request order. -- Rendering never captures a frame before declared resources are ready. -- Unsupported terminal/browser input fails clearly or emits a documented - degradation diagnostic; it never fails silently. -- Timeline conformance tests cover scene boundaries, transitions, captions, and - narration locks. - -### Phase 3 - Turn QA and Security into Release Gates - -**Goal:** detect explanation failures and sensitive-data exposure before an MP4 -is accepted. - -Progress as of 2026-08-01: - -- `seqvio-qa` exposes `baseline` and `capture` profiles. The capture profile - requires a manifest and fails before Chromium startup when steps, observed - state, timestamps, viewport data, or local browser media are invalid. -- Capture security scanning reports private-key material, provider tokens, - secret-like assignments, and sensitive URL parameters with stable source - paths. Terminal text and browser URL coverage are tested; screenshots are not - yet OCR-scanned or masked. -- Audio/caption diagnostics have stable codes for invalid or overlapping timing, - missing files, missing resolved narration, cues outside the composition, mostly - silent narration, excessive leading/trailing silence, and clipping risk. -- Browser runtime failures are classified as metadata-load or seek failures and - are written to `qa-report.json`; Chromium fixtures cover missing, corrupt, and - truncated media. -- Capture action density, pixel-identical sampled frames, and per-highlight - perceptual duration emit temporal diagnostics. -- Shared pacing policy now budgets Chinese/English narration, spreads authored - code/diagram steps to a minimum readable window, resolves synthesized cue ends - from probed audio, and emits `speech_rate_*` / `highlight_too_short` QA codes. - Captured terminal/browser timestamps remain immutable and are diagnosed rather - than rewritten. -- Resolved audio manifests retain scene-local timing. After TTS, narration cues, - track offsets, captions, scene durations, later scene starts, and global - highlight windows are reflowed from probed audio durations. Browser media seek - uses the reflowed scene-local frame rather than the composition-global frame. -- Scene extension no longer leaves the authored visual clock unchanged with only - a static tail. Each scene retains its source duration and receives a monotonic - output-to-source time map. ExplanationBeats resolve exact normalized phrases - inside TTS chunks; whole-cue character position is the explicit lower-confidence - fallback when fine timing is unavailable. Sequential chunk/highlight pairing - remains only for manifests without semantic Beats. React frame hooks, GSAP - adapters, browser media seeking, and pacing QA use this shared clock. - `scene_time_stretch_excessive` warns when narration stretches a scene beyond - the `explainer-v1` 2x threshold. -- QA rejects post-TTS unresolved anchors, reversed Beat timelines, unknown cue or - scene references, and invalid source/output frames. Low-confidence cue-character - alignment remains a promotable warning with a source repair instruction. -- `explainer-v1` is now a versioned pacing profile carried by the source IR, - generated meta, resolved audio, and QA report. `--qaConfig` supports exact - warning suppressions with mandatory reasons; errors cannot be suppressed and - stale suppressions emit `unused_qa_suppression`. -- Raw artifact retention, cue/audio duration tolerance, and configurable warning - promotion are implemented in the renderer QA CLI. Screenshot masking remains - intentionally deferred. -- A deterministic release smoke gate now covers both terminal and browser - capture manifests -> capture compiler -> `ExplainerDocument` -> generated - TSX -> locally generated narration -> scene reflow -> capture QA -> MP4 render - -> full FFmpeg decode. The browser fixture additionally exercises local media - validation and time-mapped video seeking. CI and the npm release workflow run - `npm run smoke:release-pipeline`; it has no provider/network dependency and - cleans each repository-local temporary job after completion. Passing - `--outDir output/release-pipeline-preview` retains both MP4s and their QA/audio - sidecars for human inspection. -- Terminal and Browser production commands now run capture QA after rendering. - `qa-report.json` is a standard artifact; QA errors retain diagnostics, mark - the job failed, and return exit code 3. Explicitly silent jobs do not require - a narration track, while `--withAudio` jobs do. - -Deliverables: - -- Keep existing blank-frame, overflow, font-size, contrast, and offscreen checks, - and publish their thresholds as a versioned QA profile. -- Add temporal checks for scenes with no meaningful visual change, highlights too - short to perceive, excessive action density, and captions outside scene bounds. -- Add audio checks for missing expected narration, excessive leading/trailing - silence, clipping, invalid duration, and cue/audio length mismatch. -- Add capture checks for missing steps, non-monotonic timestamps, media duration - mismatch, invalid viewport data, and absent captured state used by narration. -- Define a common redaction policy with allow/deny patterns, environment-variable - and credential detectors, URL/query handling, screenshot masking hooks, and a - machine-readable redaction report. -- Treat raw capture artifacts as sensitive by default. Document retention, - cleanup, and opt-in debug preservation behavior. -- Produce one diagnostics file with stable codes, severity, source artifact, - scene/frame/time location, and suggested repair. -- Add `--ci` profiles: `baseline` for hand-authored compositions and `capture` - for evidence-derived explanations. Errors exit non-zero; warnings remain - configurable. - -Exit criteria: - -- Deliberately broken fixtures trigger every required QA rule. -- A capture render cannot pass the release profile when narration/media is - missing, timestamps are invalid, or a high-confidence secret is detected. -- Diagnostics identify a repairable source location instead of only a rendered - frame. -- Redaction behavior is tested for both terminal text and browser artifacts. - -### Phase 4 - Stabilize Performance, Packaging, and Adoption - -**Goal:** make the reliable path fast enough and simple enough for routine agent -use without expanding Seqvio into a general video platform. - -Deliverables: - -- **Done:** create generated terminal, browser, code, and mixed-scene benchmark - compositions at 1280x720/30 fps with no network dependency. -- **Done:** record render factor, process-tree peak RSS, preparation time, cache - hit rate, throughput, and output size in three-sample local baselines and a - scheduled CI artifact. -- Profile before optimizing. Prioritize static-layer reuse, decoded media reuse, - chapter caching, and avoiding redundant browser work where determinism is - preserved. -- **Done:** define an initial compatible-host performance budget from the first - three-sample Windows reference. Different hardware reports metrics without - producing a false hard failure. -- Validate installation and smoke rendering on supported Windows, macOS, and - Linux environments, including Chromium, FFmpeg, fonts, and `node-pty`. - A three-host CI matrix now performs clean install/build/tests/package checks - and real Terminal plus Browser direct-CLI capture/QA runtime smokes. Windows - package/CLI verification also passes locally; matrix confirmation is pending. -- **Done:** provide `seqvio-doctor` / `npm run doctor` for missing dependencies, - incompatible Node versions, writable paths, an FFmpeg media probe, a real - Chromium launch, `node-pty`, and bundled-font availability. -- Promote terminal and browser capture out of pre-stable only after their Phase - 1-3 gates pass on every supported platform. - -Exit criteria: - -- Benchmark results are reproducible and stored with environment metadata. -- Performance regressions are visible before release. -- A clean supported machine can run the canonical capture-to-video workflow from - documented prerequisites without manual source edits. -- Terminal and browser packages have stable CLI contracts and compatibility - notes. - -## 5. Cross-Cutting Test Matrix - -Each phase must extend a shared matrix rather than adding isolated package tests. - -| Layer | Required tests | -| --- | --- | -| Capture contract | schema validation, timestamp ordering, artifact paths, cancellation, failure propagation | -| Adapter | real or hermetic terminal/browser fixture, redaction, cross-platform behavior | -| IR | stable validation, deterministic compilation, unsupported capability rejection | -| Scene runtime | random-access frames, reverse seek, repeated frame, asynchronous readiness, missing assets | -| Timeline/audio | narration lock, captions, transitions, silence, media-duration mismatch | -| Visual output | golden frames at scene start/middle/end and interaction boundaries | -| End to end | capture -> document -> TSX -> QA -> MP4, including failure fixtures | - -Unit tests are necessary but do not satisfy a phase exit criterion when the risk -is visible only in Chromium or the encoded video. - -## 6. Success Metrics - -Use a small set of product-level metrics rather than package activity counts. - -- **Pipeline success rate:** percentage of reference capture jobs that produce a - valid MP4 and diagnostics without manual source changes. -- **Deterministic frame rate:** percentage of sampled frames whose hashes match - across repeated and shuffled renders on the same reference environment. -- **QA escape rate:** known invalid fixtures that pass the release QA profile. -- **Capture fidelity:** percentage of recorded steps represented at the correct - time and with preserved observable result. -- **Sensitive-data escape rate:** seeded credentials or private URL data not - detected or redacted by the capture profile. -- **Render factor:** render wall time divided by output duration, tracked by scene - family and resolution. -- **Time to first successful explanation:** elapsed setup and execution time on a - clean supported environment. - -Numeric release thresholds should be set from the first reproducible baseline. -Correctness and security metrics should trend to zero escapes; performance goals -must be evidence-based rather than aspirational. - -## 7. Dependencies and Order - -```text -Phase 0: truthful contract and docs - -> Phase 1: singular IR and supported capabilities - -> Phase 2: deterministic runtime - -> Phase 3: QA and security gates - -> Phase 4: stable promotion and optimization -``` - -Performance benchmarking may begin during Phase 1, but optimization must not -precede deterministic behavior. New capture sources such as CI runs and traces -should begin only after terminal and browser satisfy the common Phase 1-3 -contracts; otherwise each adapter will create another special-case pipeline. - -## 8. Risks and Controls - -| Risk | Control | -| --- | --- | -| Duplicate capture paths return | Canonical artifact tests and one documented shared-dispatcher path | -| IR becomes a second programming language | Keep it declarative, capability-based, and limited to explainer semantics | -| Visual tests become platform-noisy | Pin fonts/browser, record environment metadata, separate semantic and pixel assertions | -| Security checks create false confidence | Treat reports as defense in depth; sensitive raw artifacts remain protected by default | -| QA rules reject intentional designs | Versioned profiles, stable diagnostics, narrowly scoped suppressions with reasons | -| Renderer optimization changes output | Require determinism and golden-frame checks for every optimization | -| Scope expands into generic video tooling | Reject work that does not improve capture fidelity, explanation clarity, verification, or required local execution | - -## 9. Explicit Non-Goals - -- A nonlinear editor or studio timeline. -- Cloud or distributed rendering. -- A larger catalog of visual styles without a capture/explanation requirement. -- Arbitrary VFX, photorealistic generation, or general post-production. -- An internal planner or mandatory model provider. -- Feature parity with general-purpose code-to-video engines. -- New capture adapters before the shared terminal/browser contract is stable. -- Pre-stable ExplainerDocument migrations or compatibility fixtures. - -## 10. First Implementation Slice - -The first slice should be small enough to review as one coherent change and -should establish the direction of later work: - -1. Correct stale pipeline statements in planning documentation. **Done.** -2. Add lifecycle labels and a supported-scene capability registry/table. **Done.** -3. Make stable validation reject placeholder-only scene types. **Done.** -4. Route terminal and browser pipelines through the shared capture dispatcher. **Done.** -5. Remove legacy `writeComposition` exports and replace their tests with canonical artifact tests. **Done.** -6. Add one end-to-end terminal fixture and one browser fixture that preserve all - intermediate artifacts and run the existing QA profile. **Done.** - -Completion of this slice does not promote capture out of pre-stable. It makes -the architecture honest and provides the test spine required for that promotion. diff --git a/docs/IMPLEMENTATION-PLAN.md b/docs/IMPLEMENTATION-PLAN.md index 5f46708..3603ea5 100644 --- a/docs/IMPLEMENTATION-PLAN.md +++ b/docs/IMPLEMENTATION-PLAN.md @@ -1,27 +1,27 @@ # Seqvio Implementation Plan > **Status:** the task-level companion to [`ROADMAP.md`](./ROADMAP.md). ROADMAP -> states *what we are betting on and in what order*; this document states *how -> each phase breaks into concrete work against the current codebase*. For +> states _what we are betting on and in what order_; this document states _how +> each phase breaks into concrete work against the current codebase_. For > positioning and scope, [`VISION.md`](./VISION.md) wins. > -> Last revised: 2026-08-01. +> Last revised: 2026-08-06. ## Current-State Inventory (what is already there) -| Capability | Current state | Disposition | -| --- | --- | --- | -| ExplainerDocument IR | Five complete scene families plus `ExplanationBeat` cues, phrase anchors, visual actions, capture evidence, validation, compilation, and pacing | Canonical interchange contract | -| chapter-render | `renderer/chapter-render.ts`: `hashRenderSettings`, `resume`, `onlyChapters`, `changedChapterIds`, `documentPath` | Reuse; incremental render built | -| render conformance | Cross-platform semantic golden plus same-host PNG hash/PSNR checks for mixed Terminal/Browser frames | Three-host CI gate with environment-tagged artifacts | -| seqvio-qa | Baseline/capture profiles cover visual, pacing, audio, media, capture-manifest, and resolved ExplanationBeat failures | Screenshot privacy masking intentionally deferred | -| `@seqvio/capture` | New: `CaptureSession` contract, `CaptureManifest` union, `compileCaptureManifestToExplainerDocument` dispatcher | Built (Phase 1.1) | -| Release/capability governance | `seqvio.release-policy.json`, package lifecycle metadata, core scene registry, docs snapshot, changesets/CI drift verifier | Built; release publication pending | -| terminal-narrator | `node-pty` + asciinema/xterm state; compiler emits capture-backed cues/Beats and audio scene timing | Production pipeline uses shared dispatcher -> IR -> TSX; legacy writer removed | -| browser-recorder | Records exact action clocks; compiler emits BrowserSceneSpec plus capture-backed cues/Beats | Production pipeline uses shared dispatcher -> IR -> TSX; legacy writer removed | -| technical components | `CodeWalkthrough{source,steps}`, `ArchitectureDiagram`, `TerminalDemo`, `ansi.ts`, `code-utils.ts` | Reuse | -| render performance | Four generated 1280x720 workloads, three-sample environment baseline, structured cache metrics, and scheduled reports | Baseline built; optimize Browser first | -| environment diagnostics | `seqvio-doctor` checks Node, Chromium launch, FFmpeg probe, bundled fonts, `node-pty`, and writable paths | Built | +| Capability | Current state | Disposition | +| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | +| ExplainerDocument IR | Five complete scene families plus `ExplanationBeat` cues, phrase anchors, visual actions, capture evidence, validation, compilation, and pacing | Canonical interchange contract | +| chapter-render | `renderer/chapter-render.ts`: `hashRenderSettings`, `resume`, `onlyChapters`, `changedChapterIds`, `documentPath` | Reuse; incremental render built | +| render conformance | Cross-platform semantic golden plus same-host PNG hash/PSNR checks for mixed Terminal/Browser frames | Three-host CI gate with environment-tagged artifacts | +| seqvio-qa | Baseline/capture profiles cover visual, pacing, audio, media, capture-manifest, resolved ExplanationBeat failures, and deterministic browser privacy masks | OCR is intentionally outside the security boundary | +| `@seqvio/capture` | New: `CaptureSession` contract, `CaptureManifest` union, `compileCaptureManifestToExplainerDocument` dispatcher | Built (Phase 1.1) | +| Release/capability governance | `seqvio.release-policy.json`, package lifecycle metadata, core scene registry, docs snapshot, changesets/CI drift verifier | Built; release publication pending | +| terminal-narrator | `node-pty` + asciinema/xterm state; compiler emits capture-backed cues/Beats and audio scene timing | Production pipeline uses shared dispatcher -> IR -> TSX; legacy writer removed | +| browser-recorder | Records exact action clocks; compiler emits BrowserSceneSpec plus capture-backed cues/Beats | Production pipeline uses shared dispatcher -> IR -> TSX; legacy writer removed | +| technical components | `CodeWalkthrough{source,steps}`, `ArchitectureDiagram`, `TerminalDemo`, `ansi.ts`, `code-utils.ts` | Reuse | +| render performance | Four generated 1280x720 workloads, three-sample environment baseline, structured cache metrics, and scheduled reports | Baseline built; optimize Browser first | +| environment diagnostics | `seqvio-doctor` checks Node, Chromium launch, FFmpeg probe, bundled fonts, `node-pty`, and writable paths | Built | ## Architectural Decisions @@ -43,6 +43,16 @@ 3. **No pre-stable IR migration program.** ExplainerDocument changes may be explicitly breaking until a stable compatibility policy is declared. Storyboard v1 remains a separate whiteboard input, not a migration obligation. +4. **DirectionPlan is a derived semantic plan, not a second source of truth.** + ExplainerDocument owns scenes, stable target ids, narration cues, and + ExplanationBeats. DirectionPlan references those ids and adds renderer- + agnostic purpose, focus, camera, pace, and transition intent before timing + resolution. It remains optional during the first rollout and is inspectable + as a generated artifact. +5. **Style Playbook is the final layer.** Style profiles are applied only after + semantic graphics, attention, direction, external media, motion grammar, + and QA contracts are stable. Style cannot change meaning, target ids, + narration timing, or evidence order. ## Phase 0 - Clear the Floor and Stop Duplicating the Closed Layer @@ -60,10 +70,12 @@ README/README.zh-CN, package READMEs, agent skill references, authoring docs, VISION/ROADMAP, and this plan are aligned with the ExplanationBeat pipeline. -### 0.3 Test baseline - DEFERRED +### 0.3 Test baseline - DONE (2026-08-06) -Characterization tests for `chapter-render`/`qa-cli`/`terminal-narrator` to be -added when those are refactored (Phase 1.2/1.3 pipeline migration). +Characterization coverage now exists for `chapter-render`, QA diagnostics, and +the terminal/browser adapter contracts. Adapter tests cover validation, +compilation, timing, CLI envelopes, and capture-session behavior after the +Phase 1.2/1.3 pipeline migration. ## Phase 1 - System Capture Adapters @@ -77,14 +89,15 @@ added when those are refactored (Phase 1.2/1.3 pipeline migration). `compileTerminalCapture` (manifest -> `TerminalSceneSpec` IR + audio manifest, ports compose.ts timing logic) + `terminalCaptureSession` (CaptureSession impl) -+ `toCaptureManifest`. The production pipeline compiles manifest -> IR -> TSX, -including visual control through `TerminalSceneSpec.renderOptions`. Each captured -step emits a cue, a phrase anchor, visual focus, and capture evidence. The -production pipeline now uses the shared dispatcher and the legacy writer is -removed. CLI contract `1.0` fixes JSON results, exit codes, progress, safe job -ids, artifact layout, per-job capture QA, and independent audio/caption options. -Windows package/CLI verification passes locally; the three-host CI matrix is -configured and must pass before promotion. + +- `toCaptureManifest`. The production pipeline compiles manifest -> IR -> TSX, + including visual control through `TerminalSceneSpec.renderOptions`. Each captured + step emits a cue, a phrase anchor, visual focus, and capture evidence. The + production pipeline now uses the shared dispatcher and the legacy writer is + removed. CLI contract `2.0` fixes JSON results, exit codes, progress, safe job + ids, artifact layout, per-job capture QA, and independent audio/caption options. + Windows package/CLI verification passes locally; the three-host CI matrix + passes for the current contract. ### 1.3 browser-recorder - DONE (core) @@ -93,69 +106,71 @@ configured and must pass before promotion. compiles manifest -> IR -> TSX. New recordings retain exact per-action start times instead of evenly distributing steps; older recording manifests keep the fallback. The production pipeline now uses the shared dispatcher and the legacy -writer is removed. CLI contract `1.0` adds direct plan execution, JSON results, -exit codes, progress, safe job ids, and artifact layout. Remaining: -per-job capture QA and audio parity. Windows package/CLI verification passes -locally; the three-host CI matrix is configured and must pass before promotion. +writer is removed. CLI contract `2.0` adds direct plan execution, JSON results, +exit codes, progress, safe job ids, and artifact layout. Per-job capture QA, + audio parity, and privacy masking are implemented. Windows package/CLI + verification passes locally; the three-host CI matrix passes for the current + contract. -### 1.5 Promote out of pre-stable (in progress) +### 1.5 Promote out of pre-stable - DONE (2026-08-06) README/skill/current-capability docs now describe the working IR path and its -pre-stable CLI status. Shared dispatcher routing and legacy writer removal are -complete. CLI contract `1.0`, per-job QA, audio parity, and independent caption -burn-in are also complete. Windows host verification passes locally; Linux and -macOS execution remains pending the configured CI matrix. Screenshot privacy -remains an explicitly deferred boundary. +stable adapter contract. Shared dispatcher routing and legacy writer removal are +complete. CLI contract `2.0`, per-job QA, audio parity, independent caption +burn-in, and deterministic selector/rectangle privacy masking are complete. The +Windows/Linux/macOS matrix passes for the current contract; OCR remains an +explicitly deferred boundary. ## Phase 2 - Generic QA Checks -### 2.1 Generic checks - DONE (partial) +### 2.1 Generic checks - DONE `seqvio-qa` checks: blank/empty/offscreen + text-overflow + font-size (12px) + contrast (WCAG AA 4.5:1). Deterministic, no LLM, `exit(1)` on error. The capture profile additionally validates manifest timing/state/media, credential-like content, narration/caption timing, narration-track presence, audio silence and clipping risk, and sampled visual change. Missing/corrupt/truncated browser media - has Chromium coverage. Speech-rate and per-highlight perceptual duration checks - are implemented; screenshot masking/OCR is intentionally deferred. Renderer QA - now retains failure artifacts, checks cue/audio duration tolerance, and supports - configurable warning promotion. - A shared core pacing policy is also used by agent authoring guidance, - ExplainerDocument timing resolution, synthesized narration retiming, and QA - speech-rate/highlight diagnostics. - TTS resolution now performs a full scene-aware timeline reflow and `seqvio-qa` - accepts `--audioManifest` so final QA evaluates the same resolved timing used - by rendering. - Reflow also preserves each authored/captured source duration and supplies a - monotonic scene-local time map. ExplanationBeats resolve exact normalized - phrases inside TTS chunks; providers without fine chunks use a lower-confidence - whole-cue character position. Chunk-order/highlight pairing remains only a - legacy fallback when semantic Beats are absent. React - frame hooks, GSAP adapters, browser media seeking, and final pacing QA consume - the same mapping. Stretch beyond the profile's 2x limit emits - `scene_time_stretch_excessive`. - `npm run smoke:release-pipeline` now exercises terminal and browser - `CaptureManifest` paths through the capture dispatcher, IR/TSX compilation, - deterministic local narration, scene reflow, capture-profile QA, MP4 rendering, - and full FFmpeg decode. The browser case also validates the captured local - video and mapped media seeking. It requires no network or TTS credentials and - removes each repository-local temporary job directory after completion. CI and - the npm release workflow run this combined gate; adapter-specific commands are - available as `smoke:release-pipeline:terminal` and - `smoke:release-pipeline:browser`. CI and release gates render at the default - 1280x720 resolution. For human review, run - `node scripts/release-pipeline-smoke.mjs --outDir output/release-pipeline-preview` - to retain the 720p MP4s and QA sidecars; the smoke gate also checks decoded - video frame count, not only container validity. `--width` and `--height` - remain available for explicit diagnostic variants. - The release contract now records `explainer-v1` end to end and accepts a - versioned `--qaConfig`; suppressions require an exact code/path and documented - reason, never apply to errors, and remain auditable in `qa-report.json`. - Terminal and Browser production jobs now run this capture profile after - rendering and include `qa-report.json` in `artifacts.json`. QA errors return - pipeline exit code 3 while retaining diagnostic artifacts. Explicitly silent - jobs still run capture/visual/pacing checks without requiring an audio track; - `--withAudio` jobs must contain valid synthesized narration. +has Chromium coverage. Speech-rate and per-highlight perceptual duration checks +are implemented; deterministic pre-capture screenshot masking is enforced for +Browser plans, while OCR remains intentionally deferred. Renderer QA +now retains failure artifacts, checks cue/audio duration tolerance, and supports +configurable warning promotion. +A shared core pacing policy is also used by agent authoring guidance, +ExplainerDocument timing resolution, synthesized narration retiming, and QA +speech-rate/highlight diagnostics. +TTS resolution now performs a full scene-aware timeline reflow and `seqvio-qa` +accepts `--audioManifest` so final QA evaluates the same resolved timing used +by rendering. +Reflow also preserves each authored/captured source duration and supplies a +monotonic scene-local time map. ExplanationBeats resolve exact normalized +phrases inside TTS chunks; providers without fine chunks use a lower-confidence +whole-cue character position. Chunk-order/highlight pairing remains only a +legacy fallback when semantic Beats are absent. React +frame hooks, GSAP adapters, browser media seeking, and final pacing QA consume +the same mapping. Stretch beyond the profile's 2x limit emits +`scene_time_stretch_excessive`. +`npm run smoke:release-pipeline` now exercises terminal and browser +`CaptureManifest` paths through the capture dispatcher, IR/TSX compilation, +deterministic local narration, scene reflow, capture-profile QA, MP4 rendering, +and full FFmpeg decode. The browser case also validates the captured local +video and mapped media seeking. It requires no network or TTS credentials and +removes each repository-local temporary job directory after completion. CI and +the npm release workflow run this combined gate; adapter-specific commands are +available as `smoke:release-pipeline:terminal` and +`smoke:release-pipeline:browser`. CI and release gates render at the default +1280x720 resolution. For human review, run +`node scripts/release-pipeline-smoke.mjs --outDir output/release-pipeline-preview` +to retain the 720p MP4s and QA sidecars; the smoke gate also checks decoded +video frame count, not only container validity. `--width` and `--height` +remain available for explicit diagnostic variants. +The release contract now records `explainer-v1` end to end and accepts a +versioned `--qaConfig`; suppressions require an exact code/path and documented +reason, never apply to errors, and remain auditable in `qa-report.json`. +Terminal and Browser production jobs now run this capture profile after +rendering and include `qa-report.json` in `artifacts.json`. QA errors return +pipeline exit code 3 while retaining diagnostic artifacts. Explicitly silent +jobs still run capture/visual/pacing checks without requiring an audio track; +`--withAudio` jobs must contain valid synthesized narration. ### Ground-truth verification - DROPPED @@ -166,6 +181,328 @@ no ground truth (whiteboard) or was dropped (code/diagram char-level checks conflate simplification with fabrication). See git history for the reverted work. +### 2.2 Product explainer production contract + +**Status: complete for the v1 production contract.** Editorial and visual +artifacts now carry the optional planning fields, and product-explainer DOM +markers activate focused text-density, template, overlap, and focal-target QA. +The retained contract validation composition passes reference-frame review. + +The first refined product video exposed recurring production risks that should +be executable rather than left as author preference: + +- Add optional planning fields for `hook`, `visualRole`, `focalTarget`, + `evidenceSource`, `onScreenTextBudget`, and `transitionIntent` to the + editorial and visual-design artifacts. +- Add QA diagnostics for repeated full-sentence overlays, excessive concurrent + primary text, repeated header/rail templates, target collisions, + title-to-graphic overlap, and scenes without a declared focal target. +- Define a scene-density policy: narration carries complete sentences; screen + text carries labels, keywords, commands, filenames, and short conclusions. + Code, terminal, and browser evidence are explicit exceptions. +- Make reference-frame review standard: capture scene starts, midpoints, + boundaries, and the final frame, retaining a contact sheet or frame index for + repair loops. +- Treat real terminal/browser output and captured media as evidence-bearing + sources. Authored diagrams may explain a model but must not imply invented + reconstruction is recorded evidence. +- Prefer focus transfer, match-object, blur crossfade, or deliberate cut; + reject decorative wipes, continuous rotation, and transitions that compete + with the explanation. +- Keep voice providers behind the resolved audio manifest. CosyVoice and other + providers may change the audio source, but the same measured voice clock, + phrase anchors, reflow, and QA contract must drive every render. +- Make brand assets explicit and transparent-background safe. Logo presence, + dimensions, contrast, and crop behavior belong in media QA. + +Exit criteria: + +- A narrated reference composition can be planned without duplicating its full + script as on-screen text. +- QA reports text-density, focal-target, collision, and template-repetition + diagnostics with stable source paths and frame numbers. +- A different resolved voice manifest does not change semantic beat order or + visual evidence order. + +## Phase 3 - Semantic Direction and Mathematical Animation + +### 3.1 `InfographicScene` semantic graphics + +**Status: complete for v1 semantic graphics.** The technical package now +exposes the shared metric, comparison, process, timeline, and relationship item +contracts with stable annotation targets and seek-safe reveal timing. +ExplainerDocument schema, capability registration, validation, and compiler +registration are wired. Bar and line charts now include addressable series, +axes, ticks, legends, units, source labels, deterministic domains, and shared +attention targets. The retained chart reference passes render QA. + +Every completed implementation increment must produce a retained verification +video under `output/`, together with its source composition, manifest, and at +least one representative frame. The artifact is part of the exit evidence for +the increment, not an optional demo. + +- Add `InfographicSceneSpec` to ExplainerDocument and register the capability in + the core scene registry. +- Implement `InfographicScene` in `@seqvio/technical` with metric, comparison, + process, timeline, chart, and relationship primitives. +- Give every datum, series, process step, and relationship a stable target id + and semantic role. +- Add `reveal`, `compare`, `trace`, `emphasize`, and `transform` visual actions + to the ExplanationBeat compiler and timing resolver. +- Provide deterministic layouts, number and unit formatting, axis and legend + helpers, data-source labels, responsive safe areas, and reduced-density rules. +- Make all animation states frame-derived and stable under direct seek, repeated + frames, reverse seek, and chapter rendering. +- Add schema, compiler, runtime, golden-frame, and end-to-end render tests. + +Exit criteria: + +- One IR document can compile and render metric, comparison, process, timeline, + and chart scenes without hand-authored pixel animation. +- Phrase anchors activate the intended data targets and semantic transitions at + the resolved audio time. +- Reference compositions pass layout, contrast, overflow, pacing, and + deterministic-frame QA at supported aspect ratios. + +### 3.2 Cross-scene attention primitives + +**Status: complete for v1 attention layout and routing.** The core package now +provides `AttentionSequenceItem`, minimum-hold and handoff metadata, a +deterministic resolver, and a renderer layer that reuses stable annotation +targets. ExplanationBeat `focus`, `highlight`, and `annotate` actions now +compile into attention items with automatic handoffs and source Beat identity. +Resolved audio `outputFrame` values reflow the same sequence through the scene +time map, with source-frame fallback when phrase resolution is unavailable. +One versioned sequence can now carry scene-owned segments, cross-scene handoff +metadata, timed/until-handoff/until-clear persistence, explicit clear frames, +and validation for invalid clears or unknown destinations. Broader primitive +kinds now include focus ring, callout, bracket, region shade, and a two-target +connector with stable endpoint validation. Guided path routing, safe-area +placement, orthogonal connector corridors, deterministic label candidates, and +title-safe callout placement are now available. Multi-label placement shares a +single deterministic layout pass, while connectors and guided paths score +occlusion-aware candidate routes. Renderer QA now reports missing +targets, offscreen labels, label collisions, and target occlusion with stable +annotation ids and frame numbers. Retained 16:9, 1:1, and 9:16 references pass QA. + +- Extend `AnnotationProvider` and `AnnotationLayer` with highlight, focus ring, + spotlight, callout, arrow, bracket, connector, region shade, and guided focus + path primitives. +- Add an `AttentionSequence` contract for activation, handoff, persistence, + priority, and clearing. +- Reuse `AnnotationTarget` ids across whiteboard, code, diagram, terminal, + browser, product-demo, infographic, and external-animation scene components. +- Add safe-area placement, connector routing, label collision resolution, + target visibility checks, and a deterministic stacking policy. +- Map ExplanationBeat attention actions to the shared sequence runtime and + expose the same contract to hand-authored TSX. +- Add QA diagnostics for unknown targets, occluded labels, offscreen geometry, + insufficient contrast, overlapping callouts, and short attention holds. + The first geometry diagnostics are shipped and exercised by + `attention-routing-validation.tsx`. +- Add interaction-boundary golden frames plus random-access and reverse-seek + tests for every primitive. + +Exit criteria: + +- A single attention sequence can guide focus across targets owned by different + scene packages without scene-specific overlay code. +- Attention geometry remains readable at 16:9, 9:16, and 1:1 output sizes. +- ExplanationBeat timing, annotations, captions, and scene actions remain + synchronized after TTS timeline reflow. + +### 3.3 Minimal `DirectionPlan` + +**Status: complete.** +The core package now derives a renderer-agnostic `DirectionPlan` sidecar from +scene and ExplanationBeat ids, preserves capture-step references, validates +scene, target, beat, capture, and transition references, and compiles into scene +actions, attention entries, and timing hints. Generated TSX exposes the compiled +result through `meta.direction`, and `npm run direction:generate` writes a +reviewable JSON sidecar without renderer properties. +Cross-scene transitions now pair source and destination target ids, conflicting +focus/camera intents produce stable diagnostics, and semantic plans remain +identical across duration and chapter reflow. + +- Add a versioned, renderer-agnostic `DirectionPlan` schema that references + existing scene ids, target ids, ExplanationBeats, and capture evidence. +- Support only the first useful intent vocabulary: `purpose`, `pace`, `focus`, + `camera`, and `transition`. +- Define explicit camera intents such as `overview`, `follow-target`, + `focus-transfer`, and `hold`; define transition intents such as `cut`, + `crossfade`, `focus-transfer`, and `match-object`. +- Compile DirectionPlan into existing scene actions, AttentionSequence entries, + and timing hints. Do not expose `opacity`, `scale`, `translate`, or renderer + APIs in the contract. +- Validate unknown targets, invalid scene references, unsupported intent values, + conflicting focus paths, and transitions without a shared target. +- Keep DirectionPlan optional for hand-authored TSX and emit it as an + inspectable sidecar for agent-generated compositions. + +Exit criteria: + +- One ExplainerDocument can produce a DirectionPlan that a human can review + without reading renderer code. +- The same plan compiles deterministically to the current TSX renderer and + remains stable under audio reflow and chapter rendering. + +### 3.4 External `@seqvio/manim-adapter` + +**Status: complete for the v1 external adapter.** The +package exposes `ManimSceneSpec`, `ManimRenderManifest`, deterministic command +construction, validation, and machine-readable Python/Manim capability checks. +Execution now supports structured progress, cancellation, timeout, retained +logs, source/runtime/asset content hashes, and validated cache lookup. Actual +rendering remains environment-dependent and is reported explicitly. `ManimClip` is now +available in `@seqvio/technical` with deterministic media seeking and named +marker targets. The experimental `manim` ExplainerDocument scene is registered, +validated, and compiled into `ManimClip`. Named markers can reference a local +ExplanationBeat and reflow from authored `sourceFrame` to resolved TTS +`outputFrame`. The first real Manim equation fixture now renders through the +adapter and the ExplainerDocument pipeline, with a valid cached second run and +machine-readable media probe. Real equation, graph, and proof fixtures are +retained, `seqvio-doctor` detects the optional local environment, and QA checks +seek and named-marker alignment. A CosyVoice-narrated graph/proof composition +passes the end-to-end render and QA loop. The geometric proof fixture renders +at its declared 1280x720/30fps contract, records opaque alpha, and reuses the +same content-addressed result on a second adapter run. + +- Create `@seqvio/manim-adapter` with versioned `ManimSceneSpec` and + `ManimRenderManifest` schemas. +- Add local Python and Manim preflight checks with machine-readable version and + capability diagnostics. +- Implement process execution with structured progress, cancellation, timeout, + logs, and repository-local temporary artifacts. +- Content-address renders from source, scene class, assets, render settings, + Python version, and Manim version; reuse validated cached outputs. +- Record rendered media path, duration, frame rate, dimensions, alpha mode, + source hash, runtime versions, and named timeline markers in the manifest. +- Add `ManimClip` to `@seqvio/technical` and compile Manim-backed scenes to the + existing media and semantic-time-map runtime. +- Resolve named Manim markers against narration phrase anchors so + ExplanationBeats can synchronize equations, graphs, geometry, and shared + attention primitives. +- Extend `seqvio-doctor` and `seqvio-qa` with adapter availability, media decode, + duration, dimensions, frame-rate, alpha, marker, and seek checks. +- Ship equation-derivation, graph-transformation, and geometric-proof fixtures + plus one narrated end-to-end composition. + +Exit criteria: + +- A clean supported environment with Python and Manim installed can render a + declared scene through Seqvio and reuse the result on an identical second run. +- Named markers align with resolved narration and remain correct after direct + seek, chapter render, and final media mux. +- Adapter failures produce stable diagnostics and retain the artifacts required + to reproduce the failed command. + +### 3.5 Motion Grammar + +**Status: complete.** The +core package now supports the nine semantic actions and compiles them into +ExplanationBeat visual actions, AttentionSequence items, and DirectionPlan +segments. Target and relationship validation is deterministic; the first +question/reveal/compare/answer, compare/merge, problem/fix, and +process/verification fixtures have retained renders. `trace` preserves its full +guided path and explicit frame timing under repeated and reverse seek. + +- Define a deliberately small semantic grammar over the existing visual action + vocabulary: `question`, `pause`, `reveal`, `trace`, `compare`, `emphasize`, + `transform`, `answer`, and `summarize`. +- Map grammar steps to ExplanationBeat visual actions, AttentionSequence + operations, and DirectionPlan intents rather than pixel-level animation. +- Add grammar fixtures for question/answer, compare/merge, problem/fix, and + process/verification explainers. +- Validate that every grammar step references an explainable target or a + declared semantic relation. + +Exit criteria: + +- Common explainer structures can be expressed without hand-authoring motion + primitives while preserving deterministic frame behavior. +- Grammar output remains inspectable as semantic actions and does not become a + general-purpose animation DSL. + +### 3.6 Director Skills + +**Status: complete.** +`npm run director:pass` derives and validates a DirectionPlan and writes a +versioned receipt with the input hash, segment count, and diagnostics. +`npm run director:task` emits versioned generate or repair tasks for a host +agent, with stable suggestions and input/candidate/output hashes. Planning stays +outside the renderer and returned artifacts are validated locally. + +- Add host-agent skills that generate or repair DirectionPlan, + AttentionSequence, and Motion Grammar from an approved ExplainerDocument. +- Keep planning decisions outside the Seqvio renderer; skills emit versioned + artifacts and receipts that Seqvio validates locally. +- Add transition, focus, camera, and rhythm skills only after their output + contracts have reference fixtures and deterministic QA. + +Exit criteria: + +- An agent can request a director pass and receive a reviewable semantic plan, + not opaque renderer code. +- The renderer never needs an LLM call to execute the plan. + +### 3.7 Style Playbook (final phase) + +**Status: complete.** A versioned profile schema and invariant checker protect +target ids, beat identity, timing, and evidence order. The runtime applies +typography, palette, spacing, motion density, camera, transition, and attention +persistence policies. `clean-technical`, `editorial-explainer`, and +`terminal-first` share one reference composition and twelve deterministic +visual-regression frames. + +- Add a versioned style-profile schema only after semantic actions, + DirectionPlan, Motion Grammar, Manim media, and QA are stable. +- Use descriptive profiles such as `clean-technical`, `editorial-explainer`, + `chalk-process`, and `terminal-first`; do not ship brand-name imitation + presets as the public contract. +- Let a style profile control typography hierarchy, motion density, camera + policy, transition policy, attention persistence, spacing, and palette roles. +- Prove that changing a style profile does not change narration timing, target + ids, evidence order, or semantic actions. +- Ship only two or three profiles backed by reference compositions and visual + regression fixtures; style count is not a roadmap goal. + +Exit criteria: + +- The same ExplainerDocument renders in multiple profiles with identical + semantic timing and QA target coverage. +- Style changes are reviewable as data and cannot silently introduce dense + text, competing focal points, or unsafe layout. + +### 3.8 Integrated authoring and QA contract + +**Status: complete.** The repository +now contains a network-free IR fixture combining InfographicScene, +ExplanationBeat attention, derived DirectionPlan, a real Manim media manifest, +and a style profile. Director receipt, compiled TSX, retained MP4, and contact +sheet are generated together. Network-free release smoke replaces external +media with a local FFmpeg fixture, then compiles, runs QA, renders, and decodes +the same IR. The semantic benchmark records cache hit rate, attention and +direction diagnostics, marker confidence, text density, and focal coverage. + +- Update authoring references with selection guidance for infographic, + annotation, captured-media, DirectionPlan, Motion Grammar, and Manim-backed + scenes. +- Add IR examples that combine an infographic scene, a cross-scene attention + sequence, a DirectionPlan, a Manim clip, and a style profile under one + narration manifest. +- Extend the release smoke with a network-free semantic-graphics composition and + a fixture-backed Manim manifest playback case. +- Track render factor, cache hit rate, attention-layout diagnostics, marker + alignment confidence, text-density diagnostics, and focal-target coverage in + benchmark and QA reports. + +Exit criteria: + +- The canonical ExplainerDocument pipeline validates, compiles, renders, and + checks all three capabilities through one ExplanationBeat timeline. +- Generated diagnostics identify the source scene, target or marker, resolved + frame, and suggested repair. + ## Phase 4 - Performance and Host Readiness ### 4.1 Reproducible render baseline - DONE (local reference) @@ -194,6 +531,10 @@ Linux/macOS confirmation remains owned by the configured host CI runs. Phase 0 ─> 1.1 (contract) ─┬─> 1.2 (terminal) ─┐ └─> 1.3 (browser) ┴─> 1.5 (promote) Phase 2.1 (generic QA) - independent +Phase 2.1/2.2 -> 3.1 (infographic) -> 3.2 (attention) -> 3.3 (direction) + \-> 3.4 (Manim) -> 3.5 (grammar) + -> 3.6 (director skills) -> 3.7 (style) + -> 3.8 (integration) ``` ## Risks @@ -202,7 +543,8 @@ Phase 2.1 (generic QA) - independent pipelines and release smoke use the shared dispatcher, canonical artifact tests cover both adapters, and legacy `writeComposition` exports are removed. - **Scope.** Capture, joint ExplanationBeat authoring, post-TTS semantic timing, - release QA, and CLI/artifact contract `1.0` are implemented. Remaining: - CI confirmation on Linux/macOS and lifecycle promotion. Windows host package - and CLI verification passes locally. Screenshot privacy remains explicitly + release QA, and CLI/artifact contract `2.0` are implemented. Remaining: npm + release publication. Windows host package and CLI verification passes locally, + and the Windows/Linux/macOS matrix plus lifecycle promotion are complete. + Deterministic privacy masking is implemented; OCR remains explicitly deferred. diff --git a/docs/LIFECYCLE.md b/docs/LIFECYCLE.md index b64b2ce..f8f9e45 100644 --- a/docs/LIFECYCLE.md +++ b/docs/LIFECYCLE.md @@ -6,13 +6,13 @@ and `npm run verify:contracts` enforces it after packages are built. ## Feature And Package Lifecycle -| Lifecycle | Contract | -| --- | --- | -| `public` | Documented, compiled, tested, and covered by compatibility notes. Breaking changes require a declared release change. | -| `experimental` | Reachable and tested, but its API or packaging may change before promotion. | -| `internal` | Repository implementation detail; not a supported consumer contract. | -| `deprecated` | Still available during a declared removal window, with a supported replacement. | -| `removed` | Rejected by current validation and unavailable from new public exports. | +| Lifecycle | Contract | +| -------------- | --------------------------------------------------------------------------------------------------------------------- | +| `public` | Documented, compiled, tested, and covered by compatibility notes. Breaking changes require a declared release change. | +| `experimental` | Reachable and tested, but its API or packaging may change before promotion. | +| `internal` | Repository implementation detail; not a supported consumer contract. | +| `deprecated` | Still available during a declared removal window, with a supported replacement. | +| `removed` | Rejected by current validation and unavailable from new public exports. | Every workspace package must declare `seqvio.lifecycle`. The private root is `internal`; the stable rendering/component train is `public`; capture and its @@ -44,9 +44,11 @@ its matching tag must exist on the remote repository. This tolerates stale npm metadata causing a duplicate-publish response, while still rejecting a zero exit when any package version or remote tag is missing. -Capture, Browser Recorder, and Terminal Narrator use independent pre-1.0 -versions. Independent versioning does not weaken dependency checks: references -to another local Seqvio package must still use its exact current version. +Capture, Browser Recorder, Terminal Narrator, and the Manim adapter use +independent pre-1.0 versions. Independent versioning does not weaken dependency +checks: references to another local Seqvio package must still use its exact +current version. The Manim adapter remains experimental and is not part of the +stable component/renderer release train. ## Capture Adapter Compatibility @@ -57,9 +59,10 @@ to another local Seqvio package must still use its exact current version. - `stable`: the declared CLI/artifact contract and compatibility notes have passed on every supported host. -Terminal and Browser currently remain `experimental` packages with -`adapterLifecycle: pre-stable`. Promotion requires the Windows/Linux/macOS -runtime matrix to pass; it is not inferred from a version number. +Terminal and Browser remain `experimental` packages with +`adapterLifecycle: stable`. The Windows/Linux/macOS runtime matrix has passed +for the current contract; package lifecycle promotion remains a separate +versioning and release decision. ## CI Enforcement diff --git a/docs/MANIM-INTEGRATION.md b/docs/MANIM-INTEGRATION.md new file mode 100644 index 0000000..d1e3073 --- /dev/null +++ b/docs/MANIM-INTEGRATION.md @@ -0,0 +1,212 @@ +# Manim Integration + +Seqvio integrates the Python package `manim` as an optional external renderer +for mathematical animation. `@seqvio/manim-adapter` is a TypeScript/Node.js +adapter; it does not reimplement Manim in JavaScript and it does not run Python +inside the browser renderer. + +The integration has two stages: + +```text +Python scene (`.py`) + -> @seqvio/manim-adapter -> rendered video + ManimRenderManifest + -> ExplainerDocument `manim` scene or @seqvio/technical `ManimClip` + -> ExplanationBeats + seqvio-qa + seqvio-render + -> final MP4 +``` + +Use Manim when an explanation needs equation transformation, graph animation, +geometric construction, or another visual that is best authored in the Python +Manim ecosystem. Keep the rest of the explanation in Seqvio so narration, +attention, annotations, captured evidence, and final rendering share one +timeline. + +## Lifecycle and requirements + +`@seqvio/manim-adapter` is currently an experimental workspace package. Its API +or packaging may change before lifecycle promotion. `ManimClip` is part of the +public `@seqvio/technical` package. + +Required for adapter rendering: + +- Node.js 18 or newer and a built Seqvio repository checkout +- Python with the `manim` package installed +- Manim's platform dependencies, including FFmpeg +- `ffprobe` for validating the rendered media contract + +The repository fixtures are validated with `manim==0.20.1`. A project may use a +different compatible version, but the Python and Manim versions participate in +the render cache key and are recorded in the manifest. + +## Install a local Manim environment + +Create a repository-local virtual environment from the repository root: + +```powershell +# Windows PowerShell +python -m venv .venv-manim +.\.venv-manim\Scripts\python.exe -m pip install manim==0.20.1 +.\.venv-manim\Scripts\python.exe -m manim --version +``` + +```bash +# macOS or Linux +python3 -m venv .venv-manim +./.venv-manim/bin/python -m pip install manim==0.20.1 +./.venv-manim/bin/python -m manim --version +``` + +Manim may require additional OS packages. Follow the Manim installation +instructions for the target platform when `pip install` reports a missing +native dependency. + +`seqvio-doctor` automatically checks `.venv-manim` in the current repository. +To use another interpreter, set `SEQVIO_MANIM_PYTHON` to its executable path: + +```powershell +$env:SEQVIO_MANIM_PYTHON = "C:\path\to\venv\Scripts\python.exe" +npm run doctor +``` + +```bash +export SEQVIO_MANIM_PYTHON=/path/to/venv/bin/python +npm run doctor +``` + +The adapter API and repository render helper also accept an explicit +`pythonCommand` or `--python` value. + +## Render through the adapter + +Build the workspaces first: + +```bash +npm ci +npm run build +``` + +Declare the external render in a JSON file such as `temp/equation.scene.json`: + +```json +{ + "format": "seqvio-manim-scene", + "version": "1.0", + "id": "equation", + "pythonFile": "examples/manim/equation.py", + "className": "EquationDerivation", + "width": 1280, + "height": 720, + "fps": 30, + "quality": "medium", + "args": ["--media_dir", "output/manim-media"], + "assets": [] +} +``` + +The Manim command and its `--media_dir` argument determine the output location. +Pass that exact expected MP4 path to the adapter helper: + +```powershell +node scripts/manim-adapter-render.mjs ` + --scene temp/equation.scene.json ` + --expectedOutput output/manim-media/videos/equation/720p30/EquationDerivation.mp4 ` + --manifest output/manim-media/equation.manifest.json ` + --python .\.venv-manim\Scripts\python.exe +``` + +```bash +node scripts/manim-adapter-render.mjs \ + --scene temp/equation.scene.json \ + --expectedOutput output/manim-media/videos/equation/720p30/EquationDerivation.mp4 \ + --manifest output/manim-media/equation.manifest.json \ + --python ./.venv-manim/bin/python +``` + +Set Manim-specific output arguments in `args` when the Python scene does not +already write to the expected location. The helper exits nonzero when Python or +Manim is unavailable, the child process fails, or the expected media cannot be +probed. + +The adapter manifest records the command, source and asset hashes, runtime +versions, dimensions, frame rate, duration, alpha mode, diagnostics, and cache +identity. Identical source, assets, settings, and runtimes reuse the validated +cached output. + +## Add the rendered media to a Seqvio explanation + +An `ExplainerDocument` references the pre-rendered media rather than the Python +source: + +```json +{ + "type": "manim", + "id": "equation", + "sourceVideo": "file:///absolute/path/to/EquationDerivation.mp4", + "sourceManifest": "output/manim-media/equation.manifest.json", + "mediaWidth": 1280, + "mediaHeight": 720, + "mediaFps": 30, + "duration": 150, + "fit": "contain", + "markers": [ + { "id": "initial-form", "frame": 0 }, + { + "id": "result", + "frame": 90, + "beatId": "show-result", + "targetId": "equation-result" + } + ] +} +``` + +The compiler turns this scene into `ManimClip`. A marker with `beatId` follows +the resolved `ExplanationBeat` frame after narration synthesis; a marker without +`beatId` remains at its authored media frame. `targetId` exposes a stable target +for shared annotation and attention primitives. + +For hand-authored TSX, import the component directly: + +```tsx +import { ManimClip } from "@seqvio/technical"; + +; +``` + +`ManimClip` is seekable and deterministic: Seqvio maps the current composition +frame to the external video instead of letting it free-play. Run `seqvio-qa` +after compilation to validate media readability, dimensions, timing, markers, +and seek behavior. + +## Repository examples + +- [`../examples/manim/`](../examples/manim/) contains equation, graph, symbolic + proof, and geometric proof Python scenes. +- [`../examples/ir/manim-real-validation.explainer.json`](../examples/ir/manim-real-validation.explainer.json) + shows the `ExplainerDocument` scene contract. Its absolute `sourceVideo` is a + local validation artifact and must be replaced after rendering on another + machine. +- [`../examples/compositions/manim-end-to-end-validation.tsx`](../examples/compositions/manim-end-to-end-validation.tsx) + demonstrates narrated graph and proof playback through `ManimClip`. + +## Troubleshooting + +- `python_not_found`: pass the correct interpreter through + `SEQVIO_MANIM_PYTHON`, `pythonCommand`, or `--python`. +- `manim_not_found`: install `manim` into the same interpreter selected above. +- `missing_asset:`: add or correct every external file declared in + `assets`; declared asset contents participate in cache invalidation. +- `missing_or_unreadable_media`: make `--expectedOutput` match Manim's actual + output path and confirm `ffprobe` can read it. +- `incomplete_media_contract`: verify that the generated file reports width, + height, frame rate, duration, and pixel format. +- A stale-looking result: Python, Manim, source files, declared assets, and + render settings are hashed. Declare every file read by the Python scene in + `assets` so changes invalidate the cache. diff --git a/docs/README.md b/docs/README.md index 5b6a87a..e35dd78 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,53 +6,58 @@ This folder separates **current usage** from **historical archive**. Use this in If you are trying to use Seqvio today, read these first: -| File | Why it matters | -| --- | --- | -| [`../README.md`](../README.md) | Top-level overview, skill-first quick start, and current capabilities | -| [`../skills/seqvio/SKILL.md`](../skills/seqvio/SKILL.md) | Agent production loop for authoring, narration, and rendering | -| [`AUTHORING-ARTIFACTS.md`](./AUTHORING-ARTIFACTS.md) | Human-readable editorial and visual design stages before executable IR | -| [`COMPOSITION-AUTHORING.md`](./COMPOSITION-AUTHORING.md) | Canonical TSX authoring and rendering contract | -| [`EXPLANATION-BEAT-TIMING.md`](./EXPLANATION-BEAT-TIMING.md) | Joint narration/visual authoring and post-TTS alignment contract | -| [`RENDER-LIFECYCLE.md`](./RENDER-LIFECYCLE.md) | Deterministic adapter stages, readiness barriers, timeouts, disposal, and diagnostics | -| [`RENDER-CONFORMANCE.md`](./RENDER-CONFORMANCE.md) | Cross-platform semantic goldens, same-host pixel checks, and CI artifacts | -| [`../benchmarks/README.md`](../benchmarks/README.md) | Reproducible 720p render workloads, metrics, and compatible-host budget checks | -| [`CAPTURE-CLI-CONTRACT.md`](./CAPTURE-CLI-CONTRACT.md) | Terminal/Browser commands, JSON results, exit codes, and artifact layout | -| [`TROUBLESHOOTING.md`](./TROUBLESHOOTING.md) | Common setup, render, and audio workflow failures | -| [`../examples/compositions/README.md`](../examples/compositions/README.md) | Composition examples and render command | -| [`../packages/whiteboard/README.md`](../packages/whiteboard/README.md) | Whiteboard component API and examples | -| [`../packages/product-demo/package.json`](../packages/product-demo/package.json) | Product walkthrough component package metadata | -| [`marketing/POSITIONING.md`](./marketing/POSITIONING.md) | Public positioning and product boundary | -| [`marketing/FEATURE-STATUS.md`](./marketing/FEATURE-STATUS.md) | Public and experimental capability wording | +| File | Why it matters | +| -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| [`../README.md`](../README.md) | Top-level overview, skill-first quick start, and current capabilities | +| [`../skills/seqvio/SKILL.md`](../skills/seqvio/SKILL.md) | Agent production loop for authoring, narration, and rendering | +| [`AUTHORING-ARTIFACTS.md`](./AUTHORING-ARTIFACTS.md) | Human-readable editorial and visual design stages before executable IR | +| [`COMPOSITION-AUTHORING.md`](./COMPOSITION-AUTHORING.md) | Canonical TSX authoring and rendering contract | +| [`EXPLANATION-BEAT-TIMING.md`](./EXPLANATION-BEAT-TIMING.md) | Joint narration/visual authoring and post-TTS alignment contract | +| [`STYLE-PLAYBOOK.md`](./STYLE-PLAYBOOK.md) | Versioned visual profiles and semantic invariants | +| [`RENDER-LIFECYCLE.md`](./RENDER-LIFECYCLE.md) | Deterministic adapter stages, readiness barriers, timeouts, disposal, and diagnostics | +| [`RENDER-CONFORMANCE.md`](./RENDER-CONFORMANCE.md) | Cross-platform semantic goldens, same-host pixel checks, and CI artifacts | +| [`../benchmarks/README.md`](../benchmarks/README.md) | Reproducible 720p render workloads, metrics, and compatible-host budget checks | +| [`CAPTURE-CLI-CONTRACT.md`](./CAPTURE-CLI-CONTRACT.md) | Terminal/Browser commands, JSON results, exit codes, and artifact layout | +| [`MANIM-INTEGRATION.md`](./MANIM-INTEGRATION.md) | Optional Python Manim setup, adapter rendering, cache/manifests, and `ManimClip` timeline integration | +| [`TROUBLESHOOTING.md`](./TROUBLESHOOTING.md) | Common setup, render, and audio workflow failures | +| [`../examples/compositions/README.md`](../examples/compositions/README.md) | Composition examples and render command | +| [`../packages/whiteboard/README.md`](../packages/whiteboard/README.md) | Whiteboard component API and examples | +| [`../packages/product-demo/package.json`](../packages/product-demo/package.json) | Product walkthrough component package metadata | +| [`../packages/manim-adapter/README.md`](../packages/manim-adapter/README.md) | Experimental Python Manim adapter API and runtime boundary | +| [`marketing/POSITIONING.md`](./marketing/POSITIONING.md) | Public positioning and product boundary | +| [`marketing/FEATURE-STATUS.md`](./marketing/FEATURE-STATUS.md) | Public and experimental capability wording | ## Current Source of Truth These docs describe the repository as it exists today: -| File | Scope | -| --- | --- | -| [`AUTHORING-ARTIFACTS.md`](./AUTHORING-ARTIFACTS.md) | EditorialPlan, VisualDesignBrief, and ExplainerDocument boundaries | -| [`COMPOSITION-AUTHORING.md`](./COMPOSITION-AUTHORING.md) | TSX authoring model, render contract, transitions, and timing rules | -| [`EXPLANATION-BEAT-TIMING.md`](./EXPLANATION-BEAT-TIMING.md) | ExplanationBeat schema, scene integration, timing resolution, and QA repair | -| [`RENDER-LIFECYCLE.md`](./RENDER-LIFECYCLE.md) | Stateful/asynchronous render lifecycle and timeout contract | -| [`RENDER-CONFORMANCE.md`](./RENDER-CONFORMANCE.md) | Windows/Linux/macOS render conformance and golden-update policy | -| [`scene-capabilities.json`](./scene-capabilities.json) | Machine-readable public scene/compiler/package/lifecycle/QA capability snapshot | -| [`LIFECYCLE.md`](./LIFECYCLE.md) | Package/feature lifecycle labels, release-train policy, adapter promotion, and CI enforcement | -| [`CAPTURE-CLI-CONTRACT.md`](./CAPTURE-CLI-CONTRACT.md) | Versioned capture CLI and job-artifact contract | -| [`../benchmarks/README.md`](../benchmarks/README.md) | Render benchmark profile, baseline policy, and commands | -| [`../examples/compositions/README.md`](../examples/compositions/README.md) | End-to-end examples | -| [`../packages/whiteboard/AI-USAGE.md`](../packages/whiteboard/AI-USAGE.md) | AI-assistant-oriented usage for the whiteboard package | -| [`../skills/seqvio/references/production-techniques.md`](../skills/seqvio/references/production-techniques.md) | Voice-first timing, reference-style analysis, and visual QA checklist | +| File | Scope | +| -------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| [`AUTHORING-ARTIFACTS.md`](./AUTHORING-ARTIFACTS.md) | EditorialPlan, VisualDesignBrief, and ExplainerDocument boundaries | +| [`COMPOSITION-AUTHORING.md`](./COMPOSITION-AUTHORING.md) | TSX authoring model, render contract, transitions, and timing rules | +| [`EXPLANATION-BEAT-TIMING.md`](./EXPLANATION-BEAT-TIMING.md) | ExplanationBeat schema, scene integration, timing resolution, and QA repair | +| [`STYLE-PLAYBOOK.md`](./STYLE-PLAYBOOK.md) | Current profile schema, runtime application, and visual regression fixtures | +| [`RENDER-LIFECYCLE.md`](./RENDER-LIFECYCLE.md) | Stateful/asynchronous render lifecycle and timeout contract | +| [`RENDER-CONFORMANCE.md`](./RENDER-CONFORMANCE.md) | Windows/Linux/macOS render conformance and golden-update policy | +| [`scene-capabilities.json`](./scene-capabilities.json) | Machine-readable public scene/compiler/package/lifecycle/QA capability snapshot | +| [`LIFECYCLE.md`](./LIFECYCLE.md) | Package/feature lifecycle labels, release-train policy, adapter promotion, and CI enforcement | +| [`CAPTURE-CLI-CONTRACT.md`](./CAPTURE-CLI-CONTRACT.md) | Versioned capture CLI and job-artifact contract | +| [`MANIM-INTEGRATION.md`](./MANIM-INTEGRATION.md) | Current external Python Manim render and Seqvio playback workflow | +| [`../benchmarks/README.md`](../benchmarks/README.md) | Render benchmark profile, baseline policy, and commands | +| [`../examples/compositions/README.md`](../examples/compositions/README.md) | End-to-end examples | +| [`../packages/whiteboard/AI-USAGE.md`](../packages/whiteboard/AI-USAGE.md) | AI-assistant-oriented usage for the whiteboard package | +| [`../skills/seqvio/references/production-techniques.md`](../skills/seqvio/references/production-techniques.md) | Voice-first timing, reference-style analysis, and visual QA checklist | ## Product and Planning Docs -| File | Notes | -| --- | --- | -| [`VISION.md`](./VISION.md) | Product positioning, target users, scope, and design principles (slow-changing source of truth) | -| [`ROADMAP.md`](./ROADMAP.md) | Directional bet and phase ordering; what to build next and why. Subordinate to `VISION.md` | -| [`EXPLAINER-FRAMEWORK-IMPROVEMENT-PLAN.md`](./EXPLAINER-FRAMEWORK-IMPROVEMENT-PLAN.md) | Active implementation plan for capture, ExplanationBeat timing, deterministic playback, QA, and release readiness | -| [`marketing/LANDING-COPY.md`](./marketing/LANDING-COPY.md) | Versioned landing page copy | -| [`marketing/PRODUCT-HUNT.md`](./marketing/PRODUCT-HUNT.md) | Product Hunt tagline, description, and demo promise | -| [`marketing/DEMO-SCRIPT.md`](./marketing/DEMO-SCRIPT.md) | Current 60-75 second product demo sequence | +| File | Notes | +| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [`VISION.md`](./VISION.md) | Product positioning, target users, scope, and design principles (slow-changing source of truth) | +| [`ROADMAP.md`](./ROADMAP.md) | Directional bet and phase ordering; what to build next and why. Subordinate to `VISION.md` | +| [`IMPLEMENTATION-PLAN.md`](./IMPLEMENTATION-PLAN.md) | Task-level implementation plan for capture, ExplanationBeat timing, explanatory graphics, attention guidance, Manim integration, QA, and release readiness | +| [`marketing/LANDING-COPY.md`](./marketing/LANDING-COPY.md) | Versioned landing page copy | +| [`marketing/PRODUCT-HUNT.md`](./marketing/PRODUCT-HUNT.md) | Product Hunt tagline, description, and demo promise | +| [`marketing/DEMO-SCRIPT.md`](./marketing/DEMO-SCRIPT.md) | Current 60-75 second product demo sequence | Active, time-boxed work items live in GitHub issues and milestones, not in a docs file. @@ -60,9 +65,9 @@ Active, time-boxed work items live in GitHub issues and milestones, not in a doc Historical logs and milestone summaries now live under `docs/archive/`: -| File | Notes | -| --- | --- | -| [`archive/README.md`](./archive/README.md) | Archive index | +| File | Notes | +| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| [`archive/README.md`](./archive/README.md) | Archive index | | [`archive/PRODUCT-PLAN-2026-07.md`](./archive/PRODUCT-PLAN-2026-07.md) | Archived July 2026 product plan snapshot; retained for competitive analysis and roadmap history (superseded by [`VISION.md`](./VISION.md)) | ## Guidance for Contributors diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 16a1e94..df2026b 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,12 +1,12 @@ # Seqvio Roadmap -> **Status:** directional. This file states *what we are betting on and in what -> order*, not dated deliverables. Time-boxed work items live in GitHub issues and +> **Status:** directional. This file states _what we are betting on and in what +> order_, not dated deliverables. Time-boxed work items live in GitHub issues and > milestones. For positioning and scope, read [`VISION.md`](./VISION.md) - when > the two disagree, `VISION.md` wins. > -> Last revised: 2026-08-01 (implementation status aligned with the current -> ExplanationBeat and terminal/browser IR pipelines). +> Last revised: 2026-08-05 (ordered semantic direction before the final style +> layer and added product-explainer production contracts). ## What Changed in This Revision @@ -27,8 +27,8 @@ The reordering moves effort to the layer HyperFrames does not have and is architecturally poorly positioned to add: **system capture.** This is not a change to `VISION.md` - it is `VISION.md`'s own "depth before breadth" and coding-agent-centric positioning applied to a changed landscape. The framing -stays: *give an agent a visual output channel for things that already happened -in a real system.* What changes is which bet earns the first dollars. +stays: _give an agent a visual output channel for things that already happened +in a real system._ What changes is which bet earns the first dollars. ## The Threat We Are Planning Against @@ -43,11 +43,11 @@ is needed at all. Two things that threat does not solve: 2. **The closed layer is now a commodity.** A composition-to-MP4 pipeline with static linting and regression is no longer a differentiator - HyperFrames has it, with cloud rendering and a studio on top. Seqvio cannot win there and - should not try. The asymmetry that remains is the *opening* layer: turning + should not try. The asymmetry that remains is the _opening_ layer: turning real system activity into a composition. -The strategic move is the one `VISION.md` already names: from *"help an agent -generate a video"* to **"give an agent a visual output channel for things that +The strategic move is the one `VISION.md` already names: from _"help an agent +generate a video"_ to **"give an agent a visual output channel for things that already happened in a real system."** This revision just stops pretending the closed layer is part of the moat. @@ -73,11 +73,11 @@ describes itself. Three feature tracks are implemented and tested but unreachable end to end: -| Track | Status | Resolution | -| --- | --- | --- | -| `packages/renderer/src/shader-transitions/` | Five GLSL transitions compiled, but nothing in the repo ever set `window.__seqvio_shaderTransitions`, so the four reads in `runtime.tsx` never saw a value. `compositor.ts` drew an `Image` synchronously after assigning `src` without awaiting `decode()`, yielding empty textures. | **Deleted** (closed-layer render effect, not on the differentiation path). | -| `packages/renderer/src/audio/ducking.ts` + `volume-envelope.ts` | `audio-mux.ts` honors `options.autoDuck`, but no CLI passes it. | **Marked `@internal`** (wire-up is medium cost; Phase 2 audio checks don't need it). | -| `packages/core/src/clock.ts` (`TransportClock`) | Nothing imports it outside its own test. | **Marked `@internal`** (retained as Phase 5 preview seed). | +| Track | Status | Resolution | +| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | +| `packages/renderer/src/shader-transitions/` | Five GLSL transitions compiled, but nothing in the repo ever set `window.__seqvio_shaderTransitions`, so the four reads in `runtime.tsx` never saw a value. `compositor.ts` drew an `Image` synchronously after assigning `src` without awaiting `decode()`, yielding empty textures. | **Deleted** (closed-layer render effect, not on the differentiation path). | +| `packages/renderer/src/audio/ducking.ts` + `volume-envelope.ts` | `audio-mux.ts` honors `options.autoDuck`, but no CLI passes it. | **Marked `@internal`** (wire-up is medium cost; Phase 2 audio checks don't need it). | +| `packages/core/src/clock.ts` (`TransportClock`) | Nothing imports it outside its own test. | **Marked `@internal`** (retained as Phase 5 preview seed). | For each: wire it up, mark it explicitly internal/unreleased, or delete it. Shipping code that no user can reach is worse than a stale doc - it violates @@ -130,8 +130,8 @@ have never been able to explain clearly: The differentiator inside capture is **temporal fidelity**, not a screenshot montage. `terminal-narrator` replays a real pty session from a cast (keystrokes and output in their real timing); `website-to-video`-style DOM snapshots are a -different, weaker thing. Hold that line: capture is *what actually happened, in -the order it happened*, not a reconstruction. +different, weaker thing. Hold that line: capture is _what actually happened, in +the order it happened_, not a reconstruction. This layer needs integrations, permissions, format adapters, redaction (the `redact.ts` in `terminal-narrator` already proves the repo has tasted this), and @@ -147,8 +147,9 @@ floor, contrast (WCAG AA), and offscreen elements. Its baseline/capture profiles also validate narration/captions, speech rate, highlight hold time, audio health, capture media/state, semantic time maps, unresolved phrase anchors, and reversed ExplanationBeats. These checks are deterministic, use no LLM, and ship as a -`--ci` mode that exits non-zero. Screenshot privacy masking remains explicitly -deferred. +`--ci` mode that exits non-zero. Browser capture plans also support deterministic +selector/rectangle privacy masks before screenshots are taken; OCR is not a +security boundary. Ground-truth verification (code vs source AST, diagram edges vs dependency graph, terminal vs real stdout) was considered and dropped: capture-produced IR is @@ -156,6 +157,101 @@ faithful (same-source as the real recording, so verification is self-consistent) and hand-written IR either has no ground truth (whiteboard) or was dropped (code/diagram char-level checks conflate simplification with fabrication). +## Phase 3 - Semantic Direction and Mathematical Animation + +Build a semantic visual-expression layer for moments where captured footage, +code, and architecture diagrams do not fully explain the idea. These connected +capabilities share the same ExplanationBeat timing contract. + +### Explainable motion graphics + +Add `InfographicScene` to `@seqvio/technical` with reusable metric, comparison, +process, timeline, chart, and relationship primitives. Motion must communicate +an explicit semantic change: reveal a quantity, compare values, trace a process, +show a state transition, or connect cause and effect. Every addressable datum and +relationship receives a stable target id so `ExplanationBeat.visualActions` can +drive `reveal`, `compare`, `trace`, `emphasize`, and `transform` actions from +phrase anchors. + +Ship deterministic layouts, shared number/chart formatting, data-source labels, +responsive safe areas, and seek-safe animation states. Add reference scenes for +a metric explanation, a before/after comparison, a multi-step process, and a +time-series change. + +### Reusable attention guidance + +Extend the shared annotation system into a cross-scene attention vocabulary: +highlight, focus ring, spotlight, callout, arrow, bracket, connector, region +shade, and guided focus path. An `AttentionSequence` coordinates activation, +handoff, persistence, and clearing across stable annotation targets while the +layout layer keeps labels and connectors inside safe areas and resolves common +collisions. + +Make the same primitives usable across whiteboard, code, diagram, terminal, +browser, product-demo, infographic, and externally rendered animation scenes. +Connect attention actions to phrase-anchored ExplanationBeats and include +random-access, reverse-seek, overlap, contrast, and minimum-hold QA coverage. + +### DirectionPlan + +Add a minimal, versioned, renderer-agnostic direction plan that references +scenes, stable targets, `ExplanationBeat`s, and capture manifests. It records +purpose, focal target, pace, camera intent, and transition intent without +encoding CSS, opacity, scale, or renderer APIs. Compile it into the existing +actions, `AttentionSequence`, and timing system, and validate unknown +references and conflicting intents before rendering. + +### Manim adapter + +**Implemented as an experimental external adapter.** +`@seqvio/manim-adapter` invokes a selected local Python interpreter and its +installed `manim` package; it is not a JavaScript implementation of Manim. Its +versioned external-render `ManimSceneSpec` describes the Python source, scene +class, render settings, and assets. Execution produces a content-addressed +`ManimRenderManifest` containing the rendered-media contract, source and asset +hashes, runtime versions, cache identity, logs, and diagnostics. + +`ManimClip` and the ExplainerDocument `manim` compiler path consume the +pre-rendered video as deterministic seekable media. Named markers can align +with narration phrases and ExplanationBeats and expose annotation targets. +Runtime preflight, cache reuse, progress events, cancellation, timeouts, media +probing, marker/seek QA, and equation, graph, symbolic-proof, and geometric-proof +fixtures are implemented. See [`MANIM-INTEGRATION.md`](./MANIM-INTEGRATION.md) +for the current usage contract. + +### Motion Grammar + +Define a small semantic motion vocabulary - `question`, `pause`, `reveal`, +`trace`, `compare`, `emphasize`, `transform`, `answer`, and `summarize` - that +maps to semantic actions, `AttentionSequence`, and `DirectionPlan`. Ship +question/answer, compare/merge, problem/fix, and process/verification fixtures. +This is a constrained explanation grammar, not a general animation DSL. + +### Director Skills + +Add host-agent skills that generate and repair reviewable `DirectionPlan`, +`AttentionSequence`, and Motion Grammar artifacts. Keep planning outside the +renderer, store versioned plans and receipts, and use the same deterministic +validation and frame QA as hand-authored compositions. + +### Style Playbook (final phase) + +Only after semantic actions, direction, motion grammar, Manim integration, and +QA are stable, add a versioned style profile. Profiles such as +`clean-technical`, `editorial-explainer`, `chalk-process`, and `terminal-first` +control typography, motion density, camera behavior, transitions, attention +persistence, spacing, and palette. They must not change narration timing, +target identity, evidence order, or semantic actions; avoid brand-name +imitation and limit the first release to two or three profiles. + +### Integrated authoring and QA + +Exercise one authoring path across captured terminal/browser media, authored +diagrams, infographic scenes, annotation primitives, DirectionPlan, Motion +Grammar, and Manim clips. Release smoke must cover voice-first timing, focal +target coverage, text-density and collision diagnostics, transition intent, +seek/reverse behavior, marker alignment, and content-addressed cache reuse. + ## Operational Stabilization This is supporting infrastructure, not a new product layer. Seqvio now carries @@ -168,13 +264,13 @@ Cross-host confirmation remains part of the lifecycle-promotion gate. ## Deprioritized -Restating what is *not* getting first dollars, and why: +Restating what is _not_ getting first dollars, and why: - **Multi-target IR output.** The hedge is real but opportunistic; it waits until capture has a reason to emit non-MP4 targets. - **Preview / Studio** (previous Phase 5). HyperFrames has a studio. Seqvio's review surface stays the minimum - scene list, diagnostics, partial re-render - - and explicitly *not* a nonlinear editor, per `VISION.md`. + - and explicitly _not_ a nonlinear editor, per `VISION.md`. - **Whiteboard polish and a fourth/fifth style.** `whiteboard` is the contract reference; it stays correct and maintained, not a feature target. No style race. diff --git a/docs/STYLE-PLAYBOOK.md b/docs/STYLE-PLAYBOOK.md new file mode 100644 index 0000000..d883f95 --- /dev/null +++ b/docs/STYLE-PLAYBOOK.md @@ -0,0 +1,71 @@ +# Style Playbook + +Seqvio style profiles are versioned visual policy data. They control visual +treatment while preserving scene ids, target ids, ExplanationBeats, capture +evidence, narration timing, and semantic action order. + +## Profiles + +The reference profiles are: + +| Profile | Treatment | +| --- | --- | +| `clean-technical` | Restrained dark technical presentation with semantic focus transfers | +| `editorial-explainer` | Light editorial typography, airy spacing, and crossfades | +| `terminal-first` | Compact dark presentation, restrained motion, and evidence-follow camera | + +Profile JSON files live in `examples/styles/`. Each profile specifies +typography, motion density, camera policy, transition policy, attention +persistence, spacing, and the background/ink/accent/muted palette roles. + +## ExplainerDocument + +Attach a validated profile to `styleProfile`: + +```json +{ + "format": "seqvio-explainer", + "schemaVersion": "1.0", + "id": "styled-explainer", + "styleProfile": { + "format": "seqvio-style-profile", + "version": "1.0", + "id": "clean-technical", + "label": "Clean Technical", + "typography": { + "headingFamily": "Inter", + "bodyFamily": "Inter", + "monoFamily": "JetBrains Mono", + "scale": "editorial" + }, + "motionDensity": "restrained", + "cameraPolicy": "semantic-focus", + "transitionPolicy": "focus-transfer", + "attentionPersistence": "until-handoff", + "spacing": "comfortable", + "paletteRoles": { + "background": "#0f172a", + "ink": "#f8fafc", + "accent": "#38bdf8", + "muted": "#94a3b8" + } + }, + "scenes": [] +} +``` + +`compileExplainerDocumentToTsx` emits a `StyleProfileProvider` automatically. +For hand-authored TSX, wrap the composition with the same provider. + +## Verification + +The three retained reference compositions are +`examples/compositions/style-playbook-*.tsx`. Their videos live under +`output/style-playbook-*-v1.mp4`, and their committed golden frames live under +`tests/visual-snapshots/baseline/`. + +Run the focused regression check with: + +```bash +node scripts/visual-regression.mjs --case style-clean-technical,style-editorial-explainer,style-terminal-first +``` diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 6e4c20a..57e4681 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -218,6 +218,24 @@ Check that you are rendering with: Also verify that the manifest `tracks[].src` values still point to existing MP3 files. +## Python Manim Integration + +### `seqvio-doctor` cannot find Manim + +The adapter uses the Python interpreter selected for the current process. A +repository checkout first checks `.venv-manim`; otherwise it falls back to +`python`. Set `SEQVIO_MANIM_PYTHON` to the full interpreter path when Manim is +installed in another environment. Confirm that the same executable can run +`-m manim --version`. + +### The adapter rendered, but reports unreadable or incomplete media + +Make `--expectedOutput` match Manim's actual MP4 path and confirm `ffprobe` can +read it. The adapter requires dimensions, frame rate, duration, and pixel-format +metadata before accepting a render. See +[`MANIM-INTEGRATION.md`](./MANIM-INTEGRATION.md) for commands, cache behavior, +diagnostic codes, and the `ManimClip` playback contract. + ## Docs and Repo Hygiene ### Which doc is the source of truth? diff --git a/docs/assets/brand/seqvio-icon-1024.png b/docs/assets/brand/seqvio-icon-1024.png new file mode 100644 index 0000000..711d8d2 Binary files /dev/null and b/docs/assets/brand/seqvio-icon-1024.png differ diff --git a/docs/assets/brand/seqvio-mark.svg b/docs/assets/brand/seqvio-mark.svg new file mode 100644 index 0000000..3905ca9 --- /dev/null +++ b/docs/assets/brand/seqvio-mark.svg @@ -0,0 +1,30 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/assets/videos/seqvio-product-hunt-en.mp4 b/docs/assets/videos/seqvio-product-hunt-en.mp4 index c2ca3f8..f0b06f5 100644 Binary files a/docs/assets/videos/seqvio-product-hunt-en.mp4 and b/docs/assets/videos/seqvio-product-hunt-en.mp4 differ diff --git a/docs/marketing/FEATURE-STATUS.md b/docs/marketing/FEATURE-STATUS.md index 96e55d7..af2880c 100644 --- a/docs/marketing/FEATURE-STATUS.md +++ b/docs/marketing/FEATURE-STATUS.md @@ -1,15 +1,19 @@ # Feature Status For Public Materials -| Area | Status | Public wording | -| --- | --- | --- | -| `ExplainerDocument` and five scene compilers | Public | Complete authoring and rendering path | -| EditorialPlan / VisualDesignBrief | Public | Reviewable planning artifacts before executable IR | -| ExplanationBeat timing and post-TTS alignment | Public | Narration, visual action, and evidence share one timing contract | -| `seqvio-qa` | Public | Deterministic audio, pacing, media, semantic, and key-frame checks | -| `@seqvio/terminal-narrator` | Experimental | Real terminal capture with platform-specific prerequisites | -| `@seqvio/browser-recorder` | Experimental | Chromium action capture with recorded evidence and timing | -| `@seqvio/capture` | Experimental | Shared capture contracts and artifacts | -| Screenshot privacy masking | Deferred | Do not promise this in marketing copy | +| Area | Status | Public wording | +| --------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------ | +| Public `ExplainerDocument` scene compilers | Public | Complete authoring and rendering path for whiteboard, code, diagram, terminal, and browser | +| EditorialPlan / VisualDesignBrief | Public | Reviewable planning artifacts before executable IR | +| ExplanationBeat timing and post-TTS alignment | Public | Narration, visual action, and evidence share one timing contract | +| `seqvio-qa` | Public | Deterministic audio, pacing, media, semantic, and key-frame checks | +| `@seqvio/terminal-narrator` | Experimental | Real terminal capture with platform-specific prerequisites | +| `@seqvio/browser-recorder` | Experimental | Chromium action capture with recorded evidence and timing | +| `@seqvio/capture` | Experimental | Shared capture contracts and artifacts | +| `InfographicScene` / `infographic` compiler | Experimental | Explanatory metrics, comparison, process, timeline, relationship, and chart scenes | +| `@seqvio/manim-adapter` / `manim` compiler | Experimental | Optional external Python Manim rendering with validated media and timeline markers | +| Screenshot privacy masking | Available for browser capture plans | Requires explicit selector/rectangle declarations; OCR is not a security boundary | Marketing demos must identify whether a scene is authored or capture-derived. Do not describe experimental adapters as a universal one-click workflow. +When mentioning Manim, state that Seqvio invokes an independently installed +Python `manim` runtime and consumes the pre-rendered media through `ManimClip`. diff --git a/docs/scene-capabilities.json b/docs/scene-capabilities.json index c321ab1..fc6f4a3 100644 --- a/docs/scene-capabilities.json +++ b/docs/scene-capabilities.json @@ -25,6 +25,14 @@ "agentAuthoring": true, "qaRules": ["visual-layout", "pacing", "audio-timeline", "explanation-beats"] }, + { + "type": "infographic", + "compiler": "complete", + "requiredPackage": "@seqvio/technical", + "lifecycle": "experimental", + "agentAuthoring": true, + "qaRules": ["visual-layout", "text-density", "focal-target", "pacing", "audio-timeline", "explanation-beats"] + }, { "type": "terminal", "compiler": "complete", @@ -40,6 +48,14 @@ "lifecycle": "public", "agentAuthoring": false, "qaRules": ["visual-layout", "capture-state", "capture-media", "pacing", "audio-timeline", "explanation-beats"] + }, + { + "type": "manim", + "compiler": "complete", + "requiredPackage": "@seqvio/manim-adapter", + "lifecycle": "experimental", + "agentAuthoring": true, + "qaRules": ["visual-layout", "external-media", "marker-alignment", "pacing", "audio-timeline", "explanation-beats"] } ] } diff --git a/examples/capture-plans/browser-privacy-mask.json b/examples/capture-plans/browser-privacy-mask.json new file mode 100644 index 0000000..cb99ba6 --- /dev/null +++ b/examples/capture-plans/browser-privacy-mask.json @@ -0,0 +1,49 @@ +{ + "version": "1.0", + "name": "Browser privacy mask verification", + "startUrl": "http://127.0.0.1:4176/demo", + "viewport": { "width": 1280, "height": 720 }, + "captureFps": 12, + "renderFps": 30, + "maxZoom": 2.0, + "privacy": { + "masks": [ + { + "id": "project-secret", + "selector": "#project-name", + "padding": 5, + "color": "#111827" + } + ] + }, + "actions": [ + { + "id": "secret", + "type": "fill", + "label": "Enter protected project token", + "selector": "#project-name", + "value": "MASKED-DEMO-VALUE", + "afterMs": 700 + }, + { + "id": "template", + "type": "click", + "label": "Choose tutorial template", + "selector": "[data-template=\"tutorial\"]", + "afterMs": 500 + }, + { + "id": "create", + "type": "click", + "label": "Create protected recording task", + "selector": "#create-task", + "afterMs": 900 + }, + { + "id": "hold", + "type": "wait", + "label": "Verify masked result", + "durationMs": 900 + } + ] +} diff --git a/examples/compositions/README.md b/examples/compositions/README.md index 889779d..148566e 100644 --- a/examples/compositions/README.md +++ b/examples/compositions/README.md @@ -28,24 +28,29 @@ node packages/renderer/dist/cli.js \ `explanation.beats`; post-TTS timing is resolved into a semantic scene time map. - **Manual TSX audio**: direct `meta.audio.narration` remains supported but is a lower-level contract without automatic phrase-level visual alignment. +- **Manim media**: render Python Manim scenes externally through + `@seqvio/manim-adapter`, then use `ManimClip` or an ExplainerDocument `manim` + scene. See [`docs/MANIM-INTEGRATION.md`](../../docs/MANIM-INTEGRATION.md). - **No templates**: layout and copy are explicit in TSX (Remotion-style). ## Examples -| File | Description | -|------|-------------| -| `seqvio-overview-en.tsx` | Narrated English product overview | -| `seqvio-overview-zh.tsx` | Narrated Chinese product overview | -| `seqvio-intro.tsx` | 4-scene framework intro (EN + partial Chinese) | -| `seqvio-intro-zh.tsx` | 4-scene all-Chinese framework intro | -| `seqvio-audio-demo.tsx` | Audio and caption metadata demo | -| `seqvio-alignment-demo.tsx` | Scene-duration derivation from resolved audio | -| `seqvio-style-manifest-demo.tsx` | Whiteboard style preset manifest demo | -| `seqvio-product-demo-preview.tsx` | Product walkthrough component demo | -| `seqvio-product-demo-validation.tsx` | Short product-demo renderer validation composition | -| `seqvio-scatterbrain.tsx` | Sticky-note / workshop style demo | -| `loop-engineering-explainer.tsx` | Long-form narrated explainer composition | -| `generated/style-layout-demo.tsx` | Storyboard IR compiled TSX demo | +| File | Description | +| ------------------------------------ | -------------------------------------------------------- | +| `seqvio-overview-en.tsx` | Narrated English product overview | +| `seqvio-overview-zh.tsx` | Narrated Chinese product overview | +| `seqvio-intro.tsx` | 4-scene framework intro (EN + partial Chinese) | +| `seqvio-intro-zh.tsx` | 4-scene all-Chinese framework intro | +| `seqvio-audio-demo.tsx` | Audio and caption metadata demo | +| `seqvio-alignment-demo.tsx` | Scene-duration derivation from resolved audio | +| `seqvio-style-manifest-demo.tsx` | Whiteboard style preset manifest demo | +| `seqvio-product-demo-preview.tsx` | Product walkthrough component demo | +| `seqvio-product-demo-validation.tsx` | Short product-demo renderer validation composition | +| `seqvio-scatterbrain.tsx` | Sticky-note / workshop style demo | +| `loop-engineering-explainer.tsx` | Long-form narrated explainer composition | +| `manim-clip-validation.tsx` | Minimal seekable `ManimClip` and named-marker validation | +| `manim-end-to-end-validation.tsx` | Narrated playback of real graph and proof Manim renders | +| `generated/style-layout-demo.tsx` | Storyboard IR compiled TSX demo | More single-scene samples: `packages/whiteboard/examples/`. diff --git a/examples/compositions/attention-cross-scene-validation.tsx b/examples/compositions/attention-cross-scene-validation.tsx new file mode 100644 index 0000000..4b9583d --- /dev/null +++ b/examples/compositions/attention-cross-scene-validation.tsx @@ -0,0 +1,100 @@ +import React from 'react'; +import type { AttentionSequenceItem, RenderableMeta } from '@seqvio/core'; +import { Scene, Transition, VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const SCENE_DURATION = 120; +const TRANSITION_DURATION = 12; +const DURATION = SCENE_DURATION * 2 + TRANSITION_DURATION; + +const attentionPlan: AttentionSequenceItem[] = [ + { + id: 'inspect-signal', + sceneId: 'observe', + targetId: 'signal', + kind: 'spotlight', + start: 10, + duration: 24, + persistence: 'until-clear', + clearAt: 102, + minHoldFrames: 36, + handoffTo: 'decision', + handoffToSceneId: 'decide', + }, + { + id: 'explain-decision', + sceneId: 'decide', + targetId: 'decision', + kind: 'box', + start: 8, + duration: 70, + persistence: 'timed', + minHoldFrames: 48, + }, +]; + +function ObserveScene() { + return ( + + + + ); +} + +function DecideScene() { + return ( + + + + ); +} + +export default function AttentionCrossSceneValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, + duration: DURATION, + width: W, + height: H, +}; diff --git a/examples/compositions/attention-layout-validation-16x9.tsx b/examples/compositions/attention-layout-validation-16x9.tsx new file mode 100644 index 0000000..7cd2d50 --- /dev/null +++ b/examples/compositions/attention-layout-validation-16x9.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { AttentionLayoutValidationComposition, attentionLayoutValidationMeta } from './attention-layout-validation-shared'; +export default function AttentionLayoutValidation16x9() { return ; } +export const meta = attentionLayoutValidationMeta(1280, 720); diff --git a/examples/compositions/attention-layout-validation-1x1.tsx b/examples/compositions/attention-layout-validation-1x1.tsx new file mode 100644 index 0000000..8019de8 --- /dev/null +++ b/examples/compositions/attention-layout-validation-1x1.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { AttentionLayoutValidationComposition, attentionLayoutValidationMeta } from './attention-layout-validation-shared'; +export default function AttentionLayoutValidation1x1() { return ; } +export const meta = attentionLayoutValidationMeta(720, 720); diff --git a/examples/compositions/attention-layout-validation-9x16.tsx b/examples/compositions/attention-layout-validation-9x16.tsx new file mode 100644 index 0000000..2811062 --- /dev/null +++ b/examples/compositions/attention-layout-validation-9x16.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { AttentionLayoutValidationComposition, attentionLayoutValidationMeta } from './attention-layout-validation-shared'; +export default function AttentionLayoutValidation9x16() { return ; } +export const meta = attentionLayoutValidationMeta(720, 1280); diff --git a/examples/compositions/attention-layout-validation-shared.tsx b/examples/compositions/attention-layout-validation-shared.tsx new file mode 100644 index 0000000..2416a81 --- /dev/null +++ b/examples/compositions/attention-layout-validation-shared.tsx @@ -0,0 +1,43 @@ +import React from 'react'; +import { AnnotationTarget, VideoComposition, type AnnotationItem, type RenderableMeta } from '@seqvio/core'; +import { TechnicalScene } from '@seqvio/technical'; + +const FPS = 30; +const DURATION = 180; + +export function attentionLayoutValidationMeta(width: number, height: number): RenderableMeta { + return { width, height, fps: FPS, duration: DURATION }; +} + +export function AttentionLayoutValidationComposition({ width, height, id }: { width: number; height: number; id: string }) { + const compact = width < height; + const boxWidth = compact ? Math.min(190, width * 0.32) : Math.min(210, width * 0.18); + const boxHeight = compact ? 74 : 82; + const y = compact ? height * 0.38 : height * 0.43; + const positions = compact ? [width * 0.08, width * 0.36, width * 0.64] : [width * 0.1, width * 0.41, width * 0.72]; + const annotations: AnnotationItem[] = [ + { id: 'label-input', targetId: 'input', kind: 'callout', start: 8, duration: 150, label: 'source event' }, + { id: 'label-model', targetId: 'model', kind: 'callout', start: 8, duration: 150, label: 'causal model' }, + { id: 'label-result', targetId: 'result', kind: 'callout', start: 8, duration: 150, label: 'verified result' }, + { id: 'route-around-model', targetId: 'input', toTargetId: 'result', kind: 'connector', start: 52, duration: 106, label: 'evidence flow' }, + ]; + return ( + + +
Attention stays readable
+
Labels share one layout pass. Routes avoid occupied regions.
+ {[ + ['input', 'Input', 'events', '#4f8cff'], + ['model', 'Model', 'meaning', '#f0a43c'], + ['result', 'Result', 'verified', '#21a179'], + ].map(([targetId, title, detail, color], index) => ( + +
{title}
+
{detail}
+ + ))} +
Deterministic at 16:9, 1:1, and 9:16
+ + + ); +} diff --git a/examples/compositions/attention-primitives-validation.tsx b/examples/compositions/attention-primitives-validation.tsx new file mode 100644 index 0000000..82354a3 --- /dev/null +++ b/examples/compositions/attention-primitives-validation.tsx @@ -0,0 +1,53 @@ +import React from 'react'; +import type { AttentionSequenceItem, RenderableMeta } from '@seqvio/core'; +import { VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const DURATION = 210; + +const attention: AttentionSequenceItem[] = [ + { id: 'ring', sceneId: 'primitives', targetId: 'input', kind: 'focus-ring', start: 6, duration: 32, persistence: 'timed', priority: 10 }, + { id: 'callout', sceneId: 'primitives', targetId: 'model', kind: 'callout', start: 42, duration: 34, persistence: 'timed', label: 'The key transformation', priority: 20 }, + { id: 'bracket', sceneId: 'primitives', targetId: 'process-explain', kind: 'bracket', start: 82, duration: 34, persistence: 'timed', label: 'semantic step', priority: 10 }, + { id: 'shade', sceneId: 'primitives', targetId: 'comparison', kind: 'region-shade', start: 122, duration: 80, persistence: 'timed', priority: 1 }, + { id: 'connector', sceneId: 'primitives', targetId: 'input', toTargetId: 'result', kind: 'connector', start: 162, duration: 40, persistence: 'timed', label: 'cause to outcome', priority: 30 }, +]; + +export default function AttentionPrimitivesValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, + duration: DURATION, + width: W, + height: H, +}; diff --git a/examples/compositions/attention-routing-validation.tsx b/examples/compositions/attention-routing-validation.tsx new file mode 100644 index 0000000..bb23571 --- /dev/null +++ b/examples/compositions/attention-routing-validation.tsx @@ -0,0 +1,46 @@ +import React from 'react'; +import type { AttentionSequenceItem, RenderableMeta } from '@seqvio/core'; +import { VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const DURATION = 210; + +const attention: AttentionSequenceItem[] = [ + { id: 'safe-callout', sceneId: 'routing', targetId: 'input', kind: 'callout', start: 4, duration: 42, persistence: 'timed', label: 'Placed inside the safe area' }, + { id: 'routed-connector', sceneId: 'routing', targetId: 'input', toTargetId: 'result', kind: 'connector', start: 54, duration: 44, persistence: 'timed', label: 'orthogonal route' }, + { id: 'guided-explanation', sceneId: 'routing', targetId: 'input', pathTargetIds: ['input', 'model', 'process-explain', 'result'], kind: 'guided-path', start: 108, duration: 88, persistence: 'timed' }, +]; + +export default function AttentionRoutingValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { fps: FPS, duration: DURATION, width: W, height: H }; diff --git a/examples/compositions/direction-plan-validation.tsx b/examples/compositions/direction-plan-validation.tsx new file mode 100644 index 0000000..a0874f3 --- /dev/null +++ b/examples/compositions/direction-plan-validation.tsx @@ -0,0 +1,69 @@ +import React from 'react'; +import type { AttentionSequenceItem, RenderableMeta } from '@seqvio/core'; +import { Scene, Transition, VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const SCENE_DURATION = 105; +const TRANSITION_DURATION = 15; +const DURATION = SCENE_DURATION * 2 + TRANSITION_DURATION; + +const attention: AttentionSequenceItem[] = [ + { id: 'follow-input', sceneId: 'model', targetId: 'input', kind: 'box', start: 10, duration: 28, persistence: 'timed' }, + { + id: 'match-output', sceneId: 'model', targetId: 'output', kind: 'spotlight', start: 45, duration: 48, + persistence: 'until-clear', clearAt: 100, handoffTo: 'resolved', handoffToSceneId: 'result', minHoldFrames: 36, + }, + { id: 'confirm-resolved', sceneId: 'result', targetId: 'resolved', kind: 'spotlight', start: 5, duration: 52, persistence: 'timed', minHoldFrames: 36 }, +]; + +function ModelScene() { + return ( + + + + ); +} + +function ResultScene() { + return ( + + + + ); +} + +export default function DirectionPlanValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { fps: FPS, duration: DURATION, width: W, height: H }; diff --git a/examples/compositions/director-skills-validation.tsx b/examples/compositions/director-skills-validation.tsx new file mode 100644 index 0000000..b970995 --- /dev/null +++ b/examples/compositions/director-skills-validation.tsx @@ -0,0 +1,55 @@ +import React from 'react'; +import type { AttentionSequenceItem, RenderableMeta } from '@seqvio/core'; +import { Scene, Transition, VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const SCENE_DURATION = 96; +const TRANSITION_DURATION = 10; +const DURATION = SCENE_DURATION * 3 + TRANSITION_DURATION * 2; + +const attention: AttentionSequenceItem[] = [ + { id: 'approved-focus', sceneId: 'approved', targetId: 'approved-ir', kind: 'spotlight', start: 12, duration: 42, persistence: 'timed' }, + { id: 'task-path', sceneId: 'host-task', targetId: 'semantic-task', pathTargetIds: ['semantic-task', 'reviewable-plan', 'versioned-result'], kind: 'guided-path', start: 16, duration: 64, persistence: 'timed' }, + { id: 'receipt-link', sceneId: 'local-validation', targetId: 'artifact-hash', toTargetId: 'receipt', kind: 'connector', start: 18, duration: 56, persistence: 'timed' }, +]; + +function Approved() { + return ; +} + +function HostTask() { + return ; +} + +function LocalValidation() { + return ; +} + +export default function DirectorSkillsValidation() { + return + + + + + + ; +} + +export const meta: RenderableMeta = { fps: FPS, duration: DURATION, width: W, height: H }; diff --git a/examples/compositions/generated/infographic-attention-validation.tsx b/examples/compositions/generated/infographic-attention-validation.tsx new file mode 100644 index 0000000..1dd0e8e --- /dev/null +++ b/examples/compositions/generated/infographic-attention-validation.tsx @@ -0,0 +1,424 @@ +// AUTO-GENERATED from a Seqvio ExplainerDocument. Safe to edit by hand. +import React from 'react'; +import type { RenderableMeta } from '@seqvio/core'; +import { VideoComposition, Scene, Transition } from '@seqvio/core'; + +import { + TechnicalScene, + AnnotationTarget, + CodeWalkthrough, + ArchitectureDiagram, + InfographicScene, + TerminalXtermDemo, +} from '@seqvio/technical'; + + +const W = 1280; +const H = 720; +const FPS = 30; + + +function ResultsScene0() { + return ( + + + + ); +} + +export default function InfographicAttentionValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, + duration: 244, + width: W, + height: H, + pacing: { profile: "explainer-v1", highlights: [ + { + "id": "results.throughput-beat", + "source": "beat", + "startFrame": 0, + "endFrame": 39, + "minDurationFrames": 27 + }, + { + "id": "results.latency-beat", + "source": "beat", + "startFrame": 39, + "endFrame": 78, + "minDurationFrames": 27 + }, + { + "id": "results.comparison-beat", + "source": "beat", + "startFrame": 78, + "endFrame": 123, + "minDurationFrames": 27 + }, + { + "id": "results.verify-beat", + "source": "beat", + "startFrame": 123, + "endFrame": 165, + "minDurationFrames": 27 + } +] }, + direction: { + "sceneActions": [ + { + "segmentId": "results.throughput-beat", + "sceneId": "results", + "purpose": "establish-model", + "pace": "hold", + "camera": "follow-target" + }, + { + "segmentId": "results.latency-beat", + "sceneId": "results", + "purpose": "establish-model", + "pace": "hold", + "camera": "follow-target" + }, + { + "segmentId": "results.comparison-beat", + "sceneId": "results", + "purpose": "establish-model", + "pace": "hold", + "camera": "follow-target" + }, + { + "segmentId": "results.verify-beat", + "sceneId": "results", + "purpose": "establish-model", + "pace": "hold", + "camera": "follow-target", + "transition": "cut" + } + ], + "attention": [ + { + "segmentId": "results.throughput-beat", + "sceneId": "results", + "targetId": "throughput", + "sourceBeatId": "throughput-beat", + "start": 0, + "duration": 1 + }, + { + "segmentId": "results.latency-beat", + "sceneId": "results", + "targetId": "latency", + "sourceBeatId": "latency-beat", + "start": 1, + "duration": 1 + }, + { + "segmentId": "results.comparison-beat", + "sceneId": "results", + "targetId": "build-time", + "sourceBeatId": "comparison-beat", + "start": 2, + "duration": 1 + }, + { + "segmentId": "results.verify-beat", + "sceneId": "results", + "targetId": "verify", + "sourceBeatId": "verify-beat", + "start": 3, + "duration": 1 + } + ], + "timingHints": [ + { + "segmentId": "results.throughput-beat", + "pace": "hold", + "minHoldFrames": 30 + }, + { + "segmentId": "results.latency-beat", + "pace": "hold", + "minHoldFrames": 30 + }, + { + "segmentId": "results.comparison-beat", + "pace": "hold", + "minHoldFrames": 30 + }, + { + "segmentId": "results.verify-beat", + "pace": "hold", + "minHoldFrames": 30 + } + ] +}, + audio: { + fps: FPS, + lockToAudio: true, + pacingProfile: "explainer-v1", + sceneTimings: [ + { + "sceneId": "results", + "startFrame": 0, + "durationFrames": 244, + "sourceDurationFrames": 244, + "transitionAfterFrames": 0, + "highlights": [ + { + "id": "results.throughput-beat", + "source": "beat", + "startFrame": 0, + "endFrame": 39, + "minDurationFrames": 27 + }, + { + "id": "results.latency-beat", + "source": "beat", + "startFrame": 39, + "endFrame": 78, + "minDurationFrames": 27 + }, + { + "id": "results.comparison-beat", + "source": "beat", + "startFrame": 78, + "endFrame": 123, + "minDurationFrames": 27 + }, + { + "id": "results.verify-beat", + "source": "beat", + "startFrame": 123, + "endFrame": 165, + "minDurationFrames": 27 + } + ] + } +], + explanationBeats: [ + { + "id": "results.throughput-beat", + "sceneId": "results", + "cueId": "results.guided", + "anchor": { + "text": "Throughput rises first" + }, + "sourceFrame": 0, + "visuals": [ + { + "targetId": "throughput", + "action": "reveal" + }, + { + "targetId": "throughput", + "action": "focus", + "minHoldMs": 1300 + } + ] + }, + { + "id": "results.latency-beat", + "sceneId": "results", + "cueId": "results.guided", + "anchor": { + "text": "Latency is the key measure" + }, + "sourceFrame": 39, + "visuals": [ + { + "targetId": "latency", + "action": "reveal" + }, + { + "targetId": "latency", + "action": "highlight", + "minHoldMs": 1300 + } + ] + }, + { + "id": "results.comparison-beat", + "sceneId": "results", + "cueId": "results.guided", + "anchor": { + "text": "compare the build time" + }, + "sourceFrame": 78, + "visuals": [ + { + "targetId": "build-time", + "action": "reveal" + }, + { + "targetId": "build-time", + "action": "annotate", + "minHoldMs": 1500 + } + ] + }, + { + "id": "results.verify-beat", + "sceneId": "results", + "cueId": "results.guided", + "anchor": { + "text": "verify the result" + }, + "sourceFrame": 123, + "visuals": [ + { + "targetId": "verify", + "action": "reveal" + }, + { + "targetId": "verify", + "action": "focus", + "minHoldMs": 1400 + } + ] + } +], + narration: [ + { + "id": "results.guided", + "sceneId": "results", + "text": "Throughput rises first. Latency is the key measure. Then compare the build time and verify the result.", + "startMs": 0, + "endMs": 7520 + } +], + }, +}; diff --git a/examples/compositions/generated/integrated-semantic-validation.tsx b/examples/compositions/generated/integrated-semantic-validation.tsx new file mode 100644 index 0000000..375c127 --- /dev/null +++ b/examples/compositions/generated/integrated-semantic-validation.tsx @@ -0,0 +1,318 @@ +// AUTO-GENERATED from a Seqvio ExplainerDocument. Safe to edit by hand. +import React from 'react'; +import type { RenderableMeta } from '@seqvio/core'; +import { StyleProfileProvider, VideoComposition, Scene, Transition } from '@seqvio/core'; + +import { + TechnicalScene, + AnnotationTarget, + CodeWalkthrough, + ArchitectureDiagram, + InfographicScene, + ManimClip, + TerminalXtermDemo, +} from '@seqvio/technical'; + + +const W = 1280; +const H = 720; +const FPS = 15; +const STYLE_PROFILE = { + "format": "seqvio-style-profile", + "version": "1.0", + "id": "clean-technical", + "label": "Clean Technical", + "typography": { + "headingFamily": "Inter", + "bodyFamily": "Inter", + "monoFamily": "JetBrains Mono", + "scale": "editorial" + }, + "motionDensity": "restrained", + "cameraPolicy": "semantic-focus", + "transitionPolicy": "focus-transfer", + "attentionPersistence": "until-handoff", + "spacing": "comfortable", + "paletteRoles": { + "background": "#0f172a", + "ink": "#f8fafc", + "accent": "#38bdf8", + "muted": "#94a3b8" + } +}; + + +function OverviewScene0() { + return ( + + + + ); +} + +function EquationScene1() { + return ( + + + + ); +} + +export default function IntegratedSemanticValidation() { + return ( + + + + + + + + + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, + duration: 147, + width: W, + height: H, + pacing: { profile: "explainer-v1", highlights: [ + { + "id": "overview.evidence-beat", + "source": "beat", + "startFrame": 0, + "endFrame": 14, + "minDurationFrames": 14 + }, + { + "id": "overview.model-beat", + "source": "beat", + "startFrame": 14, + "endFrame": 28, + "minDurationFrames": 14 + } +] }, + direction: { + "sceneActions": [ + { + "segmentId": "overview.evidence-beat", + "sceneId": "overview", + "purpose": "hook", + "pace": "steady", + "camera": "follow-target", + "transitionTargetId": "evidence", + "transitionToTargetId": "model" + }, + { + "segmentId": "overview.model-beat", + "sceneId": "overview", + "purpose": "hook", + "pace": "steady", + "camera": "follow-target", + "transition": "cut" + }, + { + "segmentId": "equation.overview", + "sceneId": "equation", + "purpose": "summarize", + "pace": "steady", + "camera": "overview", + "transition": "cut" + } + ], + "attention": [ + { + "segmentId": "overview.evidence-beat", + "sceneId": "overview", + "targetId": "evidence", + "sourceBeatId": "evidence-beat", + "start": 0, + "duration": 1 + }, + { + "segmentId": "overview.model-beat", + "sceneId": "overview", + "targetId": "model", + "sourceBeatId": "model-beat", + "start": 1, + "duration": 1 + } + ], + "timingHints": [ + { + "segmentId": "overview.evidence-beat", + "pace": "steady", + "minHoldFrames": 18 + }, + { + "segmentId": "overview.model-beat", + "pace": "steady", + "minHoldFrames": 18 + }, + { + "segmentId": "equation.overview", + "pace": "steady", + "minHoldFrames": 18 + } + ] +}, + audio: { + fps: FPS, + lockToAudio: true, + pacingProfile: "explainer-v1", + sceneTimings: [ + { + "sceneId": "overview", + "startFrame": 0, + "durationFrames": 75, + "sourceDurationFrames": 75, + "transitionAfterFrames": 12, + "highlights": [ + { + "id": "overview.evidence-beat", + "source": "beat", + "startFrame": 0, + "endFrame": 14, + "minDurationFrames": 14 + }, + { + "id": "overview.model-beat", + "source": "beat", + "startFrame": 14, + "endFrame": 28, + "minDurationFrames": 14 + } + ] + }, + { + "sceneId": "equation", + "startFrame": 87, + "durationFrames": 60, + "sourceDurationFrames": 60, + "transitionAfterFrames": 0, + "highlights": [] + } +], + explanationBeats: [ + { + "id": "overview.evidence-beat", + "sceneId": "overview", + "cueId": "overview.overview-cue", + "anchor": { + "text": "Evidence" + }, + "sourceFrame": 0, + "visuals": [ + { + "targetId": "evidence", + "action": "focus", + "minHoldMs": 800 + } + ] + }, + { + "id": "overview.model-beat", + "sceneId": "overview", + "cueId": "overview.overview-cue", + "anchor": { + "text": "explanation" + }, + "sourceFrame": 14, + "visuals": [ + { + "targetId": "model", + "action": "highlight", + "minHoldMs": 800 + } + ] + } +], + narration: [ + { + "id": "overview.overview-cue", + "sceneId": "overview", + "text": "Evidence becomes an explanation.", + "startMs": 0, + "endMs": 1840 + } +], + }, +}; diff --git a/examples/compositions/generated/manim-clip-validation.tsx b/examples/compositions/generated/manim-clip-validation.tsx new file mode 100644 index 0000000..11b0f26 --- /dev/null +++ b/examples/compositions/generated/manim-clip-validation.tsx @@ -0,0 +1,96 @@ +// AUTO-GENERATED from a Seqvio ExplainerDocument. Safe to edit by hand. +import React from 'react'; +import type { RenderableMeta } from '@seqvio/core'; +import { VideoComposition, Scene, Transition } from '@seqvio/core'; + +import { + TechnicalScene, + AnnotationTarget, + CodeWalkthrough, + ArchitectureDiagram, + InfographicScene, + ManimClip, + TerminalXtermDemo, +} from '@seqvio/technical'; + + +const W = 1280; +const H = 720; +const FPS = 30; + + +function ExternalAnimationScene0() { + return ( + + + + ); +} + +export default function ManimIrValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, + duration: 180, + width: W, + height: H, + pacing: { profile: "explainer-v1", highlights: [] }, + direction: { + "sceneActions": [ + { + "segmentId": "external-animation.overview", + "sceneId": "external-animation", + "purpose": "establish-model", + "pace": "steady", + "camera": "overview", + "transition": "cut" + } + ], + "attention": [], + "timingHints": [ + { + "segmentId": "external-animation.overview", + "pace": "steady", + "minHoldFrames": 18 + } + ] +}, +}; diff --git a/examples/compositions/generated/manim-real-validation.tsx b/examples/compositions/generated/manim-real-validation.tsx new file mode 100644 index 0000000..c92c4a0 --- /dev/null +++ b/examples/compositions/generated/manim-real-validation.tsx @@ -0,0 +1,85 @@ +// AUTO-GENERATED from a Seqvio ExplainerDocument. Safe to edit by hand. +import React from 'react'; +import type { RenderableMeta } from '@seqvio/core'; +import { VideoComposition, Scene, Transition } from '@seqvio/core'; + +import { + TechnicalScene, + AnnotationTarget, + CodeWalkthrough, + ArchitectureDiagram, + InfographicScene, + ManimClip, + TerminalXtermDemo, +} from '@seqvio/technical'; + + +const W = 1280; +const H = 720; +const FPS = 15; + + +function EquationScene0() { + return ( + + + + ); +} + +export default function ManimRealValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, + duration: 60, + width: W, + height: H, + pacing: { profile: "explainer-v1", highlights: [] }, + direction: { + "sceneActions": [ + { + "segmentId": "equation.overview", + "sceneId": "equation", + "purpose": "establish-model", + "pace": "steady", + "camera": "overview", + "transition": "cut" + } + ], + "attention": [], + "timingHints": [ + { + "segmentId": "equation.overview", + "pace": "steady", + "minHoldFrames": 18 + } + ] +}, +}; diff --git a/examples/compositions/infographic-chart-validation.tsx b/examples/compositions/infographic-chart-validation.tsx new file mode 100644 index 0000000..6f5d418 --- /dev/null +++ b/examples/compositions/infographic-chart-validation.tsx @@ -0,0 +1,49 @@ +import React from 'react'; +import { VideoComposition, type RenderableMeta } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const WIDTH = 1280; +const HEIGHT = 720; +const FPS = 30; +const DURATION = 210; + +export default function InfographicChartValidation() { + return ( + + + + + + ); +} + +export const meta: RenderableMeta = { width: WIDTH, height: HEIGHT, fps: FPS, duration: DURATION }; diff --git a/examples/compositions/infographic-responsive-validation-16x9.tsx b/examples/compositions/infographic-responsive-validation-16x9.tsx new file mode 100644 index 0000000..670b27a --- /dev/null +++ b/examples/compositions/infographic-responsive-validation-16x9.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { InfographicResponsiveValidation, infographicResponsiveMeta } from './infographic-responsive-validation-shared'; +export default function Validation() { return ; } +export const meta = infographicResponsiveMeta(1280, 720); diff --git a/examples/compositions/infographic-responsive-validation-1x1.tsx b/examples/compositions/infographic-responsive-validation-1x1.tsx new file mode 100644 index 0000000..40a0ecb --- /dev/null +++ b/examples/compositions/infographic-responsive-validation-1x1.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { InfographicResponsiveValidation, infographicResponsiveMeta } from './infographic-responsive-validation-shared'; +export default function Validation() { return ; } +export const meta = infographicResponsiveMeta(720, 720); diff --git a/examples/compositions/infographic-responsive-validation-9x16.tsx b/examples/compositions/infographic-responsive-validation-9x16.tsx new file mode 100644 index 0000000..2aee1a0 --- /dev/null +++ b/examples/compositions/infographic-responsive-validation-9x16.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { InfographicResponsiveValidation, infographicResponsiveMeta } from './infographic-responsive-validation-shared'; +export default function Validation() { return ; } +export const meta = infographicResponsiveMeta(720, 1280); diff --git a/examples/compositions/infographic-responsive-validation-shared.tsx b/examples/compositions/infographic-responsive-validation-shared.tsx new file mode 100644 index 0000000..7c1580e --- /dev/null +++ b/examples/compositions/infographic-responsive-validation-shared.tsx @@ -0,0 +1,31 @@ +import React from 'react'; +import { VideoComposition, type RenderableMeta } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const FPS = 15; +const DURATION = 105; + +export function infographicResponsiveMeta(width: number, height: number): RenderableMeta { + return { width, height, fps: FPS, duration: DURATION }; +} + +export function InfographicResponsiveValidation({ width, height, id }: { width: number; height: number; id: string }) { + return + + + + ; +} diff --git a/examples/compositions/infographic-validation.tsx b/examples/compositions/infographic-validation.tsx new file mode 100644 index 0000000..f5688f3 --- /dev/null +++ b/examples/compositions/infographic-validation.tsx @@ -0,0 +1,68 @@ +import React from 'react'; +import type { RenderableMeta } from '@seqvio/core'; +import { VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const DURATION = 240; + +function ValidationScene() { + return ( + + + + ); +} + +export default function InfographicValidation() { + return ( + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, + duration: DURATION, + width: W, + height: H, +}; diff --git a/examples/compositions/manim-clip-validation.tsx b/examples/compositions/manim-clip-validation.tsx new file mode 100644 index 0000000..27902bd --- /dev/null +++ b/examples/compositions/manim-clip-validation.tsx @@ -0,0 +1,40 @@ +import React from 'react'; +import type { RenderableMeta } from '@seqvio/core'; +import { VideoComposition } from '@seqvio/core'; +import { ManimClip } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const DURATION = 180; + +export default function ManimClipValidation() { + return ( + + + + ); +} + +export const meta: RenderableMeta = { + fps: FPS, duration: DURATION, width: W, height: H, + audio: { + fps: FPS, + lockToAudio: false, + explanationBeats: [ + { id: 'external-animation.explain-beat', sceneId: 'external-animation', cueId: 'narration', anchor: { text: 'explain' }, sourceFrame: 60, outputFrame: 82, visuals: [] }, + { id: 'external-animation.result-beat', sceneId: 'external-animation', cueId: 'narration', anchor: { text: 'result' }, sourceFrame: 140, outputFrame: 155, visuals: [] }, + ], + }, +}; diff --git a/examples/compositions/manim-end-to-end-validation.tsx b/examples/compositions/manim-end-to-end-validation.tsx new file mode 100644 index 0000000..b94feee --- /dev/null +++ b/examples/compositions/manim-end-to-end-validation.tsx @@ -0,0 +1,60 @@ +import React from 'react'; +import { Scene, VideoComposition, type RenderableMeta } from '@seqvio/core'; +import { ManimClip, TechnicalScene } from '@seqvio/technical'; + +const WIDTH = 1280; +const HEIGHT = 720; +const FPS = 30; +const GRAPH_DURATION = 213; +const PROOF_DURATION = 222; +const GRAPH = 'file:///D:/video-agent/seqvio/output/manim-fixtures/videos/graph/480p30/GraphExplanation.mp4'; +const PROOF = 'file:///D:/video-agent/seqvio/output/manim-fixtures/videos/proof/480p30/AlgebraProof.mp4'; + +function ClipFrame({ label, detail, children }: { label: string; detail: string; children: React.ReactNode }) { + return ( + +
+
{label}
+
{detail}
+
+
{children}
+ + ); +} + +export default function ManimEndToEndValidation() { + return ( + + + + + + + + + + + + + ); +} + +export const meta: RenderableMeta = { + width: WIDTH, + height: HEIGHT, + fps: FPS, + duration: GRAPH_DURATION + PROOF_DURATION, + audio: { + fps: FPS, + duration: GRAPH_DURATION + PROOF_DURATION, + lockToAudio: true, + narration: [ + { id: 'graph.voice', sceneId: 'graph', text: '先看曲线。点沿着曲线移动，让增长速度越来越快这件事直接可见。', startMs: 0, endMs: 5600 }, + { id: 'proof.voice', sceneId: 'proof', text: '再看代数证明。每一步只做等价变换，最后把勾股关系清楚地保留下来。', startMs: 6000, endMs: 11800 }, + ], + sceneTimings: [ + { sceneId: 'graph', startFrame: 0, durationFrames: GRAPH_DURATION, sourceDurationFrames: 180 }, + { sceneId: 'proof', startFrame: GRAPH_DURATION, durationFrames: PROOF_DURATION, sourceDurationFrames: 180 }, + ], + }, +}; diff --git a/examples/compositions/manim-geometric-proof-validation.tsx b/examples/compositions/manim-geometric-proof-validation.tsx new file mode 100644 index 0000000..1edb0b3 --- /dev/null +++ b/examples/compositions/manim-geometric-proof-validation.tsx @@ -0,0 +1,25 @@ +import React from 'react'; +import type { RenderableMeta } from '@seqvio/core'; +import { VideoComposition } from '@seqvio/core'; +import { ManimClip, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const DURATION = 123; + +export default function ManimGeometricProofValidation() { + return + + + + ; +} + +export const meta: RenderableMeta = { width: W, height: H, fps: FPS, duration: DURATION }; diff --git a/examples/compositions/motion-grammar-validation.tsx b/examples/compositions/motion-grammar-validation.tsx new file mode 100644 index 0000000..d8bef07 --- /dev/null +++ b/examples/compositions/motion-grammar-validation.tsx @@ -0,0 +1,81 @@ +import React from 'react'; +import type { MotionGrammar, RenderableMeta } from '@seqvio/core'; +import { compileMotionGrammar, Scene, Transition, VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +const W = 1280; +const H = 720; +const FPS = 30; +const SCENE_DURATION = 108; +const TRANSITION_DURATION = 12; +const DURATION = SCENE_DURATION * 3 + TRANSITION_DURATION * 2; + +const grammar: MotionGrammar = { + format: 'seqvio-motion-grammar', version: '1.0', id: 'explanation-patterns', + steps: [ + { id: 'compare-paths', sceneId: 'compare-merge', action: 'compare', targetId: 'manual', relatedTargetId: 'guided', startFrame: 22, holdFrames: 42 }, + { id: 'merge-result', sceneId: 'compare-merge', action: 'transform', targetId: 'merged', startFrame: 70, holdFrames: 28 }, + { id: 'show-problem', sceneId: 'problem-fix', action: 'emphasize', targetId: 'problem', startFrame: 12, holdFrames: 30 }, + { id: 'apply-fix', sceneId: 'problem-fix', action: 'transform', targetId: 'fix', startFrame: 48, holdFrames: 30 }, + { id: 'confirm-fix', sceneId: 'problem-fix', action: 'answer', targetId: 'verified', startFrame: 80, holdFrames: 22 }, + { id: 'trace-process', sceneId: 'process-verification', action: 'trace', targetId: 'observe', pathTargetIds: ['observe', 'decide', 'verify'], startFrame: 16, holdFrames: 62 }, + { id: 'show-verification', sceneId: 'process-verification', action: 'emphasize', targetId: 'verified-result', startFrame: 82, holdFrames: 22 }, + ], +}; + +const compiled = compileMotionGrammar(grammar); + +function CompareMerge() { + return ( + + + + ); +} + +function ProblemFix() { + return ( + + + + ); +} + +function ProcessVerification() { + return ( + + + + ); +} + +export default function MotionGrammarValidation() { + return ( + + + + + + + + ); +} + +export const meta: RenderableMeta = { fps: FPS, duration: DURATION, width: W, height: H }; diff --git a/examples/compositions/product-explainer-contract-validation.tsx b/examples/compositions/product-explainer-contract-validation.tsx new file mode 100644 index 0000000..82334b9 --- /dev/null +++ b/examples/compositions/product-explainer-contract-validation.tsx @@ -0,0 +1,72 @@ +import React from 'react'; +import { VideoComposition, useCurrentFrame, type RenderableMeta } from '@seqvio/core'; + +const WIDTH = 1280; +const HEIGHT = 720; +const FPS = 30; +const DURATION = 180; + +function ProductContractScene() { + const frame = useCurrentFrame(); + const progress = Math.min(1, Math.max(0, (frame - 18) / 72)); + const result = Math.min(1, Math.max(0, (frame - 92) / 44)); + return ( +
+
+
Product explainer contract
+

+ One beat. One focal point. +

+
+
+ verified-run.log +
+
+
$ seqvio verify contract
+
34 ? 1 : 0.15 }}>planning fields  ready
+
70 ? 1 : 0.15 }}>visual QA       ready
+
+
+
+
+ {[ + ['Hook', 'A concrete opening promise'], + ['Evidence', 'Truthful captured source'], + ['Transition', 'Focus follows meaning'], + ].map(([label, detail], index) => ( +
+ {label} + {detail} +
+ ))} +
+
+ + Planning and QA share one executable contract +
+
+
+ ); +} + +export default function ProductExplainerContractValidation() { + return ( + + + + ); +} + +export const meta: RenderableMeta = { width: WIDTH, height: HEIGHT, fps: FPS, duration: DURATION }; diff --git a/examples/compositions/seqvio-overview-shared.tsx b/examples/compositions/seqvio-overview-shared.tsx index d80bd23..508c382 100644 --- a/examples/compositions/seqvio-overview-shared.tsx +++ b/examples/compositions/seqvio-overview-shared.tsx @@ -19,28 +19,34 @@ import { Callout, ProductDemoScene, } from '@seqvio/product-demo'; -import seqvioIcon from '../../docs/assets/brand/seqvio-icon.svg'; -import seqvioIconSmall from '../../docs/assets/brand/seqvio-icon-small.svg'; +import seqvioMark from '../../docs/assets/brand/seqvio-mark.svg'; const W = 1280; const H = 720; export const OVERVIEW_FPS = 30; const C = { - navy: '#0B1020', - navy2: '#131C35', - paper: '#F6F8FC', - white: '#FFFFFF', - ink: '#101828', - muted: '#667085', - cyan: '#5EE7FF', - blue: '#38B6FF', - indigo: '#6E7BFF', - amber: '#F4B740', - green: '#32D583', - rose: '#FF6B7A', + navy: '#0C1118', + navy2: '#151C26', + paper: '#F3F5F7', + white: '#FCFDFE', + ink: '#171A1F', + muted: '#68717D', + cyan: '#78DCF4', + blue: '#3F9CF4', + indigo: '#7A7FF2', + amber: '#E8A23A', + green: '#36B98A', + rose: '#DF6D78', }; +const DARK_SURFACE = '#141B24'; +const DARK_SURFACE_RAISED = '#19222E'; +const DARK_BORDER = 'rgba(184, 205, 228, 0.16)'; +const LIGHT_BORDER = '#D9DEE5'; +const PANEL_RADIUS = 8; +const PANEL_SHADOW = '0 22px 60px rgba(7, 13, 22, 0.18)'; + const UI_STACK = 'Inter, "Segoe UI", Arial, sans-serif'; const MONO_STACK = '"JetBrains Mono", "Cascadia Code", Consolas, monospace'; const LONG_CANG_STACK = '"Long Cang", "Noto Sans SC", "Microsoft YaHei", cursive'; @@ -75,7 +81,7 @@ export interface OverviewCopy { hookTeaser?: string; /** Badge shown on the scene-4 player reveal, e.g. 'RENDERED BY SEQVIO · UNEDITED'. */ outputBadge?: string; - /** File name shown in the scene-4 player chrome, e.g. 'rag-explainer.mp4'. */ + /** File name shown in the scene-4 player chrome, e.g. 'technical-explainer.mp4'. */ playerFile?: string; /** Install command shown on the closing scene, e.g. 'npm install -g @seqvio/renderer'. */ ctaInstall?: string; @@ -131,62 +137,20 @@ function reveal(frame: number, start: number, duration = 18, distance = 20): CSS const p = easeOut((frame - start) / duration); return { opacity: p, - transform: `translateY(${(1 - p) * distance}px) scale(${0.97 + p * 0.03})`, + transform: `translateY(${(1 - p) * distance}px) scale(${0.985 + p * 0.015})`, + filter: `blur(${(1 - p) * 5}px)`, }; } function BrandBug({ light = false }: { light?: boolean }) { return ( -
- +
+ Seqvio
); } -function Rail({ children, light = false }: { children: ReactNode; light?: boolean }) { - return ( -
- {children} -
- ); -} - -function SceneMeta({ - index, - label, - light = false, -}: { - index: string; - label: string; - light?: boolean; -}) { - const color = light ? C.ink : '#D9E8FF'; - const line = light ? '#B9C5D8' : 'rgba(94,231,255,0.42)'; - return ( -
- {index} - {label} - -
- ); -} - function DarkStage({ children, dense = false, @@ -195,21 +159,16 @@ function DarkStage({ dense?: boolean; }) { const frame = useCurrentFrame(); - const drift = Math.sin(frame / 34) * 10; - const sweep = (frame * 1.8) % 420; + const drift = Math.sin(frame / 52) * 6; return (
-
-
+
+
{dense ? ( <> -
-
-
-
- {[0, 1, 2, 3, 4].map((index) => ( - - ))} +
+
+
) : null} @@ -228,13 +187,13 @@ function LightStage({ const frame = useCurrentFrame(); return (
-
+
+
{dense ? ( <> -
-
-
-
+
+
+
) : null} @@ -243,131 +202,122 @@ function LightStage({ ); } +function SoftSceneReveal({ light = false }: { light?: boolean }) { + const frame = useCurrentFrame(); + const p = easeOut(frame / 10); + return ( +
+ ); +} + function HookScene({ copy, duration, enhanced }: { copy: OverviewCopy; duration: number; enhanced: boolean }) { const frame = useCurrentFrame() * 126 / Math.max(1, duration); - const fragments = [ - { label: 'CLIP 01', x: 112, y: 180, color: C.indigo, delay: 0 }, - { label: 'B-ROLL', x: 922, y: 144, color: C.amber, delay: 6 }, - { label: 'MOTION', x: 850, y: 430, color: C.blue, delay: 12 }, - { label: 'FX', x: 160, y: 452, color: C.rose, delay: 18 }, - ]; - const focus = easeOut((frame - 52) / 28); + const terminalP = easeOut((frame - 2) / 18); + const commandP = easeOut((frame - 36) / 16); + const outputP = easeOut((frame - 66) / 24); + const progress = Math.min(100, Math.max(0, (frame - 58) * 2.1)); + const cursorOn = Math.floor(frame / 8) % 2 === 0; return ( - - {enhanced ? ( - <> - -
- {[18, 34, 22, 48, 30, 58, 26, 42, 20, 52, 32, 62].map((height, index) => ( - - ))} +
+
+
+
+ {[C.rose, C.amber, C.green].map((color) => )} + agent-session / completed task + ● VERIFIED +
+
+
$ codex run technical-workflow
+
+
✓ task complete
+
✓ evidence captured
+
✓ result verified
-
- +
+ $ /seqvio explain +
- {copy.hookTeaser ? ( -
-
{copy.hookTeaser}
-
-
- {[C.rose, C.amber, C.green].map((color) => )} - {copy.playerFile ?? 'rag-explainer.mp4'} -
-
- {[C.blue, C.indigo, C.amber, C.green].map((color, index) => ( - - - {index < 3 ? : null} - - ))} - - - -
-
- - - - 0:09 -
-
-
- ) : ( -
- 00:00/00:42 +
+
+
+
+
+
explanation.preview
+
+
+ WORK + + EXPLANATION
- )} - - ) : null} - {fragments.map((item, index) => { - const p = easeOut((frame - item.delay) / 22) * (1 - focus * 0.78); - const float = Math.sin((frame + index * 13) / 15) * 8; - return ( -
- {item.label}
- ); - })} -
-
{copy.hookTop}
-
{copy.hookBottom}
+
- {copy.hookRail} - + {enhanced ?
{copy.hookTeaser ?? 'MADE BY AN AGENT'}
: null} +
); } function PromiseScene({ copy, duration, enhanced }: { copy: OverviewCopy; duration: number; enhanced: boolean }) { const frame = useCurrentFrame() * 168 / Math.max(1, duration); - const iconP = easeOut(frame / 30); - const nodes = [ - { label: copy.vocabulary[0], x: 142, y: 270, color: C.amber, start: 42 }, - { label: copy.vocabulary[1], x: 920, y: 210, color: C.green, start: 62 }, - { label: copy.vocabulary[2], x: 890, y: 430, color: C.rose, start: 82 }, + const flowP = easeOut((frame - 18) / 110); + const previewP = easeOut((frame - 82) / 30); + const stages = [ + { label: copy.vocabulary[0], color: C.amber, start: 24 }, + { label: copy.vocabulary[1], color: C.blue, start: 52 }, + { label: copy.vocabulary[2], color: C.green, start: 78 }, ]; return ( - {enhanced ? : null} -
-
{copy.promiseTitle}
+
AGENT WORK
+
+
agent-session.log
+
+
✓ task complete
+
✓ browser captured
+
✓ result verified
+
/seqvio explain
+
- - {enhanced ? ( - <> - - - {[0, 1, 2, 3, 4, 5].map((index) => { - const angle = frame * 0.012 + index * Math.PI / 3; - return ; - })} - - ) : null} - {nodes.map((node) => { - const p = easeOut((frame - node.start) / 26); - return ; +
+ {stages.map((stage, index) => { + const p = easeOut((frame - stage.start) / 22); + return ( +
+ {String(index + 1).padStart(2, '0')} + {stage.label} +
+ ); })} - - - {enhanced ? ( - <> -
AGENT INPUT
-
VISUAL OUTPUT
- - ) : null} - {nodes.map((node) => ( -
- {node.label} +
+
+
+
technical-explainer.mp4
+
+ +
+ INTENTEVIDENCECLARITY +
+
- ))} - {copy.promiseRail} +
WORK → EXPLANATION
+
); } function TerminalWindow({ children }: { children: ReactNode }) { return ( -
-
+
+
{[C.rose, C.amber, C.green].map((color) => )} agent / seqvio
@@ -380,15 +330,13 @@ function PromptScene({ copy, duration, enhanced }: { copy: OverviewCopy; duratio const frame = useCurrentFrame() * 228 / Math.max(1, duration); return ( - {enhanced ? : null} -
{copy.promptLabel}
$ codex
> {copy.promptText}
/seqvio  planning explainer structure...
-
-
+
+
@@ -399,19 +347,18 @@ function PromptScene({ copy, duration, enhanced }: { copy: OverviewCopy; duratio ) : null} -
+
{copy.files.map((file, index) => ( -
- {String(index + 1).padStart(2, '0')}{file} +
+ {String(index + 1).padStart(2, '0')}{file}
))}
{enhanced ? ( -
- PLAN●AUTHOR●ALIGN●RENDER +
+ TASKEXPLAINER DOCUMENT
) : null} - {copy.promptRail} ); } @@ -425,7 +372,7 @@ function RagExplanationScene({ copy, duration, enhanced }: { copy: OverviewCopy; const t = (frame: number) => Math.max(1, Math.round(frame * drawDuration / 330)); const pull = enhanced ? easeOut((frame - (drawDuration - 18)) / 40) : 0; const badgeP = enhanced ? easeOut((frame - (drawDuration + 4)) / 16) : 0; - const boardScale = 1 - pull * 0.26; + const boardScale = 1 - pull * 0.34; const boardW = W * boardScale; const boardH = H * boardScale; const boardX = (W - boardW) / 2; @@ -435,6 +382,7 @@ function RagExplanationScene({ copy, duration, enhanced }: { copy: OverviewCopy; ? ['观察', '追踪', '诊断', '验证'] : ['OBSERVE', 'TRACE', 'DIAGNOSE', 'VERIFY']; const icons = ['lightbulb', 'plus', 'document', 'check']; + const visualLink = copy.lang === 'zh' ? '旁白 ↔ 视觉动作' : 'VOICE ↔ VISUAL ACTION'; return (
-
+
{[C.rose, C.amber, C.green].map((color) => )} - {copy.playerFile ?? 'rag-explainer.mp4'} + {copy.playerFile ?? 'technical-explainer.mp4'} 1080p
-
+
@@ -623,17 +573,19 @@ function RagExplanationScene({ copy, duration, enhanced }: { copy: OverviewCopy; style={{ position: 'absolute', left: boardX + boardW - 330, - top: boardY - 74, + top: boardY + boardH - 96, zIndex: 30, padding: '10px 16px', - background: C.green, - color: C.navy, + border: `1px solid ${C.green}55`, + borderRadius: 4, + background: '#E8F7F1', + color: '#157052', fontFamily: MONO_STACK, fontSize: 15, fontWeight: 800, letterSpacing: 0.6, whiteSpace: 'nowrap', - transform: `rotate(-4deg) translateY(${(1 - badgeP) * 14}px)`, + transform: `translateY(${(1 - badgeP) * 14}px)`, opacity: badgeP, boxShadow: '0 14px 30px rgba(16,24,40,0.3)', }} @@ -642,7 +594,6 @@ function RagExplanationScene({ copy, duration, enhanced }: { copy: OverviewCopy;
) : null} - {enhanced ? : null}
); } @@ -751,19 +702,19 @@ function MiniProductDemo() { position={{ x: 18, y: 38 }} width={308} height={248} - url="ci-run.local" + url="workflow.local" title="captured evidence" start={18} duration={22} >
- Native module load + Captured workflow
- {['BUILD', 'PTY', 'OK'].map((item, index) => ( + {['STEP', 'CHECK', 'OK'].map((item, index) => (
, color: C.blue, width: 370, offset: 0, mark: '01' }, - { label: copy.styleLabels[1], child: , color: C.amber, width: 340, offset: 26, mark: '02' }, - { label: copy.styleLabels[2], child: , color: C.green, width: 370, offset: 0, mark: '03' }, + const views = [ + { label: copy.styleLabels[0], short: 'MODEL', child: , color: C.blue, start: 16 }, + { label: copy.styleLabels[1], short: 'PLAN', child: , color: C.amber, start: 88 }, + { label: copy.styleLabels[2], short: 'EVIDENCE', child: , color: C.green, start: 160 }, ]; return ( - {enhanced ? : null} -
- {copy.stylesTitle} +
ONE IDEA. ONE VISUAL.
+
+ {views.map((view, index) => { + const enter = easeOut((frame - view.start) / 20); + const exit = index < views.length - 1 ? easeOut((frame - (view.start + 58)) / 14) : 0; + const opacity = enter * (1 - exit); + return ( +
+
{view.child}
+
+ ); + })}
-
- {cards.map((card, index) => ( -
-
- {enhanced ?
: null} -
{card.child}
-
- {card.label} -
+
+ {views.map((view, index) => { + const p = easeOut((frame - view.start) / 20); + const active = frame >= view.start && (index === views.length - 1 || frame < views[index + 1].start); + return ( +
+ +
{view.short}
+
{view.label}
- {enhanced ? ( - {card.mark} - ) : null} -
- ))} + ); + })} +
- {copy.stylesRail} ); } @@ -877,35 +788,26 @@ function StylesScene({ copy, duration, enhanced }: { copy: OverviewCopy; duratio function ProofScene({ copy, duration, enhanced }: { copy: OverviewCopy; duration: number; enhanced: boolean }) { const frame = useCurrentFrame() * 246 / Math.max(1, duration); const commands = [ - 'seqvio-generate validate --ir explainer.json --ir-format explainer', - 'seqvio-audio synthesize --manifest audio.json', - 'seqvio-qa --frames 0,120,240', - 'seqvio-render --component rag-explainer.tsx', + 'seqvio validate explainer.json', + 'seqvio align --voice audio.json', + 'seqvio qa --layout --media --evidence', + 'seqvio render technical-explainer.tsx', ]; return ( - {enhanced ? : null} -
- {copy.proofTitle} -
+
LOCAL RENDER / AUTOMATED CHECKS
- real Seqvio CLI - {enhanced ? RUN 04/04 : null} + seqvio / render + {enhanced ? RUNNING LOCALLY : null}
-
- {enhanced ?
: null} +
+ {enhanced ?
: null} {commands.map((command, index) => (
{enhanced ? {index + 1} : null} $ {command} ✓ - {copy.proofOutputs ? ( -
{copy.proofOutputs[index]}
- ) : null}
))}
-
-
+
+
{[0, 1, 2, 3].map((index) => (
-
-
- {enhanced ? ✓ : null} -
-
- {enhanced ? [22, 34, 18, 40, 28, 38].map((height, barIndex) => ) : null} -
-
- {copy.checks[index]} -
+ ✓ + {copy.checks[index]}
))}
{enhanced ? : null} - output/seqvio-rag.mp4 - {copy.proofOutputs ? ( -
1920×1080 · 0:09 · H.264
- ) : null} + output/technical-explainer.mp4 +
1920×1080 · H.264 · VERIFIED
- {enhanced ? ( -
- {Array.from({ length: 32 }).map((_, index) => )} -
- ) : null} - {copy.proofRail} ); } @@ -1012,31 +891,20 @@ function ProofScene({ copy, duration, enhanced }: { copy: OverviewCopy; duration function ClosingScene({ copy, duration, enhanced }: { copy: OverviewCopy; duration: number; enhanced: boolean }) { const frame = useCurrentFrame() * 228 / Math.max(1, duration); const p = easeOut((frame - 18) / 30); + const raysP = easeOut((frame - 44) / 28); return ( - {enhanced ? : null}
- {copy.closeKicker} -
-
- {copy.closeRail} -
-
-
+
$ {copy.ctaInstall}
- {copy.ctaStar ? ( -
- ★ - {copy.ctaStar} -
- ) : null}
) : null}
@@ -1097,45 +951,29 @@ function ClosingScene({ copy, duration, enhanced }: { copy: OverviewCopy; durati style={{ position: 'absolute', right: 92, - top: 166, + top: 174, width: 310, height: 310, - transform: `scale(${0.82 + p * 0.18}) rotate(${(1 - p) * 9}deg)`, + transform: `scale(${0.82 + p * 0.18})`, opacity: p, }} > {enhanced ? ( {[0, 1, 2, 3, 4, 5, 6, 7].map((index) => { - const angle = index * Math.PI / 4 + frame * 0.006; - return ; + const angle = index * Math.PI / 4; + const inner = 166 + raysP * 10; + const outer = 174 + raysP * 30; + return ; })} ) : null} -
-
- +
{enhanced && !copy.ctaInstall ? (
{copy.styleLabels.map((label, index) => ( - {label} + {label} ))}
) : null} @@ -1145,8 +983,8 @@ function ClosingScene({ copy, duration, enhanced }: { copy: OverviewCopy; durati left: 76, right: 76, bottom: 46, - height: 2, - background: 'linear-gradient(90deg, #5EE7FF, #6E7BFF, #32D583)', + height: 1, + background: DARK_BORDER, }} /> @@ -1179,22 +1017,22 @@ export function SeqvioOverview({ - + - + - + - + - + - + ); diff --git a/examples/compositions/seqvio-product-hunt-en.tsx b/examples/compositions/seqvio-product-hunt-en.tsx index e1d7de6..b9ce076 100644 --- a/examples/compositions/seqvio-product-hunt-en.tsx +++ b/examples/compositions/seqvio-product-hunt-en.tsx @@ -12,39 +12,39 @@ const SCENES = [62, 123, 178, 313, 196, 190, 223] as const; const copy: OverviewCopy = { lang: 'en', - hookTop: 'A video can move', - hookBottom: 'and still explain nothing.', - hookRail: 'Motion is not the same as explanation.', - promiseTitle: 'Explain what really happened.', - promiseRail: 'Seqvio turns technical ideas and system evidence into narrated video.', - vocabulary: ['PLAN', 'EVIDENCE', 'BEATS'], - promptLabel: 'A REAL TECHNICAL TASK, INSIDE A CODING AGENT', - promptText: 'Using /seqvio, explain why a native module failed in CI and how the repair was verified.', - promptRail: 'The host agent makes the editorial decisions. Seqvio makes them executable and checkable.', + hookTop: 'Your coding agent did the work.', + hookBottom: 'Now let it explain what actually happened.', + hookRail: 'Seqvio turns technical work into a clear, verifiable video.', + promiseTitle: 'A visual language for technical explanation.', + promiseRail: 'Built around intent, evidence, and understanding.', + vocabulary: ['INTENT', 'EVIDENCE', 'CLARITY'], + promptLabel: 'A TECHNICAL TASK, DOCUMENT, OR WORKFLOW', + promptText: 'Using /seqvio, explain how this technical workflow works and what viewers need to understand.', + promptRail: 'The agent decides what viewers need to learn. Seqvio makes the explanation executable and checkable.', files: ['EDITORIAL.md', 'explainer.json', 'qa-report.json'], - ragTitle: 'Why the native module failed in CI', - ragSteps: ['SYMPTOM', 'LOAD PATH', 'ROOT CAUSE', 'VERIFIED'], - ragRail: 'The claim stays connected to the observed evidence.', - stylesTitle: 'The right evidence for each claim.', - stylesRail: 'Use authored scenes for models and capture-derived scenes for what actually happened.', - styleLabels: ['AUTHORED MODEL', 'REVIEWABLE PLAN', 'CAPTURE EVIDENCE'], - styleNotes: ['expected path', 'claim', 'evidence', 'repair'], - proofTitle: 'Real workflow. Real output.', - proofRail: 'Author, validate, align, check, and render with the tools that ship today.', - checks: ['structure', 'voice timing', 'key frame', 'final render'], + ragTitle: 'How technical work becomes an explanation', + ragSteps: ['QUESTION', 'MODEL', 'EVIDENCE', 'VERIFIED'], + ragRail: 'Each spoken point stays connected to a meaningful visual action.', + stylesTitle: 'Use the right visual for each idea.', + stylesRail: 'Explain ideas with authored scenes. Show real activity with terminal and browser capture.', + styleLabels: ['AUTHORED MODEL', 'EXPLANATION PLAN', 'CAPTURE EVIDENCE'], + styleNotes: ['concept', 'intent', 'evidence', 'result'], + proofTitle: 'Aligned, checked, and rendered locally.', + proofRail: 'Timing, layout, media, and evidence are checked before delivery.', + checks: ['timing', 'layout', 'media', 'evidence'], closeKicker: 'OPEN SOURCE / AGENT-NATIVE EXPLAINERS', - closeTitle: 'Teach your agent to explain.', - closeRail: 'Not just generated motion. A reviewable explanation grounded in evidence.', + closeTitle: 'Turn work into explanation.', + closeRail: "Turn your agent's work into an explanation people can follow.", cta: 'github.com/makesynt/seqvio', hookTeaser: 'MADE BY AN AGENT', - outputBadge: 'RENDERED BY SEQVIO · UNEDITED', - playerFile: 'native-module-ci.mp4', + outputBadge: 'RENDERED BY SEQVIO · VERIFIED', + playerFile: 'technical-explainer.mp4', ctaInstall: 'npm install -g @seqvio/renderer', ctaStar: 'Star on GitHub', proofOutputs: [ - 'plan reviewed · omissions preserved', + 'intent reviewed · learning goal preserved', 'beats resolved · voice is the clock', - 'QA passed · evidence and highlights checked', + 'QA passed · layout and evidence checked', 'local MP4 · reproducible render artifacts', ], }; @@ -53,37 +53,37 @@ const narration = [ { id: 'hook', sceneId: 'hook', - text: 'A video can move, and still explain nothing.', + text: 'Your coding agent did the work. Now let it explain what actually happened. Seqvio turns technical work into a clear, verifiable video.', }, { id: 'promise', sceneId: 'promise', - text: 'Seqvio gives coding agents a visual language for explaining ideas.', + text: 'Seqvio gives agents a visual language built around intent, evidence, and understanding.', }, { id: 'prompt', sceneId: 'prompt', - text: 'Give your agent a real topic: explain how retrieval augmented generation turns a question into an answer.', + text: 'Start with a task, document, or workflow. The agent decides what viewers need to learn.', }, { id: 'explanation', sceneId: 'explanation', - text: 'The agent reviews the explanation plan and visual direction. Seqvio compiles them into an ExplainerDocument, then binds each spoken phrase to a visual action and observed evidence.', + text: 'The plan becomes an ExplainerDocument. ExplanationBeats connect spoken phrases to visual actions and captured evidence.', }, { id: 'styles', sceneId: 'styles', - text: 'Use whiteboard, code, and diagram scenes for authored models. Use terminal or browser capture when the explanation depends on what actually happened.', + text: 'Use code and diagrams for ideas. Use terminal and browser sessions to show what actually happened.', }, { id: 'proof', sceneId: 'proof', - text: 'Seqvio resolves post-TTS timing, checks semantic beats and key frames, reports capture problems, and renders the final video locally.', + text: 'Seqvio aligns scenes to the final voice, checks timing, layout, media, and evidence, then renders locally.', }, { id: 'closing', sceneId: 'closing', - text: 'Not just generated motion. A reviewable explanation grounded in evidence. Teach your agent to explain with Seqvio. Open source on GitHub.', + text: "No disconnected motion. No timeline guesswork. Turn your agent's work into an explanation people can follow.", }, ]; diff --git a/examples/compositions/style-playbook-clean-technical.tsx b/examples/compositions/style-playbook-clean-technical.tsx new file mode 100644 index 0000000..aed9c54 --- /dev/null +++ b/examples/compositions/style-playbook-clean-technical.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { cleanTechnical, StylePlaybookValidation, stylePlaybookMeta } from './style-playbook-validation-shared'; +export default function Validation() { return ; } +export const meta = stylePlaybookMeta; diff --git a/examples/compositions/style-playbook-editorial-explainer.tsx b/examples/compositions/style-playbook-editorial-explainer.tsx new file mode 100644 index 0000000..ae7e621 --- /dev/null +++ b/examples/compositions/style-playbook-editorial-explainer.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { editorialExplainer, StylePlaybookValidation, stylePlaybookMeta } from './style-playbook-validation-shared'; +export default function Validation() { return ; } +export const meta = stylePlaybookMeta; diff --git a/examples/compositions/style-playbook-terminal-first.tsx b/examples/compositions/style-playbook-terminal-first.tsx new file mode 100644 index 0000000..c83d79b --- /dev/null +++ b/examples/compositions/style-playbook-terminal-first.tsx @@ -0,0 +1,4 @@ +import React from 'react'; +import { StylePlaybookValidation, stylePlaybookMeta, terminalFirst } from './style-playbook-validation-shared'; +export default function Validation() { return ; } +export const meta = stylePlaybookMeta; diff --git a/examples/compositions/style-playbook-validation-shared.tsx b/examples/compositions/style-playbook-validation-shared.tsx new file mode 100644 index 0000000..3f194c2 --- /dev/null +++ b/examples/compositions/style-playbook-validation-shared.tsx @@ -0,0 +1,63 @@ +import React from 'react'; +import type { AttentionSequenceItem, RenderableMeta, StyleProfile } from '@seqvio/core'; +import { Scene, StyleProfileProvider, Transition, VideoComposition } from '@seqvio/core'; +import { InfographicScene, TechnicalScene } from '@seqvio/technical'; + +export const cleanTechnical: StyleProfile = { + format: 'seqvio-style-profile', version: '1.0', id: 'clean-technical', label: 'Clean Technical', + typography: { headingFamily: 'Inter', bodyFamily: 'Inter', monoFamily: 'JetBrains Mono', scale: 'editorial' }, + motionDensity: 'restrained', cameraPolicy: 'semantic-focus', transitionPolicy: 'focus-transfer', + attentionPersistence: 'until-handoff', spacing: 'comfortable', + paletteRoles: { background: '#0f172a', ink: '#f8fafc', accent: '#38bdf8', muted: '#94a3b8' }, +}; +export const editorialExplainer: StyleProfile = { + format: 'seqvio-style-profile', version: '1.0', id: 'editorial-explainer', label: 'Editorial Explainer', + typography: { headingFamily: 'Georgia', bodyFamily: 'Inter', monoFamily: 'JetBrains Mono', scale: 'large' }, + motionDensity: 'balanced', cameraPolicy: 'static', transitionPolicy: 'crossfade', + attentionPersistence: 'until-handoff', spacing: 'airy', + paletteRoles: { background: '#f7f8fa', ink: '#17202a', accent: '#2563eb', muted: '#64748b' }, +}; +export const terminalFirst: StyleProfile = { + format: 'seqvio-style-profile', version: '1.0', id: 'terminal-first', label: 'Terminal First', + typography: { headingFamily: 'JetBrains Mono', bodyFamily: 'Inter', monoFamily: 'JetBrains Mono', scale: 'compact' }, + motionDensity: 'restrained', cameraPolicy: 'evidence-follow', transitionPolicy: 'cut', + attentionPersistence: 'timed', spacing: 'tight', + paletteRoles: { background: '#101418', ink: '#e6edf3', accent: '#2dd4bf', muted: '#8b9aaa' }, +}; + +const W = 1280; +const H = 720; +const FPS = 30; +const SCENE_DURATION = 84; +const TRANSITION_DURATION = 12; +const DURATION = SCENE_DURATION * 2 + TRANSITION_DURATION; +const attention: AttentionSequenceItem[] = [ + { id: 'evidence-focus', sceneId: 'evidence', targetId: 'evidence', kind: 'focus-ring', start: 16, duration: 48, handoffTo: 'verified', handoffToSceneId: 'result' }, + { id: 'result-focus', sceneId: 'result', targetId: 'verified', kind: 'spotlight', start: 12, duration: 52 }, +]; + +export function StylePlaybookValidation({ profile }: { profile: StyleProfile }) { + return + + + + + + ; +} + +export const stylePlaybookMeta: RenderableMeta = { width: W, height: H, fps: FPS, duration: DURATION }; diff --git a/examples/direction/flow.direction.json b/examples/direction/flow.direction.json new file mode 100644 index 0000000..9db904b --- /dev/null +++ b/examples/direction/flow.direction.json @@ -0,0 +1,19 @@ +{ + "format": "seqvio-direction-plan", + "version": "1.0", + "id": "flow-direction", + "segments": [ + { + "id": "explain-flow", + "sceneId": "routing", + "purpose": "explain-mechanism", + "pace": "steady", + "focus": "sequence", + "focusSpec": { + "targetIds": ["input", "model", "result"] + }, + "camera": "follow-target", + "transition": "cut" + } + ] +} diff --git a/examples/direction/focus-transfer.direction.json b/examples/direction/focus-transfer.direction.json new file mode 100644 index 0000000..57f733a --- /dev/null +++ b/examples/direction/focus-transfer.direction.json @@ -0,0 +1,29 @@ +{ + "format": "seqvio-direction-plan", + "version": "1.0", + "id": "focus-transfer-direction", + "segments": [ + { + "id": "trace-model", + "sceneId": "model", + "purpose": "explain-mechanism", + "pace": "build", + "focus": "sequence", + "focusSpec": { "targetIds": ["input", "output"] }, + "camera": "focus-transfer", + "transition": "match-object", + "transitionTargetId": "output", + "transitionToTargetId": "resolved" + }, + { + "id": "confirm-result", + "sceneId": "result", + "purpose": "summarize", + "pace": "resolve", + "focus": "target", + "focusSpec": { "targetId": "resolved" }, + "camera": "follow-target", + "transition": "cut" + } + ] +} diff --git a/examples/director/invalid-candidate.json b/examples/director/invalid-candidate.json new file mode 100644 index 0000000..a0a9676 --- /dev/null +++ b/examples/director/invalid-candidate.json @@ -0,0 +1,43 @@ +{ + "directionPlan": { + "format": "seqvio-direction-plan", + "version": "1.0", + "id": "integrated-semantic-validation.candidate", + "segments": [ + { + "id": "overview-conflict", + "sceneId": "overview", + "purpose": "explain-mechanism", + "focus": "target", + "focusSpec": { "targetIds": ["evidence", "missing"] }, + "camera": "overview", + "transition": "cut" + } + ] + }, + "attentionSequence": [ + { + "id": "unbounded-focus", + "sceneId": "overview", + "targetId": "evidence", + "kind": "spotlight", + "start": 12, + "duration": 24, + "persistence": "until-clear" + } + ], + "motionGrammar": { + "format": "seqvio-motion-grammar", + "version": "1.0", + "id": "integrated-semantic-validation.candidate-motion", + "steps": [ + { + "id": "incomplete-trace", + "sceneId": "overview", + "action": "trace", + "targetId": "evidence", + "pathTargetIds": ["evidence"] + } + ] + } +} diff --git a/examples/ir/infographic-attention-validation.explainer.json b/examples/ir/infographic-attention-validation.explainer.json new file mode 100644 index 0000000..b314479 --- /dev/null +++ b/examples/ir/infographic-attention-validation.explainer.json @@ -0,0 +1,73 @@ +{ + "format": "seqvio-explainer", + "schemaVersion": "1.0", + "id": "infographic-attention-validation", + "width": 1280, + "height": 720, + "fps": 30, + "scenes": [ + { + "type": "infographic", + "id": "results", + "title": "One explanation, guided attention", + "duration": 240, + "metrics": [ + { "id": "throughput", "label": "Throughput", "value": "4.8x", "detail": "after batching", "color": "#38bdf8" }, + { "id": "latency", "label": "Latency", "value": "120ms", "detail": "p95 response", "color": "#34d399" }, + { "id": "errors", "label": "Errors", "value": "0.7%", "detail": "verified run", "color": "#fbbf24" } + ], + "comparisons": [ + { "id": "build-time", "label": "Build time", "before": 18, "after": 7, "beforeLabel": "Before", "afterLabel": "After" }, + { "id": "review-steps", "label": "Review steps", "before": 9, "after": 4, "beforeLabel": "Manual", "afterLabel": "Guided" } + ], + "process": [ + { "id": "capture", "label": "Capture", "detail": "record the real event" }, + { "id": "explain", "label": "Explain", "detail": "show the causal model" }, + { "id": "verify", "label": "Verify", "detail": "check the result" } + ], + "explanation": { + "cues": [ + { "id": "guided", "text": "Throughput rises first. Latency is the key measure. Then compare the build time and verify the result." } + ], + "beats": [ + { + "id": "throughput-beat", + "cueId": "guided", + "anchor": { "text": "Throughput rises first" }, + "visuals": [ + { "targetId": "throughput", "action": "reveal" }, + { "targetId": "throughput", "action": "focus", "minHoldMs": 1300 } + ] + }, + { + "id": "latency-beat", + "cueId": "guided", + "anchor": { "text": "Latency is the key measure" }, + "visuals": [ + { "targetId": "latency", "action": "reveal" }, + { "targetId": "latency", "action": "highlight", "minHoldMs": 1300 } + ] + }, + { + "id": "comparison-beat", + "cueId": "guided", + "anchor": { "text": "compare the build time" }, + "visuals": [ + { "targetId": "build-time", "action": "reveal" }, + { "targetId": "build-time", "action": "annotate", "minHoldMs": 1500 } + ] + }, + { + "id": "verify-beat", + "cueId": "guided", + "anchor": { "text": "verify the result" }, + "visuals": [ + { "targetId": "verify", "action": "reveal" }, + { "targetId": "verify", "action": "focus", "minHoldMs": 1400 } + ] + } + ] + } + } + ] +} diff --git a/examples/ir/integrated-semantic-validation.explainer.json b/examples/ir/integrated-semantic-validation.explainer.json new file mode 100644 index 0000000..971e9fd --- /dev/null +++ b/examples/ir/integrated-semantic-validation.explainer.json @@ -0,0 +1,52 @@ +{ + "format": "seqvio-explainer", + "schemaVersion": "1.0", + "id": "integrated-semantic-validation", + "width": 1280, + "height": 720, + "fps": 15, + "backgroundColor": "#0f172a", + "styleProfile": { + "format": "seqvio-style-profile", + "version": "1.0", + "id": "clean-technical", + "label": "Clean Technical", + "typography": { "headingFamily": "Inter", "bodyFamily": "Inter", "monoFamily": "JetBrains Mono", "scale": "editorial" }, + "motionDensity": "restrained", + "cameraPolicy": "semantic-focus", + "transitionPolicy": "focus-transfer", + "attentionPersistence": "until-handoff", + "spacing": "comfortable", + "paletteRoles": { "background": "#0f172a", "ink": "#f8fafc", "accent": "#38bdf8", "muted": "#94a3b8" } + }, + "scenes": [ + { + "type": "infographic", + "id": "overview", + "title": "From evidence to explanation", + "duration": 75, + "metrics": [ + { "id": "evidence", "label": "Evidence", "value": "Recorded", "detail": "source state", "color": "#38bdf8", "at": 0 }, + { "id": "model", "label": "Model", "value": "Explained", "detail": "semantic path", "color": "#a78bfa", "at": 20 } + ], + "explanation": { + "cues": [{ "id": "overview-cue", "text": "Evidence becomes an explanation." }], + "beats": [ + { "id": "evidence-beat", "cueId": "overview-cue", "anchor": { "text": "Evidence" }, "visuals": [{ "targetId": "evidence", "action": "focus", "minHoldMs": 800 }] }, + { "id": "model-beat", "cueId": "overview-cue", "anchor": { "text": "explanation" }, "visuals": [{ "targetId": "model", "action": "highlight", "minHoldMs": 800 }] } + ] + } + }, + { + "type": "manim", + "id": "equation", + "sourceVideo": "file:///D:/video-agent/seqvio/output/manim-adapter-media-v2/videos/equation/480p15/EquationDerivation.mp4", + "sourceManifest": "output/manim-adapter-cache-v2/a237aaba1ecb8dc77804e88439eccc34337a8427721e75dd75ce91a38bcb7b8a.json", + "mediaWidth": 854, + "mediaHeight": 480, + "mediaFps": 15, + "duration": 60, + "markers": [{ "id": "equation-result", "frame": 30 }] + } + ] +} diff --git a/examples/ir/manim-clip-validation.explainer.json b/examples/ir/manim-clip-validation.explainer.json new file mode 100644 index 0000000..ed12b4d --- /dev/null +++ b/examples/ir/manim-clip-validation.explainer.json @@ -0,0 +1,25 @@ +{ + "format": "seqvio-explainer", + "schemaVersion": "1.0", + "id": "manim-ir-validation", + "width": 1280, + "height": 720, + "fps": 30, + "scenes": [ + { + "type": "manim", + "id": "external-animation", + "sourceVideo": "file:///D:/video-agent/seqvio/output/direction-plan-validation-v2.mp4", + "sourceManifest": "output/manim-execution-validation-v2.dry-run.json", + "mediaWidth": 1280, + "mediaHeight": 720, + "mediaFps": 30, + "duration": 180, + "markers": [ + { "id": "start", "frame": 0, "targetId": "animation-start" }, + { "id": "explain", "frame": 60, "targetId": "animation-explain" }, + { "id": "result", "frame": 140, "targetId": "animation-result" } + ] + } + ] +} diff --git a/examples/ir/manim-real-validation.explainer.json b/examples/ir/manim-real-validation.explainer.json new file mode 100644 index 0000000..0947365 --- /dev/null +++ b/examples/ir/manim-real-validation.explainer.json @@ -0,0 +1,23 @@ +{ + "format": "seqvio-explainer", + "schemaVersion": "1.0", + "id": "manim-real-validation", + "width": 1280, + "height": 720, + "fps": 15, + "scenes": [ + { + "type": "manim", + "id": "equation", + "sourceVideo": "file:///D:/video-agent/seqvio/output/manim-adapter-media/videos/equation/480p15/EquationDerivation.mp4", + "sourceManifest": "output/manim-adapter-cache/747706299f8a8da24eb5505755c8bf7a4c12171ef444ecede5c5453ec7adcc3a.json", + "mediaWidth": 854, + "mediaHeight": 480, + "mediaFps": 15, + "duration": 60, + "markers": [ + { "id": "equation-written", "frame": 30 } + ] + } + ] +} diff --git a/examples/manim/README.md b/examples/manim/README.md new file mode 100644 index 0000000..f586af0 --- /dev/null +++ b/examples/manim/README.md @@ -0,0 +1,38 @@ +# Manim validation scenes + +These Python scenes validate Seqvio's optional integration with the Python +package `manim`. The adapter is TypeScript/Node.js, but rendering happens in the +external Python runtime. Generated MP4 files are then consumed as seekable media +by `ManimClip`. + +The repository-local validation environment is `.venv-manim`, and the fixtures +are currently validated with `manim==0.20.1`. + +```powershell +# Windows PowerShell, from the repository root +python -m venv .venv-manim +.\.venv-manim\Scripts\python.exe -m pip install manim==0.20.1 +.\.venv-manim\Scripts\python.exe -m manim -ql examples\manim\equation.py EquationDerivation --media_dir output\manim-media +.\.venv-manim\Scripts\python.exe -m manim -ql --fps 30 examples\manim\graph.py GraphExplanation --media_dir output\manim-media +.\.venv-manim\Scripts\python.exe -m manim -ql --fps 30 examples\manim\proof.py AlgebraProof --media_dir output\manim-media +.\.venv-manim\Scripts\python.exe -m manim -ql --fps 30 -r 1280,720 examples\manim\geometric-proof.py GeometricPythagoreanProof --media_dir output\manim-geometric-proof-media +``` + +```bash +# macOS or Linux, from the repository root +python3 -m venv .venv-manim +./.venv-manim/bin/python -m pip install manim==0.20.1 +./.venv-manim/bin/python -m manim -ql examples/manim/equation.py EquationDerivation --media_dir output/manim-media +./.venv-manim/bin/python -m manim -ql --fps 30 examples/manim/graph.py GraphExplanation --media_dir output/manim-media +./.venv-manim/bin/python -m manim -ql --fps 30 examples/manim/proof.py AlgebraProof --media_dir output/manim-media +./.venv-manim/bin/python -m manim -ql --fps 30 -r 1280,720 examples/manim/geometric-proof.py GeometricPythagoreanProof --media_dir output/manim-geometric-proof-media +``` + +The adapter accepts the same executable through `pythonCommand` or the +repository helper's `--python` option. `seqvio-doctor` discovers `.venv-manim` +automatically; set `SEQVIO_MANIM_PYTHON` when using another environment. + +The graph, symbolic proof, and geometric proof fixtures cover coordinate +animation, symbolic transformation, and a triangle-area construction. See the +[Manim integration guide](../../docs/MANIM-INTEGRATION.md) for adapter rendering, +manifests, cache behavior, `ExplainerDocument`, and `ExplanationBeat` markers. diff --git a/examples/manim/equation.py b/examples/manim/equation.py new file mode 100644 index 0000000..b562b34 --- /dev/null +++ b/examples/manim/equation.py @@ -0,0 +1,8 @@ +from manim import MathTex, Scene, Write + + +class EquationDerivation(Scene): + def construct(self): + equation = MathTex(r"a^2 + b^2 = c^2") + self.play(Write(equation)) + self.wait(0.5) diff --git a/examples/manim/equation.scene.json b/examples/manim/equation.scene.json new file mode 100644 index 0000000..4f25911 --- /dev/null +++ b/examples/manim/equation.scene.json @@ -0,0 +1,11 @@ +{ + "format": "seqvio-manim-scene", + "version": "1.0", + "id": "equation", + "pythonFile": "examples/manim/equation.py", + "className": "EquationDerivation", + "width": 1280, + "height": 720, + "fps": 30, + "quality": "medium" +} diff --git a/examples/manim/geometric-proof.py b/examples/manim/geometric-proof.py new file mode 100644 index 0000000..e7dc406 --- /dev/null +++ b/examples/manim/geometric-proof.py @@ -0,0 +1,32 @@ +import numpy as np +from manim import Create, FadeIn, MathTex, Polygon, Scene, Text, VGroup + + +def square_on(p1, p2, side): + edge = p2 - p1 + normal = np.array([-edge[1], edge[0], 0.0]) + normal = normal / np.linalg.norm(normal) * np.linalg.norm(edge) * side + return Polygon(p1, p2, p2 + normal, p1 + normal) + + +class GeometricPythagoreanProof(Scene): + def construct(self): + a = np.array([-1.6, -0.9, 0.0]) + b = np.array([1.2, -0.9, 0.0]) + c = np.array([-1.6, 1.2, 0.0]) + triangle = Polygon(a, b, c, color="#e2e8f0", fill_color="#1e293b", fill_opacity=0.65) + square_a = square_on(a, c, 1).set_fill("#38bdf8", opacity=0.28).set_stroke("#38bdf8") + square_b = square_on(a, b, -1).set_fill("#34d399", opacity=0.28).set_stroke("#34d399") + square_c = square_on(b, c, -1).set_fill("#fbbf24", opacity=0.24).set_stroke("#fbbf24") + labels = VGroup( + MathTex("a^2", color="#38bdf8").move_to(square_a.get_center()), + MathTex("b^2", color="#34d399").move_to(square_b.get_center()), + MathTex("c^2", color="#fbbf24").move_to(square_c.get_center()), + ) + title = Text("Areas on a right triangle", font_size=28).to_edge(np.array([0, 1, 0])) + result = MathTex("a^2+b^2=c^2", color="#e2e8f0").to_edge(np.array([0, -1, 0])) + self.play(FadeIn(title), Create(triangle), run_time=0.8) + self.play(Create(square_a), Create(square_b), run_time=1.0) + self.play(Create(square_c), FadeIn(labels), run_time=1.1) + self.play(FadeIn(result), run_time=0.7) + self.wait(0.5) diff --git a/examples/manim/geometric-proof.scene.json b/examples/manim/geometric-proof.scene.json new file mode 100644 index 0000000..bfe4351 --- /dev/null +++ b/examples/manim/geometric-proof.scene.json @@ -0,0 +1,13 @@ +{ + "format": "seqvio-manim-scene", + "version": "1.0", + "id": "geometric-proof", + "pythonFile": "examples/manim/geometric-proof.py", + "className": "GeometricPythagoreanProof", + "width": 1280, + "height": 720, + "fps": 30, + "quality": "low", + "assets": [], + "args": ["--media_dir", "output/manim-geometric-proof-media"] +} diff --git a/examples/manim/graph.py b/examples/manim/graph.py new file mode 100644 index 0000000..413acc2 --- /dev/null +++ b/examples/manim/graph.py @@ -0,0 +1,20 @@ +from manim import Axes, Create, Dot, FadeIn, Scene, Text + + +class GraphExplanation(Scene): + def construct(self): + axes = Axes( + x_range=[0, 4, 1], + y_range=[0, 8, 2], + x_length=8, + y_length=4.5, + tips=False, + ) + curve = axes.plot(lambda x: 0.5 * x * x, x_range=[0, 4], color="#4f8cff") + dot = Dot(axes.c2p(1, 0.5), color="#21a179") + label = Text("growth accelerates", font_size=28).next_to(axes, direction=[0, -1, 0]) + self.play(Create(axes), run_time=0.8) + self.play(Create(curve), FadeIn(dot), run_time=1.2) + self.play(dot.animate.move_to(axes.c2p(3.6, 0.5 * 3.6 * 3.6)), run_time=1.2) + self.play(FadeIn(label), run_time=0.5) + self.wait(0.4) diff --git a/examples/manim/graph.scene.json b/examples/manim/graph.scene.json new file mode 100644 index 0000000..08b42b0 --- /dev/null +++ b/examples/manim/graph.scene.json @@ -0,0 +1,11 @@ +{ + "format": "seqvio-manim-scene", + "version": "1.0", + "id": "graph", + "pythonFile": "examples/manim/graph.py", + "className": "GraphExplanation", + "width": 1280, + "height": 720, + "fps": 30, + "quality": "low" +} diff --git a/examples/manim/proof.py b/examples/manim/proof.py new file mode 100644 index 0000000..8bc9854 --- /dev/null +++ b/examples/manim/proof.py @@ -0,0 +1,13 @@ +from manim import MathTex, Scene, SurroundingRectangle, TransformMatchingTex, Write + + +class AlgebraProof(Scene): + def construct(self): + first = MathTex(r"(a+b)^2 = a^2 + 2ab + b^2") + second = MathTex(r"a^2 + 2ab + b^2 = c^2 + 2ab") + result = MathTex(r"a^2 + b^2 = c^2", color="#4f8cff") + self.play(Write(first), run_time=0.9) + self.play(TransformMatchingTex(first, second), run_time=1.1) + self.play(TransformMatchingTex(second, result), run_time=1.1) + self.play(Write(SurroundingRectangle(result, color="#21a179")), run_time=0.5) + self.wait(0.5) diff --git a/examples/manim/proof.scene.json b/examples/manim/proof.scene.json new file mode 100644 index 0000000..cba8071 --- /dev/null +++ b/examples/manim/proof.scene.json @@ -0,0 +1,11 @@ +{ + "format": "seqvio-manim-scene", + "version": "1.0", + "id": "proof", + "pythonFile": "examples/manim/proof.py", + "className": "AlgebraProof", + "width": 1280, + "height": 720, + "fps": 30, + "quality": "low" +} diff --git a/examples/motion-grammar/compare-merge.motion.json b/examples/motion-grammar/compare-merge.motion.json new file mode 100644 index 0000000..4655cd4 --- /dev/null +++ b/examples/motion-grammar/compare-merge.motion.json @@ -0,0 +1,11 @@ +{ + "format": "seqvio-motion-grammar", + "version": "1.0", + "id": "compare-merge", + "steps": [ + { "id": "frame-question", "sceneId": "compare-merge", "action": "question" }, + { "id": "show-before", "sceneId": "compare-merge", "action": "reveal", "targetId": "manual", "startFrame": 8 }, + { "id": "compare-paths", "sceneId": "compare-merge", "action": "compare", "targetId": "manual", "relatedTargetId": "guided", "startFrame": 30, "holdFrames": 34 }, + { "id": "merge-result", "sceneId": "compare-merge", "action": "transform", "targetId": "merged", "startFrame": 68, "holdFrames": 28 } + ] +} diff --git a/examples/motion-grammar/compare.motion.json b/examples/motion-grammar/compare.motion.json new file mode 100644 index 0000000..c209f07 --- /dev/null +++ b/examples/motion-grammar/compare.motion.json @@ -0,0 +1,11 @@ +{ + "format": "seqvio-motion-grammar", + "version": "1.0", + "id": "compare-flow", + "steps": [ + { "id": "question", "sceneId": "results", "action": "question" }, + { "id": "reveal-before", "sceneId": "results", "action": "reveal", "targetId": "before" }, + { "id": "compare", "sceneId": "results", "action": "compare", "targetId": "before", "relatedTargetId": "after", "startFrame": 45, "holdFrames": 36 }, + { "id": "answer", "sceneId": "results", "action": "answer", "targetId": "after", "startFrame": 96, "holdFrames": 36 } + ] +} diff --git a/examples/motion-grammar/problem-fix.motion.json b/examples/motion-grammar/problem-fix.motion.json new file mode 100644 index 0000000..36dde26 --- /dev/null +++ b/examples/motion-grammar/problem-fix.motion.json @@ -0,0 +1,11 @@ +{ + "format": "seqvio-motion-grammar", + "version": "1.0", + "id": "problem-fix", + "steps": [ + { "id": "state-problem", "sceneId": "problem-fix", "action": "question" }, + { "id": "show-problem", "sceneId": "problem-fix", "action": "emphasize", "targetId": "problem", "startFrame": 12, "holdFrames": 30 }, + { "id": "apply-fix", "sceneId": "problem-fix", "action": "transform", "targetId": "fix", "startFrame": 48, "holdFrames": 30 }, + { "id": "confirm-fix", "sceneId": "problem-fix", "action": "answer", "targetId": "verified", "startFrame": 78, "holdFrames": 22 } + ] +} diff --git a/examples/motion-grammar/process-verification.motion.json b/examples/motion-grammar/process-verification.motion.json new file mode 100644 index 0000000..c3dda9a --- /dev/null +++ b/examples/motion-grammar/process-verification.motion.json @@ -0,0 +1,11 @@ +{ + "format": "seqvio-motion-grammar", + "version": "1.0", + "id": "process-verification", + "steps": [ + { "id": "open-process", "sceneId": "process-verification", "action": "question" }, + { "id": "trace-process", "sceneId": "process-verification", "action": "trace", "targetId": "observe", "pathTargetIds": ["observe", "decide", "verify"], "startFrame": 18, "holdFrames": 56 }, + { "id": "show-verification", "sceneId": "process-verification", "action": "emphasize", "targetId": "verified", "startFrame": 78, "holdFrames": 24 }, + { "id": "close-process", "sceneId": "process-verification", "action": "summarize", "targetId": "verified", "startFrame": 94, "holdFrames": 20 } + ] +} diff --git a/examples/styles/clean-technical.style.json b/examples/styles/clean-technical.style.json new file mode 100644 index 0000000..6a1ca61 --- /dev/null +++ b/examples/styles/clean-technical.style.json @@ -0,0 +1,13 @@ +{ + "format": "seqvio-style-profile", + "version": "1.0", + "id": "clean-technical", + "label": "Clean Technical", + "typography": { "headingFamily": "Inter", "bodyFamily": "Inter", "monoFamily": "JetBrains Mono", "scale": "editorial" }, + "motionDensity": "restrained", + "cameraPolicy": "semantic-focus", + "transitionPolicy": "focus-transfer", + "attentionPersistence": "until-handoff", + "spacing": "comfortable", + "paletteRoles": { "background": "#0f172a", "ink": "#f8fafc", "accent": "#38bdf8", "muted": "#94a3b8" } +} diff --git a/examples/styles/editorial-explainer.style.json b/examples/styles/editorial-explainer.style.json new file mode 100644 index 0000000..fa4cb3f --- /dev/null +++ b/examples/styles/editorial-explainer.style.json @@ -0,0 +1,13 @@ +{ + "format": "seqvio-style-profile", + "version": "1.0", + "id": "editorial-explainer", + "label": "Editorial Explainer", + "typography": { "headingFamily": "Georgia", "bodyFamily": "Inter", "monoFamily": "JetBrains Mono", "scale": "large" }, + "motionDensity": "balanced", + "cameraPolicy": "static", + "transitionPolicy": "crossfade", + "attentionPersistence": "until-handoff", + "spacing": "airy", + "paletteRoles": { "background": "#f7f8fa", "ink": "#17202a", "accent": "#2563eb", "muted": "#64748b" } +} diff --git a/examples/styles/terminal-first.style.json b/examples/styles/terminal-first.style.json new file mode 100644 index 0000000..3f4d738 --- /dev/null +++ b/examples/styles/terminal-first.style.json @@ -0,0 +1,13 @@ +{ + "format": "seqvio-style-profile", + "version": "1.0", + "id": "terminal-first", + "label": "Terminal First", + "typography": { "headingFamily": "JetBrains Mono", "bodyFamily": "Inter", "monoFamily": "JetBrains Mono", "scale": "compact" }, + "motionDensity": "restrained", + "cameraPolicy": "evidence-follow", + "transitionPolicy": "cut", + "attentionPersistence": "timed", + "spacing": "tight", + "paletteRoles": { "background": "#101418", "ink": "#e6edf3", "accent": "#2dd4bf", "muted": "#8b9aaa" } +} diff --git a/package-lock.json b/package-lock.json index c3032b1..847f551 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1087,6 +1087,10 @@ "resolved": "packages/core", "link": true }, + "node_modules/@seqvio/manim-adapter": { + "resolved": "packages/manim-adapter", + "link": true + }, "node_modules/@seqvio/product-demo": { "resolved": "packages/product-demo", "link": true @@ -3922,6 +3926,15 @@ "react": "^18.0.0 || ^19.0.0" } }, + "packages/manim-adapter": { + "name": "@seqvio/manim-adapter", + "version": "0.1.0", + "license": "MIT", + "devDependencies": { + "@types/node": "^20.11.0", + "typescript": "^5.3.0" + } + }, "packages/product-demo": { "name": "@seqvio/product-demo", "version": "0.8.0", diff --git a/package.json b/package.json index 8ce7eba..f586c1d 100644 --- a/package.json +++ b/package.json @@ -41,7 +41,7 @@ }, "scripts": { "dev": "npm run dev --workspaces --if-present --include-workspace-root=false", - "build": "npm run build -w @seqvio/core && npm run build -w @seqvio/capture && npm run build -w @seqvio/whiteboard && npm run build -w @seqvio/scatterbrain && npm run build -w @seqvio/product-demo && npm run build -w @seqvio/technical && npm run build -w @seqvio/renderer && npm run build -w @seqvio/browser-recorder && npm run build -w @seqvio/terminal-narrator", + "build": "npm run build -w @seqvio/core && npm run build -w @seqvio/capture && npm run build -w @seqvio/whiteboard && npm run build -w @seqvio/scatterbrain && npm run build -w @seqvio/product-demo && npm run build -w @seqvio/technical && npm run build -w @seqvio/manim-adapter && npm run build -w @seqvio/renderer && npm run build -w @seqvio/browser-recorder && npm run build -w @seqvio/terminal-narrator", "test": "npm run test --workspaces --if-present --include-workspace-root=false", "test:release-tools": "node --test scripts/tests/*.test.mjs", "test:browser": "npm run test:browser -w @seqvio/renderer", @@ -50,6 +50,7 @@ "smoke:authoring-workflow": "node scripts/authoring-workflow-smoke.mjs", "smoke:release-pipeline:terminal": "node scripts/release-pipeline-smoke.mjs --kind terminal", "smoke:release-pipeline:browser": "node scripts/release-pipeline-smoke.mjs --kind browser", + "smoke:release-pipeline:semantic": "node scripts/release-pipeline-smoke.mjs --kind semantic", "verify:capture-host": "node scripts/verify-capture-host.mjs", "smoke:capture-host-runtime": "node scripts/capture-host-runtime-smoke.mjs", "prepare:native": "node scripts/prepare-native-dependencies.mjs", @@ -63,6 +64,9 @@ "benchmark:render:check": "node scripts/render-benchmark.mjs --check", "benchmark:render:update": "node scripts/render-benchmark.mjs --update-baseline", "audio:technical-explainer": "node scripts/technical-explainer-audio-loop.mjs", + "direction:generate": "node scripts/generate-direction-sidecar.mjs", + "director:pass": "node scripts/director-pass.mjs", + "director:task": "node scripts/director-task.mjs", "smoke:terminal:native": "node packages/terminal-narrator/dist/cli.js record --sample --outputDir temp/terminal-smoke-native", "smoke:terminal:claude": "node packages/terminal-narrator/dist/cli.js record --sample-claude --skill /help --outputDir temp/terminal-smoke-claude", "lint": "npm run lint --workspaces --if-present --include-workspace-root=false", diff --git a/packages/browser-recorder/README.md b/packages/browser-recorder/README.md index 0f6e055..1ebedbc 100644 --- a/packages/browser-recorder/README.md +++ b/packages/browser-recorder/README.md @@ -1,6 +1,6 @@ # Seqvio Browser Capture Adapter -A local, pre-stable browser walkthrough adapter that executes a validated action +A local browser walkthrough adapter with a stable capture contract that executes a validated action plan, captures the page, records cursor/focus/click metadata and exact action start times, then compiles the capture through `ExplainerDocument`. @@ -55,7 +55,7 @@ node packages/browser-recorder/dist/cli.js record --plan plan.json --jobId narra - `qa-report.json` — capture, visual, pacing, media, and audio diagnostics - `artifacts.json` — versioned status and relative artifact paths -CLI contract `1.0` provides structured JSON output, stable exit codes, monotonic +CLI contract `2.0` provides structured JSON output, stable exit codes, monotonic progress, and overwrite protection. See [`docs/CAPTURE-CLI-CONTRACT.md`](../../docs/CAPTURE-CLI-CONTRACT.md). @@ -70,6 +70,15 @@ progress, and overwrite protection. See "captureFps": 15, "renderFps": 30, "maxZoom": 2.2, + "privacy": { + "masks": [ + { "id": "account-email", "selector": "#account-email" }, + { + "id": "api-token", + "rect": { "x": 920, "y": 24, "width": 320, "height": 44 } + } + ] + }, "actions": [ { "id": "project-name", @@ -90,6 +99,13 @@ progress, and overwrite protection. See Supported actions: `click`, `fill`, `scroll`, `wait`, `navigate`, and `press`. +`privacy.masks` applies opaque overlays before page frames are captured. A mask +targets either a live CSS selector or a fixed viewport rectangle. Selector masks +are required by default: if one never matches during the recording, the job +fails instead of silently producing an unprotected video. Use stable, +non-sensitive mask ids because ids and match evidence are retained in +`recording-manifest.json`; selectors and page content are not copied there. + ## AI planner adapter The recorder does not force one model provider. Set `BROWSER_RECORDER_PLANNER_URL` to an HTTP endpoint. The endpoint receives the task, start URL, viewport, a compact list of interactive DOM elements, and the output contract. Return either a plan object or `{ "plan": ... }`. @@ -107,6 +123,8 @@ The UI enables **AI 规划** only when the planner URL is configured. Generated - Chromium web pages only; no desktop application control. - CSS selectors are the deterministic execution contract. +- Browser privacy protection is declarative and deterministic; automatic OCR is + not treated as a security boundary. - Authentication can be scripted, but reusable browser profiles are not included yet. - Captures page video only. Microphone and system audio are not recorded. Narration is synthesized from the generated cues when direct CLI jobs use `--withAudio`. - Popups, downloads, cross-origin iframes, CAPTCHA, and two-factor authentication require later adapters or human takeover. diff --git a/packages/browser-recorder/package.json b/packages/browser-recorder/package.json index 82ef59c..3650266 100644 --- a/packages/browser-recorder/package.json +++ b/packages/browser-recorder/package.json @@ -1,7 +1,7 @@ { "name": "@seqvio/browser-recorder", "version": "0.3.0", - "description": "Pre-stable browser capture-to-explanation adapter for Seqvio", + "description": "Experimental package with a stable browser capture-to-explanation adapter for Seqvio", "main": "dist/index.js", "types": "dist/index.d.ts", "bin": { @@ -11,7 +11,7 @@ }, "seqvio": { "lifecycle": "experimental", - "adapterLifecycle": "pre-stable", + "adapterLifecycle": "stable", "versionPolicy": "independent", "cliContractVersion": "2.0", "artifactManifestVersion": "1.0" diff --git a/packages/browser-recorder/src/cli.ts b/packages/browser-recorder/src/cli.ts index 61ea7fd..d271d4f 100644 --- a/packages/browser-recorder/src/cli.ts +++ b/packages/browser-recorder/src/cli.ts @@ -1,6 +1,6 @@ #!/usr/bin/env node -import * as fs from 'node:fs'; -import * as path from 'node:path'; +import * as fs from "node:fs"; +import * as path from "node:path"; import { CAPTURE_ADAPTER_LIFECYCLE, CAPTURE_CLI_CONTRACT_VERSION, @@ -11,11 +11,15 @@ import { createCaptureJobId, validateCaptureJobId, writeCaptureArtifactManifest, -} from '@seqvio/capture'; -import { runPipeline } from './pipeline'; -import { createRecorderServer } from './server'; -import type { BrowserPipelineResult, PipelineProgress, TtsProvider } from './types'; -import { validatePlan } from './validate'; +} from "@seqvio/capture"; +import { runPipeline } from "./pipeline"; +import { createRecorderServer } from "./server"; +import type { + BrowserPipelineResult, + PipelineProgress, + TtsProvider, +} from "./types"; +import { validatePlan } from "./validate"; interface CliIo { stdout(message: string): void; @@ -23,7 +27,7 @@ interface CliIo { } interface ParsedArgs { - command: 'serve' | 'record'; + command: "serve" | "record"; values: Map; } @@ -34,8 +38,24 @@ export interface BrowserCliDependencies { now?: () => number; } -const BOOLEAN_FLAGS = new Set(['help', 'version', 'json', 'withAudio', 'burnCaptions']); -const VALUE_FLAGS = new Set(['port', 'host', 'outputDir', 'output', 'plan', 'jobId', 'provider', 'voice', 'qaConfig']); +const BOOLEAN_FLAGS = new Set([ + "help", + "version", + "json", + "withAudio", + "burnCaptions", +]); +const VALUE_FLAGS = new Set([ + "port", + "host", + "outputDir", + "output", + "plan", + "jobId", + "provider", + "voice", + "qaConfig", +]); function defaultIo(): CliIo { return { stdout: console.log, stderr: console.error }; @@ -64,20 +84,23 @@ The legacy --output alias is accepted for serve and maps to --outputDir.`; } function packageVersion(): string { - const packagePath = path.resolve(__dirname, '..', 'package.json'); - return (JSON.parse(fs.readFileSync(packagePath, 'utf8')) as { version: string }).version; + const packagePath = path.resolve(__dirname, "..", "package.json"); + return ( + JSON.parse(fs.readFileSync(packagePath, "utf8")) as { version: string } + ).version; } function parseArgs(argv: string[]): ParsedArgs { const args = [...argv]; - const command = args[0] && !args[0].startsWith('--') ? args.shift() : 'serve'; - if (command !== 'serve' && command !== 'record') { + const command = args[0] && !args[0].startsWith("--") ? args.shift() : "serve"; + if (command !== "serve" && command !== "record") { throw new Error(`Unknown command "${command}"`); } const values = new Map(); for (let index = 0; index < args.length; index += 1) { const token = args[index]; - if (!token.startsWith('--')) throw new Error(`Unexpected argument "${token}"`); + if (!token.startsWith("--")) + throw new Error(`Unexpected argument "${token}"`); const name = token.slice(2); if (values.has(name)) throw new Error(`Duplicate option "--${name}"`); if (BOOLEAN_FLAGS.has(name)) { @@ -86,33 +109,51 @@ function parseArgs(argv: string[]): ParsedArgs { } if (!VALUE_FLAGS.has(name)) throw new Error(`Unknown option "--${name}"`); const value = args[index + 1]; - if (!value || value.startsWith('--')) throw new Error(`Missing value for --${name}`); + if (!value || value.startsWith("--")) + throw new Error(`Missing value for --${name}`); values.set(name, value); index += 1; } return { command, values }; } -function stringValue(values: ParsedArgs['values'], name: string, fallback?: string): string | undefined { +function stringValue( + values: ParsedArgs["values"], + name: string, + fallback?: string, +): string | undefined { const value = values.get(name); - return typeof value === 'string' ? value : fallback; + return typeof value === "string" ? value : fallback; } -function outputRoot(values: ParsedArgs['values']): string { +function outputRoot(values: ParsedArgs["values"]): string { return path.resolve( - stringValue(values, 'outputDir') ?? - stringValue(values, 'output') ?? - 'output/browser-recorder' + stringValue(values, "outputDir") ?? + stringValue(values, "output") ?? + "output/browser-recorder", ); } function parseProvider(value: string | undefined): TtsProvider | undefined { if (!value) return undefined; - if (value === 'elevenlabs' || value === 'minimax' || value === 'edge-tts' || value === 'openai') return value; - throw new Error(`Unsupported provider "${value}". Use one of: elevenlabs, minimax, edge-tts, openai.`); + if ( + value === "elevenlabs" || + value === "minimax" || + value === "edge-tts" || + value === "openai" + ) + return value; + throw new Error( + `Unsupported provider "${value}". Use one of: elevenlabs, minimax, edge-tts, openai.`, + ); } -function writeResult(io: CliIo, json: boolean, result: unknown, humanLines: string[]): void { +function writeResult( + io: CliIo, + json: boolean, + result: unknown, + humanLines: string[], +): void { if (json) { io.stdout(JSON.stringify(result)); return; @@ -120,25 +161,29 @@ function writeResult(io: CliIo, json: boolean, result: unknown, humanLines: stri humanLines.forEach((line) => io.stdout(line)); } -async function runServe(args: ParsedArgs, io: CliIo, json: boolean): Promise { - const portValue = stringValue(args.values, 'port', '4175')!; +async function runServe( + args: ParsedArgs, + io: CliIo, + json: boolean, +): Promise { + const portValue = stringValue(args.values, "port", "4175")!; const port = Number(portValue); if (!Number.isInteger(port) || port < 1 || port > 65535) { - throw new CliUsageError('port must be an integer between 1 and 65535'); + throw new CliUsageError("port must be an integer between 1 and 65535"); } - const host = stringValue(args.values, 'host', '127.0.0.1')!; + const host = stringValue(args.values, "host", "127.0.0.1")!; const outputDir = outputRoot(args.values); const server = createRecorderServer({ port, host, outputDir }); await new Promise((resolve, reject) => { - server.once('error', reject); + server.once("error", reject); server.listen(port, host, () => resolve()); }); const result = { ok: true, cliContractVersion: CAPTURE_CLI_CONTRACT_VERSION, lifecycle: CAPTURE_ADAPTER_LIFECYCLE, - adapter: 'browser', - command: 'serve', + adapter: "browser", + command: "serve", url: `http://${host}:${port}`, outputDir, }; @@ -152,17 +197,18 @@ async function runServe(args: ParsedArgs, io: CliIo, json: boolean): Promise { - const requestedJson = argv.includes('--json'); + const requestedJson = argv.includes("--json"); let parsed: ParsedArgs; try { parsed = parseArgs(argv); } catch (error) { const message = error instanceof Error ? error.message : String(error); const failure = captureCliFailure({ - adapter: 'browser', exitCode: CaptureCliExitCode.usage, - error: { code: 'invalid_arguments', message }, + adapter: "browser", + exitCode: CaptureCliExitCode.usage, + error: { code: "invalid_arguments", message }, }); if (requestedJson) io.stdout(JSON.stringify(failure)); else { @@ -171,43 +217,75 @@ export async function runBrowserCli( } return CaptureCliExitCode.usage; } - const json = parsed.values.has('json'); - if (parsed.values.has('help')) { - writeResult(io, json, { - ok: true, - cliContractVersion: CAPTURE_CLI_CONTRACT_VERSION, - lifecycle: CAPTURE_ADAPTER_LIFECYCLE, - adapter: 'browser', - command: 'help', - usage: usage(), - }, [usage()]); + const json = parsed.values.has("json"); + if (parsed.values.has("help")) { + writeResult( + io, + json, + { + ok: true, + cliContractVersion: CAPTURE_CLI_CONTRACT_VERSION, + lifecycle: CAPTURE_ADAPTER_LIFECYCLE, + adapter: "browser", + command: "help", + usage: usage(), + }, + [usage()], + ); return CaptureCliExitCode.success; } - if (parsed.values.has('version')) { - writeResult(io, json, { - packageVersion: packageVersion(), - cliContractVersion: CAPTURE_CLI_CONTRACT_VERSION, - lifecycle: CAPTURE_ADAPTER_LIFECYCLE, - }, [`${packageVersion()} (CLI contract ${CAPTURE_CLI_CONTRACT_VERSION}, ${CAPTURE_ADAPTER_LIFECYCLE})`]); + if (parsed.values.has("version")) { + writeResult( + io, + json, + { + packageVersion: packageVersion(), + cliContractVersion: CAPTURE_CLI_CONTRACT_VERSION, + lifecycle: CAPTURE_ADAPTER_LIFECYCLE, + }, + [ + `${packageVersion()} (CLI contract ${CAPTURE_CLI_CONTRACT_VERSION}, ${CAPTURE_ADAPTER_LIFECYCLE})`, + ], + ); return CaptureCliExitCode.success; } - if (parsed.command === 'serve') { + if (parsed.command === "serve") { try { - if (['plan', 'jobId', 'withAudio', 'burnCaptions', 'provider', 'voice', 'qaConfig'].some((name) => parsed.values.has(name))) { - throw new CliUsageError('recording, audio, and QA options are only valid with record'); + if ( + [ + "plan", + "jobId", + "withAudio", + "burnCaptions", + "provider", + "voice", + "qaConfig", + ].some((name) => parsed.values.has(name)) + ) { + throw new CliUsageError( + "recording, audio, and QA options are only valid with record", + ); } - if (parsed.values.has('outputDir') && parsed.values.has('output')) { - throw new CliUsageError('Use only one of --outputDir or --output'); + if (parsed.values.has("outputDir") && parsed.values.has("output")) { + throw new CliUsageError("Use only one of --outputDir or --output"); } return await runServe(parsed, io, json); } catch (error) { const message = error instanceof Error ? error.message : String(error); - const exitCode = error instanceof CliUsageError - ? CaptureCliExitCode.usage - : CaptureCliExitCode.pipeline; + const exitCode = + error instanceof CliUsageError + ? CaptureCliExitCode.usage + : CaptureCliExitCode.pipeline; const failure = captureCliFailure({ - adapter: 'browser', exitCode, - error: { code: exitCode === CaptureCliExitCode.usage ? 'invalid_input' : 'server_failed', message }, + adapter: "browser", + exitCode, + error: { + code: + exitCode === CaptureCliExitCode.usage + ? "invalid_input" + : "server_failed", + message, + }, }); (json ? io.stdout : io.stderr)(json ? JSON.stringify(failure) : message); return exitCode; @@ -217,26 +295,36 @@ export async function runBrowserCli( let jobId: string | undefined; let jobDir: string | undefined; try { - const planPath = stringValue(parsed.values, 'plan'); - if (!planPath) throw new Error('record requires --plan '); - if (parsed.values.has('port') || parsed.values.has('host')) { - throw new Error('--port and --host are only valid with serve'); + const planPath = stringValue(parsed.values, "plan"); + if (!planPath) throw new Error("record requires --plan "); + if (parsed.values.has("port") || parsed.values.has("host")) { + throw new Error("--port and --host are only valid with serve"); } - if (parsed.values.has('outputDir') && parsed.values.has('output')) { - throw new Error('Use only one of --outputDir or --output'); + if (parsed.values.has("outputDir") && parsed.values.has("output")) { + throw new Error("Use only one of --outputDir or --output"); } - const plan = validatePlan(JSON.parse(fs.readFileSync(path.resolve(planPath), 'utf8'))); - const withAudio = parsed.values.has('withAudio'); - const audioProvider = parseProvider(stringValue(parsed.values, 'provider')); - const audioVoice = stringValue(parsed.values, 'voice'); - if ((audioProvider || audioVoice) && !withAudio) throw new Error('--provider and --voice require --withAudio'); - if (parsed.values.has('burnCaptions') && !withAudio) throw new Error('--burnCaptions requires --withAudio'); + const plan = validatePlan( + JSON.parse(fs.readFileSync(path.resolve(planPath), "utf8")), + ); + const withAudio = parsed.values.has("withAudio"); + const audioProvider = parseProvider(stringValue(parsed.values, "provider")); + const audioVoice = stringValue(parsed.values, "voice"); + if ((audioProvider || audioVoice) && !withAudio) + throw new Error("--provider and --voice require --withAudio"); + if (parsed.values.has("burnCaptions") && !withAudio) + throw new Error("--burnCaptions requires --withAudio"); jobId = validateCaptureJobId( - stringValue(parsed.values, 'jobId') ?? createCaptureJobId(dependencies.now?.()) + stringValue(parsed.values, "jobId") ?? + createCaptureJobId(dependencies.now?.()), ); const candidateJobDir = path.join(outputRoot(parsed.values), jobId); - if (fs.existsSync(candidateJobDir) && fs.readdirSync(candidateJobDir).length > 0) { - throw new Error(`Job directory already exists and is not empty: ${candidateJobDir}`); + if ( + fs.existsSync(candidateJobDir) && + fs.readdirSync(candidateJobDir).length > 0 + ) { + throw new Error( + `Job directory already exists and is not empty: ${candidateJobDir}`, + ); } jobDir = candidateJobDir; fs.mkdirSync(jobDir, { recursive: true }); @@ -244,17 +332,19 @@ export async function runBrowserCli( const message = `[${progress.phase}] ${Math.round(progress.percent)}% - ${progress.message}`; (json ? io.stderr : io.stdout)(message); }; - const result: BrowserPipelineResult = await (dependencies.pipeline ?? runPipeline)( - plan, jobDir, onProgress, { - withAudio, - burnCaptions: parsed.values.has('burnCaptions'), - audioProvider, - audioVoice, - qaConfig: stringValue(parsed.values, 'qaConfig'), - } - ); + const result: BrowserPipelineResult = await ( + dependencies.pipeline ?? runPipeline + )(plan, jobDir, onProgress, { + withAudio, + burnCaptions: parsed.values.has("burnCaptions"), + audioProvider, + audioVoice, + qaConfig: stringValue(parsed.values, "qaConfig"), + }); const success = captureCliSuccess({ - adapter: 'browser', jobId, jobDir, + adapter: "browser", + jobId, + jobDir, artifactManifestPath: result.artifactManifestPath, outputVideoPath: result.outputVideoPath, }); @@ -267,19 +357,28 @@ export async function runBrowserCli( } catch (error) { const message = error instanceof Error ? error.message : String(error); const pipelineStarted = Boolean(jobDir); - const exitCode = pipelineStarted ? CaptureCliExitCode.pipeline : CaptureCliExitCode.usage; + const exitCode = pipelineStarted + ? CaptureCliExitCode.pipeline + : CaptureCliExitCode.usage; let artifactManifestPath: string | undefined; if (jobDir && jobId) { artifactManifestPath = writeCaptureArtifactManifest(jobDir, { - adapter: 'browser', status: 'failed', - artifacts: collectCaptureArtifacts(jobDir, 'browser'), - error: { code: 'pipeline_failed', message }, + adapter: "browser", + status: "failed", + artifacts: collectCaptureArtifacts(jobDir, "browser"), + error: { code: "pipeline_failed", message }, }); } const failure = captureCliFailure({ - adapter: 'browser', exitCode, - error: { code: pipelineStarted ? 'pipeline_failed' : 'invalid_input', message }, - jobId, jobDir, artifactManifestPath, + adapter: "browser", + exitCode, + error: { + code: pipelineStarted ? "pipeline_failed" : "invalid_input", + message, + }, + jobId, + jobDir, + artifactManifestPath, }); (json ? io.stdout : io.stderr)(json ? JSON.stringify(failure) : message); return exitCode; @@ -288,10 +387,12 @@ export async function runBrowserCli( if (require.main === module) { runBrowserCli(process.argv.slice(2)).then( - (exitCode) => { process.exit(exitCode); }, + (exitCode) => { + process.exitCode = exitCode; + }, (error) => { console.error(error instanceof Error ? error.message : String(error)); - process.exit(CaptureCliExitCode.internal); - } + process.exitCode = CaptureCliExitCode.internal; + }, ); } diff --git a/packages/browser-recorder/src/index.ts b/packages/browser-recorder/src/index.ts index ecae48e..8918cbf 100644 --- a/packages/browser-recorder/src/index.ts +++ b/packages/browser-recorder/src/index.ts @@ -1,11 +1,12 @@ -export * from './capture-session'; -export * from './compile-to-ir'; -export * from './composition'; -export * from './inspect'; -export * from './pipeline'; -export * from './planner'; -export * from './record'; -export * from './sample'; -export * from './server'; -export * from './types'; -export * from './validate'; +export * from "./capture-session"; +export * from "./compile-to-ir"; +export * from "./composition"; +export * from "./inspect"; +export * from "./pipeline"; +export * from "./planner"; +export * from "./privacy"; +export * from "./record"; +export * from "./sample"; +export * from "./server"; +export * from "./types"; +export * from "./validate"; diff --git a/packages/browser-recorder/src/privacy.ts b/packages/browser-recorder/src/privacy.ts new file mode 100644 index 0000000..85ae09a --- /dev/null +++ b/packages/browser-recorder/src/privacy.ts @@ -0,0 +1,176 @@ +import type { Page } from "puppeteer"; +import type { BrowserPrivacyMask, BrowserPrivacyPolicy } from "./types"; + +const MASK_ROOT_ID = "__seqvio-privacy-masks"; +const MASK_STATE_KEY = "__seqvioPrivacyMaskState"; + +export interface PrivacyMaskStatus { + foundIds: string[]; +} + +export function normalizePrivacyPolicy( + value: unknown, +): BrowserPrivacyPolicy | undefined { + if (value === undefined) return undefined; + if (!value || typeof value !== "object") + throw new Error("privacy must be an object"); + const masks = (value as BrowserPrivacyPolicy).masks; + if (!Array.isArray(masks) || masks.length === 0) { + throw new Error("privacy.masks must contain at least one mask"); + } + if (masks.length > 100) + throw new Error("privacy.masks cannot exceed 100 items"); + const ids = new Set(); + return { + masks: masks.map((mask, index) => normalizeMask(mask, index, ids)), + }; +} + +function normalizeMask( + mask: BrowserPrivacyMask, + index: number, + ids: Set, +): BrowserPrivacyMask { + if (!mask || typeof mask !== "object") + throw new Error(`privacy.masks[${index}] must be an object`); + if ( + typeof mask.id !== "string" || + !/^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/.test(mask.id) + ) { + throw new Error( + `privacy.masks[${index}].id must be a safe 1-64 character identifier`, + ); + } + if (ids.has(mask.id)) + throw new Error(`privacy mask id is duplicated: ${mask.id}`); + ids.add(mask.id); + const hasSelector = + typeof mask.selector === "string" && mask.selector.trim().length > 0; + const hasRect = mask.rect !== undefined; + if (hasSelector === hasRect) { + throw new Error( + `privacy mask ${mask.id} must define exactly one of selector or rect`, + ); + } + let rect: BrowserPrivacyMask["rect"]; + if (hasRect) { + const candidate = mask.rect!; + if ( + ![candidate.x, candidate.y, candidate.width, candidate.height].every( + Number.isFinite, + ) + ) { + throw new Error( + `privacy mask ${mask.id}.rect values must be finite numbers`, + ); + } + if (candidate.width <= 0 || candidate.height <= 0) { + throw new Error( + `privacy mask ${mask.id}.rect width and height must be positive`, + ); + } + rect = { ...candidate }; + } + const padding = mask.padding ?? 4; + if (!Number.isFinite(padding) || padding < 0 || padding > 64) { + throw new Error(`privacy mask ${mask.id}.padding must be between 0 and 64`); + } + const color = mask.color ?? "#111827"; + if (!/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/i.test(color)) { + throw new Error( + `privacy mask ${mask.id}.color must be a 6 or 8 digit hex color`, + ); + } + return { + id: mask.id, + ...(hasSelector ? { selector: mask.selector!.trim() } : { rect }), + padding, + color, + required: mask.required ?? true, + }; +} + +export async function installPrivacyMasks( + page: Page, + policy?: BrowserPrivacyPolicy, +): Promise { + if (!policy) return; + await page.evaluate( + ({ masks, rootId, stateKey }) => { + type PageMaskState = { timer?: number; foundIds: string[] }; + const runtime = window as typeof window & + Record; + const previous = runtime[stateKey]; + if (previous?.timer !== undefined) window.clearInterval(previous.timer); + document.getElementById(rootId)?.remove(); + + const root = document.createElement("div"); + root.id = rootId; + root.setAttribute("aria-hidden", "true"); + root.style.cssText = + "position:fixed;inset:0;pointer-events:none;z-index:2147483647;overflow:hidden"; + document.documentElement.appendChild(root); + const found = new Set(); + const overlays = masks.map((mask) => { + const overlay = document.createElement("div"); + overlay.dataset.seqvioPrivacyMask = mask.id; + overlay.style.cssText = `position:fixed;display:none;background:${mask.color};pointer-events:none`; + root.appendChild(overlay); + return { mask, overlay }; + }); + const update = () => { + for (const { mask, overlay } of overlays) { + let rect = mask.rect; + if (mask.selector) { + try { + const target = document.querySelector(mask.selector); + if (target) { + const live = target.getBoundingClientRect(); + if (live.width > 0 && live.height > 0) { + rect = { + x: live.x, + y: live.y, + width: live.width, + height: live.height, + }; + found.add(mask.id); + } + } + } catch { + rect = undefined; + } + } else { + found.add(mask.id); + } + if (!rect) { + overlay.style.display = "none"; + continue; + } + const padding = mask.padding ?? 0; + overlay.style.display = "block"; + overlay.style.left = `${Math.max(0, rect.x - padding)}px`; + overlay.style.top = `${Math.max(0, rect.y - padding)}px`; + overlay.style.width = `${Math.max(0, rect.width + padding * 2)}px`; + overlay.style.height = `${Math.max(0, rect.height + padding * 2)}px`; + } + const state = runtime[stateKey]; + if (state) state.foundIds = [...found]; + }; + const state: PageMaskState = { foundIds: [] }; + runtime[stateKey] = state; + update(); + state.timer = window.setInterval(update, 50); + }, + { masks: policy.masks, rootId: MASK_ROOT_ID, stateKey: MASK_STATE_KEY }, + ); +} + +export async function readPrivacyMaskStatus( + page: Page, +): Promise { + return page.evaluate((stateKey) => { + const runtime = window as typeof window & + Record; + return { foundIds: [...(runtime[stateKey]?.foundIds ?? [])] }; + }, MASK_STATE_KEY); +} diff --git a/packages/browser-recorder/src/record.ts b/packages/browser-recorder/src/record.ts index e58af33..cec3b06 100644 --- a/packages/browser-recorder/src/record.ts +++ b/packages/browser-recorder/src/record.ts @@ -1,8 +1,12 @@ -import { spawn } from 'node:child_process'; -import * as fs from 'node:fs'; -import * as path from 'node:path'; -import ffmpegPath from '@ffmpeg-installer/ffmpeg'; -import puppeteer, { type ElementHandle, type KeyInput, type Page } from 'puppeteer'; +import { spawn } from "node:child_process"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import ffmpegPath from "@ffmpeg-installer/ffmpeg"; +import puppeteer, { + type ElementHandle, + type KeyInput, + type Page, +} from "puppeteer"; import type { BrowserAction, BrowserRecordingPlan, @@ -10,25 +14,56 @@ import type { RecordedFocusTarget, RecordingManifest, TimedPoint, -} from './types'; +} from "./types"; +import { installPrivacyMasks, readPrivacyMaskStatus } from "./privacy"; const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); -async function encodeFrames(framesDir: string, fps: number, output: string): Promise { +async function encodeFrames( + framesDir: string, + fps: number, + output: string, +): Promise { await new Promise((resolve, reject) => { - const ffmpeg = spawn(ffmpegPath.path, [ - '-y', '-framerate', String(fps), '-i', path.join(framesDir, '%06d.jpg'), - '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-crf', '18', '-preset', 'fast', - '-movflags', '+faststart', output, - ], { windowsHide: true }); - let stderr = ''; - ffmpeg.stderr.on('data', (chunk) => { stderr += chunk.toString(); }); - ffmpeg.on('error', reject); - ffmpeg.on('close', (code) => code === 0 ? resolve() : reject(new Error(stderr || `FFmpeg exited ${code}`))); + const ffmpeg = spawn( + ffmpegPath.path, + [ + "-y", + "-framerate", + String(fps), + "-i", + path.join(framesDir, "%06d.jpg"), + "-c:v", + "libx264", + "-pix_fmt", + "yuv420p", + "-crf", + "18", + "-preset", + "fast", + "-movflags", + "+faststart", + output, + ], + { windowsHide: true }, + ); + let stderr = ""; + ffmpeg.stderr.on("data", (chunk) => { + stderr += chunk.toString(); + }); + ffmpeg.on("error", reject); + ffmpeg.on("close", (code) => + code === 0 + ? resolve() + : reject(new Error(stderr || `FFmpeg exited ${code}`)), + ); }); } -async function resolveElement(page: Page, selector: string): Promise> { +async function resolveElement( + page: Page, + selector: string, +): Promise> { const element = await page.$(selector); if (!element) throw new Error(`Element not found: ${selector}`); return element; @@ -38,18 +73,34 @@ export async function recordPlan( plan: BrowserRecordingPlan, jobDir: string, onProgress?: (progress: PipelineProgress) => void, -): Promise<{ manifest: RecordingManifest; manifestPath: string; rawVideoPath: string }> { - const framesDir = path.join(jobDir, 'frames'); +): Promise<{ + manifest: RecordingManifest; + manifestPath: string; + rawVideoPath: string; +}> { + const framesDir = path.join(jobDir, "frames"); fs.mkdirSync(framesDir, { recursive: true }); const captureFps = plan.captureFps ?? 15; const renderFps = plan.renderFps ?? 30; const browser = await puppeteer.launch({ headless: true, - args: ['--no-sandbox', '--disable-setuid-sandbox', '--disable-dev-shm-usage'], + args: [ + "--no-sandbox", + "--disable-setuid-sandbox", + "--disable-dev-shm-usage", + ], }); const page = await browser.newPage(); await page.setViewport({ ...plan.viewport, deviceScaleFactor: 1 }); - await page.goto(plan.startUrl, { waitUntil: 'networkidle2', timeout: 60000 }); + await page.goto(plan.startUrl, { waitUntil: "networkidle2", timeout: 60000 }); + await installPrivacyMasks(page, plan.privacy); + + const matchedPrivacyMaskIds = new Set(); + const collectPrivacyStatus = async () => { + if (!plan.privacy) return; + const status = await readPrivacyMaskStatus(page); + status.foundIds.forEach((id) => matchedPrivacyMaskIds.add(id)); + }; const startedAt = Date.now(); const now = () => Date.now() - startedAt; @@ -67,8 +118,11 @@ export async function recordPlan( while (!stopCapture) { try { await page.screenshot({ - path: path.join(framesDir, `${String(frameCount).padStart(6, '0')}.jpg`), - type: 'jpeg', + path: path.join( + framesDir, + `${String(frameCount).padStart(6, "0")}.jpg`, + ), + type: "jpeg", quality: 88, optimizeForSpeed: true, }); @@ -76,7 +130,8 @@ export async function recordPlan( const waitMs = startedAt + frameCount * frameInterval - Date.now(); if (waitMs > 0) await delay(waitMs); } catch (error) { - captureError = error instanceof Error ? error : new Error(String(error)); + captureError = + error instanceof Error ? error : new Error(String(error)); stopCapture = true; } } @@ -88,32 +143,56 @@ export async function recordPlan( for (let step = 1; step <= steps; step += 1) { const t = step / steps; const eased = t * t * (3 - 2 * t); - cursor = { x: from.x + (x - from.x) * eased, y: from.y + (y - from.y) * eased }; + cursor = { + x: from.x + (x - from.x) * eased, + y: from.y + (y - from.y) * eased, + }; await page.mouse.move(cursor.x, cursor.y); cursorPoints.push({ timeMs: now(), ...cursor }); await delay(durationMs / steps); } }; - const focusElement = async (action: BrowserAction, element: ElementHandle) => { + const focusElement = async ( + action: BrowserAction, + element: ElementHandle, + ) => { let box = await element.evaluate((node) => { const rect = node.getBoundingClientRect(); return { x: rect.x, y: rect.y, width: rect.width, height: rect.height }; }); - const outsideViewport = box.width <= 0 || box.height <= 0 - || box.x + box.width < 0 || box.y + box.height < 0 - || box.x > plan.viewport.width || box.y > plan.viewport.height; + const outsideViewport = + box.width <= 0 || + box.height <= 0 || + box.x + box.width < 0 || + box.y + box.height < 0 || + box.x > plan.viewport.width || + box.y > plan.viewport.height; if (outsideViewport) { - await element.evaluate((node) => node.scrollIntoView({ block: 'center', inline: 'center', behavior: 'auto' })); + await element.evaluate((node) => + node.scrollIntoView({ + block: "center", + inline: "center", + behavior: "auto", + }), + ); await delay(350); box = await element.evaluate((node) => { const rect = node.getBoundingClientRect(); return { x: rect.x, y: rect.y, width: rect.width, height: rect.height }; }); } - if (box.width <= 0 || box.height <= 0) throw new Error(`Element is not visible: ${action.selector}`); + if (box.width <= 0 || box.height <= 0) + throw new Error(`Element is not visible: ${action.selector}`); if (action.focus !== false) { - focusTargets.push({ id: `focus-${action.id}`, timeMs: now(), x: box.x, y: box.y, width: box.width, height: box.height }); + focusTargets.push({ + id: `focus-${action.id}`, + timeMs: now(), + x: box.x, + y: box.y, + width: box.width, + height: box.height, + }); } await moveCursor(box.x + box.width / 2, box.y + box.height / 2); return box; @@ -132,43 +211,53 @@ export async function recordPlan( }; try { - onProgress?.({ phase: 'recording', percent: 3, message: 'Browser ready' }); + onProgress?.({ phase: "recording", percent: 3, message: "Browser ready" }); for (let index = 0; index < plan.actions.length; index += 1) { const action = plan.actions[index]; actionTimings.push({ id: action.id, timeMs: now() }); onProgress?.({ - phase: 'recording', + phase: "recording", percent: 5 + Math.round((index / plan.actions.length) * 60), message: action.label, }); - if (action.type === 'click') { + if (action.type === "click") { const element = await resolveElement(page, action.selector!); await focusElement(action, element); clicks.push({ timeMs: now(), ...cursor }); await page.mouse.click(cursor.x, cursor.y); - } else if (action.type === 'fill') { + } else if (action.type === "fill") { const element = await resolveElement(page, action.selector!); await focusElement(action, element); clicks.push({ timeMs: now(), ...cursor }); await page.mouse.click(cursor.x, cursor.y, { count: 3 }); - await page.keyboard.press('Backspace'); - await page.keyboard.type(action.value ?? '', { delay: 42 }); - } else if (action.type === 'scroll') { + await page.keyboard.press("Backspace"); + await page.keyboard.type(action.value ?? "", { delay: 42 }); + } else if (action.type === "scroll") { resetFocus(action.id); const x = action.x ?? 0; const y = action.y ?? 520; - await page.evaluate(({ dx, dy }) => window.scrollBy({ left: dx, top: dy, behavior: 'smooth' }), { dx: x, dy: y }); + await page.evaluate( + ({ dx, dy }) => + window.scrollBy({ left: dx, top: dy, behavior: "smooth" }), + { dx: x, dy: y }, + ); await delay(action.durationMs ?? 700); - } else if (action.type === 'navigate') { + } else if (action.type === "navigate") { + await collectPrivacyStatus(); resetFocus(action.id); - await page.goto(action.value!, { waitUntil: 'networkidle2', timeout: 60000 }); - } else if (action.type === 'press') { + await page.goto(action.value!, { + waitUntil: "networkidle2", + timeout: 60000, + }); + await installPrivacyMasks(page, plan.privacy); + } else if (action.type === "press") { resetFocus(action.id); await page.keyboard.press(action.key! as KeyInput); - } else if (action.type === 'wait') { + } else if (action.type === "wait") { await delay(action.durationMs ?? 1000); } await delay(action.afterMs ?? 500); + await collectPrivacyStatus(); } resetFocus(); await delay(700); @@ -178,14 +267,28 @@ export async function recordPlan( await browser.close(); } if (captureError) throw captureError; - if (frameCount === 0) throw new Error('Recorder captured no frames'); + if (frameCount === 0) throw new Error("Recorder captured no frames"); + const missingPrivacyMasks = (plan.privacy?.masks ?? []) + .filter( + (mask) => mask.required !== false && !matchedPrivacyMaskIds.has(mask.id), + ) + .map((mask) => mask.id); + if (missingPrivacyMasks.length > 0) { + throw new Error( + `Required privacy masks never matched: ${missingPrivacyMasks.join(", ")}`, + ); + } - onProgress?.({ phase: 'encoding', percent: 70, message: `Encoding ${frameCount} captured frames` }); - const rawVideoPath = path.join(jobDir, 'raw.mp4'); + onProgress?.({ + phase: "encoding", + percent: 70, + message: `Encoding ${frameCount} captured frames`, + }); + const rawVideoPath = path.join(jobDir, "raw.mp4"); await encodeFrames(framesDir, captureFps, rawVideoPath); const durationMs = Math.round((frameCount / captureFps) * 1000); const manifest: RecordingManifest = { - version: '1.0', + version: "1.0", name: plan.name, sourceVideo: rawVideoPath, recordingWidth: plan.viewport.width, @@ -199,9 +302,21 @@ export async function recordPlan( focusTargets, clicks, actionTimings, + ...(plan.privacy + ? { + privacy: { + maskIds: plan.privacy.masks.map((mask) => mask.id), + matchedMaskIds: [...matchedPrivacyMaskIds].sort(), + }, + } + : {}), }; - const manifestPath = path.join(jobDir, 'recording-manifest.json'); - fs.writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, 'utf8'); + const manifestPath = path.join(jobDir, "recording-manifest.json"); + fs.writeFileSync( + manifestPath, + `${JSON.stringify(manifest, null, 2)}\n`, + "utf8", + ); fs.rmSync(framesDir, { recursive: true, force: true }); return { manifest, manifestPath, rawVideoPath }; } diff --git a/packages/browser-recorder/src/types.ts b/packages/browser-recorder/src/types.ts index 15c28ea..9d2ba7b 100644 --- a/packages/browser-recorder/src/types.ts +++ b/packages/browser-recorder/src/types.ts @@ -1,10 +1,10 @@ export type BrowserActionType = - | 'click' - | 'fill' - | 'scroll' - | 'wait' - | 'navigate' - | 'press'; + | "click" + | "fill" + | "scroll" + | "wait" + | "navigate" + | "press"; export interface BrowserAction { id: string; @@ -20,14 +20,34 @@ export interface BrowserAction { focus?: boolean; } +export interface BrowserPrivacyMask { + /** Stable, non-sensitive identifier written to recording metadata. */ + id: string; + /** Mask a live DOM element. Exactly one of selector or rect is required. */ + selector?: string; + /** Mask a fixed viewport region. Exactly one of selector or rect is required. */ + rect?: { x: number; y: number; width: number; height: number }; + /** Extra pixels around the matched element or rectangle. */ + padding?: number; + /** Solid CSS hex color. Defaults to #111827. */ + color?: string; + /** Fail the recording when a selector never matches. Defaults to true. */ + required?: boolean; +} + +export interface BrowserPrivacyPolicy { + masks: BrowserPrivacyMask[]; +} + export interface BrowserRecordingPlan { - version: '1.0'; + version: "1.0"; name: string; startUrl: string; viewport: { width: number; height: number }; captureFps?: number; renderFps?: number; maxZoom?: number; + privacy?: BrowserPrivacyPolicy; actions: BrowserAction[]; } @@ -45,7 +65,7 @@ export interface RecordedFocusTarget extends TimedPoint { } export interface RecordingManifest { - version: '1.0'; + version: "1.0"; name: string; sourceVideo: string; recordingWidth: number; @@ -60,15 +80,26 @@ export interface RecordingManifest { clicks: TimedPoint[]; /** Exact action start times captured from the recording clock. */ actionTimings?: Array<{ id: string; timeMs: number }>; + /** Privacy evidence without retaining selectors or page content. */ + privacy?: { maskIds: string[]; matchedMaskIds: string[] }; } export interface PipelineProgress { - phase: 'queued' | 'recording' | 'encoding' | 'composing' | 'synthesizing' | 'rendering' | 'qa' | 'done' | 'failed'; + phase: + | "queued" + | "recording" + | "encoding" + | "composing" + | "synthesizing" + | "rendering" + | "qa" + | "done" + | "failed"; percent: number; message: string; } -export type TtsProvider = 'elevenlabs' | 'minimax' | 'edge-tts' | 'openai'; +export type TtsProvider = "elevenlabs" | "minimax" | "edge-tts" | "openai"; export interface BrowserPipelineOptions { withAudio?: boolean; diff --git a/packages/browser-recorder/src/validate.ts b/packages/browser-recorder/src/validate.ts index 58603ba..66d7152 100644 --- a/packages/browser-recorder/src/validate.ts +++ b/packages/browser-recorder/src/validate.ts @@ -1,44 +1,94 @@ -import type { BrowserAction, BrowserRecordingPlan } from './types'; +import type { BrowserAction, BrowserRecordingPlan } from "./types"; +import { normalizePrivacyPolicy } from "./privacy"; -const ACTION_TYPES = new Set(['click', 'fill', 'scroll', 'wait', 'navigate', 'press']); +const ACTION_TYPES = new Set([ + "click", + "fill", + "scroll", + "wait", + "navigate", + "press", +]); function assert(condition: unknown, message: string): asserts condition { if (!condition) throw new Error(message); } function validateAction(action: BrowserAction, index: number): void { - assert(action && typeof action === 'object', `actions[${index}] must be an object`); - assert(typeof action.id === 'string' && action.id.length > 0, `actions[${index}].id is required`); - assert(ACTION_TYPES.has(action.type), `actions[${index}].type is unsupported`); - assert(typeof action.label === 'string' && action.label.length > 0, `actions[${index}].label is required`); - if (action.type === 'click' || action.type === 'fill') { - assert(typeof action.selector === 'string' && action.selector.length > 0, `${action.id}.selector is required`); + assert( + action && typeof action === "object", + `actions[${index}] must be an object`, + ); + assert( + typeof action.id === "string" && action.id.length > 0, + `actions[${index}].id is required`, + ); + assert( + ACTION_TYPES.has(action.type), + `actions[${index}].type is unsupported`, + ); + assert( + typeof action.label === "string" && action.label.length > 0, + `actions[${index}].label is required`, + ); + if (action.type === "click" || action.type === "fill") { + assert( + typeof action.selector === "string" && action.selector.length > 0, + `${action.id}.selector is required`, + ); } - if (action.type === 'fill') { - assert(typeof action.value === 'string', `${action.id}.value is required`); + if (action.type === "fill") { + assert(typeof action.value === "string", `${action.id}.value is required`); } - if (action.type === 'navigate') { - assert(typeof action.value === 'string' && action.value.length > 0, `${action.id}.value is required`); + if (action.type === "navigate") { + assert( + typeof action.value === "string" && action.value.length > 0, + `${action.id}.value is required`, + ); } - if (action.type === 'press') { - assert(typeof action.key === 'string' && action.key.length > 0, `${action.id}.key is required`); + if (action.type === "press") { + assert( + typeof action.key === "string" && action.key.length > 0, + `${action.id}.key is required`, + ); } } export function validatePlan(value: unknown): BrowserRecordingPlan { - assert(value && typeof value === 'object', 'Plan must be an object'); + assert(value && typeof value === "object", "Plan must be an object"); const plan = value as BrowserRecordingPlan; - assert(plan.version === '1.0', 'plan.version must be "1.0"'); - assert(typeof plan.name === 'string' && plan.name.trim().length > 0, 'plan.name is required'); - assert(typeof plan.startUrl === 'string', 'plan.startUrl is required'); + assert(plan.version === "1.0", 'plan.version must be "1.0"'); + assert( + typeof plan.name === "string" && plan.name.trim().length > 0, + "plan.name is required", + ); + assert(typeof plan.startUrl === "string", "plan.startUrl is required"); const url = new URL(plan.startUrl); - assert(['http:', 'https:', 'file:'].includes(url.protocol), 'startUrl must use http, https, or file'); - assert(plan.viewport && Number.isFinite(plan.viewport.width), 'viewport.width is required'); - assert(plan.viewport && Number.isFinite(plan.viewport.height), 'viewport.height is required'); - assert(plan.viewport.width >= 640 && plan.viewport.width <= 3840, 'viewport.width must be 640-3840'); - assert(plan.viewport.height >= 360 && plan.viewport.height <= 2160, 'viewport.height must be 360-2160'); - assert(Array.isArray(plan.actions) && plan.actions.length > 0, 'actions must contain at least one action'); - assert(plan.actions.length <= 100, 'actions cannot exceed 100 items'); + assert( + ["http:", "https:", "file:"].includes(url.protocol), + "startUrl must use http, https, or file", + ); + assert( + plan.viewport && Number.isFinite(plan.viewport.width), + "viewport.width is required", + ); + assert( + plan.viewport && Number.isFinite(plan.viewport.height), + "viewport.height is required", + ); + assert( + plan.viewport.width >= 640 && plan.viewport.width <= 3840, + "viewport.width must be 640-3840", + ); + assert( + plan.viewport.height >= 360 && plan.viewport.height <= 2160, + "viewport.height must be 360-2160", + ); + assert( + Array.isArray(plan.actions) && plan.actions.length > 0, + "actions must contain at least one action", + ); + assert(plan.actions.length <= 100, "actions cannot exceed 100 items"); plan.actions.forEach(validateAction); return { ...plan, @@ -46,5 +96,6 @@ export function validatePlan(value: unknown): BrowserRecordingPlan { captureFps: Math.max(5, Math.min(30, plan.captureFps ?? 15)), renderFps: Math.max(24, Math.min(60, plan.renderFps ?? 30)), maxZoom: Math.max(1, Math.min(3, plan.maxZoom ?? 2.2)), + privacy: normalizePrivacyPolicy(plan.privacy), }; } diff --git a/packages/browser-recorder/tests/privacy.test.mjs b/packages/browser-recorder/tests/privacy.test.mjs new file mode 100644 index 0000000..a63e9ca --- /dev/null +++ b/packages/browser-recorder/tests/privacy.test.mjs @@ -0,0 +1,51 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import puppeteer from 'puppeteer'; +import { installPrivacyMasks, readPrivacyMaskStatus } from '../dist/privacy.js'; + +test('privacy masks cover selectors and fixed regions without intercepting input', async (t) => { + const browser = await puppeteer.launch({ + headless: true, + args: ['--no-sandbox', '--disable-setuid-sandbox', '--disable-dev-shm-usage'], + }); + t.after(() => browser.close()); + const page = await browser.newPage(); + await page.setViewport({ width: 640, height: 360 }); + await page.setContent(''); + await installPrivacyMasks(page, { masks: [ + { id: 'token', selector: '#secret', padding: 4, color: '#123456', required: true }, + { id: 'account', rect: { x: 20, y: 24, width: 60, height: 30 }, padding: 0, color: '#000000', required: true }, + ] }); + + const result = await page.evaluate(() => { + const selectorMask = document.querySelector('[data-seqvio-privacy-mask="token"]'); + const regionMask = document.querySelector('[data-seqvio-privacy-mask="account"]'); + const selectorRect = selectorMask.getBoundingClientRect(); + const regionRect = regionMask.getBoundingClientRect(); + return { + selector: { + x: selectorRect.x, y: selectorRect.y, width: selectorRect.width, height: selectorRect.height, + background: getComputedStyle(selectorMask).backgroundColor, + pointerEvents: getComputedStyle(selectorMask).pointerEvents, + }, + region: { x: regionRect.x, y: regionRect.y, width: regionRect.width, height: regionRect.height }, + }; + }); + assert.deepEqual(result.selector, { + x: 96, y: 76, width: 228, height: 44, + background: 'rgb(18, 52, 86)', pointerEvents: 'none', + }); + assert.deepEqual(result.region, { x: 20, y: 24, width: 60, height: 30 }); + assert.deepEqual((await readPrivacyMaskStatus(page)).foundIds.sort(), ['account', 'token']); +}); + +test('privacy mask status does not report selectors that never matched', async (t) => { + const browser = await puppeteer.launch({ headless: true, args: ['--no-sandbox'] }); + t.after(() => browser.close()); + const page = await browser.newPage(); + await page.setContent('
Public content
'); + await installPrivacyMasks(page, { masks: [ + { id: 'missing-secret', selector: '#does-not-exist', padding: 4, color: '#111827', required: true }, + ] }); + assert.deepEqual((await readPrivacyMaskStatus(page)).foundIds, []); +}); diff --git a/packages/browser-recorder/tests/serve-lifecycle.test.mjs b/packages/browser-recorder/tests/serve-lifecycle.test.mjs new file mode 100644 index 0000000..50b95e8 --- /dev/null +++ b/packages/browser-recorder/tests/serve-lifecycle.test.mjs @@ -0,0 +1,47 @@ +import assert from 'node:assert/strict'; +import { spawn } from 'node:child_process'; +import fs from 'node:fs'; +import http from 'node:http'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; + +function reservePort() { + return new Promise((resolve, reject) => { + const server = http.createServer(); + server.once('error', reject); + server.listen(0, '127.0.0.1', () => { + const address = server.address(); + server.close((error) => error ? reject(error) : resolve(address.port)); + }); + }); +} + +async function waitForDemo(port, child) { + for (let attempt = 0; attempt < 100; attempt += 1) { + if (child.exitCode !== null) throw new Error(`serve process exited early with ${child.exitCode}`); + try { + const response = await fetch(`http://127.0.0.1:${port}/demo`); + if (response.ok) return response.text(); + } catch {} + await new Promise((resolve) => setTimeout(resolve, 100)); + } + throw new Error('serve process did not become ready'); +} + +test('CLI serve entrypoint remains alive and responds until terminated', async (t) => { + const port = await reservePort(); + const outputDir = fs.mkdtempSync(path.join(os.tmpdir(), 'seqvio-browser-serve-')); + const cli = path.resolve('dist/cli.js'); + const child = spawn(process.execPath, [cli, 'serve', '--port', String(port), '--outputDir', outputDir], { + stdio: 'ignore', + windowsHide: true, + }); + t.after(() => { + if (child.exitCode === null) child.kill(); + }); + const html = await waitForDemo(port, child); + assert.match(html, /Seqvio Recorder/); + assert.equal(child.exitCode, null); + child.kill(); +}); diff --git a/packages/browser-recorder/tests/validate.test.mjs b/packages/browser-recorder/tests/validate.test.mjs index 573b0d6..34f2ab1 100644 --- a/packages/browser-recorder/tests/validate.test.mjs +++ b/packages/browser-recorder/tests/validate.test.mjs @@ -24,3 +24,39 @@ test('validatePlan rejects executable URL protocols', () => { actions: [{ id: 'wait', type: 'wait', label: 'Wait' }], }), /http, https, or file/); }); + +test('validatePlan normalizes deterministic privacy masks', () => { + const plan = validatePlan({ + version: '1.0', + name: 'Private demo', + startUrl: 'https://example.com', + viewport: { width: 1280, height: 720 }, + privacy: { + masks: [ + { id: 'api-token', selector: '#token' }, + { id: 'account', rect: { x: 20, y: 30, width: 200, height: 40 }, required: false }, + ], + }, + actions: [{ id: 'wait', type: 'wait', label: 'Wait' }], + }); + assert.deepEqual(plan.privacy.masks, [ + { id: 'api-token', selector: '#token', padding: 4, color: '#111827', required: true }, + { id: 'account', rect: { x: 20, y: 30, width: 200, height: 40 }, padding: 4, color: '#111827', required: false }, + ]); +}); + +test('validatePlan rejects ambiguous and unsafe privacy masks', () => { + const base = { + version: '1.0', name: 'Private demo', startUrl: 'https://example.com', + viewport: { width: 1280, height: 720 }, + actions: [{ id: 'wait', type: 'wait', label: 'Wait' }], + }; + assert.throws(() => validatePlan({ + ...base, + privacy: { masks: [{ id: 'ambiguous', selector: '#token', rect: { x: 0, y: 0, width: 1, height: 1 } }] }, + }), /exactly one/); + assert.throws(() => validatePlan({ + ...base, + privacy: { masks: [{ id: 'bad color', selector: '#token', color: 'transparent' }] }, + }), /safe 1-64 character identifier/); +}); diff --git a/packages/capture/src/cli-contract.ts b/packages/capture/src/cli-contract.ts index d83db23..459ba9f 100644 --- a/packages/capture/src/cli-contract.ts +++ b/packages/capture/src/cli-contract.ts @@ -2,7 +2,7 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; export const CAPTURE_CLI_CONTRACT_VERSION = '2.0' as const; -export const CAPTURE_ADAPTER_LIFECYCLE = 'pre-stable' as const; +export const CAPTURE_ADAPTER_LIFECYCLE = 'stable' as const; export const CAPTURE_ARTIFACT_MANIFEST_VERSION = '1.0' as const; export const CaptureCliExitCode = { diff --git a/packages/core/src/annotation/AnnotationLayer.tsx b/packages/core/src/annotation/AnnotationLayer.tsx index 4cc9aea..aa6983f 100644 --- a/packages/core/src/annotation/AnnotationLayer.tsx +++ b/packages/core/src/annotation/AnnotationLayer.tsx @@ -8,6 +8,7 @@ import React, { useState, } from 'react'; import { useCurrentFrame } from '../frame'; +import { resolveSafeLabelPlacement, resolveSafeLabelPlacements, routeConnector, routeGuidedPath, type SafeLabelPlacement } from './routing'; export interface TargetRect { x: number; @@ -16,6 +17,10 @@ export interface TargetRect { height: number; } +function sameRect(left: TargetRect, right: TargetRect): boolean { + return left.x === right.x && left.y === right.y && left.width === right.width && left.height === right.height; +} + interface AnnotationTargetRegistry { register: (id: string, node: Element | null) => void; getRect: (id: string) => TargetRect | undefined; @@ -154,36 +159,52 @@ export type AnnotationKind = | 'circle' | 'box' | 'underline' - | 'spotlight'; + | 'spotlight' + | 'focus-ring' + | 'callout' + | 'bracket' + | 'connector' + | 'region-shade' + | 'guided-path'; export interface AnnotationItem { id: string; targetId: string; + toTargetId?: string; + pathTargetIds?: string[]; kind: AnnotationKind; start: number; duration: number; label?: string; + /** Higher-priority annotations render above lower-priority annotations. */ + priority?: number; } export interface AnnotationLayerProps { annotations: AnnotationItem[]; } -function annotationOpacity(frame: number, start: number, duration: number): number { +export function annotationOpacity(frame: number, start: number, duration: number): number { if (frame < start) return 0; if (frame >= start + duration) return 1; const t = (frame - start) / Math.max(1, duration); return Math.min(1, t * 1.4); } +export function orderAnnotationsForStacking>(annotations: T[]): T[] { + return [...annotations].sort((left, right) => (left.priority ?? 0) - (right.priority ?? 0) || left.id.localeCompare(right.id)); +} + export const AnnotationLayer: React.FC = ({ annotations }) => { const registry = useAnnotationTargetRegistry(); const frame = useCurrentFrame(); const readyId = useId(); const [registryVersion, bump] = useState(0); const [rects, setRects] = useState>(() => new Map()); - const annotationKey = annotations - .map((annotation) => `${annotation.id}:${annotation.targetId}:${annotation.start}:${annotation.duration}`) + const [canvasSize, setCanvasSize] = useState({ width: 1280, height: 720 }); + const orderedAnnotations = orderAnnotationsForStacking(annotations); + const annotationKey = orderedAnnotations + .map((annotation) => `${annotation.id}:${annotation.targetId}:${annotation.toTargetId ?? ''}:${annotation.pathTargetIds?.join(',') ?? ''}:${annotation.start}:${annotation.duration}:${annotation.priority ?? 0}`) .join('|'); useLayoutEffect(() => { @@ -226,10 +247,20 @@ export const AnnotationLayer: React.FC = ({ annotations }) const measureHandle = requestAnimationFrame(() => { if (cancelled) return; const next = new Map(); - for (const annotation of annotations) { + const container = registry.getContainer(); + if (container) setCanvasSize({ width: container.clientWidth, height: container.clientHeight }); + for (const annotation of orderedAnnotations) { if (annotationOpacity(frame, annotation.start, annotation.duration) <= 0) continue; const rect = registry.getRect(annotation.targetId); if (rect) next.set(annotation.id, rect); + if (annotation.toTargetId) { + const toRect = registry.getRect(annotation.toTargetId); + if (toRect) next.set(`${annotation.id}:to`, toRect); + } + for (const [index, targetId] of (annotation.pathTargetIds ?? []).entries()) { + const pathRect = registry.getRect(targetId); + if (pathRect) next.set(`${annotation.id}:path:${index}`, pathRect); + } } setRects(next); resolveReady?.(); @@ -243,24 +274,51 @@ export const AnnotationLayer: React.FC = ({ annotations }) if (!registry) return null; + const measuredTargets = [...rects.entries()] + .filter(([id]) => !id.includes(':to') && !id.includes(':path:')) + .map(([, rect]) => rect) + .filter((rect, index, all) => all.findIndex((candidate) => candidate.x === rect.x && candidate.y === rect.y && candidate.width === rect.width && candidate.height === rect.height) === index); + const labelPlacements = resolveSafeLabelPlacements( + orderedAnnotations.filter((annotation) => annotation.kind === 'callout' && annotation.label && rects.has(annotation.id)).map((annotation) => ({ + id: annotation.id, + target: rects.get(annotation.id)!, + width: 180, + height: 38, + })), + canvasSize.width, + canvasSize.height, + measuredTargets, + 72, + ); + const labelPlacementMap = new Map(labelPlacements.map((placement) => [placement.id, placement])); + const routeObstacles: TargetRect[] = [...measuredTargets, ...labelPlacements]; + return (
- {annotations.map((annotation) => { + {orderedAnnotations.map((annotation) => { const opacity = annotationOpacity(frame, annotation.start, annotation.duration); if (opacity <= 0) return null; const rect = rects.get(annotation.id); - if (!rect) return null; + if (!rect) return ; return (
rects.get(`${annotation.id}:path:${index}`)).filter((value): value is TargetRect => Boolean(value))} + canvasWidth={canvasSize.width} + canvasHeight={canvasSize.height} opacity={opacity} + labelPlacement={labelPlacementMap.get(annotation.id)} + routeObstacles={routeObstacles} />
); @@ -272,8 +330,14 @@ export const AnnotationLayer: React.FC = ({ annotations }) const AnnotationOverlay: React.FC<{ annotation: AnnotationItem; rect: TargetRect; + toRect?: TargetRect; + pathRects: TargetRect[]; + canvasWidth: number; + canvasHeight: number; opacity: number; -}> = ({ annotation, rect, opacity }) => { + labelPlacement?: SafeLabelPlacement; + routeObstacles: TargetRect[]; +}> = ({ annotation, rect, toRect, pathRects, canvasWidth, canvasHeight, opacity, labelPlacement, routeObstacles }) => { const pad = 8; const common: React.CSSProperties = { position: 'absolute', @@ -298,7 +362,7 @@ const AnnotationOverlay: React.FC<{ ); } - if (annotation.kind === 'circle') { + if (annotation.kind === 'circle' || annotation.kind === 'focus-ring') { return (
; + } + + if (annotation.kind === 'bracket') { + return ( +
+ {annotation.label ?
{annotation.label}
: null} +
+ ); + } + + if (annotation.kind === 'callout') { + const label = labelPlacement ?? resolveSafeLabelPlacement(rect, 180, 38, canvasWidth, canvasHeight, [rect], 72); + const route = routeConnector(rect, label, canvasWidth, canvasHeight, 24, routeObstacles.filter((obstacle) => !sameRect(obstacle, rect) && !sameRect(obstacle, label))); + return ( + <> `${point.x},${point.y}`).join(' ')} fill="none" stroke="#7dd3fc" strokeWidth="2" strokeLinejoin="round" />
{annotation.label ?? 'Note'}
+ ); + } + + if (annotation.kind === 'connector' && toRect) { + const points = routeConnector(rect, toRect, canvasWidth, canvasHeight, 24, routeObstacles.filter((obstacle) => !sameRect(obstacle, rect) && !sameRect(obstacle, toRect))); + const middle = points[Math.floor(points.length / 2)]; + return ( + + `${point.x},${point.y}`).join(' ')} fill="none" stroke="#38bdf8" strokeWidth="3" strokeLinecap="round" strokeLinejoin="round" /> + {points.length ? <> : null} + {annotation.label && middle ? {annotation.label} : null} + + ); + } + + if (annotation.kind === 'guided-path' && pathRects.length >= 2) { + const points = routeGuidedPath(pathRects, canvasWidth, canvasHeight, 24, routeObstacles.filter((obstacle) => !pathRects.includes(obstacle))); + return `${point.x},${point.y}`).join(' ')} fill="none" stroke="#fbbf24" strokeWidth="3" strokeDasharray="8 7" strokeLinecap="round" strokeLinejoin="round" />{points.map((point, index) => )}; + } + if (annotation.kind === 'arrow') { return (
diff --git a/packages/core/src/annotation/attention.tsx b/packages/core/src/annotation/attention.tsx new file mode 100644 index 0000000..07943d1 --- /dev/null +++ b/packages/core/src/annotation/attention.tsx @@ -0,0 +1,116 @@ +import React from 'react'; +import { AnnotationLayer, type AnnotationItem, type AnnotationKind } from './AnnotationLayer'; +import { mapSceneOutputFrameToSource } from '../frame'; +import { useCurrentFrame } from '../frame'; +import type { AudioSceneTiming } from '../audio'; +import { useStyleProfile } from '../style-profile-runtime'; + +export interface AttentionSequenceItem extends AnnotationItem { + /** Scene that owns the target. Omit for legacy scene-local sequences. */ + sceneId?: string; + /** Optional stable id of the next focus target for an explicit handoff. */ + handoffTo?: string; + /** Minimum visible hold in frames before a handoff may clear this item. */ + minHoldFrames?: number; + /** Composition-unique ExplanationBeat id that produced this attention item. */ + sourceBeatId?: string; + /** Controls how long the focus remains active. */ + persistence?: 'timed' | 'until-handoff' | 'until-clear'; + /** Explicit scene-local clear frame, required for until-clear persistence. */ + clearAt?: number; + /** Destination scene for a cross-scene semantic handoff. */ + handoffToSceneId?: string; +} + +export interface ResolvedAttentionItem extends AttentionSequenceItem { + active: boolean; + handoff: boolean; +} + +export interface AttentionSequenceIssue { + itemId: string; + code: 'missing_clear_frame' | 'invalid_clear_frame' | 'unknown_handoff_target'; + message: string; +} + +export function validateAttentionSequence(sequence: AttentionSequenceItem[]): AttentionSequenceIssue[] { + const issues: AttentionSequenceIssue[] = []; + const targets = new Set(sequence.map((item) => `${item.sceneId ?? ''}:${item.targetId}`)); + for (const item of sequence) { + if (item.persistence === 'until-clear' && item.clearAt === undefined) { + issues.push({ itemId: item.id, code: 'missing_clear_frame', message: 'until-clear persistence requires clearAt' }); + } else if (item.clearAt !== undefined && item.clearAt <= item.start) { + issues.push({ itemId: item.id, code: 'invalid_clear_frame', message: 'clearAt must be greater than start' }); + } + if (item.handoffTo) { + const destination = `${item.handoffToSceneId ?? item.sceneId ?? ''}:${item.handoffTo}`; + if (!targets.has(destination)) { + issues.push({ itemId: item.id, code: 'unknown_handoff_target', message: `handoff target ${destination} does not exist` }); + } + } + } + return issues; +} + +export function resolveAttentionSequence( + sequence: AttentionSequenceItem[], + frame: number, +): ResolvedAttentionItem[] { + return sequence.map((item, index) => { + const next = sequence[index + 1]; + const timedEnd = item.start + Math.max(item.duration, item.minHoldFrames ?? 0); + const holdUntil = item.persistence === 'until-clear' + ? Math.max(item.start, item.clearAt ?? timedEnd) + : item.persistence === 'until-handoff' + ? Math.max(item.start, next?.start ?? timedEnd) + : timedEnd; + const active = frame >= item.start && frame < holdUntil; + return { + ...item, + active, + handoff: Boolean(next && item.handoffTo === next.targetId && frame >= next.start), + }; + }); +} + +export function selectAttentionForScene( + sequence: AttentionSequenceItem[], + sceneId: string, +): AttentionSequenceItem[] { + return sequence.filter((item) => item.sceneId === undefined || item.sceneId === sceneId); +} + +export function resolveAttentionSequenceAtOutputFrame( + sequence: AttentionSequenceItem[], + outputFrame: number, + sourceDuration: number, + outputDuration: number, + timeMap?: AudioSceneTiming['timeMap'], +): ResolvedAttentionItem[] { + const sourceFrame = mapSceneOutputFrameToSource( + outputFrame, + sourceDuration, + outputDuration, + timeMap, + ); + return resolveAttentionSequence(sequence, sourceFrame); +} + +export interface AttentionSequenceLayerProps { + sequence: AttentionSequenceItem[]; + sceneId?: string; +} + +/** Renders a semantic focus sequence through the shared annotation renderer. */ +export const AttentionSequenceLayer: React.FC = ({ sequence, sceneId }) => { + const frame = useCurrentFrame(); + const styleProfile = useStyleProfile(); + const styledSequence = sequence.map((item) => item.persistence ? item : { ...item, persistence: styleProfile?.attentionPersistence ?? 'timed' }); + const sceneSequence = sceneId ? selectAttentionForScene(styledSequence, sceneId) : styledSequence; + const active = resolveAttentionSequence(sceneSequence, frame) + .filter((item) => item.active) + .map(({ active: _active, handoff: _handoff, ...item }) => item); + return ; +}; + +export type AttentionKind = AnnotationKind; diff --git a/packages/core/src/annotation/index.ts b/packages/core/src/annotation/index.ts index 33346e4..2fd591a 100644 --- a/packages/core/src/annotation/index.ts +++ b/packages/core/src/annotation/index.ts @@ -2,8 +2,26 @@ export { AnnotationLayer, AnnotationProvider, AnnotationTarget, + annotationOpacity, + orderAnnotationsForStacking, useAnnotationTargetRegistry, } from './AnnotationLayer'; +export { + AttentionSequenceLayer, + resolveAttentionSequence, + resolveAttentionSequenceAtOutputFrame, + selectAttentionForScene, + validateAttentionSequence, +} from './attention'; +export { routeConnector, routeGuidedPath, resolveSafeLabelPlacement, resolveSafeLabelPlacements } from './routing'; +export type { LabelPlacementRequest, PlacedLabel, RoutePoint, SafeLabelPlacement } from './routing'; +export type { + AttentionKind, + AttentionSequenceItem, + AttentionSequenceLayerProps, + ResolvedAttentionItem, + AttentionSequenceIssue, +} from './attention'; export type { AnnotationItem, AnnotationKind, diff --git a/packages/core/src/annotation/routing.ts b/packages/core/src/annotation/routing.ts new file mode 100644 index 0000000..b028ba1 --- /dev/null +++ b/packages/core/src/annotation/routing.ts @@ -0,0 +1,160 @@ +import type { TargetRect } from './AnnotationLayer'; + +export interface RoutePoint { + x: number; + y: number; +} + +export interface SafeLabelPlacement extends RoutePoint { + width: number; + height: number; + position: 'above' | 'below' | 'right' | 'left'; +} + +export interface LabelPlacementRequest { + id: string; + target: TargetRect; + width: number; + height: number; +} + +export interface PlacedLabel extends SafeLabelPlacement { + id: string; +} + +function clamp(value: number, min: number, max: number): number { + return Math.max(min, Math.min(max, value)); +} + +function center(rect: TargetRect): RoutePoint { + return { x: rect.x + rect.width / 2, y: rect.y + rect.height / 2 }; +} + +export function routeConnector( + from: TargetRect, + to: TargetRect, + width: number, + height: number, + margin = 24, + obstacles: TargetRect[] = [], +): RoutePoint[] { + const a = center(from); + const b = center(to); + const horizontal = Math.abs(b.x - a.x) >= Math.abs(b.y - a.y); + const corridorValues = horizontal + ? [Math.min(from.y, to.y) - 18, Math.max(from.y + from.height, to.y + to.height) + 18, margin, height - margin] + : [Math.min(from.x, to.x) - 18, Math.max(from.x + from.width, to.x + to.width) + 18, margin, width - margin]; + const routes = [...new Set(corridorValues.map((value) => horizontal + ? clamp(value, margin, Math.max(margin, height - margin)) + : clamp(value, margin, Math.max(margin, width - margin))))].map((corridor) => { + const points = horizontal + ? [a, { x: a.x, y: corridor }, { x: b.x, y: corridor }, b] + : [a, { x: corridor, y: a.y }, { x: corridor, y: b.y }, b]; + return points.map((point) => ({ x: clamp(point.x, margin, Math.max(margin, width - margin)), y: clamp(point.y, margin, Math.max(margin, height - margin)) })); + }); + const score = (points: RoutePoint[]) => { + let hits = 0; + let length = 0; + for (let index = 1; index < points.length; index++) { + const start = points[index - 1]; + const end = points[index]; + length += Math.abs(end.x - start.x) + Math.abs(end.y - start.y); + hits += obstacles.filter((obstacle) => segmentIntersectsRect(start, end, obstacle, 8)).length; + } + return hits * 1_000_000 + length; + }; + return routes.sort((left, right) => score(left) - score(right))[0] ?? [a, b]; +} + +function overlaps(a: TargetRect, b: TargetRect): boolean { + return a.x < b.x + b.width && a.x + a.width > b.x && a.y < b.y + b.height && a.y + a.height > b.y; +} + +function overlapArea(a: TargetRect, b: TargetRect): number { + const width = Math.max(0, Math.min(a.x + a.width, b.x + b.width) - Math.max(a.x, b.x)); + const height = Math.max(0, Math.min(a.y + a.height, b.y + b.height) - Math.max(a.y, b.y)); + return width * height; +} + +function segmentIntersectsRect(start: RoutePoint, end: RoutePoint, rect: TargetRect, padding = 0): boolean { + const left = rect.x - padding; + const right = rect.x + rect.width + padding; + const top = rect.y - padding; + const bottom = rect.y + rect.height + padding; + if (start.x === end.x) return start.x >= left && start.x <= right && Math.max(start.y, end.y) >= top && Math.min(start.y, end.y) <= bottom; + if (start.y === end.y) return start.y >= top && start.y <= bottom && Math.max(start.x, end.x) >= left && Math.min(start.x, end.x) <= right; + return false; +} + +export function resolveSafeLabelPlacement( + target: TargetRect, + labelWidth: number, + labelHeight: number, + canvasWidth: number, + canvasHeight: number, + occupied: TargetRect[] = [], + margin = 18, +): SafeLabelPlacement { + const rawCandidates: SafeLabelPlacement[] = [ + { x: target.x + target.width / 2 - labelWidth / 2, y: target.y - labelHeight - 14, width: labelWidth, height: labelHeight, position: 'above' }, + { x: target.x + target.width / 2 - labelWidth / 2, y: target.y + target.height + 14, width: labelWidth, height: labelHeight, position: 'below' }, + { x: target.x + target.width + 14, y: target.y + target.height / 2 - labelHeight / 2, width: labelWidth, height: labelHeight, position: 'right' }, + { x: target.x - labelWidth - 14, y: target.y + target.height / 2 - labelHeight / 2, width: labelWidth, height: labelHeight, position: 'left' }, + ]; + const inBounds = rawCandidates.filter((candidate) => + candidate.x >= margin && candidate.y >= margin && + candidate.x + candidate.width <= canvasWidth - margin && + candidate.y + candidate.height <= canvasHeight - margin + ); + const ordered = [...inBounds, ...rawCandidates.filter((candidate) => !inBounds.includes(candidate))]; + const candidates: SafeLabelPlacement[] = ordered.map((candidate) => ({ + ...candidate, + x: clamp(candidate.x, margin, Math.max(margin, canvasWidth - labelWidth - margin)), + y: clamp(candidate.y, margin, Math.max(margin, canvasHeight - labelHeight - margin)), + })); + return candidates.find((candidate) => !occupied.some((rect) => overlaps(candidate, rect))) ?? candidates[0]; +} + +export function resolveSafeLabelPlacements( + requests: LabelPlacementRequest[], + canvasWidth: number, + canvasHeight: number, + obstacles: TargetRect[] = [], + margin = 18, +): PlacedLabel[] { + const placed: PlacedLabel[] = []; + for (const request of requests) { + const { target, width, height } = request; + const offsets = [0, 1, -1, 2, -2].map((step) => step * (height + 8)); + const raw = offsets.flatMap((offset) => [ + { x: target.x + target.width / 2 - width / 2 + offset, y: target.y - height - 14, width, height, position: 'above' }, + { x: target.x + target.width / 2 - width / 2 + offset, y: target.y + target.height + 14, width, height, position: 'below' }, + { x: target.x + target.width + 14, y: target.y + target.height / 2 - height / 2 + offset, width, height, position: 'right' }, + { x: target.x - width - 14, y: target.y + target.height / 2 - height / 2 + offset, width, height, position: 'left' }, + ]); + const candidates = raw.map((candidate) => ({ + ...candidate, + x: clamp(candidate.x, margin, Math.max(margin, canvasWidth - width - margin)), + y: clamp(candidate.y, margin, Math.max(margin, canvasHeight - height - margin)), + })).filter((candidate, index, all) => all.findIndex((item) => item.x === candidate.x && item.y === candidate.y) === index); + const occupied: TargetRect[] = [...obstacles, ...placed]; + const score = (candidate: SafeLabelPlacement) => { + const collision = occupied.reduce((sum, rect) => sum + overlapArea(candidate, rect), 0); + const distance = Math.abs(candidate.x + width / 2 - (target.x + target.width / 2)) + Math.abs(candidate.y + height / 2 - (target.y + target.height / 2)); + return collision * 1_000_000 + distance; + }; + const choice = candidates.sort((left, right) => score(left) - score(right))[0] + ?? resolveSafeLabelPlacement(target, width, height, canvasWidth, canvasHeight, occupied, margin); + placed.push({ id: request.id, ...choice }); + } + return placed; +} + +export function routeGuidedPath(rects: TargetRect[], width: number, height: number, margin = 24, obstacles: TargetRect[] = []): RoutePoint[] { + if (rects.length === 0) return []; + const points: RoutePoint[] = [center(rects[0])]; + for (let index = 1; index < rects.length; index++) { + points.push(...routeConnector(rects[index - 1], rects[index], width, height, margin, obstacles).slice(1)); + } + return points; +} diff --git a/packages/core/src/audio.ts b/packages/core/src/audio.ts index fccd8f5..e07fb97 100644 --- a/packages/core/src/audio.ts +++ b/packages/core/src/audio.ts @@ -113,6 +113,8 @@ export interface RenderableMeta { minDurationFrames: number; }>; }; + /** Reviewable renderer-agnostic direction compiled from semantic ids. */ + direction?: import('./direction').CompiledDirectionPlan; } export function resolveCompositionAudioManifest( diff --git a/packages/core/src/authoring/markdown.ts b/packages/core/src/authoring/markdown.ts index 3e5b644..c332272 100644 --- a/packages/core/src/authoring/markdown.ts +++ b/packages/core/src/authoring/markdown.ts @@ -37,17 +37,33 @@ export function formatEditorialPlanMarkdown(plan: EditorialPlan): string { `- Concepts: ${section.conceptIds.map((id) => `\`${id}\``).join(', ')}`, `- Audience outcome: ${section.expectedOutcome}`, ]; + if (section.evidenceSource !== undefined) lines.push(`- Evidence source: ${section.evidenceSource}`); if (section.targetSeconds !== undefined) lines.push(`- Target: ${section.targetSeconds}s`); return lines.join('\n'); }).join('\n\n'); - return `---\nformat: ${plan.format}\nid: ${plan.id}\nduration_budget_sec: ${plan.durationBudgetSec}\n---\n\n# Editorial Plan: ${plan.title}\n\n## Objective\n\n${plan.objective}\n\n## Audience\n\n${plan.audience.description}\n\n### Prior Knowledge\n\n${bullets(plan.audience.priorKnowledge)}\n\n### Likely Misconceptions\n\n${bullets(plan.audience.likelyMisconceptions)}\n\n## Thesis\n\n${plan.thesis}\n\n## Explanation Strategy\n\n${strategy}\n\n## Content Decisions\n\n${concepts}\n\n## Explanation Structure\n\n${sections}\n`; + return `---\nformat: ${plan.format}\nid: ${plan.id}\nduration_budget_sec: ${plan.durationBudgetSec}\n---\n\n# Editorial Plan: ${plan.title}\n\n## Objective\n\n${plan.objective}\n\n## Audience\n\n${plan.audience.description}\n\n### Prior Knowledge\n\n${bullets(plan.audience.priorKnowledge)}\n\n### Likely Misconceptions\n\n${bullets(plan.audience.likelyMisconceptions)}\n\n## Thesis\n\n${plan.thesis}\n\n## Hook\n\n${plan.hook ?? 'Not declared.'}\n\n## Explanation Strategy\n\n${strategy}\n\n## Content Decisions\n\n${concepts}\n\n## Explanation Structure\n\n${sections}\n`; } export function formatVisualDesignBriefMarkdown(brief: VisualDesignBrief): string { const palette = brief.palette.map((item) => `| ${item.role} | \`${item.value}\` | ${item.use} |`).join('\n'); const typography = brief.typography.map((item) => `| ${item.role} | ${item.family} | ${item.sizePx}px | ${item.weight ?? 'normal'} | ${item.use} |`).join('\n'); - const treatments = brief.sceneTreatments?.map((item) => `### ${item.sectionId}\n\n- Scenes: ${item.sceneIds.map((id) => `\`${id}\``).join(', ')}\n- Visual form: ${item.visualForm}\n- Composition: ${item.composition}\n- Emphasis: ${item.emphasis}`).join('\n\n') ?? 'No section-specific treatments.'; + const treatments = brief.sceneTreatments?.map((item) => { + const lines = [ + `### ${item.sectionId}`, + '', + `- Scenes: ${item.sceneIds.map((id) => `\`${id}\``).join(', ')}`, + `- Visual form: ${item.visualForm}`, + `- Composition: ${item.composition}`, + `- Emphasis: ${item.emphasis}`, + ]; + if (item.visualRole !== undefined) lines.push(`- Visual role: ${item.visualRole}`); + if (item.focalTarget !== undefined) lines.push(`- Focal target: \`${item.focalTarget}\``); + if (item.evidenceSource !== undefined) lines.push(`- Evidence source: ${item.evidenceSource}`); + if (item.onScreenTextBudget !== undefined) lines.push(`- On-screen text budget: ${item.onScreenTextBudget} words`); + if (item.transitionIntent !== undefined) lines.push(`- Transition intent: ${item.transitionIntent}`); + return lines.join('\n'); + }).join('\n\n') ?? 'No section-specific treatments.'; return `---\nformat: ${brief.format}\nid: ${brief.id}\ncanvas: ${brief.canvas.width}x${brief.canvas.height}\n---\n\n# Visual Design Brief: ${brief.title}\n\n## Direction\n\n${brief.direction}\n\n## Canvas\n\n- Size: ${brief.canvas.width} x ${brief.canvas.height}\n- Background: \`${brief.canvas.background}\`\n- Safe area: ${brief.canvas.safeAreaPx ?? 'not specified'} px\n\n## Palette\n\n| Role | Value | Use |\n|---|---|---|\n${palette}\n\n## Typography\n\n| Role | Family | Size | Weight | Use |\n|---|---|---:|---:|---|\n${typography}\n\n## Layout Rules\n\n${bullets(brief.layoutRules)}\n\n## Motion Rules\n\n${bullets(brief.motionRules)}\n\n## Section Treatments\n\n${treatments}\n\n## Avoid\n\n${bullets(brief.avoid)}\n`; } diff --git a/packages/core/src/authoring/schema.ts b/packages/core/src/authoring/schema.ts index c43f75b..312878c 100644 --- a/packages/core/src/authoring/schema.ts +++ b/packages/core/src/authoring/schema.ts @@ -41,6 +41,15 @@ export type EditorialSectionPurpose = | 'correct-misconception' | 'summarize'; +export type EvidenceSource = + | 'authored' + | 'terminal-capture' + | 'browser-capture' + | 'recorded-media'; + +export type VisualRole = 'hook' | 'model' | 'mechanism' | 'evidence' | 'result'; +export type TransitionIntent = 'cut' | 'crossfade' | 'focus-transfer' | 'match-object'; + export interface EditorialPlan { format: typeof EDITORIAL_PLAN_FORMAT; id: string; @@ -52,6 +61,8 @@ export interface EditorialPlan { likelyMisconceptions?: string[]; }; thesis: string; + /** Opening promise or tension; narration-sized, not a required title card. */ + hook?: string; durationBudgetSec: number; /** Optional, composable structural guidance. It does not constrain executable IR. */ explanationStrategy?: EditorialExplanationStrategy; @@ -70,6 +81,7 @@ export interface EditorialPlan { purpose: EditorialSectionPurpose; conceptIds: string[]; expectedOutcome: string; + evidenceSource?: EvidenceSource; targetSeconds?: number; }>; } @@ -101,7 +113,13 @@ export interface VisualDesignBrief { sceneTreatments?: Array<{ sectionId: string; sceneIds: string[]; - visualForm: 'whiteboard' | 'code' | 'diagram' | 'terminal' | 'browser'; + visualForm: 'whiteboard' | 'code' | 'diagram' | 'infographic' | 'terminal' | 'browser' | 'manim'; + visualRole?: VisualRole; + focalTarget?: string; + evidenceSource?: EvidenceSource; + /** Maximum words visible as primary explanatory text at once. */ + onScreenTextBudget?: number; + transitionIntent?: TransitionIntent; composition: string; emphasis: string; }>; diff --git a/packages/core/src/authoring/validate.ts b/packages/core/src/authoring/validate.ts index e2fb215..0a2dd61 100644 --- a/packages/core/src/authoring/validate.ts +++ b/packages/core/src/authoring/validate.ts @@ -32,6 +32,9 @@ export function validateEditorialPlan(plan: EditorialPlan): AuthoringIssue[] { if (!Number.isFinite(plan.durationBudgetSec) || plan.durationBudgetSec <= 0) { issues.push({ severity: 'error', path: 'durationBudgetSec', code: 'invalid_duration_budget', message: 'Duration budget must be greater than zero.' }); } + if (plan.hook !== undefined && !plan.hook.trim()) { + issues.push({ severity: 'warning', path: 'hook', code: 'empty_hook', message: 'Omit hook or state a concrete opening promise.' }); + } const strategy = plan.explanationStrategy; if (strategy) { @@ -193,6 +196,17 @@ export function validateVisualDesignBrief(brief: VisualDesignBrief, plan?: Edito if (!sectionIds.has(treatment.sectionId)) { issues.push({ severity: 'error', path: 'sceneTreatments', code: 'unknown_section', message: `Visual treatment references unknown section "${treatment.sectionId}".` }); } + if (treatment.onScreenTextBudget !== undefined && (!Number.isInteger(treatment.onScreenTextBudget) || treatment.onScreenTextBudget <= 0)) { + issues.push({ severity: 'error', path: `sceneTreatments.${treatment.sectionId}.onScreenTextBudget`, code: 'invalid_text_budget', message: 'On-screen text budget must be a positive whole-word count.' }); + } + const usesProductionContract = treatment.visualRole !== undefined + || treatment.focalTarget !== undefined + || treatment.evidenceSource !== undefined + || treatment.onScreenTextBudget !== undefined + || treatment.transitionIntent !== undefined; + if (usesProductionContract && !treatment.focalTarget) { + issues.push({ severity: 'warning', path: `sceneTreatments.${treatment.sectionId}.focalTarget`, code: 'missing_focal_target', message: 'A production-contract treatment should declare one focal target.' }); + } } } return issues; diff --git a/packages/core/src/composition.tsx b/packages/core/src/composition.tsx index 5105e74..5f0cdbb 100644 --- a/packages/core/src/composition.tsx +++ b/packages/core/src/composition.tsx @@ -19,6 +19,7 @@ import { import { resolveCompositionDurationFrames } from './time'; import { getTransitionProgress, getTransitionStyle } from './transitions'; import { FpsProvider, SceneLocalFrameProvider } from './frame'; +import { useStyleProfile } from './style-profile-runtime'; export interface CompositionConfig { id: string; @@ -185,6 +186,7 @@ export const Scene: React.FC = ({ id, duration, children }) => { const { timeline, layout } = useComposition(); const frame = timeline.getCurrentFrame(); const registration = getSceneRegistration(layout, id); + const styleProfile = useStyleProfile(); if (!registration) return null; @@ -213,6 +215,13 @@ export const Scene: React.FC = ({ id, duration, children }) => { let transitionRole: 'outgoing' | 'incoming' | undefined; let transitionProgress: number | undefined; + if (registration && styleProfile && styleProfile.cameraPolicy !== 'static') { + const sceneProgress = Math.min(1, Math.max(0, (frame - registration.globalStart) / Math.max(1, registration.duration))); + const cameraScale = styleProfile?.cameraPolicy === 'evidence-follow' ? 0.02 : 0.008; + sceneStyle.transform = `scale(${1 + sceneProgress * cameraScale})`; + sceneStyle.transformOrigin = '50% 50%'; + } + if (activeTransition && (isOutgoing || isIncoming)) { const progress = getTransitionProgress( frame, @@ -220,17 +229,20 @@ export const Scene: React.FC = ({ id, duration, children }) => { activeTransition.duration ); transitionProgress = progress; - const style = getTransitionStyle(activeTransition.type, progress); + const effectiveTransition = styleProfile?.transitionPolicy === 'cut' ? undefined + : styleProfile?.transitionPolicy === 'focus-transfer' ? 'cinematic-zoom' + : styleProfile?.transitionPolicy === 'crossfade' ? 'fade' : activeTransition.type; + const style = effectiveTransition ? getTransitionStyle(effectiveTransition, progress) : undefined; if (isOutgoing) { transitionRole = 'outgoing'; - sceneStyle.opacity = style.outgoingOpacity; + sceneStyle.opacity = style?.outgoingOpacity ?? 0; sceneStyle.zIndex = 1; } if (isIncoming) { transitionRole = 'incoming'; - sceneStyle.opacity = style.incomingOpacity; - sceneStyle.transform = style.incomingTransform; - sceneStyle.clipPath = style.overlayClipPath; + sceneStyle.opacity = style?.incomingOpacity ?? 1; + sceneStyle.transform = style?.incomingTransform; + sceneStyle.clipPath = style?.overlayClipPath; sceneStyle.zIndex = 2; } } diff --git a/packages/core/src/direction/derive.ts b/packages/core/src/direction/derive.ts new file mode 100644 index 0000000..669908d --- /dev/null +++ b/packages/core/src/direction/derive.ts @@ -0,0 +1,58 @@ +import type { ExplainerDocument, SceneSpec } from '../explainer-document/schema'; +import { + DIRECTION_PLAN_FORMAT, + DIRECTION_PLAN_VERSION, + type DirectionCamera, + type DirectionPace, + type DirectionPlan, + type DirectionPurpose, + type DirectionSegment, +} from './schema'; + +function purposeFor(scene: SceneSpec, index: number, count: number): DirectionPurpose { + if (index === 0 && count > 1) return 'hook'; + if (index === count - 1 && count > 1) return 'summarize'; + if (scene.type === 'terminal' || scene.type === 'browser') return 'demonstrate'; + return index === 0 ? 'establish-model' : 'explain-mechanism'; +} + +function paceFor(targetCount: number, minHoldMs: number): DirectionPace { + if (minHoldMs >= 1000) return 'hold'; + if (targetCount > 1) return 'build'; + return 'steady'; +} + +function cameraFor(targetCount: number): DirectionCamera { + if (targetCount === 0) return 'overview'; + if (targetCount === 1) return 'follow-target'; + return 'focus-transfer'; +} + +/** Derive direction only from semantic ids already owned by the document. */ +export function deriveDirectionPlan(document: ExplainerDocument): DirectionPlan { + const segments = document.scenes.flatMap((scene, sceneIndex) => { + const beats = scene.explanation?.beats ?? []; + if (beats.length === 0) { + return [{ + id: `${scene.id}.overview`, sceneId: scene.id, + purpose: purposeFor(scene, sceneIndex, document.scenes.length), + pace: 'steady' as const, focus: 'overview' as const, + camera: 'overview' as const, transition: 'cut' as const, + }]; + } + return beats.map((beat, beatIndex) => { + const targetIds = [...new Set(beat.visuals.map((visual) => visual.targetId))]; + const minHoldMs = Math.max(0, ...beat.visuals.map((visual) => visual.minHoldMs ?? 0)); + return { + id: `${scene.id}.${beat.id}`, sceneId: scene.id, + purpose: purposeFor(scene, sceneIndex, document.scenes.length), + pace: paceFor(targetIds.length, minHoldMs), + focus: targetIds.length > 1 ? 'sequence' as const : 'target' as const, + focusSpec: { targetIds, beatId: beat.id, captureStepId: beat.evidence?.captureStepId }, + camera: cameraFor(targetIds.length), + transition: beatIndex === beats.length - 1 ? 'cut' as const : undefined, + }; + }); + }); + return { format: DIRECTION_PLAN_FORMAT, version: DIRECTION_PLAN_VERSION, id: `${document.id}.direction`, segments }; +} diff --git a/packages/core/src/direction/index.ts b/packages/core/src/direction/index.ts new file mode 100644 index 0000000..cf54287 --- /dev/null +++ b/packages/core/src/direction/index.ts @@ -0,0 +1,3 @@ +export * from './schema'; +export * from './validate'; +export * from './derive'; diff --git a/packages/core/src/direction/schema.ts b/packages/core/src/direction/schema.ts new file mode 100644 index 0000000..169c0ef --- /dev/null +++ b/packages/core/src/direction/schema.ts @@ -0,0 +1,74 @@ +import type { ExplainerDocument } from '../explainer-document/schema'; + +export const DIRECTION_PLAN_FORMAT = 'seqvio-direction-plan' as const; +export const DIRECTION_PLAN_VERSION = '1.0' as const; + +export type DirectionPurpose = + | 'hook' + | 'establish-model' + | 'explain-mechanism' + | 'demonstrate' + | 'summarize'; +export type DirectionPace = 'hold' | 'steady' | 'build' | 'resolve'; +export type DirectionFocus = 'overview' | 'target' | 'sequence' | 'result'; +export type DirectionCamera = 'overview' | 'follow-target' | 'focus-transfer' | 'hold'; +export type DirectionTransition = 'cut' | 'crossfade' | 'focus-transfer' | 'match-object'; + +export interface DirectionFocusSpec { + targetId?: string; + targetIds?: string[]; + beatId?: string; + captureStepId?: string; +} + +export interface DirectionSegment { + id: string; + sceneId: string; + purpose: DirectionPurpose; + pace?: DirectionPace; + focus?: DirectionFocus; + focusSpec?: DirectionFocusSpec; + camera?: DirectionCamera; + transition?: DirectionTransition; + transitionTargetId?: string; + transitionToTargetId?: string; +} + +export interface DirectionPlan { + format: typeof DIRECTION_PLAN_FORMAT; + version: typeof DIRECTION_PLAN_VERSION; + id: string; + segments: DirectionSegment[]; +} + +export interface DirectionPlanIssue { + severity: 'error' | 'warning'; + code: string; + path: string; + message: string; +} + +export interface CompiledDirectionPlan { + sceneActions: Array<{ + segmentId: string; + sceneId: string; + purpose: DirectionPurpose; + pace?: DirectionPace; + camera?: DirectionCamera; + transition?: DirectionTransition; + transitionTargetId?: string; + transitionToTargetId?: string; + }>; + attention: Array<{ + segmentId: string; + sceneId: string; + targetId: string; + sourceBeatId?: string; + sourceCaptureStepId?: string; + start: number; + duration: number; + }>; + timingHints: Array<{ segmentId: string; pace: DirectionPace; minHoldFrames: number }>; +} + +export type DirectionDocument = Pick; diff --git a/packages/core/src/direction/validate.ts b/packages/core/src/direction/validate.ts new file mode 100644 index 0000000..1a38c92 --- /dev/null +++ b/packages/core/src/direction/validate.ts @@ -0,0 +1,111 @@ +import type { SceneSpec } from '../explainer-document/schema'; +import { + DIRECTION_PLAN_FORMAT, + DIRECTION_PLAN_VERSION, + type DirectionDocument, + type DirectionPlan, + type DirectionPlanIssue, +} from './schema'; + +const PURPOSES = new Set(['hook', 'establish-model', 'explain-mechanism', 'demonstrate', 'summarize']); +const PACES = new Set(['hold', 'steady', 'build', 'resolve']); +const FOCUSES = new Set(['overview', 'target', 'sequence', 'result']); +const CAMERAS = new Set(['overview', 'follow-target', 'focus-transfer', 'hold']); +const TRANSITIONS = new Set(['cut', 'crossfade', 'focus-transfer', 'match-object']); + +function issue(issues: DirectionPlanIssue[], path: string, code: string, message: string, severity: 'error' | 'warning' = 'error') { + issues.push({ severity, path, code, message }); +} + +function sceneTargets(scene: SceneSpec): Set { + const ids = new Set(); + const add = (value: unknown) => { if (typeof value === 'string' && value) ids.add(value); }; + add(scene.id); + if (scene.type === 'diagram') { scene.nodes.forEach((n) => add(n.id)); scene.edges.forEach((e) => add(e.id)); scene.steps.forEach((s) => add(s.id)); } + if (scene.type === 'whiteboard') scene.elements.forEach((e) => add(e.id)); + if (scene.type === 'code') scene.steps.forEach((s) => add(s.id)); + if (scene.type === 'infographic') [...(scene.metrics ?? []), ...(scene.comparisons ?? []), ...(scene.process ?? []), ...(scene.timeline ?? []), ...(scene.relationshipNodes ?? []), ...(scene.relationships ?? []), ...(scene.charts ?? [])].forEach((item) => add(item.id)); + if (scene.type === 'terminal' || scene.type === 'browser') (scene.steps ?? []).forEach((s) => add(s.id)); + if (scene.type === 'manim') (scene.markers ?? []).forEach((marker) => { add(marker.id); add(marker.targetId); }); + return ids; +} + +export function validateDirectionPlan(input: unknown, document?: DirectionDocument): DirectionPlanIssue[] { + const issues: DirectionPlanIssue[] = []; + if (!input || typeof input !== 'object') { issue(issues, '$', 'expected_plan_object', 'DirectionPlan must be an object'); return issues; } + const plan = input as Partial; + if (plan.format !== DIRECTION_PLAN_FORMAT) issue(issues, 'format', 'unsupported_direction_format', `format must be "${DIRECTION_PLAN_FORMAT}"`); + if (plan.version !== DIRECTION_PLAN_VERSION) issue(issues, 'version', 'unsupported_direction_version', `version must be "${DIRECTION_PLAN_VERSION}"`); + if (typeof plan.id !== 'string' || !plan.id) issue(issues, 'id', 'missing_direction_id', 'id must be a non-empty string'); + if (!Array.isArray(plan.segments) || plan.segments.length === 0) { issue(issues, 'segments', 'missing_direction_segments', 'segments must be a non-empty array'); return issues; } + + const sceneMap = new Map>((document?.scenes ?? []).map((scene) => [scene.id, sceneTargets(scene)])); + const beatMap = new Map((document?.scenes ?? []).map((scene) => [scene.id, new Set((scene.explanation?.beats ?? []).map((beat) => beat.id))])); + const captureStepMap = new Map((document?.scenes ?? []).map((scene) => [ + scene.id, + new Set(scene.type === 'terminal' || scene.type === 'browser' ? (scene.steps ?? []).map((step) => step.id) : []), + ])); + const segmentIds = new Set(); + plan.segments.forEach((segment, index) => { + const path = `segments[${index}]`; + if (!segment || typeof segment !== 'object') { issue(issues, path, 'invalid_direction_segment', `${path} must be an object`); return; } + if (typeof segment.id !== 'string' || !segment.id) issue(issues, `${path}.id`, 'missing_direction_segment_id', 'segment id is required'); + else if (segmentIds.has(segment.id)) issue(issues, `${path}.id`, 'duplicate_direction_segment_id', `Duplicate segment id "${segment.id}"`); else segmentIds.add(segment.id); + if (typeof segment.sceneId !== 'string' || !segment.sceneId) issue(issues, `${path}.sceneId`, 'missing_direction_scene', 'sceneId is required'); + const targets = sceneMap.get(segment.sceneId ?? ''); + if (document && !targets) issue(issues, `${path}.sceneId`, 'unknown_direction_scene', `Unknown scene id "${segment.sceneId}"`); + if (!PURPOSES.has(String(segment.purpose))) issue(issues, `${path}.purpose`, 'unsupported_direction_purpose', `Unsupported purpose "${segment.purpose}"`); + if (segment.pace !== undefined && !PACES.has(segment.pace)) issue(issues, `${path}.pace`, 'unsupported_direction_pace', `Unsupported pace "${segment.pace}"`); + if (segment.focus !== undefined && !FOCUSES.has(segment.focus)) issue(issues, `${path}.focus`, 'unsupported_direction_focus', `Unsupported focus "${segment.focus}"`); + if (segment.camera !== undefined && !CAMERAS.has(segment.camera)) issue(issues, `${path}.camera`, 'unsupported_direction_camera', `Unsupported camera "${segment.camera}"`); + if (segment.transition !== undefined && !TRANSITIONS.has(segment.transition)) issue(issues, `${path}.transition`, 'unsupported_direction_transition', `Unsupported transition "${segment.transition}"`); + const focus = segment.focusSpec; + const focusIds = [...(focus?.targetIds ?? []), ...(focus?.targetId ? [focus.targetId] : [])]; + if (segment.focus === 'overview' && focusIds.length > 0) issue(issues, `${path}.focusSpec`, 'conflicting_direction_focus', 'Overview focus cannot also name focal targets'); + if (segment.focus === 'target' && focusIds.length !== 1) issue(issues, `${path}.focusSpec`, 'conflicting_direction_focus', 'Target focus requires exactly one focal target'); + if (segment.focus === 'sequence' && focusIds.length < 2) issue(issues, `${path}.focusSpec`, 'conflicting_direction_focus', 'Sequence focus requires at least two focal targets'); + if (segment.camera === 'overview' && focusIds.length > 0) issue(issues, `${path}.camera`, 'conflicting_direction_camera', 'Overview camera cannot follow focal targets'); + if (targets) focusIds.forEach((targetId) => { if (!targets.has(targetId)) issue(issues, `${path}.focusSpec`, 'unknown_direction_target', `Unknown target id "${targetId}" in scene "${segment.sceneId}"`); }); + if (targets && segment.transitionTargetId && !targets.has(segment.transitionTargetId)) issue(issues, `${path}.transitionTargetId`, 'unknown_direction_transition_target', `Unknown transition target id "${segment.transitionTargetId}" in scene "${segment.sceneId}"`); + if (focus?.beatId && !beatMap.get(segment.sceneId ?? '')?.has(focus.beatId)) issue(issues, `${path}.focusSpec.beatId`, 'unknown_direction_beat', `Unknown ExplanationBeat id "${focus.beatId}" in scene "${segment.sceneId}"`); + if (focus?.captureStepId && !captureStepMap.get(segment.sceneId ?? '')?.has(focus.captureStepId)) issue(issues, `${path}.focusSpec.captureStepId`, 'unknown_direction_capture_step', `Unknown capture step id "${focus.captureStepId}" in scene "${segment.sceneId}"`); + if (segment.transition && segment.transition !== 'cut') { + const next = plan.segments?.[index + 1]; + const nextFocusIds = [...(next?.focusSpec?.targetIds ?? []), ...(next?.focusSpec?.targetId ? [next.focusSpec.targetId] : [])]; + const sourceTargetId = segment.transitionTargetId ?? focusIds.at(-1); + const destinationTargetId = segment.transitionToTargetId ?? nextFocusIds[0]; + if (!sourceTargetId) issue(issues, `${path}.transitionTargetId`, 'transition_without_shared_target', 'Non-cut transitions require a source target'); + if (!next) issue(issues, `${path}.transition`, 'transition_without_destination', 'Non-cut transitions require a following segment'); + else if (!destinationTargetId) issue(issues, `${path}.transitionToTargetId`, 'transition_without_destination_target', 'Non-cut transitions require a destination target'); + const nextTargets = next ? sceneMap.get(next.sceneId ?? '') : undefined; + if (nextTargets && destinationTargetId && !nextTargets.has(destinationTargetId)) issue(issues, `${path}.transitionToTargetId`, 'unknown_direction_transition_destination', `Unknown destination target id "${destinationTargetId}" in scene "${next?.sceneId}"`); + } + }); + return issues; +} + +export function compileDirectionPlan(plan: DirectionPlan): import('./schema').CompiledDirectionPlan { + const sceneActions = plan.segments.map((segment, index) => { + const targets = [...(segment.focusSpec?.targetIds ?? []), ...(segment.focusSpec?.targetId ? [segment.focusSpec.targetId] : [])]; + const next = plan.segments[index + 1]; + const nextTargets = [...(next?.focusSpec?.targetIds ?? []), ...(next?.focusSpec?.targetId ? [next.focusSpec.targetId] : [])]; + return { + segmentId: segment.id, + sceneId: segment.sceneId, + purpose: segment.purpose, + pace: segment.pace, + camera: segment.camera, + transition: segment.transition, + transitionTargetId: segment.transitionTargetId ?? (segment.transition !== 'cut' ? targets.at(-1) : undefined), + transitionToTargetId: segment.transitionToTargetId ?? (segment.transition !== 'cut' ? nextTargets[0] : undefined), + }; + }); + const attention: import('./schema').CompiledDirectionPlan['attention'] = []; + const timingHints: import('./schema').CompiledDirectionPlan['timingHints'] = []; + plan.segments.forEach((segment, index) => { + const targets = [...(segment.focusSpec?.targetIds ?? []), ...(segment.focusSpec?.targetId ? [segment.focusSpec.targetId] : [])]; + targets.forEach((targetId) => attention.push({ segmentId: segment.id, sceneId: segment.sceneId, targetId, sourceBeatId: segment.focusSpec?.beatId, sourceCaptureStepId: segment.focusSpec?.captureStepId, start: index, duration: 1 })); + if (segment.pace) timingHints.push({ segmentId: segment.id, pace: segment.pace, minHoldFrames: segment.pace === 'hold' ? 30 : segment.pace === 'build' ? 12 : 18 }); + }); + return { sceneActions, attention, timingHints }; +} diff --git a/packages/core/src/director/index.ts b/packages/core/src/director/index.ts new file mode 100644 index 0000000..fff53bd --- /dev/null +++ b/packages/core/src/director/index.ts @@ -0,0 +1,95 @@ +import type { AttentionSequenceItem } from '../annotation'; +import { validateAttentionSequence } from '../annotation'; +import type { ExplainerDocument } from '../explainer-document'; +import { deriveDirectionPlan, type DirectionPlan, validateDirectionPlan } from '../direction'; +import type { MotionGrammar } from '../motion-grammar'; +import { validateMotionGrammar } from '../motion-grammar'; + +export const DIRECTOR_TASK_FORMAT = 'seqvio-director-task' as const; +export const DIRECTOR_TASK_VERSION = '1.0' as const; + +export type DirectorTaskMode = 'generate' | 'repair'; +export type DirectorArtifactKind = 'directionPlan' | 'attentionSequence' | 'motionGrammar'; + +export interface DirectorArtifactBundle { + directionPlan?: DirectionPlan; + attentionSequence?: AttentionSequenceItem[]; + motionGrammar?: MotionGrammar; +} + +export interface DirectorDiagnostic { + artifact: DirectorArtifactKind; + code: string; + path: string; + message: string; + suggestion: string; +} + +export interface DirectorTask { + format: typeof DIRECTOR_TASK_FORMAT; + version: typeof DIRECTOR_TASK_VERSION; + id: string; + mode: DirectorTaskMode; + approvedDocumentId: string; + requestedArtifacts: DirectorArtifactKind[]; + constraints: string[]; + baseline: DirectorArtifactBundle; + candidate?: DirectorArtifactBundle; + diagnostics: DirectorDiagnostic[]; + outputContract: { + format: 'seqvio-director-result'; + version: '1.0'; + artifacts: DirectorArtifactKind[]; + }; +} + +const suggestionFor = (code: string): string => { + if (code.includes('target')) return 'Reference a stable target id that exists in the owning scene.'; + if (code.includes('transition')) return 'Pair the source and destination targets and keep a following segment.'; + if (code.includes('focus') || code.includes('camera')) return 'Use one compatible focus path and camera intent.'; + if (code.includes('trace')) return 'Provide an ordered path with at least two stable target ids.'; + if (code.includes('compare')) return 'Provide both the primary and related comparison targets.'; + if (code.includes('clear') || code.includes('handoff')) return 'Add an explicit clear or a valid semantic handoff target.'; + return 'Repair the versioned artifact while preserving approved document ids and evidence order.'; +}; + +export function validateDirectorArtifacts(bundle: DirectorArtifactBundle, document: ExplainerDocument): DirectorDiagnostic[] { + const diagnostics: DirectorDiagnostic[] = []; + for (const item of bundle.directionPlan ? validateDirectionPlan(bundle.directionPlan, document) : []) { + diagnostics.push({ artifact: 'directionPlan', code: item.code, path: item.path, message: item.message, suggestion: suggestionFor(item.code) }); + } + for (const item of bundle.attentionSequence ? validateAttentionSequence(bundle.attentionSequence) : []) { + diagnostics.push({ artifact: 'attentionSequence', code: item.code, path: item.itemId, message: item.message, suggestion: suggestionFor(item.code) }); + } + for (const item of bundle.motionGrammar ? validateMotionGrammar(bundle.motionGrammar, document) : []) { + diagnostics.push({ artifact: 'motionGrammar', code: item.code, path: item.path, message: item.message, suggestion: suggestionFor(item.code) }); + } + return diagnostics; +} + +export function createDirectorTask(document: ExplainerDocument, mode: DirectorTaskMode, candidate?: DirectorArtifactBundle): DirectorTask { + const requestedArtifacts: DirectorArtifactKind[] = ['directionPlan', 'attentionSequence', 'motionGrammar']; + const baselineAttention = document.scenes.flatMap((scene) => scene.type === 'infographic' ? (scene.attention ?? []) : []); + const baseline: DirectorArtifactBundle = { + directionPlan: deriveDirectionPlan(document), + attentionSequence: baselineAttention, + }; + return { + format: DIRECTOR_TASK_FORMAT, + version: DIRECTOR_TASK_VERSION, + id: `${document.id}.director.${mode}`, + mode, + approvedDocumentId: document.id, + requestedArtifacts, + constraints: [ + 'Preserve scene ids, target ids, ExplanationBeat ids, and evidence order.', + 'Express intent as semantic artifacts; do not emit renderer code or pixel-level animation.', + 'Keep every action deterministic at an arbitrary seek frame.', + 'Return only the versioned director result contract for local validation.', + ], + baseline, + candidate, + diagnostics: mode === 'repair' && candidate ? validateDirectorArtifacts(candidate, document) : [], + outputContract: { format: 'seqvio-director-result', version: '1.0', artifacts: requestedArtifacts }, + }; +} diff --git a/packages/core/src/explainer-document/capabilities.ts b/packages/core/src/explainer-document/capabilities.ts index 9a5206f..30dc39f 100644 --- a/packages/core/src/explainer-document/capabilities.ts +++ b/packages/core/src/explainer-document/capabilities.ts @@ -39,6 +39,12 @@ export const SCENE_CAPABILITIES = { authoringSummary: 'nodes, edges, steps = reveal | connect | trace | emphasize', qaRules: ['visual-layout', 'pacing', 'audio-timeline', 'explanation-beats'], }, + infographic: { + type: 'infographic', schemaVersion: '1.0', compiler: 'complete', + requiredPackage: '@seqvio/technical', lifecycle: 'experimental', agentAuthoring: true, + authoringSummary: 'metrics, comparisons, process, timeline, and relationship primitives', + qaRules: ['visual-layout', 'text-density', 'focal-target', 'pacing', 'audio-timeline', 'explanation-beats'], + }, terminal: { type: 'terminal', schemaVersion: '1.0', compiler: 'complete', requiredPackage: '@seqvio/technical', lifecycle: 'public', agentAuthoring: false, @@ -51,6 +57,12 @@ export const SCENE_CAPABILITIES = { authoringSummary: 'capture-derived video, cursor, focus, click, and recorded steps', qaRules: ['visual-layout', 'capture-state', 'capture-media', 'pacing', 'audio-timeline', 'explanation-beats'], }, + manim: { + type: 'manim', schemaVersion: '1.0', compiler: 'complete', + requiredPackage: '@seqvio/manim-adapter', lifecycle: 'experimental', agentAuthoring: true, + authoringSummary: 'pre-rendered mathematical animation, render manifest, and named markers', + qaRules: ['visual-layout', 'external-media', 'marker-alignment', 'pacing', 'audio-timeline', 'explanation-beats'], + }, } as const satisfies Record; export type SceneType = keyof typeof SCENE_CAPABILITIES; diff --git a/packages/core/src/explainer-document/compile.ts b/packages/core/src/explainer-document/compile.ts index d57322a..f11b05a 100644 --- a/packages/core/src/explainer-document/compile.ts +++ b/packages/core/src/explainer-document/compile.ts @@ -20,6 +20,8 @@ import { type CodeSceneSpec, type ExplainerDocument, type DiagramSceneSpec, + type InfographicSceneSpec, + type ManimSceneSpec, type SceneSpec, } from './schema'; import { sceneDurationFrames } from './timeline'; @@ -29,6 +31,12 @@ import { resolvePacingProfile, resolveScenePacing, } from '../pacing'; +import { + compileDirectionPlan, + deriveDirectionPlan, + type CompiledDirectionPlan, + type DirectionPlan, +} from '../direction'; function resolved(doc: ExplainerDocument) { return { @@ -137,6 +145,31 @@ function compileDiagramScene(scene: DiagramSceneSpec, componentName: string): st }`; } +function compileInfographicScene(scene: InfographicSceneSpec, componentName: string): string { + const annotations = serializeAnnotations(scene.annotations); + return `function ${componentName}() { + return ( + + + + ); +}`; +} + function compileTerminalScene(scene: Extract, componentName: string): string { const annotations = serializeAnnotations(scene.annotations); const legacyEvents = (scene.commands ?? []).map((command, index) => ({ @@ -217,6 +250,25 @@ function compileBrowserScene( }`; } +function compileManimScene(scene: ManimSceneSpec, componentName: string): string { + const annotations = serializeAnnotations(scene.annotations); + return `function ${componentName}() { + return ( + + + + ); +}`; +} + function unsupportedSceneType(scene: never): never { const type = (scene as { type?: unknown }).type; throw new Error(`Unsupported ExplainerDocument scene type: ${String(type)}`); @@ -234,10 +286,14 @@ function compileSceneComponent( return compileCodeScene(scene, componentName); case 'diagram': return compileDiagramScene(scene, componentName); + case 'infographic': + return compileInfographicScene(scene, componentName); case 'terminal': return compileTerminalScene(scene, componentName); case 'browser': return compileBrowserScene(scene, componentName); + case 'manim': + return compileManimScene(scene, componentName); } return unsupportedSceneType(scene); @@ -249,6 +305,8 @@ function sceneDurationFramesForCompile(scene: SceneSpec, fps: number): number { export interface CompileCompositionResult { code: string; + directionPlan: DirectionPlan; + compiledDirection: CompiledDirectionPlan; } export function compileExplainerDocumentToTsx( @@ -256,6 +314,8 @@ export function compileExplainerDocumentToTsx( ): CompileCompositionResult { const pacingProfile = resolvePacingProfile(doc.pacingProfile); const pacedDoc = resolveCompositionPacing(doc, pacingProfile.policy); + const directionPlan = deriveDirectionPlan(pacedDoc); + const compiledDirection = compileDirectionPlan(directionPlan); const r = resolved(pacedDoc); const sceneNames = pacedDoc.scenes.map((scene, index) => sceneComponentName(scene.id, index) @@ -363,6 +423,8 @@ export function compileExplainerDocumentToTsx( AnnotationTarget, CodeWalkthrough, ArchitectureDiagram, + InfographicScene, + ManimClip, TerminalXtermDemo, } from '@seqvio/technical';` : ''; @@ -383,7 +445,7 @@ const STYLE = getSeqvioStylePreset(STYLE_ID) ?? { const code = `// AUTO-GENERATED from a Seqvio ExplainerDocument. Safe to edit by hand. import React from 'react'; import type { RenderableMeta } from '@seqvio/core'; -import { VideoComposition, Scene, Transition } from '@seqvio/core'; +import { StyleProfileProvider, VideoComposition, Scene, Transition } from '@seqvio/core'; ${whiteboardImports} ${technicalImports} ${productDemoImports} @@ -391,21 +453,24 @@ ${productDemoImports} const W = ${r.width}; const H = ${r.height}; const FPS = ${r.fps}; +const STYLE_PROFILE = ${JSON.stringify(pacedDoc.styleProfile, null, 2)}; ${styleBlock} ${sceneFns} export default function ${pascalId(r.id)}() { return ( + ${sceneTree} + ); } @@ -417,6 +482,7 @@ ${ width: W, height: H, pacing: { profile: ${JSON.stringify(pacingProfile.id)}, highlights: ${JSON.stringify(pacingHighlights, null, 2)} }, + direction: ${JSON.stringify(compiledDirection, null, 2)}, audio: { fps: FPS, lockToAudio: ${r.lockToAudio}, @@ -432,9 +498,10 @@ ${ width: W, height: H, pacing: { profile: ${JSON.stringify(pacingProfile.id)}, highlights: ${JSON.stringify(pacingHighlights, null, 2)} }, + direction: ${JSON.stringify(compiledDirection, null, 2)}, };` } `; - return { code }; + return { code, directionPlan, compiledDirection }; } diff --git a/packages/core/src/explainer-document/schema.ts b/packages/core/src/explainer-document/schema.ts index 8e774bb..3213fcc 100644 --- a/packages/core/src/explainer-document/schema.ts +++ b/packages/core/src/explainer-document/schema.ts @@ -23,7 +23,13 @@ export type AnnotationKind = | 'circle' | 'box' | 'underline' - | 'spotlight'; + | 'spotlight' + | 'focus-ring' + | 'callout' + | 'bracket' + | 'connector' + | 'region-shade' + | 'guided-path'; export const ANNOTATION_KINDS: AnnotationKind[] = [ 'arrow', @@ -31,13 +37,27 @@ export const ANNOTATION_KINDS: AnnotationKind[] = [ 'box', 'underline', 'spotlight', + 'focus-ring', + 'callout', + 'bracket', + 'connector', + 'region-shade', + 'guided-path', ]; export interface AddressableElement { id: string; } -export type VisualBeatActionKind = 'reveal' | 'highlight' | 'focus' | 'annotate'; +export type VisualBeatActionKind = + | 'reveal' + | 'highlight' + | 'focus' + | 'annotate' + | 'compare' + | 'trace' + | 'emphasize' + | 'transform'; export interface ExplanationCueSpec { id: string; @@ -55,6 +75,8 @@ export interface ExplanationBeatAnchorSpec { export interface VisualBeatAction { targetId: string; action: VisualBeatActionKind; + relatedTargetId?: string; + pathTargetIds?: string[]; /** Negative values reveal before speech; positive values delay the action. */ offsetMs?: number; minHoldMs?: number; @@ -76,10 +98,13 @@ export interface SceneExplanationSpec { export interface AnnotationSpec extends AddressableElement { targetId: string; + toTargetId?: string; + pathTargetIds?: string[]; kind: AnnotationKind; start: number; duration: number; label?: string; + priority?: number; } export interface LineRange { @@ -159,6 +184,115 @@ export interface DiagramSceneSpec { annotations?: AnnotationSpec[]; } +export interface InfographicMetricSpec extends AddressableElement { + label: string; + value: string; + detail?: string; + color?: string; + at?: number; +} + +export interface InfographicComparisonSpec extends AddressableElement { + label: string; + before: number; + after: number; + beforeLabel?: string; + afterLabel?: string; + at?: number; +} + +export interface InfographicProcessStepSpec extends AddressableElement { + label: string; + detail?: string; + at?: number; +} + +export interface InfographicTimelineEventSpec extends AddressableElement { + label: string; + detail?: string; + at?: number; +} + +export interface InfographicRelationshipNodeSpec extends AddressableElement { + label: string; + x: number; + y: number; +} + +export interface InfographicRelationshipSpec extends AddressableElement { + from: string; + to: string; + label?: string; + at?: number; +} + +export interface InfographicChartPointSpec { + x: string; + y: number; +} + +export interface InfographicChartSeriesSpec extends AddressableElement { + label: string; + color?: string; + points: InfographicChartPointSpec[]; +} + +export interface InfographicChartAxisSpec { + label?: string; + min?: number; + max?: number; + ticks?: number; +} + +export interface InfographicChartSpec extends AddressableElement { + title: string; + kind: 'bar' | 'line'; + series: InfographicChartSeriesSpec[]; + xAxis?: InfographicChartAxisSpec; + yAxis?: InfographicChartAxisSpec; + legend?: 'none' | 'top' | 'bottom'; + unit?: string; + sourceLabel?: string; + at?: number; +} + +export interface AttentionSequenceSpec extends AddressableElement { + sceneId?: string; + targetId: string; + toTargetId?: string; + pathTargetIds?: string[]; + kind: AnnotationKind; + start: number; + duration: number; + label?: string; + handoffTo?: string; + minHoldFrames?: number; + sourceBeatId?: string; + persistence?: 'timed' | 'until-handoff' | 'until-clear'; + clearAt?: number; + handoffToSceneId?: string; + priority?: number; +} + +export interface InfographicSceneSpec { + type: 'infographic'; + id: string; + title?: string; + density?: 'auto' | 'standard' | 'reduced'; + metrics?: InfographicMetricSpec[]; + comparisons?: InfographicComparisonSpec[]; + process?: InfographicProcessStepSpec[]; + timeline?: InfographicTimelineEventSpec[]; + relationshipNodes?: InfographicRelationshipNodeSpec[]; + relationships?: InfographicRelationshipSpec[]; + charts?: InfographicChartSpec[]; + attention?: AttentionSequenceSpec[]; + narration?: string; + explanation?: SceneExplanationSpec; + duration?: number; + annotations?: AnnotationSpec[]; +} + /** Grid cell contract mirroring @seqvio/technical's runtime TerminalGridCell. */ export interface TerminalGridCellSpec { x: number; @@ -276,12 +410,37 @@ export interface BrowserSceneSpec { annotations?: AnnotationSpec[]; } +export interface ManimMarkerSpec extends AddressableElement { + frame: number; + targetId?: string; + beatId?: string; +} + +/** Pre-rendered mathematical animation produced by @seqvio/manim-adapter. */ +export interface ManimSceneSpec { + type: 'manim'; + id: string; + sourceVideo: string; + sourceManifest?: string; + markers?: ManimMarkerSpec[]; + mediaWidth?: number; + mediaHeight?: number; + mediaFps?: number; + fit?: 'contain' | 'cover' | 'fill'; + narration?: string; + explanation?: SceneExplanationSpec; + duration?: number; + annotations?: AnnotationSpec[]; +} + export type SceneSpec = | WhiteboardSceneSpec | CodeSceneSpec | DiagramSceneSpec + | InfographicSceneSpec | TerminalSceneSpec - | BrowserSceneSpec; + | BrowserSceneSpec + | ManimSceneSpec; export interface ChapterSpec { id: string; @@ -302,6 +461,8 @@ export interface ExplainerDocument { transitionDuration?: number; /** Versioned pacing policy used by authoring, timing resolution, and QA. */ pacingProfile?: string; + /** Optional visual profile; semantic ids, evidence, and timing remain unchanged. */ + styleProfile?: import('../style-profile').StyleProfile; chapters?: ChapterSpec[]; scenes: SceneSpec[]; /** Document-level annotations that may target any scene element id. */ diff --git a/packages/core/src/explainer-document/validate.ts b/packages/core/src/explainer-document/validate.ts index 8ab85dd..65ae2f6 100644 --- a/packages/core/src/explainer-document/validate.ts +++ b/packages/core/src/explainer-document/validate.ts @@ -12,6 +12,8 @@ import { EXPLAINER_DOCUMENT_SCHEMA_VERSION, type ExplainerDocument, type DiagramSceneSpec, + type InfographicSceneSpec, + type ManimSceneSpec, type SceneExplanationSpec, type SceneSpec, type WhiteboardSceneSpec, @@ -19,6 +21,7 @@ import { import { SCENE_TYPES, isSceneType } from './capabilities'; import { isPacingProfileId } from '../pacing'; import { findNarrationAnchorMatches, normalizeNarrationText } from '../narration-anchor'; +import { validateStyleProfile } from '../style-profile'; export type CompositionIssue = StoryboardIssue; @@ -292,6 +295,71 @@ function validateDiagramScene( } } +function validateInfographicScene( + scene: InfographicSceneSpec, + scenePath: string, + issues: CompositionIssue[], +): void { + if (scene.density !== undefined && !['auto', 'standard', 'reduced'].includes(scene.density)) { + issue(issues, { severity: 'error', path: `${scenePath}.density`, code: 'unsupported_infographic_density', message: `${scenePath}.density must be auto, standard, or reduced`, repairable: true }); + } + const collections = [ + ['metrics', scene.metrics], + ['comparisons', scene.comparisons], + ['process', scene.process], + ['timeline', scene.timeline], + ['relationshipNodes', scene.relationshipNodes], + ['relationships', scene.relationships], + ['charts', scene.charts], + ] as const; + const ids = new Set(); + for (const [name, items] of collections) { + if (!items) continue; + items.forEach((item, index) => { + const itemPath = `${scenePath}.${name}[${index}]`; + if (typeof item.id !== 'string' || item.id.length === 0) { + issue(issues, { severity: 'error', path: `${itemPath}.id`, code: 'missing_infographic_item_id', message: `${itemPath}.id must be a non-empty string`, repairable: true }); + } else if (ids.has(item.id)) { + issue(issues, { severity: 'error', path: `${itemPath}.id`, code: 'duplicate_infographic_item_id', message: `Duplicate infographic item id "${item.id}"`, repairable: true }); + } else ids.add(item.id); + if (name !== 'relationships' && name !== 'charts' && (!('label' in item) || typeof item.label !== 'string' || item.label.length === 0)) { + issue(issues, { severity: 'error', path: `${itemPath}.label`, code: 'missing_infographic_item_label', message: `${itemPath}.label must be a non-empty string`, repairable: true }); + } + if (name === 'charts' && (!('title' in item) || typeof item.title !== 'string' || item.title.length === 0)) { + issue(issues, { severity: 'error', path: `${itemPath}.title`, code: 'missing_infographic_chart_title', message: `${itemPath}.title must be a non-empty string`, repairable: true }); + } + if ('at' in item && item.at !== undefined && (typeof item.at !== 'number' || item.at < 0)) { + issue(issues, { severity: 'error', path: `${itemPath}.at`, code: 'invalid_infographic_item_time', message: `${itemPath}.at must be a non-negative number`, repairable: true }); + } + }); + } + const nodeIds = new Set((scene.relationshipNodes ?? []).map((node) => node.id)); + (scene.relationships ?? []).forEach((relationship, index) => { + if (!nodeIds.has(relationship.from)) issue(issues, { severity: 'error', path: `${scenePath}.relationships[${index}].from`, code: 'invalid_infographic_relationship_from', message: 'Relationship source must reference a relationship node', repairable: true }); + if (!nodeIds.has(relationship.to)) issue(issues, { severity: 'error', path: `${scenePath}.relationships[${index}].to`, code: 'invalid_infographic_relationship_to', message: 'Relationship target must reference a relationship node', repairable: true }); + }); + (scene.charts ?? []).forEach((chart, chartIndex) => { + const chartPath = `${scenePath}.charts[${chartIndex}]`; + if (chart.kind !== 'bar' && chart.kind !== 'line') issue(issues, { severity: 'error', path: `${chartPath}.kind`, code: 'invalid_infographic_chart_kind', message: 'Chart kind must be bar or line', repairable: true }); + if (!Array.isArray(chart.series) || chart.series.length === 0) { + issue(issues, { severity: 'error', path: `${chartPath}.series`, code: 'missing_infographic_chart_series', message: 'Chart must contain at least one data series', repairable: true }); + } + const seriesIds = new Set(); + (chart.series ?? []).forEach((series, seriesIndex) => { + const seriesPath = `${chartPath}.series[${seriesIndex}]`; + if (!series.id || seriesIds.has(series.id)) issue(issues, { severity: 'error', path: `${seriesPath}.id`, code: series.id ? 'duplicate_infographic_series_id' : 'missing_infographic_series_id', message: 'Chart series ids must be non-empty and unique within the chart', repairable: true }); + else seriesIds.add(series.id); + if (!series.label) issue(issues, { severity: 'error', path: `${seriesPath}.label`, code: 'missing_infographic_series_label', message: 'Chart series label is required', repairable: true }); + if (!Array.isArray(series.points) || series.points.length === 0 || series.points.some((point) => typeof point.x !== 'string' || !Number.isFinite(point.y))) { + issue(issues, { severity: 'error', path: `${seriesPath}.points`, code: 'invalid_infographic_chart_points', message: 'Chart points require a string x value and finite numeric y value', repairable: true }); + } + }); + if (chart.yAxis?.min !== undefined && chart.yAxis?.max !== undefined && chart.yAxis.min >= chart.yAxis.max) { + issue(issues, { severity: 'error', path: `${chartPath}.yAxis`, code: 'invalid_infographic_axis_domain', message: 'yAxis min must be less than max', repairable: true }); + } + }); +} + function validatePlaceholderScene( scene: SceneSpec, scenePath: string, @@ -431,6 +499,22 @@ function validatePlaceholderScene( } } +function validateManimScene(scene: ManimSceneSpec, scenePath: string, issues: CompositionIssue[]): void { + if (typeof scene.sourceVideo !== 'string' || !scene.sourceVideo.trim()) { + issue(issues, { severity: 'error', path: `${scenePath}.sourceVideo`, code: 'missing_manim_media', message: `${scenePath}.sourceVideo must reference rendered media`, repairable: true }); + } + const markerIds = new Set(); + const beatIds = new Set((scene.explanation?.beats ?? []).map((beat) => beat.id)); + (scene.markers ?? []).forEach((marker, index) => { + const path = `${scenePath}.markers[${index}]`; + if (!marker.id) issue(issues, { severity: 'error', path: `${path}.id`, code: 'missing_manim_marker_id', message: `${path}.id is required`, repairable: true }); + else if (markerIds.has(marker.id)) issue(issues, { severity: 'error', path: `${path}.id`, code: 'duplicate_manim_marker_id', message: `Duplicate Manim marker id "${marker.id}"`, repairable: true }); + else markerIds.add(marker.id); + if (!Number.isInteger(marker.frame) || marker.frame < 0) issue(issues, { severity: 'error', path: `${path}.frame`, code: 'invalid_manim_marker_frame', message: `${path}.frame must be a non-negative integer`, repairable: true }); + if (marker.beatId && !beatIds.has(marker.beatId)) issue(issues, { severity: 'error', path: `${path}.beatId`, code: 'unknown_manim_marker_beat', message: `${path}.beatId must reference an ExplanationBeat in the same scene`, repairable: true }); + }); +} + function validateAnnotation( annotation: unknown, path: string, @@ -464,6 +548,18 @@ function validateAnnotation( repairable: true, }); } + if (annotation.kind === 'connector' && (typeof annotation.toTargetId !== 'string' || annotation.toTargetId.length === 0)) { + issue(issues, { + severity: 'error', path: `${path}.toTargetId`, code: 'missing_connector_target', + message: `${path}.toTargetId must be a non-empty string for connector annotations`, repairable: true, + }); + } + if (annotation.kind === 'guided-path' && (!Array.isArray(annotation.pathTargetIds) || annotation.pathTargetIds.length < 2)) { + issue(issues, { + severity: 'error', path: `${path}.pathTargetIds`, code: 'invalid_guided_path_targets', + message: `${path}.pathTargetIds must contain at least two target ids`, repairable: true, + }); + } if ( typeof annotation.kind !== 'string' || !ANNOTATION_KINDS.includes(annotation.kind as never) @@ -514,6 +610,17 @@ function collectAddressableIds(scene: SceneSpec): string[] { for (const step of scene.steps) { if (step.id) ids.push(step.id); } + } else if (scene.type === 'infographic') { + for (const item of [ + ...(scene.metrics ?? []), + ...(scene.comparisons ?? []), + ...(scene.process ?? []), + ...(scene.timeline ?? []), + ...(scene.relationshipNodes ?? []), + ...(scene.relationships ?? []), + ...(scene.charts ?? []), + ]) ids.push(item.id); + for (const chart of scene.charts ?? []) for (const series of chart.series ?? []) ids.push(series.id); } else if (scene.type === 'terminal' || scene.type === 'browser') { for (const step of scene.steps ?? []) ids.push(step.id); if (scene.type === 'browser') { @@ -521,6 +628,11 @@ function collectAddressableIds(scene: SceneSpec): string[] { if (target.id) ids.push(target.id); } } + } else if (scene.type === 'manim') { + for (const marker of scene.markers ?? []) { + ids.push(marker.id); + if (marker.targetId) ids.push(marker.targetId); + } } for (const annotation of scene.annotations ?? []) ids.push(annotation.id); return ids; @@ -669,12 +781,26 @@ function validateExplanation( message: `${visualPath}.targetId must reference an addressable element in the scene`, repairable: true, }); } - if (!isObject(visual) || !['reveal', 'highlight', 'focus', 'annotate'].includes(String(visual.action))) { + if (!isObject(visual) || !['reveal', 'highlight', 'focus', 'annotate', 'compare', 'trace', 'emphasize', 'transform'].includes(String(visual.action))) { issue(issues, { severity: 'error', path: `${visualPath}.action`, code: 'unsupported_beat_visual_action', message: `${visualPath}.action is unsupported`, repairable: true, }); } + if (isObject(visual) && visual.action === 'compare' && + (typeof visual.relatedTargetId !== 'string' || !addressableIds.has(visual.relatedTargetId))) { + issue(issues, { + severity: 'error', path: `${visualPath}.relatedTargetId`, code: 'invalid_compare_visual_target', + message: `${visualPath}.relatedTargetId must reference an addressable element`, repairable: true, + }); + } + if (isObject(visual) && visual.action === 'trace' && + (!Array.isArray(visual.pathTargetIds) || visual.pathTargetIds.length < 2 || visual.pathTargetIds.some((id) => typeof id !== 'string' || !addressableIds.has(id)))) { + issue(issues, { + severity: 'error', path: `${visualPath}.pathTargetIds`, code: 'invalid_trace_visual_path', + message: `${visualPath}.pathTargetIds must reference at least two addressable elements`, repairable: true, + }); + } if (isObject(visual) && visual.offsetMs !== undefined && !Number.isFinite(visual.offsetMs)) { issue(issues, { severity: 'error', path: `${visualPath}.offsetMs`, code: 'invalid_beat_visual_offset', @@ -841,6 +967,18 @@ export function validateExplainerDocument(input: unknown): CompositionIssue[] { }); } + if (doc.styleProfile !== undefined) { + for (const profileIssue of validateStyleProfile(doc.styleProfile)) { + issue(issues, { + severity: profileIssue.severity, + path: `styleProfile.${profileIssue.path}`, + code: profileIssue.code, + message: profileIssue.message, + repairable: true, + }); + } + } + for (const numField of ['width', 'height', 'fps'] as const) { const value = doc[numField]; if (value !== undefined && (typeof value !== 'number' || value <= 0)) { @@ -934,6 +1072,10 @@ export function validateExplainerDocument(input: unknown): CompositionIssue[] { validateCodeScene(typedScene, scenePath, issues); } else if (typedScene.type === 'diagram') { validateDiagramScene(typedScene, scenePath, issues); + } else if (typedScene.type === 'infographic') { + validateInfographicScene(typedScene, scenePath, issues); + } else if (typedScene.type === 'manim') { + validateManimScene(typedScene, scenePath, issues); } else { validatePlaceholderScene(typedScene, scenePath, issues); } @@ -979,6 +1121,17 @@ export function validateExplainerDocument(input: unknown): CompositionIssue[] { targetId: annotation.targetId, }); } + if (isObject(annotation) && typeof annotation.toTargetId === 'string') { + annotationTargets.push({ + path: `annotations[${index}].toTargetId`, + targetId: annotation.toTargetId, + }); + } + if (isObject(annotation) && Array.isArray(annotation.pathTargetIds)) { + annotation.pathTargetIds.forEach((targetId, targetIndex) => { + if (typeof targetId === 'string') annotationTargets.push({ path: `annotations[${index}].pathTargetIds[${targetIndex}]`, targetId }); + }); + } }); } doc.scenes.forEach((scene, sceneIndex) => { @@ -990,6 +1143,17 @@ export function validateExplainerDocument(input: unknown): CompositionIssue[] { targetId: annotation.targetId, }); } + if (isObject(annotation) && typeof annotation.toTargetId === 'string') { + annotationTargets.push({ + path: `scenes[${sceneIndex}].annotations[${annotationIndex}].toTargetId`, + targetId: annotation.toTargetId, + }); + } + if (isObject(annotation) && Array.isArray(annotation.pathTargetIds)) { + annotation.pathTargetIds.forEach((targetId, targetIndex) => { + if (typeof targetId === 'string') annotationTargets.push({ path: `scenes[${sceneIndex}].annotations[${annotationIndex}].pathTargetIds[${targetIndex}]`, targetId }); + }); + } }); }); diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 26a2e6a..578be34 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -17,6 +17,17 @@ export { AnnotationLayer, AnnotationProvider, AnnotationTarget, + AttentionSequenceLayer, + resolveAttentionSequence, + resolveAttentionSequenceAtOutputFrame, + selectAttentionForScene, + validateAttentionSequence, + routeConnector, + routeGuidedPath, + resolveSafeLabelPlacement, + resolveSafeLabelPlacements, + annotationOpacity, + orderAnnotationsForStacking, useAnnotationTargetRegistry, } from './annotation'; export type { @@ -24,6 +35,15 @@ export type { AnnotationLayerProps, AnnotationTargetProps, TargetRect, + AttentionKind, + AttentionSequenceItem, + AttentionSequenceLayerProps, + ResolvedAttentionItem, + AttentionSequenceIssue, + RoutePoint, + SafeLabelPlacement, + LabelPlacementRequest, + PlacedLabel, } from './annotation'; export * from './storyboard'; export * from './authoring'; @@ -32,4 +52,8 @@ export * from './seekable'; export * from './adapters'; export * from './clock'; export * from './parsers'; - +export * from './direction'; +export * from './motion-grammar'; +export * from './style-profile'; +export * from './style-profile-runtime'; +export * from './director'; diff --git a/packages/core/src/motion-grammar/index.ts b/packages/core/src/motion-grammar/index.ts new file mode 100644 index 0000000..c9b9e21 --- /dev/null +++ b/packages/core/src/motion-grammar/index.ts @@ -0,0 +1,91 @@ +import type { AttentionSequenceItem } from '../annotation'; +import type { ExplainerDocument, VisualBeatAction } from '../explainer-document/schema'; +import type { DirectionSegment } from '../direction'; + +export const MOTION_GRAMMAR_FORMAT = 'seqvio-motion-grammar' as const; +export const MOTION_GRAMMAR_VERSION = '1.0' as const; +export type MotionGrammarAction = 'question' | 'pause' | 'reveal' | 'trace' | 'compare' | 'emphasize' | 'transform' | 'answer' | 'summarize'; + +export interface MotionGrammarStep { + id: string; + sceneId: string; + action: MotionGrammarAction; + targetId?: string; + relatedTargetId?: string; + pathTargetIds?: string[]; + beatId?: string; + startFrame?: number; + holdFrames?: number; +} + +export interface MotionGrammar { + format: typeof MOTION_GRAMMAR_FORMAT; + version: typeof MOTION_GRAMMAR_VERSION; + id: string; + steps: MotionGrammarStep[]; +} + +export interface MotionGrammarIssue { severity: 'error' | 'warning'; code: string; path: string; message: string } +export interface CompiledMotionGrammar { + visuals: Array<{ stepId: string; sceneId: string; beatId?: string; visual: VisualBeatAction }>; + attention: AttentionSequenceItem[]; + direction: DirectionSegment[]; +} + +const ACTIONS = new Set(['question', 'pause', 'reveal', 'trace', 'compare', 'emphasize', 'transform', 'answer', 'summarize']); +const TARGET_OPTIONAL = new Set(['question', 'pause', 'answer', 'summarize']); + +function targetsForScene(document: ExplainerDocument, sceneId: string): Set { + const plan = document.scenes.find((scene) => scene.id === sceneId); + if (!plan) return new Set(); + const ids = new Set([sceneId]); + const add = (id: string | undefined) => { if (id) ids.add(id); }; + if (plan.type === 'whiteboard') plan.elements.forEach((item) => add(item.id)); + if (plan.type === 'code' || plan.type === 'diagram') plan.steps.forEach((item) => add(item.id)); + if (plan.type === 'diagram') { plan.nodes.forEach((item) => add(item.id)); plan.edges.forEach((item) => add(item.id)); } + if (plan.type === 'infographic') { + [...(plan.metrics ?? []), ...(plan.comparisons ?? []), ...(plan.process ?? []), ...(plan.timeline ?? []), ...(plan.relationshipNodes ?? []), ...(plan.relationships ?? []), ...(plan.charts ?? [])].forEach((item) => add(item.id)); + (plan.charts ?? []).forEach((chart) => chart.series.forEach((series) => add(series.id))); + } + if (plan.type === 'terminal' || plan.type === 'browser') (plan.steps ?? []).forEach((item) => add(item.id)); + if (plan.type === 'manim') (plan.markers ?? []).forEach((item) => { add(item.id); add(item.targetId); }); + return ids; +} + +export function validateMotionGrammar(input: unknown, document?: ExplainerDocument): MotionGrammarIssue[] { + const issues: MotionGrammarIssue[] = []; + const add = (path: string, code: string, message: string) => issues.push({ severity: 'error' as const, path, code, message }); + if (!input || typeof input !== 'object') return [{ severity: 'error', path: '$', code: 'invalid_motion_grammar', message: 'MotionGrammar must be an object' }]; + const grammar = input as Partial; + if (grammar.format !== MOTION_GRAMMAR_FORMAT) add('format', 'unsupported_motion_grammar_format', `format must be "${MOTION_GRAMMAR_FORMAT}"`); + if (grammar.version !== MOTION_GRAMMAR_VERSION) add('version', 'unsupported_motion_grammar_version', `version must be "${MOTION_GRAMMAR_VERSION}"`); + if (!Array.isArray(grammar.steps) || grammar.steps.length === 0) { add('steps', 'missing_motion_steps', 'steps must be non-empty'); return issues; } + const ids = new Set(); + grammar.steps.forEach((step, index) => { + const path = `steps[${index}]`; + if (!step?.id) add(`${path}.id`, 'missing_motion_step_id', 'step id is required'); else if (ids.has(step.id)) add(`${path}.id`, 'duplicate_motion_step_id', `Duplicate step id "${step.id}"`); else ids.add(step.id); + if (!ACTIONS.has(step.action)) add(`${path}.action`, 'unsupported_motion_action', `Unsupported action "${step.action}"`); + const targets = document ? targetsForScene(document, step.sceneId) : undefined; + if (document && targets?.size === 0) add(`${path}.sceneId`, 'unknown_motion_scene', `Unknown scene "${step.sceneId}"`); + if (!TARGET_OPTIONAL.has(step.action) && !step.targetId) add(`${path}.targetId`, 'missing_motion_target', `Action "${step.action}" requires a target`); + for (const target of [step.targetId, step.relatedTargetId, ...(step.pathTargetIds ?? [])]) if (target && targets && !targets.has(target)) add(`${path}.targetId`, 'unknown_motion_target', `Unknown target "${target}" in scene "${step.sceneId}"`); + if (step.action === 'compare' && !step.relatedTargetId) add(`${path}.relatedTargetId`, 'missing_compare_relation', 'compare requires relatedTargetId'); + if (step.action === 'trace' && (!Array.isArray(step.pathTargetIds) || step.pathTargetIds.length < 2)) add(`${path}.pathTargetIds`, 'missing_trace_path', 'trace requires at least two pathTargetIds'); + if (step.startFrame !== undefined && (!Number.isInteger(step.startFrame) || step.startFrame < 0)) add(`${path}.startFrame`, 'invalid_motion_start', 'startFrame must be a non-negative integer'); + if (step.holdFrames !== undefined && (!Number.isInteger(step.holdFrames) || step.holdFrames <= 0)) add(`${path}.holdFrames`, 'invalid_motion_hold', 'holdFrames must be a positive integer'); + }); + return issues; +} + +export function compileMotionGrammar(grammar: MotionGrammar): CompiledMotionGrammar { + const visuals: CompiledMotionGrammar['visuals'] = []; + const attention: AttentionSequenceItem[] = []; + const direction: DirectionSegment[] = []; + grammar.steps.forEach((step, index) => { + const visualAction: VisualBeatAction['action'] = ['reveal', 'trace', 'compare', 'emphasize', 'transform'].includes(step.action) ? step.action as VisualBeatAction['action'] : 'focus'; + if (step.targetId) visuals.push({ stepId: step.id, sceneId: step.sceneId, beatId: step.beatId, visual: { targetId: step.targetId, action: visualAction, relatedTargetId: step.relatedTargetId, pathTargetIds: step.pathTargetIds } }); + if (step.targetId && step.action !== 'reveal') attention.push({ id: step.id, sceneId: step.sceneId, targetId: step.targetId, toTargetId: step.relatedTargetId, pathTargetIds: step.pathTargetIds, kind: step.action === 'compare' ? 'connector' : step.action === 'trace' ? 'guided-path' : 'focus-ring', start: step.startFrame ?? index * 30, duration: step.holdFrames ?? 24, sourceBeatId: step.beatId, persistence: 'timed' }); + direction.push({ id: step.id, sceneId: step.sceneId, purpose: step.action === 'question' ? 'hook' : step.action === 'answer' || step.action === 'summarize' ? 'summarize' : 'explain-mechanism', pace: step.action === 'pause' ? 'hold' : 'steady', focus: step.targetId ? 'target' : 'overview', focusSpec: step.targetId ? { targetId: step.targetId, beatId: step.beatId } : undefined, camera: step.targetId ? 'follow-target' : 'hold', transition: 'cut' }); + }); + return { visuals, attention, direction }; +} diff --git a/packages/core/src/pacing.ts b/packages/core/src/pacing.ts index 3420889..b3f4460 100644 --- a/packages/core/src/pacing.ts +++ b/packages/core/src/pacing.ts @@ -196,6 +196,46 @@ function applyExplanationTiming( : step; }), } as SceneSpec; + } else if (scene.type === 'infographic') { + const attention = beats.flatMap((beat) => beat.visuals + .filter((visual) => visual.action !== 'reveal') + .map((visual, visualIndex) => { + const offsetFrames = Math.round(((visual.offsetMs ?? 0) / 1000) * fps); + const start = Math.max(0, beat.sourceFrame + offsetFrames); + const duration = Math.max( + minFrames, + Math.ceil(((visual.minHoldMs ?? policy.minHighlightMs) / 1000) * fps), + ); + return { + id: `${beat.id}.attention-${visualIndex + 1}`, + targetId: visual.targetId, + kind: visual.action === 'focus' ? 'spotlight' as const + : visual.action === 'annotate' ? 'arrow' as const + : visual.action === 'compare' ? 'connector' as const + : visual.action === 'trace' ? 'guided-path' as const + : 'box' as const, + toTargetId: visual.relatedTargetId, + pathTargetIds: visual.pathTargetIds, + start, + duration, + minHoldFrames: duration, + sourceBeatId: beat.id, + sceneId: scene.id, + persistence: 'until-handoff' as const, + }; + })); + timedScene = { + ...scene, + metrics: scene.metrics?.map((item) => startByTarget.has(item.id) ? { ...item, at: startByTarget.get(item.id)! } : item), + comparisons: scene.comparisons?.map((item) => startByTarget.has(item.id) ? { ...item, at: startByTarget.get(item.id)! } : item), + process: scene.process?.map((item) => startByTarget.has(item.id) ? { ...item, at: startByTarget.get(item.id)! } : item), + timeline: scene.timeline?.map((item) => startByTarget.has(item.id) ? { ...item, at: startByTarget.get(item.id)! } : item), + relationships: scene.relationships?.map((item) => startByTarget.has(item.id) ? { ...item, at: startByTarget.get(item.id)! } : item), + attention: attention.map((item, index) => ({ + ...item, + handoffTo: attention[index + 1]?.targetId, + })), + }; } else if (scene.type === 'browser') { timedScene = { ...scene, @@ -233,6 +273,14 @@ function baseSceneDurationFrames(scene: SceneSpec, fps: number, policy: PacingPo const maxMs = Math.max(0, ...(scene.events ?? []).map((event) => event.timeMs), ...(scene.steps ?? []).map((step) => step.timeMs)); return Math.max(1, Math.ceil((maxMs / 1000) * fps) + tail); } + if (scene.type === 'infographic') { + const itemEnds = [ + ...(scene.metrics ?? []), ...(scene.comparisons ?? []), ...(scene.process ?? []), + ...(scene.timeline ?? []), ...(scene.relationships ?? []), + ].map((item) => (item.at ?? 0) + 30); + const attentionEnds = (scene.attention ?? []).map((item) => item.start + Math.max(item.duration, item.minHoldFrames ?? 0)); + return Math.max(120, ...itemEnds, ...attentionEnds) + tail; + } if (scene.type === 'browser') { const maxMs = Math.max(0, ...(scene.cursorPoints ?? []).map((point) => point.timeMs), ...(scene.focusTargets ?? []).map((point) => point.timeMs), ...(scene.clicks ?? []).map((point) => point.timeMs)); return Math.max(120, Math.ceil((maxMs / 1000) * fps) + tail); diff --git a/packages/core/src/style-profile-runtime.tsx b/packages/core/src/style-profile-runtime.tsx new file mode 100644 index 0000000..adec0f5 --- /dev/null +++ b/packages/core/src/style-profile-runtime.tsx @@ -0,0 +1,22 @@ +import React, { createContext, useContext } from 'react'; +import type { StyleProfile } from './style-profile'; + +const StyleProfileContext = createContext(undefined); + +export function useStyleProfile(): StyleProfile | undefined { + return useContext(StyleProfileContext); +} + +export function StyleProfileProvider({ profile, children }: { profile?: StyleProfile; children: React.ReactNode }) { + if (!profile) return <>{children}; + const variables = { + '--seqvio-color-background': profile.paletteRoles.background, + '--seqvio-color-ink': profile.paletteRoles.ink, + '--seqvio-color-accent': profile.paletteRoles.accent, + '--seqvio-color-muted': profile.paletteRoles.muted, + '--seqvio-font-heading': profile.typography.headingFamily, + '--seqvio-font-body': profile.typography.bodyFamily, + '--seqvio-font-mono': profile.typography.monoFamily ?? 'monospace', + } as React.CSSProperties; + return
{children}
; +} diff --git a/packages/core/src/style-profile.ts b/packages/core/src/style-profile.ts new file mode 100644 index 0000000..c93b2fb --- /dev/null +++ b/packages/core/src/style-profile.ts @@ -0,0 +1,63 @@ +export const STYLE_PROFILE_FORMAT = 'seqvio-style-profile' as const; +export const STYLE_PROFILE_VERSION = '1.0' as const; + +export type StyleMotionDensity = 'restrained' | 'balanced' | 'expressive'; +export type StyleCameraPolicy = 'static' | 'semantic-focus' | 'evidence-follow'; +export type StyleTransitionPolicy = 'cut' | 'crossfade' | 'focus-transfer'; + +export interface StyleProfile { + format: typeof STYLE_PROFILE_FORMAT; + version: typeof STYLE_PROFILE_VERSION; + id: string; + label: string; + typography: { headingFamily: string; bodyFamily: string; monoFamily?: string; scale: 'compact' | 'editorial' | 'large' }; + motionDensity: StyleMotionDensity; + cameraPolicy: StyleCameraPolicy; + transitionPolicy: StyleTransitionPolicy; + attentionPersistence: 'timed' | 'until-handoff'; + spacing: 'tight' | 'comfortable' | 'airy'; + paletteRoles: Record<'background' | 'ink' | 'accent' | 'muted', string>; +} + +export interface StyleProfileIssue { severity: 'error' | 'warning'; code: string; path: string; message: string } + +export function validateStyleProfile(input: unknown): StyleProfileIssue[] { + const issues: StyleProfileIssue[] = []; + if (!input || typeof input !== 'object') return [{ severity: 'error', code: 'invalid_style_profile', path: '$', message: 'Style profile must be an object' }]; + const profile = input as Partial; + if (profile.format !== STYLE_PROFILE_FORMAT) issues.push({ severity: 'error', code: 'unsupported_style_format', path: 'format', message: `format must be "${STYLE_PROFILE_FORMAT}"` }); + if (profile.version !== STYLE_PROFILE_VERSION) issues.push({ severity: 'error', code: 'unsupported_style_version', path: 'version', message: `version must be "${STYLE_PROFILE_VERSION}"` }); + if (!profile.id) issues.push({ severity: 'error', code: 'missing_style_id', path: 'id', message: 'id is required' }); + if (!profile.label) issues.push({ severity: 'error', code: 'missing_style_label', path: 'label', message: 'label is required' }); + if (!profile.typography?.headingFamily || !profile.typography?.bodyFamily) issues.push({ severity: 'error', code: 'missing_style_typography', path: 'typography', message: 'headingFamily and bodyFamily are required' }); + if (!['compact', 'editorial', 'large'].includes(String(profile.typography?.scale))) issues.push({ severity: 'error', code: 'unsupported_typography_scale', path: 'typography.scale', message: 'Typography scale must be compact, editorial, or large' }); + if (!['restrained', 'balanced', 'expressive'].includes(String(profile.motionDensity))) issues.push({ severity: 'error', code: 'unsupported_motion_density', path: 'motionDensity', message: 'motionDensity must be restrained, balanced, or expressive' }); + if (!['static', 'semantic-focus', 'evidence-follow'].includes(String(profile.cameraPolicy))) issues.push({ severity: 'error', code: 'unsupported_camera_policy', path: 'cameraPolicy', message: 'Unsupported camera policy' }); + if (!['cut', 'crossfade', 'focus-transfer'].includes(String(profile.transitionPolicy))) issues.push({ severity: 'error', code: 'unsupported_transition_policy', path: 'transitionPolicy', message: 'Unsupported transition policy' }); + if (!['timed', 'until-handoff'].includes(String(profile.attentionPersistence))) issues.push({ severity: 'error', code: 'unsupported_attention_persistence', path: 'attentionPersistence', message: 'Unsupported attention persistence' }); + if (!['tight', 'comfortable', 'airy'].includes(String(profile.spacing))) issues.push({ severity: 'error', code: 'unsupported_style_spacing', path: 'spacing', message: 'Spacing must be tight, comfortable, or airy' }); + for (const role of ['background', 'ink', 'accent', 'muted'] as const) { + if (typeof profile.paletteRoles?.[role] !== 'string' || !profile.paletteRoles[role]) issues.push({ severity: 'error', code: 'missing_palette_role', path: `paletteRoles.${role}`, message: `Palette role ${role} is required` }); + } + return issues; +} + +/** Compare semantic timing/identity fields before and after style application. */ +export function semanticStyleInvariant(before: unknown, after: unknown): { ok: boolean; differences: string[] } { + const project = (value: any) => { + if (!value || typeof value !== 'object') return JSON.stringify(value); + const { styleProfile: _styleProfile, ...semanticDocument } = value; + return JSON.stringify(semanticDocument); + }; + const left = project(before); const right = project(after); + return left === right ? { ok: true, differences: [] } : { ok: false, differences: ['semantic_timing_or_identity_changed'] }; +} + +export function applyStyleProfile(document: T, profile: StyleProfile): T & { styleProfile: StyleProfile } { + const errors = validateStyleProfile(profile).filter((item) => item.severity === 'error'); + if (errors.length > 0) throw new Error(`Invalid style profile: ${errors.map((item) => item.code).join(', ')}`); + const styled = { ...document, styleProfile: profile }; + const invariant = semanticStyleInvariant(document, styled); + if (!invariant.ok) throw new Error(invariant.differences.join(', ')); + return styled; +} diff --git a/packages/core/tests/annotation-determinism.test.mjs b/packages/core/tests/annotation-determinism.test.mjs new file mode 100644 index 0000000..3269988 --- /dev/null +++ b/packages/core/tests/annotation-determinism.test.mjs @@ -0,0 +1,23 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { ANNOTATION_KINDS, annotationOpacity, orderAnnotationsForStacking } from '../dist/index.js'; + +describe('annotation primitive determinism', () => { + it('orders every primitive by priority and stable id independent of input order', () => { + const annotations = ANNOTATION_KINDS.map((kind, index) => ({ id: `${kind}-${String(index).padStart(2, '0')}`, kind, priority: index % 3 })); + const expected = orderAnnotationsForStacking(annotations).map((item) => item.id); + const reversed = orderAnnotationsForStacking([...annotations].reverse()).map((item) => item.id); + const shuffled = orderAnnotationsForStacking([...annotations].sort((a, b) => (a.id.charCodeAt(1) % 5) - (b.id.charCodeAt(1) % 5))).map((item) => item.id); + assert.deepEqual(reversed, expected); + assert.deepEqual(shuffled, expected); + }); + + it('derives every primitive opacity from the requested frame under random and reverse seek', () => { + const frames = [0, 8, 17, 33, 64, 21, 9, 48]; + for (const kind of ANNOTATION_KINDS) { + const forward = new Map(frames.map((frame) => [frame, annotationOpacity(frame, 8, 24)])); + const reverse = new Map([...frames].reverse().map((frame) => [frame, annotationOpacity(frame, 8, 24)])); + for (const frame of frames) assert.equal(reverse.get(frame), forward.get(frame), `${kind} changed at frame ${frame}`); + } + }); +}); diff --git a/packages/core/tests/authoring.test.mjs b/packages/core/tests/authoring.test.mjs index 63cf8e8..24cc59b 100644 --- a/packages/core/tests/authoring.test.mjs +++ b/packages/core/tests/authoring.test.mjs @@ -155,6 +155,31 @@ describe('human-readable authoring artifacts', () => { assert.match(formatVisualDesignBriefMarkdown(design), /# Visual Design Brief:/); }); + it('formats and validates the product-explainer production contract', () => { + const productPlan = { + ...plan, + hook: 'The command succeeded, so why is the module still missing?', + sections: [{ ...plan.sections[0], evidenceSource: 'terminal-capture' }], + }; + const productDesign = { + ...design, + sceneTreatments: [{ + ...design.sceneTreatments[0], visualRole: 'evidence', focalTarget: 'blocked-script', + evidenceSource: 'terminal-capture', onScreenTextBudget: 6, transitionIntent: 'focus-transfer', + }], + }; + assert.deepEqual(validateEditorialPlan(productPlan), []); + assert.deepEqual(validateVisualDesignBrief(productDesign, productPlan), []); + assert.match(formatEditorialPlanMarkdown(productPlan), /## Hook/); + assert.match(formatVisualDesignBriefMarkdown(productDesign), /On-screen text budget: 6 words/); + const issues = validateVisualDesignBrief({ + ...productDesign, + sceneTreatments: [{ ...productDesign.sceneTreatments[0], focalTarget: undefined, onScreenTextBudget: 0 }], + }, productPlan); + assert.ok(issues.some((issue) => issue.code === 'invalid_text_budget')); + assert.ok(issues.some((issue) => issue.code === 'missing_focal_target')); + }); + it('rejects visual treatments for unknown sections', () => { const issues = validateVisualDesignBrief({ ...design, diff --git a/packages/core/tests/direction.test.mjs b/packages/core/tests/direction.test.mjs new file mode 100644 index 0000000..1476d0e --- /dev/null +++ b/packages/core/tests/direction.test.mjs @@ -0,0 +1,122 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { + compileDirectionPlan, + compileExplainerDocumentToTsx, + deriveDirectionPlan, + validateDirectionPlan, +} from '../dist/index.js'; + +const document = { + scenes: [{ + type: 'diagram', id: 'flow', + nodes: [{ id: 'input', label: 'Input' }, { id: 'output', label: 'Output' }], + edges: [{ id: 'path', from: 'input', to: 'output' }], steps: [], + }], +}; + +const plan = { + format: 'seqvio-direction-plan', version: '1.0', id: 'flow-direction', + segments: [{ + id: 'explain-flow', sceneId: 'flow', purpose: 'explain-mechanism', pace: 'steady', + focus: 'sequence', camera: 'focus-transfer', focusSpec: { targetIds: ['input', 'output'] }, + transition: 'cut', + }], +}; + +describe('DirectionPlan', () => { + it('validates and compiles a renderer-agnostic plan', () => { + assert.deepEqual(validateDirectionPlan(plan, document), []); + const compiled = compileDirectionPlan(plan); + assert.deepEqual(compiled.attention.map((item) => item.targetId), ['input', 'output']); + assert.equal(compiled.sceneActions[0].camera, 'focus-transfer'); + }); + + it('rejects unknown targets and targetless transitions', () => { + const issues = validateDirectionPlan({ + ...plan, + segments: [{ ...plan.segments[0], focusSpec: undefined, transition: 'match-object' }], + }, document); + assert.ok(issues.some((item) => item.code === 'transition_without_shared_target')); + const targetIssues = validateDirectionPlan({ + ...plan, + segments: [{ ...plan.segments[0], focusSpec: { targetId: 'missing' } }], + }, document); + assert.ok(targetIssues.some((item) => item.code === 'unknown_direction_target')); + }); + + it('pairs source and destination targets for cross-scene transitions', () => { + const twoSceneDocument = { + scenes: [ + document.scenes[0], + { type: 'diagram', id: 'result', nodes: [{ id: 'resolved', label: 'Resolved' }], edges: [], steps: [] }, + ], + }; + const paired = { + ...plan, + segments: [ + { ...plan.segments[0], transition: 'match-object', transitionTargetId: 'output', transitionToTargetId: 'resolved' }, + { id: 'show-result', sceneId: 'result', purpose: 'summarize', focus: 'target', camera: 'follow-target', focusSpec: { targetId: 'resolved' }, transition: 'cut' }, + ], + }; + assert.deepEqual(validateDirectionPlan(paired, twoSceneDocument), []); + assert.deepEqual( + compileDirectionPlan(paired).sceneActions.slice(0, 1).map(({ transitionTargetId, transitionToTargetId }) => ({ transitionTargetId, transitionToTargetId })), + [{ transitionTargetId: 'output', transitionToTargetId: 'resolved' }], + ); + const invalid = { ...paired, segments: [{ ...paired.segments[0], transitionToTargetId: 'missing' }, paired.segments[1]] }; + assert.ok(validateDirectionPlan(invalid, twoSceneDocument).some((item) => item.code === 'unknown_direction_transition_destination')); + }); + + it('diagnoses incompatible focus and camera instructions', () => { + const issues = validateDirectionPlan({ + ...plan, + segments: [{ ...plan.segments[0], focus: 'overview', camera: 'overview' }], + }, document); + assert.ok(issues.some((item) => item.code === 'conflicting_direction_focus')); + assert.ok(issues.some((item) => item.code === 'conflicting_direction_camera')); + }); + + it('derives a beat-backed sidecar and embeds compiled direction in meta', () => { + const explainer = { + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'directed-flow', + scenes: [{ + ...document.scenes[0], + explanation: { + cues: [{ id: 'cue', text: 'Follow input to output.' }], + beats: [{ + id: 'follow', cueId: 'cue', anchor: { text: 'input' }, + visuals: [{ targetId: 'input', action: 'focus' }, { targetId: 'output', action: 'highlight' }], + }], + }, + }], + }; + const derived = deriveDirectionPlan(explainer); + assert.deepEqual(validateDirectionPlan(derived, explainer), []); + assert.equal(derived.segments[0].focusSpec.beatId, 'follow'); + const compiled = compileExplainerDocumentToTsx(explainer); + assert.equal(compiled.directionPlan.id, 'directed-flow.direction'); + assert.deepEqual(compiled.compiledDirection.attention.map((item) => item.targetId), ['input', 'output']); + assert.match(compiled.code, /direction:/); + }); + + it('keeps semantic direction stable across duration and chapter reflow', () => { + const base = { + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'stable-direction', + scenes: [{ + ...document.scenes[0], duration: 120, + explanation: { + cues: [{ id: 'cue', text: 'Follow input to output.' }], + beats: [{ id: 'follow', cueId: 'cue', anchor: { text: 'input' }, visuals: [{ targetId: 'input', action: 'focus' }, { targetId: 'output', action: 'highlight' }] }], + }, + }], + chapters: [{ id: 'chapter-a', sceneIds: ['flow'] }], + }; + const reflowed = { + ...base, + scenes: [{ ...base.scenes[0], duration: 240, transitionDuration: 18 }], + chapters: [{ id: 'chapter-reflowed', sceneIds: ['flow'] }], + }; + assert.deepEqual(deriveDirectionPlan(reflowed), deriveDirectionPlan(base)); + }); +}); diff --git a/packages/core/tests/director.test.mjs b/packages/core/tests/director.test.mjs new file mode 100644 index 0000000..b1868b9 --- /dev/null +++ b/packages/core/tests/director.test.mjs @@ -0,0 +1,41 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { createDirectorTask, validateDirectorArtifacts } from '../dist/index.js'; + +const document = { + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'director-example', + scenes: [{ + type: 'infographic', id: 'results', + metrics: [{ id: 'before', label: 'Before', value: '17' }, { id: 'after', label: 'After', value: '6' }], + explanation: { cues: [{ id: 'cue', text: 'Compare before and after.' }], beats: [{ id: 'compare', cueId: 'cue', anchor: { text: 'before and after' }, visuals: [{ targetId: 'before', action: 'compare', relatedTargetId: 'after' }] }] }, + }], +}; + +describe('Director task contract', () => { + it('prepares a renderer-independent generation task with a derived baseline', () => { + const task = createDirectorTask(document, 'generate'); + assert.equal(task.format, 'seqvio-director-task'); + assert.equal(task.baseline.directionPlan.id, 'director-example.direction'); + assert.deepEqual(task.requestedArtifacts, ['directionPlan', 'attentionSequence', 'motionGrammar']); + assert.deepEqual(task.diagnostics, []); + assert.ok(task.constraints.every((value) => !value.includes('React'))); + }); + + it('turns invalid candidate artifacts into reviewable repair suggestions', () => { + const candidate = { + directionPlan: { + format: 'seqvio-direction-plan', version: '1.0', id: 'bad', + segments: [{ id: 'bad-focus', sceneId: 'results', purpose: 'summarize', focus: 'target', camera: 'overview', focusSpec: { targetIds: ['before', 'missing'] }, transition: 'cut' }], + }, + attentionSequence: [{ id: 'held', sceneId: 'results', targetId: 'before', kind: 'box', start: 10, duration: 20, persistence: 'until-clear' }], + motionGrammar: { format: 'seqvio-motion-grammar', version: '1.0', id: 'bad-motion', steps: [{ id: 'trace', sceneId: 'results', action: 'trace', targetId: 'before', pathTargetIds: ['before'] }] }, + }; + const diagnostics = validateDirectorArtifacts(candidate, document); + assert.ok(diagnostics.some((item) => item.artifact === 'directionPlan' && item.code === 'unknown_direction_target')); + assert.ok(diagnostics.some((item) => item.artifact === 'attentionSequence' && item.code === 'missing_clear_frame')); + assert.ok(diagnostics.some((item) => item.artifact === 'motionGrammar' && item.code === 'missing_trace_path')); + const task = createDirectorTask(document, 'repair', candidate); + assert.equal(task.diagnostics.length, diagnostics.length); + assert.ok(task.diagnostics.every((item) => item.suggestion.length > 0)); + }); +}); diff --git a/packages/core/tests/explainer-document.test.mjs b/packages/core/tests/explainer-document.test.mjs index 6c91837..3ebb353 100644 --- a/packages/core/tests/explainer-document.test.mjs +++ b/packages/core/tests/explainer-document.test.mjs @@ -15,16 +15,105 @@ import { SCENE_CAPABILITIES, SCENE_TYPES, listAgentAuthorableSceneCapabilities, + resolveAttentionSequence, + resolveAttentionSequenceAtOutputFrame, + selectAttentionForScene, + validateAttentionSequence, + routeConnector, + routeGuidedPath, + resolveSafeLabelPlacement, + resolveSafeLabelPlacements, } from '../dist/index.js'; +describe('attention sequence', () => { + it('resolves holds and explicit handoffs deterministically', () => { + const sequence = [ + { id: 'a', targetId: 'metric-a', kind: 'box', start: 0, duration: 10, minHoldFrames: 20, handoffTo: 'metric-b' }, + { id: 'b', targetId: 'metric-b', kind: 'circle', start: 15, duration: 20 }, + ]; + assert.strictEqual(resolveAttentionSequence(sequence, 12)[0].active, true); + assert.strictEqual(resolveAttentionSequence(sequence, 16)[0].handoff, true); + assert.strictEqual(resolveAttentionSequence(sequence, 21)[0].active, false); + }); + + it('reflows attention through a post-TTS scene time map', () => { + const sequence = [ + { id: 'a', targetId: 'metric-a', kind: 'box', start: 20, duration: 20, sourceBeatId: 'scene.beat-a' }, + { id: 'b', targetId: 'metric-b', kind: 'circle', start: 60, duration: 20, sourceBeatId: 'scene.beat-b' }, + ]; + const timeMap = [ + { outputFrame: 0, sourceFrame: 0 }, + { outputFrame: 80, sourceFrame: 20 }, + { outputFrame: 160, sourceFrame: 60 }, + { outputFrame: 240, sourceFrame: 100 }, + ]; + const atFirstPhrase = resolveAttentionSequenceAtOutputFrame(sequence, 80, 100, 240, timeMap); + const atSecondPhrase = resolveAttentionSequenceAtOutputFrame(sequence, 160, 100, 240, timeMap); + assert.strictEqual(atFirstPhrase[0].active, true); + assert.strictEqual(atSecondPhrase[1].active, true); + }); + + it('selects scene-local segments and enforces explicit clearing', () => { + const sequence = [ + { id: 'a', sceneId: 'before', targetId: 'metric-a', kind: 'spotlight', start: 0, duration: 10, persistence: 'until-clear', clearAt: 50, handoffTo: 'metric-b', handoffToSceneId: 'after' }, + { id: 'b', sceneId: 'after', targetId: 'metric-b', kind: 'box', start: 0, duration: 20 }, + ]; + assert.deepStrictEqual(selectAttentionForScene(sequence, 'before').map((item) => item.id), ['a']); + assert.deepStrictEqual(validateAttentionSequence(sequence), []); + assert.strictEqual(resolveAttentionSequence(sequence.slice(0, 1), 49)[0].active, true); + assert.strictEqual(resolveAttentionSequence(sequence.slice(0, 1), 50)[0].active, false); + }); +}); + +describe('attention routing', () => { + it('routes connectors deterministically inside the safe area', () => { + const from = { x: 10, y: 40, width: 80, height: 40 }; + const to = { x: 1110, y: 600, width: 100, height: 50 }; + const route = routeConnector(from, to, 1280, 720, 24); + assert.deepStrictEqual(route, routeConnector(from, to, 1280, 720, 24)); + assert.ok(route.every((point) => point.x >= 24 && point.x <= 1256 && point.y >= 24 && point.y <= 696)); + assert.ok(routeGuidedPath([from, to], 1280, 720).length >= 4); + }); + + it('keeps callout labels inside the canvas', () => { + const placement = resolveSafeLabelPlacement({ x: 0, y: 0, width: 60, height: 30 }, 180, 38, 1280, 720); + assert.ok(placement.x >= 18 && placement.y >= 18); + }); + + it('places multiple labels without collisions and reroutes around obstacles', () => { + const targetA = { x: 400, y: 240, width: 80, height: 50 }; + const targetB = { x: 490, y: 245, width: 80, height: 50 }; + const labels = resolveSafeLabelPlacements([ + { id: 'a', target: targetA, width: 180, height: 38 }, + { id: 'b', target: targetB, width: 180, height: 38 }, + { id: 'c', target: { x: 445, y: 330, width: 80, height: 50 }, width: 180, height: 38 }, + ], 960, 540, [targetA, targetB], 24); + assert.deepStrictEqual(labels, resolveSafeLabelPlacements([ + { id: 'a', target: targetA, width: 180, height: 38 }, + { id: 'b', target: targetB, width: 180, height: 38 }, + { id: 'c', target: { x: 445, y: 330, width: 80, height: 50 }, width: 180, height: 38 }, + ], 960, 540, [targetA, targetB], 24)); + const overlap = (a, b) => a.x < b.x + b.width && a.x + a.width > b.x && a.y < b.y + b.height && a.y + a.height > b.y; + assert.ok(labels.every((label, index) => labels.slice(index + 1).every((other) => !overlap(label, other)))); + + const from = { x: 80, y: 250, width: 80, height: 50 }; + const to = { x: 800, y: 250, width: 80, height: 50 }; + const obstacle = { x: 420, y: 180, width: 120, height: 140 }; + const unobstructed = routeConnector(from, to, 960, 540, 24); + const routed = routeConnector(from, to, 960, 540, 24, [obstacle]); + assert.notDeepStrictEqual(routed, unobstructed); + assert.ok(routed.some((point) => point.y > obstacle.y + obstacle.height)); + }); +}); + describe('scene capability registry', () => { it('is the complete source for stable scene validation and compilation metadata', () => { - assert.deepStrictEqual(SCENE_TYPES, ['whiteboard', 'code', 'diagram', 'terminal', 'browser']); + assert.deepStrictEqual(SCENE_TYPES, ['whiteboard', 'code', 'diagram', 'infographic', 'terminal', 'browser', 'manim']); for (const type of SCENE_TYPES) { const capability = SCENE_CAPABILITIES[type]; assert.strictEqual(capability.schemaVersion, '1.0'); assert.strictEqual(capability.compiler, 'complete'); - assert.strictEqual(capability.lifecycle, 'public'); + assert.ok(['public', 'experimental'].includes(capability.lifecycle)); assert.ok(capability.requiredPackage.startsWith('@seqvio/')); assert.ok(capability.qaRules.length > 0); } @@ -131,6 +220,21 @@ describe('validateExplainerDocument', () => { assert.ok(issues.some((i) => i.code === 'unknown_annotation_target')); }); + it('validates connector annotations against both stable targets', () => { + const valid = { + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'connector-valid', + scenes: [{ + type: 'infographic', id: 'summary', + metrics: [{ id: 'from', label: 'From', value: '1' }, { id: 'to', label: 'To', value: '2' }], + annotations: [{ id: 'link', targetId: 'from', toTargetId: 'to', kind: 'connector', start: 0, duration: 20 }], + }], + }; + assert.deepStrictEqual(validateExplainerDocument(valid), []); + const invalid = structuredClone(valid); + invalid.scenes[0].annotations[0].toTargetId = 'missing'; + assert.ok(validateExplainerDocument(invalid).some((issue) => issue.code === 'unknown_annotation_target')); + }); + it('rejects chapter references to unknown scenes', () => { const issues = validateExplainerDocument({ format: 'seqvio-explainer', schemaVersion: '1.0', @@ -173,11 +277,10 @@ describe('validateExplainerDocument', () => { assert.ok(issues.some((i) => i.code === 'invalid_terminal_event_kind')); }); - it('rejects removed chat, diff, and infographic scene types', () => { + it('rejects removed chat and diff scene types', () => { for (const scene of [ { type: 'chat', id: 'conversation', messages: [{ role: 'user', text: 'hello' }] }, { type: 'diff', id: 'change', before: 'old', after: 'new' }, - { type: 'infographic', id: 'summary', panels: [{ id: 'total', label: 'Total' }] }, ]) { const issues = validateExplainerDocument({ format: 'seqvio-explainer', schemaVersion: '1.0', @@ -309,8 +412,63 @@ describe('compileExplainerDocumentToTsx', () => { assert.doesNotMatch(code, /shell-command-1/); }); + it('compiles an infographic scene', () => { + const { code } = compileExplainerDocumentToTsx({ + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'infographic-compile', + scenes: [{ + type: 'infographic', id: 'summary', + metrics: [{ id: 'total', label: 'Total', value: '42' }], + comparisons: [{ id: 'delta', label: 'Delta', before: 10, after: 4 }], + charts: [{ + id: 'trend', title: 'Verified trend', kind: 'line', unit: 'ms', sourceLabel: 'fixture', + xAxis: { label: 'Run' }, yAxis: { label: 'Latency', min: 0, max: 20, ticks: 4 }, legend: 'top', + series: [{ id: 'p95', label: 'p95', points: [{ x: 'A', y: 18 }, { x: 'B', y: 7 }] }], + }], + explanation: { + cues: [{ id: 'result', text: 'The total improves, then the delta becomes clear.' }], + beats: [ + { id: 'total-beat', cueId: 'result', anchor: { text: 'total improves' }, visuals: [{ targetId: 'total', action: 'focus', minHoldMs: 1200 }] }, + { id: 'delta-beat', cueId: 'result', anchor: { text: 'delta becomes clear' }, visuals: [{ targetId: 'delta', action: 'highlight' }] }, + ], + }, + }], + }); + assert.match(code, /InfographicScene/); + assert.match(code, /"id": "total"/); + assert.match(code, /"kind": "spotlight"/); + assert.match(code, /"handoffTo": "delta"/); + assert.match(code, /"sourceBeatId": "summary.total-beat"/); + assert.match(code, /"sourceLabel": "fixture"/); + assert.match(code, /"id": "p95"/); + }); + + it('rejects malformed infographic chart data and axis domains', () => { + const issues = validateExplainerDocument({ + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'bad-chart', + scenes: [{ type: 'infographic', id: 'chart', charts: [{ id: 'trend', title: 'Trend', kind: 'line', yAxis: { min: 10, max: 2 }, series: [{ id: 'series', label: 'Series', points: [] }] }] }], + }); + const codes = issues.map((issue) => issue.code); + assert.ok(codes.includes('invalid_infographic_chart_points')); + assert.ok(codes.includes('invalid_infographic_axis_domain')); + }); + + it('validates and compiles a Manim-backed scene with named markers', () => { + const document = { + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'manim-compile', + scenes: [{ + type: 'manim', id: 'equation', sourceVideo: 'equation.mp4', mediaFps: 30, + markers: [{ id: 'start', frame: 0 }, { id: 'result', frame: 60, targetId: 'equation-result' }], + }], + }; + assert.deepEqual(validateExplainerDocument(document), []); + const { code } = compileExplainerDocumentToTsx(document); + assert.match(code, /ManimClip/); + assert.match(code, /equation\.mp4/); + assert.match(code, /"equation-result"/); + }); + it('refuses removed scene types when validation is bypassed', () => { - for (const type of ['chat', 'diff', 'infographic']) { + for (const type of ['chat', 'diff']) { assert.throws( () => compileExplainerDocumentToTsx({ format: 'seqvio-explainer', schemaVersion: '1.0', diff --git a/packages/core/tests/motion-grammar.test.mjs b/packages/core/tests/motion-grammar.test.mjs new file mode 100644 index 0000000..a37ff43 --- /dev/null +++ b/packages/core/tests/motion-grammar.test.mjs @@ -0,0 +1,56 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { compileMotionGrammar, validateMotionGrammar } from '../dist/index.js'; + +const document = { + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'compare', + scenes: [{ type: 'infographic', id: 'results', metrics: [{ id: 'before', label: 'Before', value: '17' }, { id: 'middle', label: 'Middle', value: '11' }, { id: 'after', label: 'After', value: '6' }] }], +}; +const grammar = { + format: 'seqvio-motion-grammar', version: '1.0', id: 'compare-flow', + steps: [ + { id: 'question', sceneId: 'results', action: 'question' }, + { id: 'reveal-before', sceneId: 'results', action: 'reveal', targetId: 'before' }, + { id: 'compare', sceneId: 'results', action: 'compare', targetId: 'before', relatedTargetId: 'after' }, + { id: 'answer', sceneId: 'results', action: 'answer', targetId: 'after' }, + ], +}; + +describe('Motion Grammar', () => { + it('validates and compiles semantic actions into existing contracts', () => { + assert.deepEqual(validateMotionGrammar(grammar, document), []); + const compiled = compileMotionGrammar(grammar); + assert.equal(compiled.visuals.length, 3); + assert.equal(compiled.attention.find((item) => item.id === 'compare').kind, 'connector'); + assert.equal(compiled.visuals.find((item) => item.stepId === 'compare').visual.action, 'compare'); + assert.equal(compiled.direction[0].purpose, 'hook'); + }); + + it('rejects unexplained targets and incomplete comparisons', () => { + const issues = validateMotionGrammar({ ...grammar, steps: [{ id: 'bad', sceneId: 'results', action: 'compare', targetId: 'missing' }] }, document); + assert.ok(issues.some((item) => item.code === 'unknown_motion_target')); + assert.ok(issues.some((item) => item.code === 'missing_compare_relation')); + }); + + it('preserves a complete guided path and deterministic seek timing', () => { + const traceGrammar = { + ...grammar, + steps: [{ id: 'trace', sceneId: 'results', action: 'trace', targetId: 'before', pathTargetIds: ['before', 'middle', 'after'], startFrame: 42, holdFrames: 48 }], + }; + assert.deepEqual(validateMotionGrammar(traceGrammar, document), []); + const first = compileMotionGrammar(traceGrammar); + const second = compileMotionGrammar(traceGrammar); + assert.deepEqual(second, first); + assert.deepEqual(first.attention[0].pathTargetIds, ['before', 'middle', 'after']); + assert.equal(first.attention[0].start, 42); + assert.equal(first.attention[0].duration, 48); + assert.equal(first.visuals[0].visual.action, 'trace'); + }); + + it('rejects incomplete traces and unsafe timing', () => { + const issues = validateMotionGrammar({ ...grammar, steps: [{ id: 'bad-trace', sceneId: 'results', action: 'trace', targetId: 'before', pathTargetIds: ['before'], startFrame: -1, holdFrames: 0 }] }, document); + assert.ok(issues.some((item) => item.code === 'missing_trace_path')); + assert.ok(issues.some((item) => item.code === 'invalid_motion_start')); + assert.ok(issues.some((item) => item.code === 'invalid_motion_hold')); + }); +}); diff --git a/packages/core/tests/style-profile.test.mjs b/packages/core/tests/style-profile.test.mjs new file mode 100644 index 0000000..622a4d1 --- /dev/null +++ b/packages/core/tests/style-profile.test.mjs @@ -0,0 +1,32 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { applyStyleProfile, compileExplainerDocumentToTsx, semanticStyleInvariant, validateStyleProfile } from '../dist/index.js'; + +const profile = { + format: 'seqvio-style-profile', version: '1.0', id: 'clean-technical', label: 'Clean Technical', + typography: { headingFamily: 'Inter', bodyFamily: 'Inter', monoFamily: 'JetBrains Mono', scale: 'editorial' }, + motionDensity: 'restrained', cameraPolicy: 'semantic-focus', transitionPolicy: 'focus-transfer', + attentionPersistence: 'until-handoff', spacing: 'comfortable', + paletteRoles: { background: '#0f172a', ink: '#f8fafc', accent: '#38bdf8', muted: '#94a3b8' }, +}; + +describe('Style Profile', () => { + it('validates a semantic style profile', () => assert.deepEqual(validateStyleProfile(profile), [])); + it('rejects semantic changes caused by style application', () => { + const before = { id: 'x', scenes: [{ id: 's', explanation: { beats: [{ id: 'b' }] } }] }; + assert.equal(semanticStyleInvariant(before, { ...before, scenes: [{ ...before.scenes[0], explanation: { beats: [] } }] }).ok, false); + assert.equal(semanticStyleInvariant(before, before).ok, true); + }); + it('applies visual profiles without changing semantic timing or identity', () => { + const document = { + format: 'seqvio-explainer', schemaVersion: '1.0', id: 'styled', fps: 30, + scenes: [{ type: 'infographic', id: 'scene', duration: 60, metrics: [{ id: 'result', label: 'Result', value: 'Ready' }], explanation: { cues: [{ id: 'cue', text: 'The result is ready.' }], beats: [{ id: 'beat', cueId: 'cue', anchor: { text: 'result' }, visuals: [{ targetId: 'result', action: 'focus' }] }] } }], + }; + const styled = applyStyleProfile(document, profile); + assert.equal(semanticStyleInvariant(document, styled).ok, true); + const compiled = compileExplainerDocumentToTsx(styled); + assert.match(compiled.code, /StyleProfileProvider/); + assert.match(compiled.code, /clean-technical/); + assert.deepEqual(compiled.compiledDirection.attention.map((item) => item.targetId), ['result']); + }); +}); diff --git a/packages/manim-adapter/README.md b/packages/manim-adapter/README.md new file mode 100644 index 0000000..83b658a --- /dev/null +++ b/packages/manim-adapter/README.md @@ -0,0 +1,68 @@ +# @seqvio/manim-adapter + +Experimental TypeScript/Node.js adapter for the Python package `manim`. + +The package validates a versioned `ManimSceneSpec`, invokes a selected Python +interpreter with `python -m manim`, probes the resulting media, and writes a +content-addressed `ManimRenderManifest`. It does not contain a JavaScript Manim +implementation and does not run Python in the browser. + +Use the rendered media with `ManimClip` from `@seqvio/technical` or a `manim` +scene in Seqvio's `ExplainerDocument`. + +## Requirements + +- Node.js 18 or newer +- Python with the `manim` package installed +- Manim's system dependencies and FFmpeg +- `ffprobe` for output validation + +The repository fixtures currently use `manim==0.20.1`. + +## API + +```ts +import { + executeManimScene, + preflightManim, + type ManimSceneSpec, +} from "@seqvio/manim-adapter"; + +const pythonCommand = process.env.SEQVIO_MANIM_PYTHON ?? "python"; +const scene: ManimSceneSpec = { + format: "seqvio-manim-scene", + version: "1.0", + id: "equation", + pythonFile: "examples/manim/equation.py", + className: "EquationDerivation", + width: 1280, + height: 720, + fps: 30, + quality: "medium", + args: ["--media_dir", "output/manim-media"], + assets: [], +}; + +const preflight = preflightManim(pythonCommand); +const manifest = await executeManimScene(scene, { + cwd: process.cwd(), + cacheDir: "output/manim-cache", + expectedOutputPath: + "output/manim-media/videos/equation/720p30/EquationDerivation.mp4", + pythonCommand, + preflight, + onProgress: console.log, +}); + +if (manifest.status !== "rendered") { + throw new Error(manifest.diagnostics.join(", ")); +} +``` + +`pythonCommand` may be `python`, `python3`, or an absolute virtual-environment +interpreter path. The cache key includes the scene contract, Python source, +declared assets, render settings, Python version, and Manim version. + +See the complete [Manim integration guide](../../docs/MANIM-INTEGRATION.md) for +environment setup, the repository CLI helper, `ExplainerDocument`, markers, +`ExplanationBeat` alignment, and troubleshooting. diff --git a/packages/manim-adapter/package.json b/packages/manim-adapter/package.json new file mode 100644 index 0000000..3feaaa8 --- /dev/null +++ b/packages/manim-adapter/package.json @@ -0,0 +1,15 @@ +{ + "name": "@seqvio/manim-adapter", + "version": "0.1.0", + "description": "Experimental adapter for rendering Python Manim scenes into validated Seqvio media manifests", + "license": "MIT", + "seqvio": { + "lifecycle": "experimental", + "versionPolicy": "independent" + }, + "main": "dist/index.js", + "types": "dist/index.d.ts", + "files": ["dist"], + "scripts": { "build": "tsc", "test": "node --test tests/*.test.mjs" }, + "devDependencies": { "typescript": "^5.3.0", "@types/node": "^20.11.0" } +} diff --git a/packages/manim-adapter/src/index.ts b/packages/manim-adapter/src/index.ts new file mode 100644 index 0000000..47e1acd --- /dev/null +++ b/packages/manim-adapter/src/index.ts @@ -0,0 +1,223 @@ +import { execFileSync, spawn } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { mkdir, readFile, stat, writeFile } from 'node:fs/promises'; +import path from 'node:path'; + +export const MANIM_SCENE_FORMAT = 'seqvio-manim-scene' as const; +export const MANIM_SCENE_VERSION = '1.0' as const; + +export interface ManimSceneSpec { + format: typeof MANIM_SCENE_FORMAT; + version: typeof MANIM_SCENE_VERSION; + id: string; + pythonFile: string; + className: string; + width?: number; + height?: number; + fps?: number; + quality?: 'low' | 'medium' | 'high'; + args?: string[]; + /** External files read by the scene; their content participates in the cache key. */ + assets?: string[]; +} + +export interface ManimRenderManifest { + scene: ManimSceneSpec; + command: string[]; + outputPath?: string; + status: 'planned' | 'rendered' | 'failed'; + manimVersion?: string; + pythonVersion?: string; + capabilities: string[]; + diagnostics: string[]; + cacheKey?: string; + cached?: boolean; + durationSeconds?: number; + width?: number; + height?: number; + fps?: number; + alphaMode?: 'opaque' | 'straight' | 'premultiplied'; + markers?: Array<{ id: string; frame: number }>; + logPath?: string; + sourceHash?: string; + runtimeHash?: string; + assetHashes?: Record; +} + +export interface ManimPreflightResult { + available: boolean; + python?: string; + manim?: string; + pythonVersion?: string; + manimVersion?: string; + capabilities: string[]; + diagnostics: string[]; +} + +export interface ManimMediaProbe { + width?: number; + height?: number; + fps?: number; + durationSeconds?: number; + codec?: string; + alphaMode?: 'opaque' | 'straight' | 'premultiplied'; +} + +function probe(command: string, args: string[]): string | undefined { + try { return execFileSync(command, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); } + catch { return undefined; } +} + +export function probeManimMedia(mediaPath: string, ffprobeCommand = 'ffprobe'): ManimMediaProbe | undefined { + try { + const raw = execFileSync(ffprobeCommand, ['-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=width,height,r_frame_rate,codec_name,pix_fmt:format=duration', '-of', 'json', mediaPath], { encoding: 'utf8' }); + const parsed = JSON.parse(raw) as { streams?: Array<{ width?: number; height?: number; r_frame_rate?: string; codec_name?: string; pix_fmt?: string }>; format?: { duration?: string } }; + const stream = parsed.streams?.[0]; + if (!stream) return undefined; + const [num, den] = String(stream.r_frame_rate ?? '').split('/').map(Number); + const pix = stream.pix_fmt ?? ''; + return { width: stream.width, height: stream.height, fps: den ? num / den : undefined, durationSeconds: parsed.format?.duration ? Number(parsed.format.duration) : undefined, codec: stream.codec_name, alphaMode: pix.includes('a') ? 'straight' : 'opaque' }; + } catch { return undefined; } +} + +export function validateManimScene(scene: unknown): string[] { + const errors: string[] = []; + if (!scene || typeof scene !== 'object') return ['scene must be an object']; + const value = scene as Partial; + if (value.format !== MANIM_SCENE_FORMAT) errors.push(`format must be "${MANIM_SCENE_FORMAT}"`); + if (value.version !== MANIM_SCENE_VERSION) errors.push(`version must be "${MANIM_SCENE_VERSION}"`); + if (!value.id) errors.push('id is required'); + if (!value.pythonFile) errors.push('pythonFile is required'); + if (!value.className) errors.push('className is required'); + return errors; +} + +export function buildManimCommand(scene: ManimSceneSpec): string[] { + return ['-m', 'manim', ...(scene.quality ? [`-${scene.quality === 'low' ? 'ql' : scene.quality === 'high' ? 'qh' : 'qm'}`] : []), ...(scene.fps ? ['--fps', String(scene.fps)] : []), ...(scene.width && scene.height ? ['-r', `${scene.width},${scene.height}`] : []), scene.pythonFile, scene.className, ...(scene.args ?? [])]; +} + +export function preflightManim(pythonCommand = 'python'): ManimPreflightResult { + const python = probe(pythonCommand, ['--version']); + const manim = probe(pythonCommand, ['-m', 'manim', '--version']); + const diagnostics: string[] = []; + if (!python) diagnostics.push('python_not_found'); + if (!manim) diagnostics.push('manim_not_found'); + return { available: Boolean(python && manim), python: python ? 'python' : undefined, manim: manim ? 'python -m manim' : undefined, pythonVersion: python, manimVersion: manim, capabilities: manim ? ['scene-render', 'png-frame', 'mp4-export'] : [], diagnostics }; +} + +export function createManimManifest(scene: ManimSceneSpec, preflight = preflightManim()): ManimRenderManifest { + const errors = validateManimScene(scene); + const runtimeHash = createHash('sha256').update(JSON.stringify({ python: preflight.pythonVersion, manim: preflight.manimVersion })).digest('hex'); + return { scene, command: ['python', ...buildManimCommand(scene)], status: errors.length || !preflight.available ? 'failed' : 'planned', manimVersion: preflight.manimVersion, pythonVersion: preflight.pythonVersion, runtimeHash, capabilities: preflight.capabilities, diagnostics: [...errors, ...preflight.diagnostics] }; +} + +export interface ManimExecutionOptions { + cwd?: string; + cacheDir?: string; + expectedOutputPath?: string; + timeoutMs?: number; + signal?: AbortSignal; + pythonCommand?: string; + preflight?: ManimPreflightResult; + dryRun?: boolean; + onProgress?: (event: { phase: 'planned' | 'rendering' | 'cached' | 'complete'; progress?: number; message?: string }) => void; +} + +export async function hashManimAssets(scene: ManimSceneSpec, baseDir = process.cwd()): Promise> { + const hashes: Record = {}; + for (const asset of [...(scene.assets ?? [])].sort()) { + try { hashes[asset] = createHash('sha256').update(await readFile(path.resolve(baseDir, asset))).digest('hex'); } + catch { hashes[asset] = 'missing'; } + } + return hashes; +} + +export async function hashManimSource(scene: ManimSceneSpec, baseDir = process.cwd()): Promise { + try { return createHash('sha256').update(await readFile(path.resolve(baseDir, scene.pythonFile))).digest('hex'); } + catch { return createHash('sha256').update(`missing:${scene.pythonFile}`).digest('hex'); } +} + +export async function hashManimRender(scene: ManimSceneSpec, preflight: ManimPreflightResult, baseDir = process.cwd()): Promise { + const sourceHash = await hashManimSource(scene, baseDir); + const assetHashes = await hashManimAssets(scene, baseDir); + return createHash('sha256').update(JSON.stringify({ contract: 'source-assets-runtime-v1', scene, sourceHash, assetHashes, python: preflight.pythonVersion, manim: preflight.manimVersion })).digest('hex'); +} + +export async function executeManimScene(scene: ManimSceneSpec, options: ManimExecutionOptions = {}): Promise { + const pythonCommand = options.pythonCommand ?? 'python'; + const preflight = options.preflight ?? preflightManim(pythonCommand); + const manifest = createManimManifest(scene, preflight); + const errors = validateManimScene(scene); + if (errors.length) return manifest; + const cacheKey = await hashManimRender(scene, preflight, options.cwd); + manifest.cacheKey = cacheKey; + manifest.sourceHash = await hashManimSource(scene, options.cwd); + manifest.assetHashes = await hashManimAssets(scene, options.cwd); + const missingAssets = Object.entries(manifest.assetHashes).filter(([, value]) => value === 'missing').map(([asset]) => asset); + if (missingAssets.length > 0) { + manifest.status = 'failed'; + manifest.diagnostics.push(...missingAssets.map((asset) => `missing_asset:${asset}`)); + return manifest; + } + const cacheDir = path.resolve(options.cacheDir ?? '.seqvio-cache/manim'); + const cacheManifestPath = path.join(cacheDir, `${cacheKey}.json`); + try { + const cached = JSON.parse(await readFile(cacheManifestPath, 'utf8')) as ManimRenderManifest; + if (cached.outputPath && (await stat(cached.outputPath)).isFile()) { + options.onProgress?.({ phase: 'cached', progress: 1, message: cacheKey }); + return { ...cached, cached: true }; + } + } catch { /* Cache miss. */ } + + const command = [pythonCommand, ...buildManimCommand(scene)]; + manifest.command = command; + manifest.cached = false; + if (options.dryRun) { + manifest.status = 'planned'; + options.onProgress?.({ phase: 'planned', progress: 0, message: command.join(' ') }); + return manifest; + } + if (!preflight.available) return manifest; + + await mkdir(cacheDir, { recursive: true }); + const logPath = path.join(cacheDir, `${cacheKey}.log`); + manifest.logPath = logPath; + const logs: string[] = []; + const timeoutMs = options.timeoutMs ?? 10 * 60_000; + await new Promise((resolve, reject) => { + const child = spawn(command[0], command.slice(1), { cwd: options.cwd, windowsHide: true }); + const finish = (error?: Error) => { clearTimeout(timer); options.signal?.removeEventListener('abort', abort); error ? reject(error) : resolve(); }; + const abort = () => { child.kill(); finish(new Error('manim_cancelled')); }; + const timer = setTimeout(() => { child.kill(); finish(new Error('manim_timeout')); }, timeoutMs); + options.signal?.addEventListener('abort', abort, { once: true }); + const capture = (chunk: Buffer) => { + const text = chunk.toString(); logs.push(text); + const match = text.match(/(\d{1,3})%/); + options.onProgress?.({ phase: 'rendering', progress: match ? Math.min(1, Number(match[1]) / 100) : undefined, message: text.trim() }); + }; + child.stdout.on('data', capture); child.stderr.on('data', capture); + child.on('error', finish); + child.on('exit', (code) => code === 0 ? finish() : finish(new Error(`manim_exit_${code ?? 'unknown'}`))); + }).catch((error) => { manifest.status = 'failed'; manifest.diagnostics.push(error instanceof Error ? error.message : String(error)); }); + await writeFile(logPath, logs.join(''), 'utf8'); + if (manifest.status !== 'failed') { + manifest.status = 'rendered'; manifest.outputPath = options.expectedOutputPath; + if (!manifest.outputPath || !probeManimMedia(manifest.outputPath)) { + manifest.status = 'failed'; + manifest.diagnostics.push('missing_or_unreadable_media'); + } else { + const media = probeManimMedia(manifest.outputPath); + manifest.width = media?.width; manifest.height = media?.height; manifest.fps = media?.fps; + manifest.durationSeconds = media?.durationSeconds; manifest.alphaMode = media?.alphaMode; + if (!manifest.width || !manifest.height || !manifest.fps || !manifest.alphaMode) { + manifest.status = 'failed'; + manifest.diagnostics.push('incomplete_media_contract'); + } + } + } + if (manifest.status === 'rendered') { + options.onProgress?.({ phase: 'complete', progress: 1 }); + } + await writeFile(cacheManifestPath, `${JSON.stringify(manifest, null, 2)}\n`, 'utf8'); + return manifest; +} diff --git a/packages/manim-adapter/tests/adapter.test.mjs b/packages/manim-adapter/tests/adapter.test.mjs new file mode 100644 index 0000000..f2942a2 --- /dev/null +++ b/packages/manim-adapter/tests/adapter.test.mjs @@ -0,0 +1,51 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { fileURLToPath } from 'node:url'; +import { + buildManimCommand, + executeManimScene, + hashManimAssets, + hashManimSource, + hashManimRender, + probeManimMedia, + validateManimScene, +} from '../dist/index.js'; + +const assetPath = fileURLToPath(new URL('../../../examples/manim/equation.py', import.meta.url)); +const scene = { + format: 'seqvio-manim-scene', version: '1.0', id: 'equation', + pythonFile: 'examples/manim/equation.py', className: 'EquationDerivation', + width: 1280, height: 720, fps: 30, quality: 'medium', assets: [assetPath], +}; +const unavailable = { available: false, pythonVersion: 'Python test', manimVersion: 'Manim test', capabilities: [], diagnostics: ['test_preflight'] }; + +describe('Manim adapter', () => { + it('validates and constructs a deterministic command', () => { + assert.deepEqual(validateManimScene(scene), []); + assert.deepEqual(buildManimCommand(scene), ['-m', 'manim', '-qm', '--fps', '30', '-r', '1280,720', scene.pythonFile, scene.className]); + }); + + it('hashes source and produces an inspectable dry-run manifest', async () => { + assert.equal(await hashManimRender(scene, unavailable), await hashManimRender(scene, unavailable)); + assert.match((await hashManimAssets(scene))[assetPath], /^[a-f0-9]{64}$/); + assert.match(await hashManimSource(scene), /^[a-f0-9]{64}$/); + const events = []; + const manifest = await executeManimScene(scene, { preflight: unavailable, dryRun: true, onProgress: (event) => events.push(event) }); + assert.equal(manifest.status, 'planned'); + assert.equal(manifest.cached, false); + assert.ok(manifest.cacheKey); + assert.match(manifest.sourceHash, /^[a-f0-9]{64}$/); + assert.match(manifest.runtimeHash, /^[a-f0-9]{64}$/); + assert.equal(events[0].phase, 'planned'); + }); + + it('fails before execution when a declared asset is missing', async () => { + const manifest = await executeManimScene({ ...scene, assets: ['missing.asset'] }, { preflight: unavailable }); + assert.equal(manifest.status, 'failed'); + assert.ok(manifest.diagnostics.includes('missing_asset:missing.asset')); + }); + + it('returns a machine-readable media probe failure for missing output', () => { + assert.equal(probeManimMedia('missing-manim-output.mp4'), undefined); + }); +}); diff --git a/packages/manim-adapter/tsconfig.json b/packages/manim-adapter/tsconfig.json new file mode 100644 index 0000000..2c1e1e1 --- /dev/null +++ b/packages/manim-adapter/tsconfig.json @@ -0,0 +1,8 @@ +{ + "compilerOptions": { + "target": "ES2020", "module": "commonjs", "declaration": true, + "outDir": "dist", "rootDir": "src", "strict": true, + "esModuleInterop": true, "skipLibCheck": true, "moduleResolution": "node" + }, + "include": ["src/**/*"] +} diff --git a/packages/renderer/src/agent-contract.ts b/packages/renderer/src/agent-contract.ts index 20559ae..1128514 100644 --- a/packages/renderer/src/agent-contract.ts +++ b/packages/renderer/src/agent-contract.ts @@ -63,12 +63,14 @@ The plan must contain these headings: - Explanation Strategy - Content Decisions (each item: stable id, include/omit, role, reason, prerequisites, time estimate) - Explanation Structure (each section: stable id, purpose, concept ids, audience outcome, target seconds) +- Hook (the opening narration promise or tension, not a title-card sentence) Rules: - Make omissions explicit; do not merely summarize everything in the source. - Every included essential concept must appear in the explanation structure. - Keep the section budget within the intended video length. - One section should perform one cognitive job. +- Declare whether evidence is authored, terminal capture, browser capture, or recorded media. - Select zero to two explanation patterns only when they improve the content. - If selected, use exactly one primary pattern, optionally one supporting pattern, and state the reason and any adaptations. Patterns are guidance, not templates: @@ -114,9 +116,12 @@ The brief must contain these headings: - Avoid For every editorial section, choose one supported visual form: whiteboard, code, -diagram, terminal, or browser. Terminal/browser may only be selected when real +diagram, infographic, terminal, browser, or manim. Terminal/browser may only be selected when real capture material exists; otherwise choose a truthful authored fallback. -Specify hierarchy, composition, emphasis, and motion behavior rather than vague +Declare visualRole, one focalTarget, evidenceSource, a maximum concurrent +onScreenTextBudget in words, and transitionIntent. Narration carries full +sentences; screen text carries labels, keywords, commands, filenames, and short +conclusions. Specify hierarchy, composition, emphasis, and motion behavior rather than vague mood words. Return Markdown only, beginning with "# Visual Design Brief:". ${language} diff --git a/packages/renderer/src/doctor.ts b/packages/renderer/src/doctor.ts index e8fb275..eeb9c7e 100644 --- a/packages/renderer/src/doctor.ts +++ b/packages/renderer/src/doctor.ts @@ -39,6 +39,14 @@ export interface DoctorOptions { launchBrowser?: boolean; } +export function resolveManimPythonCommand(cwd: string, configured?: string): string { + if (configured) return configured; + const local = process.platform === 'win32' + ? path.join(cwd, '.venv-manim', 'Scripts', 'python.exe') + : path.join(cwd, '.venv-manim', 'bin', 'python'); + return fs.existsSync(local) ? local : 'python'; +} + export function supportsNodeVersion(version: string): boolean { const match = /^v?(\d+)/.exec(version.trim()); return match !== null && Number(match[1]) >= 18; @@ -165,6 +173,29 @@ function checkFontAssets(requireFromProject: NodeRequire): DoctorCheck { ); } +function checkManim(requireFromProject: NodeRequire, cwd: string): DoctorCheck { + try { + let resolved: string; + try { + resolved = requireFromProject.resolve('@seqvio/manim-adapter'); + } catch { + const workspaceEntry = path.join(cwd, 'packages', 'manim-adapter', 'dist', 'index.js'); + if (!fs.existsSync(workspaceEntry)) throw new Error('Cannot resolve @seqvio/manim-adapter or its workspace build.'); + resolved = workspaceEntry; + } + const adapter = requireFromProject(resolved) as { + preflightManim?: (pythonCommand?: string) => { available: boolean; pythonVersion?: string; manimVersion?: string; diagnostics: string[] }; + }; + if (typeof adapter.preflightManim !== 'function') throw new Error('Adapter does not export preflightManim().'); + const pythonCommand = resolveManimPythonCommand(cwd, process.env.SEQVIO_MANIM_PYTHON); + const result = adapter.preflightManim(pythonCommand); + if (result.available) return check('manim', 'Manim adapter', 'pass', `${result.pythonVersion}; ${result.manimVersion}; ${resolved}`); + return check('manim', 'Manim adapter', 'warn', result.diagnostics.join(', ') || `Unavailable through ${pythonCommand}`, 'Install Manim in .venv-manim or set SEQVIO_MANIM_PYTHON to its Python executable.'); + } catch (error) { + return check('manim', 'Manim adapter', 'warn', error instanceof Error ? error.message : String(error), 'Install @seqvio/manim-adapter only when mathematical animation is required.'); + } +} + export async function runDoctor(options: DoctorOptions = {}): Promise { const cwd = path.resolve(options.cwd ?? process.cwd()); const checks: DoctorCheck[] = []; @@ -175,6 +206,7 @@ export async function runDoctor(options: DoctorOptions = {}): Promise { return page.evaluate(() => { const elements = Array.from(document.querySelectorAll('#root *')); @@ -350,6 +358,10 @@ async function inspectDom(page: import('puppeteer').Page): Promise<{ let textOverflowCount = 0; let smallFontCount = 0; let lowContrastCount = 0; + const attentionMissingTargets: string[] = []; + const attentionOffscreenLabels: string[] = []; + const attentionLabelCollisions: string[] = []; + const attentionTargetOcclusions: string[] = []; const MIN_FONT_PX = 12; const MIN_CONTRAST = 4.5; @@ -358,6 +370,7 @@ async function inspectDom(page: import('puppeteer').Page): Promise<{ if (!m) return null; const parts = m[1].split(',').map((p) => parseFloat(p.trim())); if (parts.length < 3 || parts.some((n) => !Number.isFinite(n))) return null; + if (parts.length >= 4 && parts[3] < 0.95) return null; return [parts[0], parts[1], parts[2]]; }; const luminance = (rgb: [number, number, number]): number => { @@ -412,6 +425,97 @@ async function inspectDom(page: import('puppeteer').Page): Promise<{ } } } + const wrappers = Array.from(document.querySelectorAll('[data-seqvio-annotation-id]')); + const labels: Array<{ id: string; rect: DOMRect }> = []; + for (const wrapper of wrappers) { + const id = wrapper.dataset.seqvioAnnotationId ?? 'unknown'; + if (wrapper.dataset.seqvioAnnotationTargetResolved === 'false') { + attentionMissingTargets.push(id); + continue; + } + const candidates = Array.from(wrapper.querySelectorAll('div,text')) + .filter((element) => (element.textContent ?? '').trim().length > 0); + for (const label of candidates) { + const rect = label.getBoundingClientRect(); + if (rect.width <= 0 || rect.height <= 0) continue; + labels.push({ id, rect }); + if (rect.left < 0 || rect.top < 0 || rect.right > viewport.width || rect.bottom > viewport.height) { + attentionOffscreenLabels.push(id); + } + } + } + for (let leftIndex = 0; leftIndex < labels.length; leftIndex++) { + for (let rightIndex = leftIndex + 1; rightIndex < labels.length; rightIndex++) { + const left = labels[leftIndex]; + const right = labels[rightIndex]; + if (left.id !== right.id && left.rect.left < right.rect.right && left.rect.right > right.rect.left && left.rect.top < right.rect.bottom && left.rect.bottom > right.rect.top) { + attentionLabelCollisions.push(`${left.id}:${right.id}`); + } + } + } + const targets = Array.from(document.querySelectorAll('[data-annotation-target]')); + for (const label of labels) { + const ownTarget = wrappers.find((wrapper) => wrapper.dataset.seqvioAnnotationId === label.id)?.dataset.seqvioAnnotationTargetId; + for (const target of targets) { + const targetId = target.dataset.annotationTarget; + if (!targetId || targetId === ownTarget) continue; + const rect = target.getBoundingClientRect(); + if (label.rect.left < rect.right && label.rect.right > rect.left && label.rect.top < rect.bottom && label.rect.bottom > rect.top) { + attentionTargetOcclusions.push(`${label.id}:${targetId}`); + } + } + } + const productRoot = document.querySelector('[data-seqvio-product-explainer]'); + const productFrame = productRoot ? (() => { + const visible = (element: Element): element is HTMLElement => { + const rect = element.getBoundingClientRect(); + const style = window.getComputedStyle(element); + return rect.width > 0 && rect.height > 0 && style.visibility !== 'hidden' && style.display !== 'none' && Number(style.opacity) > 0; + }; + const primary = Array.from(productRoot.querySelectorAll('h1,h2,[data-seqvio-text-role="primary"]')).filter(visible); + const evidenceSelector = '[data-seqvio-evidence-source="terminal-capture"],[data-seqvio-evidence-source="browser-capture"],[data-seqvio-evidence-source="recorded-media"]'; + const fullSentenceOverlayIds = primary.filter((element) => { + if (element.closest(evidenceSelector)) return false; + const text = (element.textContent ?? '').trim(); + const wordCount = text.split(/\s+/).filter(Boolean).length; + return wordCount >= 12 && /[.!?。！？]/.test(text); + }).map((element, index) => element.dataset.seqvioTextId ?? element.id ?? `primary-${index + 1}`); + const templateCounts = new Map(); + for (const element of Array.from(productRoot.querySelectorAll('[data-seqvio-template]')).filter(visible)) { + const id = element.dataset.seqvioTemplate; + if (id) templateCounts.set(id, (templateCounts.get(id) ?? 0) + 1); + } + const graphics = Array.from(productRoot.querySelectorAll('[data-seqvio-visual-role="graphic"]')).filter(visible); + const titleGraphicOverlaps: string[] = []; + primary.forEach((title, titleIndex) => graphics.forEach((graphic, graphicIndex) => { + const a = title.getBoundingClientRect(); + const b = graphic.getBoundingClientRect(); + if (a.left < b.right && a.right > b.left && a.top < b.bottom && a.bottom > b.top) { + titleGraphicOverlaps.push(`${title.dataset.seqvioTextId ?? title.id ?? `title-${titleIndex + 1}`}:${graphic.id || `graphic-${graphicIndex + 1}`}`); + } + })); + const declaredBudget = Number(productRoot.dataset.seqvioOnScreenTextBudget); + return { + fullSentenceOverlayIds, + primaryTextCount: primary.length, + primaryTextBudget: Number.isFinite(declaredBudget) && declaredBudget > 0 ? declaredBudget : 1, + repeatedTemplateIds: [...templateCounts.entries()].filter(([, count]) => count > 1).map(([id]) => id), + titleGraphicOverlaps: [...new Set(titleGraphicOverlaps)], + hasFocalTarget: Array.from(productRoot.querySelectorAll('[data-seqvio-focal-target]')).some(visible), + }; + })() : undefined; + const manimFrames = Array.from(document.querySelectorAll('[data-seqvio-manim-clip]')).map((clip) => { + const video = clip.querySelector('video'); + return { + id: clip.dataset.seqvioManimClip ?? 'unknown', + frame: Number(clip.dataset.seqvioManimFrame ?? 0), + fps: Number(video?.dataset.seqvioMediaFps ?? 30), + currentTime: video?.currentTime ?? 0, + duration: video && Number.isFinite(video.duration) ? video.duration : undefined, + marker: clip.dataset.seqvioManimMarker || undefined, + markerCount: Number(clip.dataset.seqvioManimMarkerCount ?? 0), + }; + }); return { elementCount: elements.length, bodyTextLength: document.body.innerText.trim().length, @@ -420,6 +524,12 @@ async function inspectDom(page: import('puppeteer').Page): Promise<{ textOverflowCount, smallFontCount, lowContrastCount, + attentionMissingTargets: [...new Set(attentionMissingTargets)], + attentionOffscreenLabels: [...new Set(attentionOffscreenLabels)], + attentionLabelCollisions: [...new Set(attentionLabelCollisions)], + attentionTargetOcclusions: [...new Set(attentionTargetOcclusions)], + productFrame, + manimFrames, }; }); } @@ -585,6 +695,12 @@ async function main(): Promise { repair: 'Increase foreground/background contrast to at least 4.5:1.', }); } + for (const id of dom.attentionMissingTargets) issues.push({ severity: 'error', code: 'attention_target_missing', path: `attention.${id}`, frame: sourceFrame, message: `Attention item "${id}" has no measurable target at frame ${sourceFrame}.`, repair: 'Use a stable target id that exists and is visible during the attention window.' }); + for (const id of dom.attentionOffscreenLabels) issues.push({ severity: 'warning', code: 'attention_label_offscreen', path: `attention.${id}`, frame: sourceFrame, message: `Attention label "${id}" extends outside the viewport.`, repair: 'Use safe-area label placement or move the target away from the edge.' }); + for (const pair of dom.attentionLabelCollisions) issues.push({ severity: 'warning', code: 'attention_label_collision', path: `attention.${pair}`, frame: sourceFrame, message: `Attention labels "${pair}" overlap.`, repair: 'Change label placement, shorten copy, or serialize the attention items.' }); + for (const pair of dom.attentionTargetOcclusions) issues.push({ severity: 'warning', code: 'attention_target_occluded', path: `attention.${pair}`, frame: sourceFrame, message: `Attention label "${pair.split(':')[0]}" occludes target "${pair.split(':')[1]}".`, repair: 'Move the label to another safe candidate or reroute its leader.' }); + issues.push(...diagnoseProductFrame(dom.productFrame, sourceFrame)); + for (const manimFrame of dom.manimFrames) issues.push(...diagnoseManimFrame(manimFrame, sourceFrame)); // narration/visual agreement: the current cue's keywords should overlap // with the visible text. Catches the common AI-video failure where the @@ -595,10 +711,13 @@ async function main(): Promise { (c) => frameTimeMs >= (c.startMs ?? 0) && frameTimeMs < (c.endMs ?? Infinity) ); if (currentCue && dom.bodyText) { - const cueWords = currentCue.text + const latinWords = currentCue.text .toLowerCase() .split(/\s+/) - .filter((w) => w.length > 3 && /[a-z一-龥]/.test(w)); + .filter((word) => word.length > 3 && /[a-z]/.test(word)); + const cjkChunks = currentCue.text.match(/[一-龥]{2,}/g) ?? []; + const cjkBigrams = cjkChunks.flatMap((chunk) => Array.from({ length: Math.max(0, chunk.length - 1) }, (_, index) => chunk.slice(index, index + 2))); + const cueWords = [...latinWords, ...cjkBigrams]; const bodyLower = dom.bodyText.toLowerCase(); const matched = cueWords.filter((w) => bodyLower.includes(w)); if (cueWords.length > 0 && matched.length === 0) { diff --git a/packages/renderer/src/qa-diagnostics.ts b/packages/renderer/src/qa-diagnostics.ts index 76b1f8c..0c7e3fa 100644 --- a/packages/renderer/src/qa-diagnostics.ts +++ b/packages/renderer/src/qa-diagnostics.ts @@ -12,6 +12,64 @@ export interface QaDiagnostic { repair?: string; } +export interface ProductFrameObservation { + fullSentenceOverlayIds: string[]; + primaryTextCount: number; + primaryTextBudget: number; + repeatedTemplateIds: string[]; + titleGraphicOverlaps: string[]; + hasFocalTarget: boolean; +} + +export interface ManimFrameObservation { + id: string; + frame: number; + fps: number; + currentTime: number; + duration?: number; + marker?: string; + markerCount: number; +} + +export function diagnoseManimFrame(observation: ManimFrameObservation, frame: number): QaDiagnostic[] { + const issues: QaDiagnostic[] = []; + const rawExpectedTime = observation.frame / Math.max(1, observation.fps); + const expectedTime = observation.duration && Number.isFinite(observation.duration) + ? Math.min(rawExpectedTime, observation.duration) + : rawExpectedTime; + if (Math.abs(observation.currentTime - expectedTime) > Math.max(0.08, 2 / Math.max(1, observation.fps))) { + issues.push({ severity: 'warning', code: 'manim_seek_misaligned', path: `manim.${observation.id}`, frame, message: `Manim clip "${observation.id}" is at ${observation.currentTime.toFixed(3)}s; expected ${expectedTime.toFixed(3)}s.`, repair: 'Verify the clip fps metadata and retain a seekable MP4 output from the adapter.' }); + } + if (observation.markerCount > 0 && observation.frame > 0 && !observation.marker) { + issues.push({ severity: 'warning', code: 'manim_marker_unresolved', path: `manim.${observation.id}.marker`, frame, message: `Manim clip "${observation.id}" has named markers but none resolves at this frame.`, repair: 'Add a frame-zero marker or align the first marker to the narration beat that introduces the clip.' }); + } + return issues; +} + +export function diagnoseProductFrame( + observation: ProductFrameObservation | undefined, + frame: number, +): QaDiagnostic[] { + if (!observation) return []; + const issues: QaDiagnostic[] = []; + for (const id of observation.fullSentenceOverlayIds) { + issues.push({ severity: 'warning', code: 'full_sentence_overlay', path: `productText.${id}`, frame, message: `Primary overlay "${id}" reads like narration rather than concise screen text.`, repair: 'Keep the sentence in narration and reduce the overlay to a label, keyword, command, filename, or short conclusion.' }); + } + if (observation.primaryTextCount > observation.primaryTextBudget) { + issues.push({ severity: 'warning', code: 'concurrent_primary_text', path: 'productText.primary', frame, message: `${observation.primaryTextCount} primary text elements exceed the frame budget of ${observation.primaryTextBudget}.`, repair: 'Choose one primary message and demote or sequence the remaining text.' }); + } + for (const id of observation.repeatedTemplateIds) { + issues.push({ severity: 'warning', code: 'repeated_scene_template', path: `productTemplate.${id}`, frame, message: `Template "${id}" is repeated concurrently.`, repair: 'Vary the composition around the visual role instead of duplicating the same header or rail.' }); + } + for (const pair of observation.titleGraphicOverlaps) { + issues.push({ severity: 'warning', code: 'title_graphic_overlap', path: `productLayout.${pair}`, frame, message: `Primary title and graphic "${pair}" overlap.`, repair: 'Reserve separate title and graphic regions or reduce the text footprint.' }); + } + if (!observation.hasFocalTarget) { + issues.push({ severity: 'warning', code: 'missing_focal_target', path: 'productLayout.focalTarget', frame, message: 'The product-explainer frame has no visible declared focal target.', repair: 'Mark the single element carrying this beat with data-seqvio-focal-target.' }); + } + return issues; +} + export function promoteQaWarnings(issues: QaDiagnostic[], codes: ReadonlySet): QaDiagnostic[] { return issues.map((issue) => issue.severity === 'warning' && (codes.has('*') || codes.has(issue.code)) diff --git a/packages/renderer/tests/doctor.test.mjs b/packages/renderer/tests/doctor.test.mjs index 291262e..7ab84ed 100644 --- a/packages/renderer/tests/doctor.test.mjs +++ b/packages/renderer/tests/doctor.test.mjs @@ -1,7 +1,7 @@ import assert from 'node:assert/strict'; import test from 'node:test'; -import { doctorExitCode, supportsNodeVersion } from '../dist/doctor.js'; +import { doctorExitCode, resolveManimPythonCommand, supportsNodeVersion } from '../dist/doctor.js'; test('doctor enforces the supported Node.js floor', () => { assert.equal(supportsNodeVersion('v18.0.0'), true); @@ -14,3 +14,7 @@ test('doctor exit code follows blocking failures', () => { assert.equal(doctorExitCode({ ok: true }), 0); assert.equal(doctorExitCode({ ok: false }), 1); }); + +test('doctor honors an explicit Manim Python command', () => { + assert.equal(resolveManimPythonCommand(process.cwd(), 'custom-python'), 'custom-python'); +}); diff --git a/packages/renderer/tests/qa-diagnostics.test.mjs b/packages/renderer/tests/qa-diagnostics.test.mjs index 6848a6e..eca5d57 100644 --- a/packages/renderer/tests/qa-diagnostics.test.mjs +++ b/packages/renderer/tests/qa-diagnostics.test.mjs @@ -1,7 +1,7 @@ import assert from 'node:assert/strict'; import test from 'node:test'; -import { classifyQaRuntimeError, diagnosePacing, expectedNarrationTrackDurationMs, promoteQaWarnings } from '../dist/qa-diagnostics.js'; +import { classifyQaRuntimeError, diagnoseManimFrame, diagnosePacing, diagnoseProductFrame, expectedNarrationTrackDurationMs, promoteQaWarnings } from '../dist/qa-diagnostics.js'; import { reflowSynthesizedTimeline, resolveSynthesizedCueTiming, validateAudioManifest } from '../dist/audio/manifest.js'; test('audio validation emits stable codes for invalid narration and captions', () => { @@ -206,6 +206,27 @@ test('warning promotion is selective and preserves other diagnostics', () => { assert.equal(issues[1].severity, 'warning'); }); +test('product-frame diagnostics expose stable production-contract codes', () => { + const issues = diagnoseProductFrame({ + fullSentenceOverlayIds: ['sentence'], primaryTextCount: 3, primaryTextBudget: 1, + repeatedTemplateIds: ['header-rail'], titleGraphicOverlaps: ['title:graphic'], hasFocalTarget: false, + }, 42); + assert.deepEqual(issues.map((issue) => issue.code), [ + 'full_sentence_overlay', 'concurrent_primary_text', 'repeated_scene_template', + 'title_graphic_overlap', 'missing_focal_target', + ]); + assert.ok(issues.every((issue) => issue.frame === 42 && issue.repair)); +}); + +test('Manim diagnostics report seek and marker alignment failures', () => { + const issues = diagnoseManimFrame({ id: 'proof', frame: 60, fps: 30, currentTime: 0.5, markerCount: 2 }, 60); + assert.deepEqual(issues.map((issue) => issue.code), ['manim_seek_misaligned', 'manim_marker_unresolved']); +}); + +test('Manim seek diagnostics accept a clip held on its final media frame', () => { + assert.deepEqual(diagnoseManimFrame({ id: 'graph', frame: 180, fps: 30, currentTime: 4.1, duration: 4.1, marker: 'result', markerCount: 2 }, 180), []); +}); + test('pacing diagnostics report fast speech and short highlights', () => { const issues = diagnosePacing({ audio: { diff --git a/packages/technical/src/InfographicScene.tsx b/packages/technical/src/InfographicScene.tsx new file mode 100644 index 0000000..1a36786 --- /dev/null +++ b/packages/technical/src/InfographicScene.tsx @@ -0,0 +1,256 @@ +import React, { useMemo } from 'react'; +import { AnnotationTarget, AttentionSequenceLayer, useCurrentFrame, useStyleProfile, type AttentionSequenceItem } from '@seqvio/core'; +import { ease } from './anim'; +import { technicalFonts, technicalPalette } from './theme'; + +export interface InfographicMetric { + id: string; + label: string; + value: string; + detail?: string; + color?: string; + at?: number; +} + +export interface InfographicComparison { + id: string; + label: string; + before: number; + after: number; + beforeLabel?: string; + afterLabel?: string; + at?: number; +} + +export interface InfographicProcessStep { + id: string; + label: string; + detail?: string; + at?: number; +} + +export interface InfographicTimelineEvent { + id: string; + label: string; + detail?: string; + at?: number; +} + +export interface InfographicRelationshipNode { + id: string; + label: string; + x: number; + y: number; +} + +export interface InfographicRelationship { + id: string; + from: string; + to: string; + label?: string; + at?: number; +} + +export interface InfographicChartPoint { x: string; y: number; } +export interface InfographicChartSeries { id: string; label: string; color?: string; points: InfographicChartPoint[]; } +export interface InfographicChartAxis { label?: string; min?: number; max?: number; ticks?: number; } +export interface InfographicChart { + id: string; + title: string; + kind: 'bar' | 'line'; + series: InfographicChartSeries[]; + xAxis?: InfographicChartAxis; + yAxis?: InfographicChartAxis; + legend?: 'none' | 'top' | 'bottom'; + unit?: string; + sourceLabel?: string; + at?: number; +} + +export interface InfographicSceneProps { + id: string; + title?: string; + density?: 'auto' | 'standard' | 'reduced'; + width?: number; + height?: number; + metrics?: InfographicMetric[]; + comparisons?: InfographicComparison[]; + process?: InfographicProcessStep[]; + timeline?: InfographicTimelineEvent[]; + relationshipNodes?: InfographicRelationshipNode[]; + relationships?: InfographicRelationship[]; + charts?: InfographicChart[]; + attention?: AttentionSequenceItem[]; +} + +export function infographicProgress(frame: number, at = 0, duration = 18): number { + return ease((frame - at) / Math.max(1, duration)); +} + +function revealStyle(frame: number, at: number | undefined, index: number, duration = 18): React.CSSProperties { + const progress = infographicProgress(frame, at ?? index * 10, duration); + return { + opacity: progress, + transform: `translateY(${(1 - progress) * 10}px)`, + }; +} + +const ItemTarget: React.FC<{ + id: string; + frame: number; + at?: number; + index: number; + children: React.ReactNode; + style?: React.CSSProperties; + revealDuration?: number; +}> = ({ id, frame, at, index, children, style, revealDuration }) => ( + + {children} + +); + +export function resolveChartDomain(chart: InfographicChart): { min: number; max: number } { + const values = chart.series.flatMap((series) => series.points.map((point) => point.y)); + const observedMin = values.length ? Math.min(...values) : 0; + const observedMax = values.length ? Math.max(...values) : 1; + const min = chart.yAxis?.min ?? Math.min(0, observedMin); + const requestedMax = chart.yAxis?.max ?? observedMax; + return { min, max: requestedMax > min ? requestedMax : min + 1 }; +} + +const CHART_COLORS = ['#4f8cff', '#21a179', '#f0a43c', '#d7658b']; + +const ChartView: React.FC<{ chart: InfographicChart; frame: number; index: number; revealDuration: number }> = ({ chart, frame, index, revealDuration }) => { + const progress = infographicProgress(frame, chart.at ?? index * 12, revealDuration + 6); + const domain = resolveChartDomain(chart); + const categories = [...new Set(chart.series.flatMap((series) => series.points.map((point) => point.x)))]; + const plot = { left: 58, top: 32, width: 424, height: 224 }; + const scaleY = (value: number) => plot.top + plot.height - ((value - domain.min) / (domain.max - domain.min)) * plot.height; + const ticks = Math.max(2, chart.yAxis?.ticks ?? 4); + const legend = chart.legend ?? 'top'; + return ( + +
+
{chart.title}
+ {chart.unit ?
Unit: {chart.unit}
: null} +
+ {legend === 'top' ? : null} + + {Array.from({ length: ticks + 1 }, (_, tick) => { + const ratio = tick / ticks; + const y = plot.top + plot.height * ratio; + const value = domain.max - (domain.max - domain.min) * ratio; + return {Number(value.toFixed(1))}; + })} + + + {categories.map((category, categoryIndex) => { + const x = plot.left + ((categoryIndex + 0.5) / Math.max(1, categories.length)) * plot.width; + return {category}; + })} + {chart.kind === 'bar' ? chart.series.map((series, seriesIndex) => { + const color = series.color ?? CHART_COLORS[seriesIndex % CHART_COLORS.length]; + const categoryWidth = plot.width / Math.max(1, categories.length); + const barWidth = Math.min(34, categoryWidth * 0.72 / Math.max(1, chart.series.length)); + return {series.points.map((point) => { + const categoryIndex = categories.indexOf(point.x); + const center = plot.left + (categoryIndex + 0.5) * categoryWidth; + const x = center - (chart.series.length * barWidth) / 2 + seriesIndex * barWidth; + const top = scaleY(domain.min + (point.y - domain.min) * progress); + return ; + })}; + }) : chart.series.map((series, seriesIndex) => { + const color = series.color ?? CHART_COLORS[seriesIndex % CHART_COLORS.length]; + const path = series.points.map((point, pointIndex) => { + const categoryIndex = categories.indexOf(point.x); + const x = plot.left + ((categoryIndex + 0.5) / Math.max(1, categories.length)) * plot.width; + const y = scaleY(point.y); + return `${pointIndex === 0 ? 'M' : 'L'} ${x} ${y}`; + }).join(' '); + return {series.points.map((point) => { const categoryIndex = categories.indexOf(point.x); const x = plot.left + ((categoryIndex + 0.5) / Math.max(1, categories.length)) * plot.width; return ; })}; + })} + {chart.xAxis?.label ? {chart.xAxis.label} : null} + {chart.yAxis?.label ? {chart.yAxis.label} : null} + + {legend === 'bottom' ? : null} + {chart.sourceLabel ?
Source: {chart.sourceLabel}
: null} + + ); +}; + +const ChartLegend: React.FC<{ chart: InfographicChart }> = ({ chart }) => ( +
+ {chart.series.map((series, index) =>
{series.label}
)} +
+); + +export const InfographicScene: React.FC = ({ + id, + title = 'Explanation', + density = 'auto', + width = 1280, + height = 720, + metrics = [], + comparisons = [], + process = [], + timeline = [], + relationshipNodes = [], + relationships = [], + charts = [], + attention = [], +}) => { + const frame = useCurrentFrame(); + const styleProfile = useStyleProfile(); + const nodeMap = useMemo(() => new Map(relationshipNodes.map((node) => [node.id, node])), [relationshipNodes]); + const portrait = width / height < 0.82; + const square = width / height >= 0.82 && width / height < 1.35; + const reduced = density === 'reduced' || (density === 'auto' && width / height < 1.35); + const spacingScale = styleProfile?.spacing === 'tight' ? 0.82 : styleProfile?.spacing === 'airy' ? 1.16 : 1; + const padding = Math.round((portrait ? 24 : square ? 30 : 40) * spacingScale); + const metricColumns = portrait ? 1 : square ? 2 : 3; + const revealDuration = styleProfile?.motionDensity === 'restrained' ? 12 : styleProfile?.motionDensity === 'expressive' ? 24 : 18; + const titleScale = styleProfile?.typography.scale === 'compact' ? 0.86 : styleProfile?.typography.scale === 'large' ? 1.15 : 1; + + return ( + +
+
{title}
+ {charts.length > 0 ?
1 && !reduced ? 'repeat(2, minmax(0, 1fr))' : 'minmax(0, 1fr)', gap: (reduced ? 14 : 20) * spacingScale }}>{charts.map((chart, index) => )}
: null} +
+ {metrics.map((metric, index) => ( + +
{metric.value}
+
{metric.label}
+ {metric.detail && !reduced ?
{metric.detail}
: null} + + ))} +
+
+ {comparisons.map((comparison, index) => { + const progress = infographicProgress(frame, comparison.at ?? index * 10); + const max = Math.max(comparison.before, comparison.after, 1); + return ( + +
{comparison.label}
+ {[['before', comparison.before, comparison.beforeLabel ?? 'Before', technicalPalette.muted], ['after', comparison.after, comparison.afterLabel ?? 'After', technicalPalette.success]].map(([key, value, label, color]) => ( +
+
{label as string}
+
+
+
+
{value as number}
+
+ ))} + + ); + })} +
+ {process.length > 0 ?
{process.map((step, index) =>
{step.label}
{step.detail && !reduced ?
{step.detail}
: null})}
: null} + {timeline.length > 0 ?
{timeline.map((event, index) =>
{event.label}
{event.detail ?
{event.detail}
: null})}
: null} + {relationshipNodes.length > 0 ? {relationships.map((relationship, index) => { const from = nodeMap.get(relationship.from); const to = nodeMap.get(relationship.to); if (!from || !to) return null; const progress = infographicProgress(frame, relationship.at ?? index * 10); return {relationship.label ?? ''}; })} : null} + {relationshipNodes.map((node, index) => {node.label})} + {attention.length > 0 ? : null} +
+ + ); +}; diff --git a/packages/technical/src/ManimClip.tsx b/packages/technical/src/ManimClip.tsx new file mode 100644 index 0000000..adf42b9 --- /dev/null +++ b/packages/technical/src/ManimClip.tsx @@ -0,0 +1,70 @@ +import React from 'react'; +import { AnnotationTarget, useComposition, useCurrentFrame, type ExplanationBeatTiming } from '@seqvio/core'; + +export interface ManimMarker { + id: string; + frame: number; + targetId?: string; + beatId?: string; +} + +export interface ManimClipProps { + id: string; + src: string; + width?: number; + height?: number; + fps?: number; + markers?: ManimMarker[]; + background?: string; + fit?: 'contain' | 'cover' | 'fill'; +} + +export function markerAtFrame(markers: ManimMarker[], frame: number): ManimMarker | undefined { + return [...markers].sort((a, b) => a.frame - b.frame).filter((marker) => marker.frame <= frame).at(-1); +} + +export function resolveManimMarkerFrames(markers: ManimMarker[], beats: ExplanationBeatTiming[] = [], sceneId?: string): ManimMarker[] { + return markers.map((marker) => { + if (!marker.beatId) return marker; + const qualified = sceneId ? `${sceneId}.${marker.beatId}` : marker.beatId; + const beat = beats.find((item) => item.id === qualified || item.id === marker.beatId); + return beat ? { ...marker, frame: beat.outputFrame ?? beat.sourceFrame } : marker; + }); +} + +export function ManimClip({ + id, src, width = 1280, height = 720, fps = 30, markers = [], + background = '#0f172a', fit = 'contain', +}: ManimClipProps) { + const frame = useCurrentFrame(); + const { config } = useComposition(); + const resolvedMarkers = resolveManimMarkerFrames(markers, config.audio?.explanationBeats, id); + const marker = markerAtFrame(resolvedMarkers, frame); + return ( + +
+
+ + ); +} diff --git a/packages/technical/src/index.ts b/packages/technical/src/index.ts index 6fb2f23..c94ffec 100644 --- a/packages/technical/src/index.ts +++ b/packages/technical/src/index.ts @@ -1,6 +1,24 @@ export { TechnicalScene } from './TechnicalScene'; export type { TechnicalSceneProps } from './TechnicalScene'; +export { InfographicScene, infographicProgress, resolveChartDomain } from './InfographicScene'; +export type { + InfographicSceneProps, + InfographicMetric, + InfographicComparison, + InfographicProcessStep, + InfographicTimelineEvent, + InfographicRelationshipNode, + InfographicRelationship, + InfographicChart, + InfographicChartPoint, + InfographicChartSeries, + InfographicChartAxis, +} from './InfographicScene'; + +export { ManimClip, markerAtFrame, resolveManimMarkerFrames } from './ManimClip'; +export type { ManimClipProps, ManimMarker } from './ManimClip'; + export { AnnotationLayer, AnnotationProvider, diff --git a/packages/technical/src/theme.ts b/packages/technical/src/theme.ts index c77fd13..c298c6f 100644 --- a/packages/technical/src/theme.ts +++ b/packages/technical/src/theme.ts @@ -1,25 +1,25 @@ import { programmingMonoFontStack } from './fonts'; export const technicalPalette = { - canvas: '#0f172a', - surface: '#111827', - panel: '#1e293b', - ink: '#e2e8f0', - muted: '#94a3b8', - accent: '#38bdf8', - accentSoft: 'rgba(56, 189, 248, 0.18)', + canvas: 'var(--seqvio-color-background, #0f172a)', + surface: 'color-mix(in srgb, var(--seqvio-color-background, #0f172a) 92%, var(--seqvio-color-ink, #e2e8f0))', + panel: 'color-mix(in srgb, var(--seqvio-color-background, #0f172a) 84%, var(--seqvio-color-ink, #e2e8f0))', + ink: 'var(--seqvio-color-ink, #e2e8f0)', + muted: 'var(--seqvio-color-muted, #94a3b8)', + accent: 'var(--seqvio-color-accent, #38bdf8)', + accentSoft: 'color-mix(in srgb, var(--seqvio-color-accent, #38bdf8) 18%, transparent)', success: '#34d399', warning: '#fbbf24', danger: '#f87171', - line: 'rgba(148, 163, 184, 0.35)', + line: 'color-mix(in srgb, var(--seqvio-color-muted, #94a3b8) 35%, transparent)', codeBg: '#0b1220', gutter: '#64748b', }; export const technicalFonts = { /** Prefer installed coding fonts, then generic monospace. */ - mono: programmingMonoFontStack(), - sans: 'Inter, "Segoe UI", system-ui, sans-serif', + mono: `var(--seqvio-font-mono, ${programmingMonoFontStack()})`, + sans: 'var(--seqvio-font-body, Inter, "Segoe UI", system-ui, sans-serif)', }; export const technicalCodeTheme = { diff --git a/packages/technical/tests/manim-clip.test.mjs b/packages/technical/tests/manim-clip.test.mjs new file mode 100644 index 0000000..967b519 --- /dev/null +++ b/packages/technical/tests/manim-clip.test.mjs @@ -0,0 +1,18 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { markerAtFrame, resolveManimMarkerFrames } from '../dist/index.js'; + +describe('ManimClip markers', () => { + it('resolves named markers deterministically for direct and reverse seek', () => { + const markers = [{ id: 'start', frame: 0 }, { id: 'proof', frame: 30 }, { id: 'result', frame: 60 }]; + assert.equal(markerAtFrame(markers, 45).id, 'proof'); + assert.equal(markerAtFrame(markers, 15).id, 'start'); + assert.equal(markerAtFrame(markers, 60).id, 'result'); + }); + + it('reflows a named marker from the resolved narration beat', () => { + const markers = [{ id: 'result', frame: 60, beatId: 'result-beat' }]; + const beats = [{ id: 'equation.result-beat', sceneId: 'equation', sourceFrame: 60, outputFrame: 92 }]; + assert.equal(resolveManimMarkerFrames(markers, beats, 'equation')[0].frame, 92); + }); +}); diff --git a/packages/technical/tests/technical.test.mjs b/packages/technical/tests/technical.test.mjs index d1a9019..e976f21 100644 --- a/packages/technical/tests/technical.test.mjs +++ b/packages/technical/tests/technical.test.mjs @@ -11,6 +11,7 @@ import { collapsedGroupsAt, layoutDiagram, } from '../dist/diagram-layout.js'; +import { infographicProgress, resolveChartDomain } from '../dist/InfographicScene.js'; describe('highlightLine / highlightSource (Shiki)', () => { it('colors keywords, types, and strings via Shiki', () => { @@ -142,3 +143,18 @@ describe('layoutDiagram', () => { assert.strictEqual(collapsed.nodes.length, 2); }); }); + +describe('InfographicScene timing', () => { + it('keeps semantic items hidden before their cue and seek-safe afterwards', () => { + assert.strictEqual(infographicProgress(9, 10), 0); + assert.ok(infographicProgress(19, 10) > 0); + assert.strictEqual(infographicProgress(40, 10), 1); + assert.strictEqual(infographicProgress(19, 10), infographicProgress(19, 10)); + }); + + it('resolves explicit and data-derived chart domains deterministically', () => { + const chart = { id: 'trend', title: 'Trend', kind: 'line', series: [{ id: 'a', label: 'A', points: [{ x: 'one', y: 4 }, { x: 'two', y: 9 }] }] }; + assert.deepStrictEqual(resolveChartDomain(chart), { min: 0, max: 9 }); + assert.deepStrictEqual(resolveChartDomain({ ...chart, yAxis: { min: 2, max: 12 } }), { min: 2, max: 12 }); + }); +}); diff --git a/packages/terminal-narrator/README.md b/packages/terminal-narrator/README.md index d97b323..fc66912 100644 --- a/packages/terminal-narrator/README.md +++ b/packages/terminal-narrator/README.md @@ -4,7 +4,7 @@ 状态回放 → `ExplainerDocument` → ExplanationBeat 音画对齐 → 可选 TTS 旁白 → QA / MP4。 -当前 CLI 属于 pre-stable；生产管线已经使用 manifest → shared capture +当前 CLI 的 capture contract 已稳定；生产管线已经使用 manifest → shared capture dispatcher → IR → TSX，legacy `writeComposition` writer 已移除。 **范围**：机制 1（外壳调教 Claude Code）+ 机制 3（日志回放渲染）。**不含** MCP（机制 2）。 diff --git a/packages/terminal-narrator/package.json b/packages/terminal-narrator/package.json index 9a9d5b9..6094180 100644 --- a/packages/terminal-narrator/package.json +++ b/packages/terminal-narrator/package.json @@ -1,7 +1,7 @@ { "name": "@seqvio/terminal-narrator", "version": "0.4.0", - "description": "Pre-stable terminal capture-to-explanation adapter for Seqvio", + "description": "Experimental package with a stable terminal capture-to-explanation adapter for Seqvio", "main": "dist/index.js", "types": "dist/index.d.ts", "bin": { @@ -10,7 +10,7 @@ }, "seqvio": { "lifecycle": "experimental", - "adapterLifecycle": "pre-stable", + "adapterLifecycle": "stable", "versionPolicy": "independent", "cliContractVersion": "2.0", "artifactManifestVersion": "1.0" diff --git a/scripts/capture-host-runtime-smoke.mjs b/scripts/capture-host-runtime-smoke.mjs index ed734e3..f3d52b0 100644 --- a/scripts/capture-host-runtime-smoke.mjs +++ b/scripts/capture-host-runtime-smoke.mjs @@ -4,18 +4,28 @@ import * as http from 'node:http'; import * as os from 'node:os'; import * as path from 'node:path'; -function runNode(args, env = {}) { - return new Promise((resolve, reject) => { - const child = spawn(process.execPath, args, { - cwd: process.cwd(), - stdio: 'inherit', - env: { ...process.env, ...env }, - }); - child.once('error', reject); - child.once('exit', (code) => code === 0 - ? resolve() - : reject(new Error(`node ${args.join(' ')} failed with exit code ${code}`))); - }); +async function runNode(args, env = {}, attempts = 1) { + let lastError; + for (let attempt = 1; attempt <= attempts; attempt += 1) { + try { + await new Promise((resolve, reject) => { + const child = spawn(process.execPath, args, { + cwd: process.cwd(), + stdio: 'inherit', + env: { ...process.env, ...env }, + }); + child.once('error', reject); + child.once('exit', (code) => code === 0 + ? resolve() + : reject(new Error(`node ${args.join(' ')} failed with exit code ${code}`))); + }); + return; + } catch (error) { + lastError = error instanceof Error ? error : new Error(String(error)); + if (attempt < attempts) await new Promise((resolve) => setTimeout(resolve, 1_000)); + } + } + throw lastError; } function assertComplete(jobDir) { @@ -64,16 +74,17 @@ try { captureFps: quick ? 5 : 10, renderFps: quick ? 10 : 30, actions: [ - { id: 'run', type: 'click', label: 'Run', selector: '#run', afterMs: quick ? 50 : 250 }, + { id: 'run', type: 'click', label: 'Run', selector: '#run', afterMs: quick ? 500 : 250 }, ], }, null, 2)}\n`, 'utf8'); if (requestedKind === 'all' || requestedKind === 'terminal') { + const terminalJobId = process.platform === 'win32' && quick ? 'terminal-quick' : 'terminal'; await runNode([ 'packages/terminal-narrator/dist/cli.js', 'record', '--sample', - '--outputDir', tempRoot, '--jobId', 'terminal', '--json', - ], quick ? { SEQVIO_CAPTURE_SMOKE_PROFILE: 'quick' } : {}); - assertComplete(path.join(tempRoot, 'terminal')); + '--outputDir', tempRoot, '--jobId', terminalJobId, '--json', + ], quick ? { SEQVIO_CAPTURE_SMOKE_PROFILE: 'quick' } : {}, process.platform === 'win32' && quick ? 2 : 1); + assertComplete(path.join(tempRoot, terminalJobId)); } if (requestedKind === 'all' || requestedKind === 'browser') { await runNode([ diff --git a/scripts/director-pass.mjs b/scripts/director-pass.mjs new file mode 100644 index 0000000..3937274 --- /dev/null +++ b/scripts/director-pass.mjs @@ -0,0 +1,27 @@ +import { createHash } from 'node:crypto'; +import { readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { deriveDirectionPlan, validateDirectionPlan } from '../packages/core/dist/index.js'; + +const args = new Map(); +for (let i = 2; i < process.argv.length; i += 2) args.set(process.argv[i], process.argv[i + 1]); +const input = args.get('--ir'); +const output = args.get('--out'); +const receiptPath = args.get('--receipt') ?? `${output}.receipt.json`; +if (!input || !output) { + console.error('Usage: node scripts/director-pass.mjs --ir --out [--receipt ]'); + process.exit(2); +} +const inputPath = path.resolve(input); +const outputPath = path.resolve(output); +const source = await readFile(inputPath, 'utf8'); +const document = JSON.parse(source); +const plan = deriveDirectionPlan(document); +const issues = validateDirectionPlan(plan, document); +const errors = issues.filter((item) => item.severity === 'error'); +const inputHash = createHash('sha256').update(source).digest('hex'); +const receipt = { format: 'seqvio-director-receipt', version: '1.0', operation: 'derive-direction-plan', input: inputPath, inputHash, output: outputPath, planId: plan.id, segmentCount: plan.segments.length, issues, status: errors.length ? 'failed' : 'complete' }; +if (errors.length) { await writeFile(receiptPath, `${JSON.stringify(receipt, null, 2)}\n`, 'utf8'); console.error(JSON.stringify(receipt, null, 2)); process.exit(1); } +await writeFile(outputPath, `${JSON.stringify(plan, null, 2)}\n`, 'utf8'); +await writeFile(receiptPath, `${JSON.stringify(receipt, null, 2)}\n`, 'utf8'); +console.log(JSON.stringify(receipt, null, 2)); diff --git a/scripts/director-task.mjs b/scripts/director-task.mjs new file mode 100644 index 0000000..ab37e70 --- /dev/null +++ b/scripts/director-task.mjs @@ -0,0 +1,37 @@ +import { createHash } from 'node:crypto'; +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { createDirectorTask } from '../packages/core/dist/index.js'; + +const args = new Map(); +for (let i = 2; i < process.argv.length; i += 2) args.set(process.argv[i], process.argv[i + 1]); +const mode = args.get('--mode'); +const input = args.get('--ir'); +const output = args.get('--out'); +const candidatePath = args.get('--candidate'); +const receiptPath = args.get('--receipt') ?? `${output}.receipt.json`; +if (!['generate', 'repair'].includes(mode) || !input || !output || (mode === 'repair' && !candidatePath)) { + console.error('Usage: node scripts/director-task.mjs --mode --ir --out [--candidate ] [--receipt ]'); + process.exit(2); +} + +const hash = (value) => createHash('sha256').update(value).digest('hex'); +const inputPath = path.resolve(input); +const outputPath = path.resolve(output); +const source = await readFile(inputPath, 'utf8'); +const document = JSON.parse(source); +const candidateSource = candidatePath ? await readFile(path.resolve(candidatePath), 'utf8') : undefined; +const candidate = candidateSource ? JSON.parse(candidateSource) : undefined; +const task = createDirectorTask(document, mode, candidate); +const serializedTask = `${JSON.stringify(task, null, 2)}\n`; +const receipt = { + format: 'seqvio-director-receipt', version: '1.0', operation: `prepare-${mode}-task`, status: 'complete', + input: inputPath, inputHash: hash(source), candidate: candidatePath ? path.resolve(candidatePath) : undefined, + candidateHash: candidateSource ? hash(candidateSource) : undefined, output: outputPath, outputHash: hash(serializedTask), + requestedArtifacts: task.requestedArtifacts, diagnosticCount: task.diagnostics.length, +}; +await mkdir(path.dirname(outputPath), { recursive: true }); +await mkdir(path.dirname(path.resolve(receiptPath)), { recursive: true }); +await writeFile(outputPath, serializedTask, 'utf8'); +await writeFile(path.resolve(receiptPath), `${JSON.stringify(receipt, null, 2)}\n`, 'utf8'); +console.log(JSON.stringify(receipt, null, 2)); diff --git a/scripts/generate-direction-sidecar.mjs b/scripts/generate-direction-sidecar.mjs new file mode 100644 index 0000000..1a11d4a --- /dev/null +++ b/scripts/generate-direction-sidecar.mjs @@ -0,0 +1,22 @@ +import { readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { deriveDirectionPlan, validateDirectionPlan } from '../packages/core/dist/index.js'; + +const [inputArg, outputArg] = process.argv.slice(2); +if (!inputArg || !outputArg) { + console.error('Usage: node scripts/generate-direction-sidecar.mjs '); + process.exit(2); +} + +const inputPath = path.resolve(inputArg); +const outputPath = path.resolve(outputArg); +const document = JSON.parse(await readFile(inputPath, 'utf8')); +const plan = deriveDirectionPlan(document); +const issues = validateDirectionPlan(plan, document); +const errors = issues.filter((item) => item.severity === 'error'); +if (errors.length > 0) { + console.error(JSON.stringify({ ok: false, issues }, null, 2)); + process.exit(1); +} +await writeFile(outputPath, `${JSON.stringify(plan, null, 2)}\n`, 'utf8'); +console.log(JSON.stringify({ ok: true, inputPath, outputPath, segments: plan.segments.length, issues }, null, 2)); diff --git a/scripts/manim-adapter-render.mjs b/scripts/manim-adapter-render.mjs new file mode 100644 index 0000000..d95beb8 --- /dev/null +++ b/scripts/manim-adapter-render.mjs @@ -0,0 +1,26 @@ +import { readFile, writeFile } from 'node:fs/promises'; +import path from 'node:path'; +import { executeManimScene, preflightManim } from '../packages/manim-adapter/dist/index.js'; + +const args = new Map(); +for (let i = 2; i < process.argv.length; i += 2) args.set(process.argv[i], process.argv[i + 1]); +const scenePath = args.get('--scene'); +const expectedOutputPath = args.get('--expectedOutput'); +const manifestPath = args.get('--manifest'); +const pythonCommand = args.get('--python') ?? 'python'; +const cacheDir = args.get('--cacheDir') ?? 'output/manim-adapter-cache'; +if (!scenePath || !expectedOutputPath || !manifestPath) { + console.error('Usage: node scripts/manim-adapter-render.mjs --scene --expectedOutput --manifest [--python ] [--cacheDir ]'); + process.exit(2); +} + +const scene = JSON.parse(await readFile(path.resolve(scenePath), 'utf8')); +const resolvedPythonCommand = /[\\/]/.test(pythonCommand) ? path.resolve(pythonCommand) : pythonCommand; +const manifest = await executeManimScene(scene, { + cwd: process.cwd(), cacheDir: path.resolve(cacheDir), expectedOutputPath: path.resolve(expectedOutputPath), + pythonCommand: resolvedPythonCommand, preflight: preflightManim(resolvedPythonCommand), + onProgress: (event) => console.log(JSON.stringify(event)), +}); +await writeFile(path.resolve(manifestPath), `${JSON.stringify(manifest, null, 2)}\n`, 'utf8'); +console.log(JSON.stringify(manifest, null, 2)); +if (manifest.status !== 'rendered') process.exit(1); diff --git a/scripts/release-pipeline-smoke.mjs b/scripts/release-pipeline-smoke.mjs index 9d14f37..dfe1a58 100644 --- a/scripts/release-pipeline-smoke.mjs +++ b/scripts/release-pipeline-smoke.mjs @@ -14,7 +14,14 @@ import { fileURLToPath, pathToFileURL } from 'node:url'; import ffmpegInstaller from '@ffmpeg-installer/ffmpeg'; import { compileBrowserCapture } from '@seqvio/browser-recorder'; import { compileCaptureManifestToExplainerDocument } from '@seqvio/capture'; -import { compileExplainerDocumentToTsx, resolveScenePacing } from '@seqvio/core'; +import { + compileExplainerDocumentToTsx, + computeDocumentTimeline, + deriveDirectionPlan, + resolveScenePacing, + validateAttentionSequence, + validateDirectionPlan, +} from '@seqvio/core'; import { compileTerminalCapture } from '@seqvio/terminal-narrator'; import puppeteer from 'puppeteer'; import { @@ -41,10 +48,10 @@ const height = numericArgument('height', quick ? 360 : 720); function requestedKinds() { const kindIndex = process.argv.indexOf('--kind'); const kind = kindIndex >= 0 ? process.argv[kindIndex + 1] : 'all'; - if (!['all', 'terminal', 'browser'].includes(kind)) { - throw new Error('--kind must be all, terminal, or browser.'); + if (!['all', 'terminal', 'browser', 'semantic'].includes(kind)) { + throw new Error('--kind must be all, terminal, browser, or semantic.'); } - return kind === 'all' ? ['terminal', 'browser'] : [kind]; + return kind === 'all' ? ['terminal', 'browser', 'semantic'] : [kind]; } function artifactOutputDir() { @@ -372,11 +379,90 @@ async function runPipeline(kind, smokeTempRoot, artifactDir) { } } +function semanticHealth(document) { + const compiled = compileExplainerDocumentToTsx(document); + const pacedAttention = document.scenes.flatMap((scene) => { + const paced = resolveScenePacing(scene, document.fps ?? fps).scene; + return paced.type === 'infographic' ? (paced.attention ?? []) : []; + }); + const markerScenes = document.scenes.filter((scene) => scene.type === 'manim'); + const markers = markerScenes.flatMap((scene) => (scene.markers ?? []).map((marker) => ({ marker, duration: scene.duration ?? 0 }))); + const validMarkers = markers.filter(({ marker, duration }) => marker.frame >= 0 && marker.frame < duration).length; + const focalTargets = new Set(document.scenes.flatMap((scene) => (scene.explanation?.beats ?? []).flatMap((beat) => beat.visuals.map((visual) => visual.targetId)))); + const addressableTargets = new Set(document.scenes.flatMap((scene) => [ + ...(scene.type === 'infographic' ? [...(scene.metrics ?? []), ...(scene.comparisons ?? []), ...(scene.process ?? []), ...(scene.timeline ?? []), ...(scene.charts ?? [])].map((item) => item.id) : []), + ...(scene.type === 'manim' ? (scene.markers ?? []).map((marker) => marker.id) : []), + ])); + const textCharacters = JSON.stringify(document).match(/[A-Za-z0-9]/g)?.length ?? 0; + const megapixels = ((document.width ?? width) * (document.height ?? height)) / 1_000_000; + const direction = deriveDirectionPlan(document); + return { + attentionDiagnosticCount: validateAttentionSequence(pacedAttention).length, + directionDiagnosticCount: validateDirectionPlan(direction, document).filter((issue) => issue.severity === 'error').length, + markerConfidence: markers.length ? Number((validMarkers / markers.length).toFixed(4)) : 1, + textDensityPerMegapixel: Number((textCharacters / Math.max(0.01, megapixels)).toFixed(2)), + focalCoverage: addressableTargets.size ? Number((focalTargets.size / addressableTargets.size).toFixed(4)) : 1, + directionSegmentCount: compiled.directionPlan.segments.length, + }; +} + +async function runSemanticPipeline(smokeTempRoot, artifactDir) { + const workDir = await mkdtemp(path.join(smokeTempRoot, 'semantic-release-pipeline-')); + try { + const mediaPath = path.join(workDir, 'manim-fixture.mp4'); + const manifestPath = path.join(workDir, 'manim-fixture.manifest.json'); + const componentPath = path.join(workDir, 'composition.tsx'); + const qaDir = path.join(workDir, 'qa'); + const outputPath = path.join(workDir, 'final.mp4'); + await runFfmpeg(['-y', '-f', 'lavfi', '-i', `testsrc2=size=${width}x${height}:rate=${fps}:duration=2`, '-an', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', mediaPath]); + const manifest = { format: 'seqvio-manim-render', version: '1.0', status: 'rendered', outputPath: mediaPath, width, height, fps, durationSeconds: 2, alphaMode: 'opaque' }; + await writeFile(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`); + const source = JSON.parse(await readFile(path.join(root, 'examples', 'ir', 'integrated-semantic-validation.explainer.json'), 'utf8')); + const document = { + ...source, width, height, fps, lockToAudio: false, transitionDuration: quick ? 2 : 6, + scenes: source.scenes.map((scene) => scene.type === 'manim' + ? { ...scene, sourceVideo: pathToFileURL(mediaPath).href, sourceManifest: manifestPath, mediaWidth: width, mediaHeight: height, mediaFps: fps, duration: quick ? 20 : 60, markers: [{ id: 'equation-result', frame: quick ? 10 : 30 }] } + : { ...scene, duration: quick ? 24 : scene.duration }), + }; + const compiled = compileExplainerDocumentToTsx(document); + await writeFile(componentPath, compiled.code, 'utf8'); + const timeline = computeDocumentTimeline(document); + await runNode(path.join(root, 'packages/renderer/dist/qa-cli.js'), [ + '--component', componentPath, '--outDir', qaDir, '--width', String(width), '--height', String(height), '--fps', String(fps), + '--frames', `0,${Math.max(0, timeline.totalFrames - 1)}`, '--allowSilentNarration', '--ci', + ]); + await runNode(path.join(root, 'packages/renderer/dist/cli.js'), [ + '--component', componentPath, '--output', outputPath, '--width', String(width), '--height', String(height), '--fps', String(fps), + '--quality', 'low', '--pixelRatio', '1', '--workers', '1', + ]); + await assertDecodedFrameCount(outputPath, timeline.totalFrames); + const metrics = semanticHealth(document); + const metricsPath = path.join(workDir, 'semantic.metrics.json'); + await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`); + if (metrics.attentionDiagnosticCount || metrics.directionDiagnosticCount || metrics.markerConfidence < 1) throw new Error(`Semantic health failed: ${JSON.stringify(metrics)}`); + console.log(`semantic release pipeline smoke passed (${timeline.totalFrames} frames, ${JSON.stringify(metrics)})`); + if (artifactDir) { + await mkdir(artifactDir, { recursive: true }); + await Promise.all([ + copyFile(outputPath, path.join(artifactDir, 'semantic.mp4')), + copyFile(path.join(qaDir, 'qa-report.json'), path.join(artifactDir, 'semantic.qa-report.json')), + copyFile(metricsPath, path.join(artifactDir, 'semantic.metrics.json')), + copyFile(manifestPath, path.join(artifactDir, 'semantic.manim-manifest.json')), + ]); + } + } finally { + await rm(workDir, { recursive: true, force: true }); + } +} + async function main() { const smokeTempRoot = path.join(root, 'temp'); await mkdir(smokeTempRoot, { recursive: true }); const artifactDir = artifactOutputDir(); - for (const kind of requestedKinds()) await runPipeline(kind, smokeTempRoot, artifactDir); + for (const kind of requestedKinds()) { + if (kind === 'semantic') await runSemanticPipeline(smokeTempRoot, artifactDir); + else await runPipeline(kind, smokeTempRoot, artifactDir); + } } main().catch((error) => { diff --git a/scripts/render-benchmark.mjs b/scripts/render-benchmark.mjs index e96a215..7dca6d3 100644 --- a/scripts/render-benchmark.mjs +++ b/scripts/render-benchmark.mjs @@ -10,7 +10,14 @@ import { fileURLToPath, pathToFileURL } from 'node:url'; import ffmpegInstaller from '@ffmpeg-installer/ffmpeg'; import puppeteer from 'puppeteer'; -import { compileExplainerDocumentToTsx, computeDocumentTimeline } from '../packages/core/dist/index.js'; +import { + compileExplainerDocumentToTsx, + computeDocumentTimeline, + deriveDirectionPlan, + resolveScenePacing, + validateAttentionSequence, + validateDirectionPlan, +} from '../packages/core/dist/index.js'; import { render } from '../packages/renderer/dist/index.js'; const execFileAsync = promisify(execFile); @@ -193,13 +200,59 @@ function documents(browserVideo) { ], }, }, + { + name: 'semantic-720p', + document: { + ...base, id: 'benchmark-semantic', transitionDuration: 6, + scenes: [ + { + type: 'infographic', id: 'overview', duration: 45, title: 'From evidence to explanation', + metrics: [ + { id: 'evidence', label: 'Evidence', value: 'Recorded', detail: 'source state', color: '#38bdf8', at: 0 }, + { id: 'model', label: 'Model', value: 'Explained', detail: 'semantic path', color: '#a78bfa', at: 15 }, + ], + explanation: { + cues: [{ id: 'overview-cue', text: 'Evidence becomes an explanation.' }], + beats: [ + { id: 'evidence-beat', cueId: 'overview-cue', anchor: { text: 'Evidence' }, visuals: [{ targetId: 'evidence', action: 'focus', minHoldMs: 500 }] }, + { id: 'model-beat', cueId: 'overview-cue', anchor: { text: 'explanation' }, visuals: [{ targetId: 'model', action: 'highlight', minHoldMs: 500 }] }, + ], + }, + }, + { type: 'manim', id: 'equation', duration: 45, sourceVideo: browserVideo, sourceManifest: 'benchmark-local-manifest.json', mediaWidth: width, mediaHeight: height, mediaFps: fps, markers: [{ id: 'equation-result', frame: 20 }] }, + ], + }, + }, ]; return definitions.map((definition) => ({ ...definition, frames: computeDocumentTimeline(definition.document).totalFrames, + semanticMetrics: definition.name === 'semantic-720p' ? semanticHealth(definition.document) : undefined, })); } +function semanticHealth(document) { + const pacedAttention = document.scenes.flatMap((scene) => { + const paced = resolveScenePacing(scene, document.fps ?? fps).scene; + return paced.type === 'infographic' ? (paced.attention ?? []) : []; + }); + const markers = document.scenes.filter((scene) => scene.type === 'manim').flatMap((scene) => (scene.markers ?? []).map((marker) => ({ marker, duration: scene.duration ?? 0 }))); + const focalTargets = new Set(document.scenes.flatMap((scene) => (scene.explanation?.beats ?? []).flatMap((beat) => beat.visuals.map((visual) => visual.targetId)))); + const addressableTargets = new Set(document.scenes.flatMap((scene) => [ + ...(scene.type === 'infographic' ? [...(scene.metrics ?? []), ...(scene.comparisons ?? []), ...(scene.process ?? []), ...(scene.timeline ?? []), ...(scene.charts ?? [])].map((item) => item.id) : []), + ...(scene.type === 'manim' ? (scene.markers ?? []).map((marker) => marker.id) : []), + ])); + const textCharacters = JSON.stringify(document).match(/[A-Za-z0-9]/g)?.length ?? 0; + const direction = deriveDirectionPlan(document); + return { + attentionDiagnosticCount: validateAttentionSequence(pacedAttention).length, + directionDiagnosticCount: validateDirectionPlan(direction, document).filter((item) => item.severity === 'error').length, + markerConfidence: markers.length ? Number((markers.filter(({ marker, duration }) => marker.frame >= 0 && marker.frame < duration).length / markers.length).toFixed(4)) : 1, + textDensityPerMegapixel: Number((textCharacters / ((width * height) / 1_000_000)).toFixed(2)), + focalCoverage: addressableTargets.size ? Number((focalTargets.size / addressableTargets.size).toFixed(4)) : 1, + }; +} + async function environment() { let chromiumVersion = 'unknown'; try { @@ -272,6 +325,7 @@ async function runCase(definition, workDir, runIndex) { function summarize(definition, samples) { const metric = (name) => median(samples.map((sample) => sample[name])); + const cacheHitRate = Number(metric('cacheHitRate').toFixed(4)); return { name: definition.name, width, height, fps, frames: definition.frames, @@ -284,13 +338,14 @@ function summarize(definition, samples) { peakProcessTreeRssMb: Number(metric('peakProcessTreeRssMb').toFixed(1)), outputBytes: Math.round(metric('outputBytes')), reusedFrames: Math.round(metric('reusedFrames')), - cacheHitRate: Number(metric('cacheHitRate').toFixed(4)), + cacheHitRate, }, p90: { totalMs: Math.round(percentile(samples.map((sample) => sample.totalMs), 0.9)), peakProcessTreeRssMb: Number(percentile(samples.map((sample) => sample.peakProcessTreeRssMb), 0.9).toFixed(1)), }, samples, + semanticMetrics: definition.semanticMetrics ? { ...definition.semanticMetrics, cacheHitRate } : undefined, }; } diff --git a/scripts/verify-contracts.mjs b/scripts/verify-contracts.mjs index 429f8e3..a9835a0 100644 --- a/scripts/verify-contracts.mjs +++ b/scripts/verify-contracts.mjs @@ -74,8 +74,8 @@ for (const name of policy.independentPackages) { for (const name of policy.adapterPackages) { const entry = packages.get(name); - if (entry?.manifest.seqvio?.adapterLifecycle !== 'pre-stable') { - fail(`${entry?.file ?? name}: capture adapters must remain pre-stable until host promotion`); + if (entry?.manifest.seqvio?.adapterLifecycle !== 'stable') { + fail(`${entry?.file ?? name}: capture adapters must declare the promoted stable lifecycle`); } } @@ -116,8 +116,10 @@ if (capabilities) { for (const scene of actual.scenes) { const required = packages.get(scene.requiredPackage)?.manifest; if (!required) fail(`Scene ${scene.type} requires missing package ${scene.requiredPackage}`); - else if (required.seqvio?.lifecycle !== 'public') { - fail(`Scene ${scene.type} requires non-public package ${scene.requiredPackage}`); + else if (scene.lifecycle === 'public' && required.seqvio?.lifecycle !== 'public') { + fail(`Public scene ${scene.type} requires non-public package ${scene.requiredPackage}`); + } else if (scene.lifecycle === 'experimental' && !['public', 'experimental'].includes(required.seqvio?.lifecycle)) { + fail(`Experimental scene ${scene.type} requires unavailable package ${scene.requiredPackage}`); } } } diff --git a/scripts/visual-regression.mjs b/scripts/visual-regression.mjs index c8e363c..5592588 100644 --- a/scripts/visual-regression.mjs +++ b/scripts/visual-regression.mjs @@ -66,6 +66,21 @@ const CASES = [ // hook whiteboard, code walkthrough, architecture diagram frames: [20, 100, 250], }, + { + name: 'style-clean-technical', + component: 'examples/compositions/style-playbook-clean-technical.tsx', + width: 1280, height: 720, frames: [24, 90, 132, 174], + }, + { + name: 'style-editorial-explainer', + component: 'examples/compositions/style-playbook-editorial-explainer.tsx', + width: 1280, height: 720, frames: [24, 90, 132, 174], + }, + { + name: 'style-terminal-first', + component: 'examples/compositions/style-playbook-terminal-first.tsx', + width: 1280, height: 720, frames: [24, 90, 132, 174], + }, ]; // PSNR below this (dB) for any frame is treated as a regression. Pixel-identical @@ -73,6 +88,8 @@ const CASES = [ const PSNR_THRESHOLD_DB = 40; const isUpdate = process.argv.includes('--update'); +const caseIndex = process.argv.indexOf('--case'); +const selectedCases = new Set(caseIndex >= 0 ? String(process.argv[caseIndex + 1] ?? '').split(',').filter(Boolean) : []); function frameFileName(caseName, frame) { return `${caseName}-f${String(frame).padStart(6, '0')}.png`; @@ -176,6 +193,7 @@ async function main() { let compared = 0; for (const testCase of CASES) { + if (selectedCases.size > 0 && !selectedCases.has(testCase.name)) continue; for (const frame of testCase.frames) { const fileName = frameFileName(testCase.name, frame); const baselinePath = path.join(baselineDir, fileName); diff --git a/seqvio.release-policy.json b/seqvio.release-policy.json index 71235d5..3ecb1db 100644 --- a/seqvio.release-policy.json +++ b/seqvio.release-policy.json @@ -14,7 +14,8 @@ "independentPackages": [ "@seqvio/capture", "@seqvio/browser-recorder", - "@seqvio/terminal-narrator" + "@seqvio/terminal-narrator", + "@seqvio/manim-adapter" ], "adapterPackages": [ "@seqvio/browser-recorder", diff --git a/skills/seqvio/references/current-capabilities.md b/skills/seqvio/references/current-capabilities.md index 685b872..8900e9d 100644 --- a/skills/seqvio/references/current-capabilities.md +++ b/skills/seqvio/references/current-capabilities.md @@ -62,9 +62,52 @@ From `@seqvio/technical`: - `AnnotationProvider` / `AnnotationTarget` / `AnnotationLayer` (also exported from `@seqvio/core` for cross-style use) - `CodeWalkthrough` — Shiki sync highlighter, stable line ids, focus / type / insert / replace / delete / annotate - `ArchitectureDiagram` — dagre layout with reveal / connect / trace / emphasize / collapse / expand +- `InfographicScene` — metrics, comparisons, process, timeline, relationships, + bar/line charts, series targets, axes, legends, units, and source labels +- `ManimClip` — seekable externally rendered mathematical animation with named, + narration-reflowable markers + +Shared attention includes spotlight, focus ring, callout, bracket, connector, +region shade, and guided path primitives. Simultaneous callouts use deterministic +collision-aware placement; connectors choose safe-area routes around occupied +targets. Explicit priority provides deterministic stacking, and every primitive +has random-access and reverse-seek coverage. + +Semantic direction is available through versioned `DirectionPlan` and Motion +Grammar contracts. `npm run director:task` prepares host-agent generate/repair +tasks and auditable receipts; the renderer executes their validated artifacts +without a model call. + +Style profiles apply typography, palette, spacing, motion density, camera, +transition, and attention persistence policies. The shipped reference profiles +are `clean-technical`, `editorial-explainer`, and `terminal-first`; semantic +timing and target identity remain invariant across them. Whiteboard drawables and product-demo chrome accept `annotationId` / element `id` so annotations can target them under a shared `AnnotationProvider`. +## External Python Manim integration + +`@seqvio/manim-adapter` is an experimental TypeScript/Node.js adapter for the +independently installed Python package `manim`. It does not implement Manim in +JavaScript. The adapter validates an external-render `ManimSceneSpec`, runs +`python -m manim`, probes the output, and records source/assets/runtime hashes, +media metadata, cache identity, progress, logs, and diagnostics in a +`ManimRenderManifest`. + +The external render and Seqvio playback contracts are separate: + +- `@seqvio/manim-adapter` turns a Python file and scene class into validated + media plus a manifest. +- The ExplainerDocument `manim` scene references that pre-rendered media and is + compiled into `@seqvio/technical` `ManimClip`. +- `ManimClip` seeks the media from the current composition frame. Named markers + may follow resolved ExplanationBeats and expose shared annotation targets. + +`seqvio-doctor` discovers a repository-local `.venv-manim` or uses +`SEQVIO_MANIM_PYTHON`. See +[`docs/MANIM-INTEGRATION.md`](../../../docs/MANIM-INTEGRATION.md) for the complete +setup and rendering workflow. + ## Core composition components From `@seqvio/core`: @@ -94,8 +137,11 @@ From `@seqvio/renderer`: - `seqvio-audio extract` — narration manifest extraction - `seqvio-audio synthesize` — TTS synthesis and resolved manifest generation - `seqvio-qa` — visual, pacing, media, capture, and resolved timing checks +- `seqvio-doctor` — environment checks including optional local Manim discovery +- `npm run director:task -- --mode generate|repair ...` — versioned host-agent + direction task and receipt generation -Capture adapters are available but pre-stable: +Capture adapters are available with a stable CLI/artifact contract: - `@seqvio/terminal-narrator` converts xterm recordings into terminal scenes, capture-backed ExplanationBeats, and optional synthesized narration. @@ -103,12 +149,13 @@ Capture adapters are available but pre-stable: start times when compiling Browser scenes and Beats. Their shared-dispatcher data path is implemented, legacy writers are removed, -and release smoke is tested at `1280x720`. CLI contract `1.0` provides direct +and release smoke is tested at `1280x720`. CLI contract `2.0` provides direct commands, JSON results, stable exit codes, monotonic progress, safe job ids, and portable `artifacts.json` paths. Windows package and real runtime verification -passes locally; the configured Windows/Linux/macOS matrix must pass before -lifecycle promotion. Screenshot privacy work is intentionally deferred and must -not be assumed. +passes locally; the configured Windows/Linux/macOS matrix and lifecycle +promotion pass for the current contract. Browser capture supports deterministic +selector/rectangle privacy masks before frame capture; OCR is not treated as a +security boundary. ## Narration providers @@ -144,17 +191,21 @@ Supported today: Preferred starting points: -| File | Use case | -| --- | --- | -| `examples/compositions/seqvio-overview-en.tsx` | Narrated English product overview | -| `examples/compositions/seqvio-overview-zh.tsx` | Narrated Chinese product overview | -| `examples/compositions/seqvio-audio-demo.tsx` | Audio and caption metadata | -| `examples/compositions/seqvio-intro.tsx` | Multi-scene framework intro | -| `examples/compositions/technical-demo.tsx` | Short technical smoke (whiteboard + code + diagram) | -| `examples/compositions/technical-explainer.tsx` | ~4.5 min technical reference composition (`lockToAudio`) | -| `examples/ir/technical-demo.explainer.json` | Short ExplainerDocument IR | -| `examples/ir/technical-explainer.explainer.json` | Full ExplainerDocument IR + chapters | -| `packages/whiteboard/examples/` | Single-scene whiteboard samples | +| File | Use case | +| -------------------------------------------------------- | -------------------------------------------------------------------- | +| `examples/compositions/seqvio-overview-en.tsx` | Narrated English product overview | +| `examples/compositions/seqvio-overview-zh.tsx` | Narrated Chinese product overview | +| `examples/compositions/seqvio-audio-demo.tsx` | Audio and caption metadata | +| `examples/compositions/seqvio-intro.tsx` | Multi-scene framework intro | +| `examples/compositions/technical-demo.tsx` | Short technical smoke (whiteboard + code + diagram) | +| `examples/compositions/technical-explainer.tsx` | ~4.5 min technical reference composition (`lockToAudio`) | +| `examples/ir/technical-demo.explainer.json` | Short ExplainerDocument IR | +| `examples/ir/technical-explainer.explainer.json` | Full ExplainerDocument IR + chapters | +| `examples/compositions/infographic-chart-validation.tsx` | Chart/series/axis/legend reference | +| `examples/compositions/manim-end-to-end-validation.tsx` | Real graph/proof Manim playback with narration | +| `examples/compositions/style-playbook-*.tsx` | Same semantic composition under three visual profiles | +| `examples/manim/` | Equation, graph, symbolic proof, and geometric proof Python fixtures | +| `packages/whiteboard/examples/` | Single-scene whiteboard samples | Narrated technical reference loop: @@ -172,21 +223,22 @@ Local render intermediates belong in `output/` / `.media/` and are gitignored. ## Repository layout -| Path | Purpose | -| --- | --- | -| `packages/whiteboard` | Whiteboard components | -| `packages/scatterbrain` | Scatterbrain sticky-note / cork-board style components | -| `packages/technical` | Technical explainer components | -| `packages/core` | Scene, transition, and IR runtime | -| `packages/renderer` | Bundler and CLIs | -| `packages/product-demo` | Product walkthrough components | -| `packages/capture` | Shared capture manifest and evidence contracts | -| `packages/terminal-narrator` | Pre-stable terminal capture adapter | -| `packages/browser-recorder` | Pre-stable browser capture adapter | -| `examples/compositions/` | Renderable compositions | -| `examples/ir/` | Storyboard / ExplainerDocument JSON examples | -| `skills/seqvio/` | Agent skill and references | -| `docs/` | Human-facing docs | +| Path | Purpose | +| ---------------------------- | ------------------------------------------------------------ | +| `packages/whiteboard` | Whiteboard components | +| `packages/scatterbrain` | Scatterbrain sticky-note / cork-board style components | +| `packages/technical` | Technical explainer components | +| `packages/core` | Scene, transition, and IR runtime | +| `packages/renderer` | Bundler and CLIs | +| `packages/product-demo` | Product walkthrough components | +| `packages/manim-adapter` | Optional external Manim execution and media manifest adapter | +| `packages/capture` | Shared capture manifest and evidence contracts | +| `packages/terminal-narrator` | Pre-stable terminal capture adapter | +| `packages/browser-recorder` | Pre-stable browser capture adapter | +| `examples/compositions/` | Renderable compositions | +| `examples/ir/` | Storyboard / ExplainerDocument JSON examples | +| `skills/seqvio/` | Agent skill and references | +| `docs/` | Human-facing docs | ## Not implemented yet @@ -195,9 +247,9 @@ Do not assume these exist just because they appear in roadmap or proposal docs: - visual editor / studio workflow - automatic custom SVG illustration generation per topic - Seqvio-side AI planning or planner API calls -- OpenMontage adapter inside Seqvio core +- Product-specific orchestration adapter inside Seqvio core - `@seqvio/education` / full LessonPlan package -- ChatTranscript, DiffReview, or infographic scene families +- ChatTranscript or DiffReview scene families - automatic screenshot privacy or redaction guarantees - formal VISION.md promise for 10-minute videos - transitions beyond `fade`, `slide`, and `wipe` diff --git a/skills/seqvio/references/planning-workflow.md b/skills/seqvio/references/planning-workflow.md index 3ad922e..56ec4b6 100644 --- a/skills/seqvio/references/planning-workflow.md +++ b/skills/seqvio/references/planning-workflow.md @@ -159,5 +159,5 @@ Keep final TSX as the editable production source after compile. - Use `plan-auto`; Seqvio no longer ships a heuristic planner. - Ask Seqvio CLI to call model APIs. - Generate placeholder art in Seqvio core. -- Put OpenMontage schemas into Seqvio core. +- Put application-specific production schemas into Seqvio core. - Split scene planning and illustration into hidden CLI passes. diff --git a/skills/seqvio/references/production-techniques.md b/skills/seqvio/references/production-techniques.md index eab87c8..084b013 100644 --- a/skills/seqvio/references/production-techniques.md +++ b/skills/seqvio/references/production-techniques.md @@ -47,6 +47,43 @@ Before copying a reference style, extract several frames and identify the visual Do not imitate only the background color or font. If the reference uses icons, symbols, arrows, and varied shapes, the Seqvio composition should also use a varied shape vocabulary. +## Product explainer production contract + +Product-video reviews exposed several failure modes that should be prevented by +the authoring contract rather than fixed by taste at the end: + +- Start from the real task and its visible result. Do not spend the opening on + an intro slide, a generic product label, or a full script on screen. +- Let narration be the master clock. Keep full sentences in the voice track; + on-screen text is limited to labels, keywords, commands, filenames, and + short conclusions. Code, terminal, and browser evidence may retain the text + needed to inspect the action. +- Declare one `focalTarget` per scene and make every reveal, emphasis, and + camera move serve that target. A scene without a clear target is a planning + error, not a style problem. +- Mark whether a visual is evidence-bearing capture or an authored conceptual + explanation. A diagram may clarify a model, but must not imply that it is a + recording of a system event. +- Vary scene composition and takeaway treatment. Repeating a header, title, + card grid, and bottom rail makes a technical video read like a slide deck; + use the scene's metaphor to determine its layout and conclusion. +- Use transitions to transfer attention: deliberate cuts, focus transfers, + match-object links, and restrained blur crossfades. Avoid horizontal wipes, + continuous rotation, and motion that has no explanatory role. +- Review reference frames at scene starts, midpoints, boundaries, and the final + frame. Record a contact sheet or frame index and check text overflow, + title-to-graphic collisions, focal-target coverage, and brand-asset + transparency. +- Keep voice providers behind the resolved audio manifest. CosyVoice and other + providers must produce the same phrase anchors, narration clock, and QA + behavior; provider choice must not alter semantic scene order. + +Implementation hooks are the planning fields `hook`, `visualRole`, +`focalTarget`, `evidenceSource`, `onScreenTextBudget`, and +`transitionIntent`; the QA layer should report their violations with a scene +id and frame number. This turns production experience into repeatable Seqvio +behavior instead of a one-off editing checklist. + ## Diagram richness Avoid explaining an entire video with only rounded rectangles plus text. Prefer semantic visual primitives: diff --git a/tests/visual-snapshots/baseline/style-clean-technical-f000024.png b/tests/visual-snapshots/baseline/style-clean-technical-f000024.png new file mode 100644 index 0000000..395cca9 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-clean-technical-f000024.png differ diff --git a/tests/visual-snapshots/baseline/style-clean-technical-f000090.png b/tests/visual-snapshots/baseline/style-clean-technical-f000090.png new file mode 100644 index 0000000..18e85b5 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-clean-technical-f000090.png differ diff --git a/tests/visual-snapshots/baseline/style-clean-technical-f000132.png b/tests/visual-snapshots/baseline/style-clean-technical-f000132.png new file mode 100644 index 0000000..6897f76 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-clean-technical-f000132.png differ diff --git a/tests/visual-snapshots/baseline/style-clean-technical-f000174.png b/tests/visual-snapshots/baseline/style-clean-technical-f000174.png new file mode 100644 index 0000000..db91996 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-clean-technical-f000174.png differ diff --git a/tests/visual-snapshots/baseline/style-editorial-explainer-f000024.png b/tests/visual-snapshots/baseline/style-editorial-explainer-f000024.png new file mode 100644 index 0000000..3d347b4 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-editorial-explainer-f000024.png differ diff --git a/tests/visual-snapshots/baseline/style-editorial-explainer-f000090.png b/tests/visual-snapshots/baseline/style-editorial-explainer-f000090.png new file mode 100644 index 0000000..91b5757 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-editorial-explainer-f000090.png differ diff --git a/tests/visual-snapshots/baseline/style-editorial-explainer-f000132.png b/tests/visual-snapshots/baseline/style-editorial-explainer-f000132.png new file mode 100644 index 0000000..8fc5b06 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-editorial-explainer-f000132.png differ diff --git a/tests/visual-snapshots/baseline/style-editorial-explainer-f000174.png b/tests/visual-snapshots/baseline/style-editorial-explainer-f000174.png new file mode 100644 index 0000000..46d8b9f Binary files /dev/null and b/tests/visual-snapshots/baseline/style-editorial-explainer-f000174.png differ diff --git a/tests/visual-snapshots/baseline/style-terminal-first-f000024.png b/tests/visual-snapshots/baseline/style-terminal-first-f000024.png new file mode 100644 index 0000000..e89c602 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-terminal-first-f000024.png differ diff --git a/tests/visual-snapshots/baseline/style-terminal-first-f000090.png b/tests/visual-snapshots/baseline/style-terminal-first-f000090.png new file mode 100644 index 0000000..218b8a1 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-terminal-first-f000090.png differ diff --git a/tests/visual-snapshots/baseline/style-terminal-first-f000132.png b/tests/visual-snapshots/baseline/style-terminal-first-f000132.png new file mode 100644 index 0000000..88c5590 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-terminal-first-f000132.png differ diff --git a/tests/visual-snapshots/baseline/style-terminal-first-f000174.png b/tests/visual-snapshots/baseline/style-terminal-first-f000174.png new file mode 100644 index 0000000..2f9e4a2 Binary files /dev/null and b/tests/visual-snapshots/baseline/style-terminal-first-f000174.png differ