SemanticVideo

Open, reusable semantic sidecar format for video understanding.

SemanticVideo analyzes media once and preserves time-aligned core content, evidence, confidence, coverage, and provenance beside the source file. AI agents and applications can reuse that understanding instead of repeatedly decoding the whole video and extracting the same screenshots. It does not replace MP4, MOV, MKV, codecs, editors, or renderers.

Raw video -> understand once -> reusable semantic manifest -> any AI/application

Important

SemanticVideo is an early specification and reference implementation. The 0.x schema may change while real-world editing workflows are validated.

Why

Traditional media containers store pixels, audio, subtitles, and technical metadata. They do not provide a common representation of time-aligned scenes, speech, subjects, events, locations, quality, evidence, or editing value. Consequently, every AI application has to inspect the original media again.

SemanticVideo persists those observations as a sidecar manifest:

GX010231.MP4
GX010231.semantic.json

The media remains the source of truth for pixels and audio. The manifest describes what is known about the media, when it is true, and where that claim came from.

Design principles

• Every observation is explicitly time-aligned.
• Structural segments and overlapping semantic annotations are separate.
• Machine-generated claims carry confidence, evidence, and provenance.
• Time is represented exactly as integer ticks and a rate, not float seconds.
• Core schemas are provider-neutral and human-editable.
• Analysis is incremental, cacheable, and replaceable.
• Semantic understanding, edit decisions, timeline interchange, and rendering remain separate responsibilities.

Current scope

Milestones 0 through 5 establish the reusable semantic cache layer and one reference rough-cut consumer:

• Pydantic models for media, streams, exact time, segments, annotations, entities, evidence, and provenance
• JSON serialization and JSON Schema export
• cross-reference and temporal validation
• an example .semantic.json manifest
• deterministic ffprobe inspection for real video files
• a scriptable semanticvideo inspect command with JSON output
• FFmpeg scene-change detection and representative-frame extraction
• provider-neutral shot descriptions with an OpenAI adapter and reviewed JSON import
• deterministic multi-frame quality, audio-level, location, similarity, and editing signals that do not require an AI service
• one editing-oriented .semantic.json containing media, shots, descriptions, provenance, and analysis parameters
• validated, explainable EditPlan objects generated from structured editing signals
• deterministic FFmpeg trim-and-concatenate rendering with atomic output handling
• adaptive representative-frame sampling for long shots
• optional persistent keyframe evidence linked from semantic claims
• video/shot summary hierarchy and time-aligned moments inside shots
• explicit capability fields and covered source ranges
• focused supplements that preserve human-confirmed claims
• rebuildable JSONL search indexes across one or many manifests
• tests, linting, typing, CI, documentation, and architectural decisions

Embeddings and OpenTimelineIO interchange remain later milestones.

Quick start

Requirements: Python 3.12+, uv, and FFmpeg's ffprobe executable for media inspection.

uv sync --all-groups
uv run pytest
uv run ruff check .
uv run mypy src
uv run semanticvideo-schema --output semanticvideo.schema.json

Inspect a real video without invoking an AI model:

uv run semanticvideo inspect GX010231.MP4
uv run semanticvideo inspect GX010231.MP4 --output GX010231.inspect.json

The command reports source identity, exact duration, container, bitrate, video/audio/ subtitle streams, codecs, dimensions, frame rate, time base, rotation, color metadata, audio layout, language, timestamps, and filesystem facts as JSON.

Generate the required editing information in one file:

uv sync --extra openai
set OPENAI_API_KEY=your_key
uv run semanticvideo analyze GX010231.MP4 --language zh-CN

No API key is required when Codex or another agent drives the analysis:

uv run semanticvideo prepare-agent GX010231.MP4 --output GX010231.task
# Ask the agent to complete GX010231.task/response.json.
uv run semanticvideo analyze GX010231.MP4 \
  --agent-response GX010231.task/response.json

The default GX010231.semantic.json always contains core media facts, contiguous shot ranges, three representative times per shot, structured scene descriptions, editing fitness, audio levels, and segment relations. The command fails instead of silently writing an incomplete manifest if description generation is unavailable.

Persist reusable evidence and increase coverage for long shots:

uv run semanticvideo analyze GX010231.MP4 `
  --adaptive-frames `
  --maximum-frame-interval 8 `
  --evidence-dir GX010231.semantic\keyframes

Before reopening the source, ask whether the manifest already covers a question:

uv run semanticvideo gaps GX010231.semantic.json `
  --field moments.actions --start 30 --duration 10

If a focused agent inspection produces a validated supplement, merge it without overwriting human-authored or human-reviewed claims:

uv run semanticvideo enrich GX010231.semantic.json response.supplement.json

Build and query a disposable index (the manifests remain authoritative):

uv run semanticvideo index *.semantic.json -o semantic.index.jsonl
uv run semanticvideo search semantic.index.jsonl "Tokyo Tower Conan"

Turn the analysis into a reviewable rough cut:

uv run semanticvideo plan GX010231.semantic.json \
  --target-duration 60 --maximum-clip-duration 8
uv run semanticvideo render GX010231.semantic.json --output GX010231.roughcut.mp4

Optional information is opt-in and remains in that same JSON:

uv run semanticvideo analyze GX010231.MP4 --include technical --include metadata
uv run semanticvideo analyze GX010231.MP4 --include checksum --include raw

Descriptions produced elsewhere or reviewed by a person can be imported from an object keyed by shot ID:

uv run semanticvideo analyze GX010231.MP4 --descriptions descriptions.json

Load and validate a manifest:

from pathlib import Path

from semanticvideo import SemanticVideoDocument

document = SemanticVideoDocument.model_validate_json(
    Path("examples/japan-trip/GX010231.semantic.json").read_text()
)
print(document.media.duration.seconds)

See agent workflow, reusable understanding, editing and rendering, video analysis, media inspection, the semantic format, architecture, and roadmap for details.

Project status

SemanticVideo is pre-alpha. Contributions and concrete media workflows are welcome, but consumers should pin an exact schema version and retain their original media.

Contributing

See CONTRIBUTING.md. Changes use Conventional Commits, for example feat(schema): add exact time ranges.

License

Licensed under the Apache License 2.0.