Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
- Repository foundation, documentation, tests, and continuous integration.
- Initial provider-neutral SemanticVideo schema and JSON Schema export.
- Example Japan trip semantic manifest.

- Deterministic `ffprobe` media inspection and `semanticvideo inspect` CLI.
- Fixture-driven parser coverage and a synthetic video integration test.
28 changes: 21 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,21 +44,24 @@ came from.

## Current scope

Milestones 0 and 1 establish the repository and the schema foundation:
Milestones 0 through 2 establish the schema foundation and technical inspection:

- Pydantic models for media, streams, exact time, segments, annotations,
entities, evidence, and provenance
- JSON serialization and JSON Schema export
- cross-reference and temporal validation
- an example `.semantic.json` manifest
- deterministic `ffprobe` inspection for real video files
- a scriptable `semanticvideo inspect` command with JSON output
- tests, linting, typing, CI, documentation, and architectural decisions

Media analysis, AI providers, search, EditPlan, OpenTimelineIO, and FFmpeg
rendering are intentionally scheduled for later milestones.
Shot detection, semantic AI providers, search, EditPlan, OpenTimelineIO, and
FFmpeg rendering are intentionally scheduled for later milestones.

## Quick start

Requirements: Python 3.12+ and [`uv`](https://docs.astral.sh/uv/).
Requirements: Python 3.12+, [`uv`](https://docs.astral.sh/uv/), and FFmpeg's
`ffprobe` executable for media inspection.

```bash
uv sync --all-groups
Expand All @@ -68,6 +71,17 @@ uv run mypy src
uv run semanticvideo-schema --output semanticvideo.schema.json
```

Inspect a real video without invoking an AI model:

```bash
uv run semanticvideo inspect GX010231.MP4
uv run semanticvideo inspect GX010231.MP4 --output GX010231.inspect.json
```

The command reports source identity, exact duration, container, bitrate, video/audio/
subtitle streams, codecs, dimensions, frame rate, time base, rotation, color
metadata, audio layout, language, timestamps, and filesystem facts as JSON.

Load and validate a manifest:

```python
Expand All @@ -81,8 +95,9 @@ document = SemanticVideoDocument.model_validate_json(
print(document.media.duration.seconds)
```

See [the semantic format](docs/semantic-format.md),
[architecture](docs/architecture.md), and [roadmap](ROADMAP.md) for details.
See [media inspection](docs/media-inspection.md),
[the semantic format](docs/semantic-format.md), [architecture](docs/architecture.md),
and [roadmap](ROADMAP.md) for details.

## Project status

Expand All @@ -98,4 +113,3 @@ example `feat(schema): add exact time ranges`.
## License

Licensed under the [Apache License 2.0](LICENSE).

21 changes: 11 additions & 10 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,25 @@ must not compromise the provider-neutral schema foundation.
- **Milestone 1 — Core schema:** exact time, media identity, streams,
structural segments, typed annotations, entities, evidence, provenance,
document validation, JSON serialization, and JSON Schema export.
- **Milestone 2 — Media inspection:** safe `ffprobe` execution, pure JSON
parsing, filesystem identity, technical stream metadata, scriptable CLI,
fixtures, and a synthetic video integration test.

## Next milestones

1. **Media inspection:** parse deterministic `ffprobe` output.
2. **Vertical editing slice:** manually authored manifests to a minimal
1. **Vertical editing slice:** manually authored manifests to a minimal
EditPlan and deterministic FFmpeg cut/concatenate renderer.
3. **Frame extraction and shot detection:** modular sampling and boundaries.
4. **Representative frames and signal quality:** local deterministic metrics.
5. **Timed transcription:** provider interface and one reference adapter.
6. **Structured visual semantics:** provider-neutral VLM adapter contracts.
7. **Semantic retrieval:** embeddings behind a replaceable local index.
8. **Editorial interchange:** validated EditPlan and OpenTimelineIO export.
9. **Japan trip demo:** semantic selection and a human-reviewable rough cut.
2. **Frame extraction and shot detection:** modular sampling and boundaries.
3. **Representative frames and signal quality:** local deterministic metrics.
4. **Timed transcription:** provider interface and one reference adapter.
5. **Structured visual semantics:** provider-neutral VLM adapter contracts.
6. **Semantic retrieval:** embeddings behind a replaceable local index.
7. **Editorial interchange:** validated EditPlan and OpenTimelineIO export.
8. **Japan trip demo:** semantic selection and a human-reviewable rough cut.

## Future exploration

OCR, face and speaker identity, GPS fusion, landmark recognition, audio events,
story segmentation, visual similarity, rights policies, content credentials,
professional editor integrations, an MCP server, and embedded container
metadata remain future ideas, not current commitments.

31 changes: 31 additions & 0 deletions docs/adr/0006-ffprobe-inspection-boundary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# 0006: Separate ffprobe execution from metadata parsing

- Status: Accepted
- Date: 2026-08-09

## Context

Technical inspection depends on an external executable, untrusted media paths,
and ffprobe JSON that varies by container and stream. Tests should not require
large binary fixtures or a particular local FFmpeg installation.

## Decision

Invoke ffprobe without a shell, with fixed arguments, captured diagnostics, and
a timeout. Keep its JSON-to-schema conversion in a pure parser. Combine local
filesystem facts only in the higher-level inspection function.

## Alternatives

- Parse human-readable ffprobe console output
- Couple subprocess execution and schema conversion in one function
- Use OpenCV as the authoritative technical metadata source
- Require a Python FFmpeg wrapper dependency

## Consequences

Fixture tests are fast and deterministic, command injection through filenames
is avoided, and other transports can reuse the parser. The project still
depends on an installed ffprobe executable for real inspection and must handle
version-specific fields conservatively.

15 changes: 14 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@ Replaceable analyzers ---> SemanticVideo manifest

None of these representations substitutes for the others.

## Media inspection boundary

Milestone 2 separates three responsibilities:

1. `run_ffprobe` invokes a configured executable with a fixed argument list,
no shell, a timeout, and captured diagnostics.
2. `parse_ffprobe_json` is a deterministic pure parser that can be tested from
checked-in fixtures without an FFmpeg installation.
3. `inspect_media` validates the local path, combines filesystem identity with
parsed metadata, and returns a core `MediaInfo` model.

The command-line interface only formats this validated model. It does not
duplicate media parsing rules.

## Core document

`SemanticVideoDocument` is the aggregate consistency boundary for one source
Expand Down Expand Up @@ -72,4 +86,3 @@ artifact references instead of bloating human-readable JSON.

Breaking core changes require a schema version change, fixtures, migration
notes, and an ADR. Provider-specific fields do not belong in core models.

75 changes: 75 additions & 0 deletions docs/media-inspection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Media inspection

Milestone 2 extracts deterministic technical metadata from a real media file.
It does not detect shots or describe the visible content.

## Requirements

Install FFmpeg and ensure `ffprobe` is on `PATH`, or pass its executable path:

```bash
semanticvideo inspect input.mp4 --ffprobe /path/to/ffprobe
```

SemanticVideo invokes ffprobe with a fixed argument list and without a shell:

```text
ffprobe -v error -print_format json -show_format -show_streams INPUT
```

The default timeout is 60 seconds and can be changed with `--timeout`.

## Usage

Pretty JSON is written to standard output:

```bash
semanticvideo inspect GX010231.MP4
```

For pipelines or files:

```bash
semanticvideo inspect GX010231.MP4 --compact
semanticvideo inspect GX010231.MP4 --output GX010231.inspect.json
```

Expected media, ffprobe, parse, and output errors use exit code 1 and a concise
diagnostic on standard error. Argument errors use argparse's exit code 2.

## Extracted information

The validated `MediaInfo` result includes:

- deterministic asset ID derived from the resolved file URI
- original input URI, file size, and filesystem modification time
- exact positive duration represented as integer ticks
- embedded creation time, container format, and bitrate when present
- video codec and bitrate, dimensions, pixel format, frame rate, time base, display
rotation, sample aspect ratio, color metadata, and a conservative VFR hint
- audio codec and bitrate, sample rate, channel count/layout, language, and time base
- subtitle codec and language
- string-valued container tags for later evidence or identity decisions

Data and attachment streams are ignored in v0.1 because the core schema does
not yet model them.

## Duration fallback

The parser uses container duration when positive. If unavailable, it selects
the longest positive stream duration. A stream may express duration directly
in seconds or as `duration_ts * time_base`. Missing or non-positive duration is
an error because the core media schema requires a bounded timeline.

## Frame-rate limitation

`variable_frame_rate` compares ffprobe's average and nominal frame rates. A
difference is useful evidence of VFR, but equality does not prove that every
frame interval is constant. Later analysis may inspect packet timestamps when
an editing workflow requires stronger guarantees.

## Identity and hashing

The asset ID is deterministic for a resolved local URI; it is not a content
hash. M2 deliberately avoids hashing large video files. A future cache strategy
will define fast fingerprints and optional SHA-256 verification separately.
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Issues = "https://github.com/TristinOrg/SemanticVideo/issues"
Repository = "https://github.com/TristinOrg/SemanticVideo.git"

[project.scripts]
semanticvideo = "semanticvideo.cli.main:main"
semanticvideo-schema = "semanticvideo.schema.export:main"

[dependency-groups]
Expand All @@ -49,6 +50,9 @@ packages = ["src/semanticvideo"]
[tool.pytest.ini_options]
addopts = "--strict-config --strict-markers --cov=semanticvideo --cov-report=term-missing --cov-fail-under=90"
testpaths = ["tests"]
markers = [
"integration: requires ffmpeg and ffprobe executables",
]

[tool.ruff]
line-length = 88
Expand Down
49 changes: 42 additions & 7 deletions semanticvideo.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,19 @@
"additionalProperties": false,
"description": "Technical metadata for one audio stream.",
"properties": {
"bit_rate": {
"anyOf": [
{
"minimum": 0,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"title": "Bit Rate"
},
"channel_layout": {
"anyOf": [
{
Expand Down Expand Up @@ -826,6 +839,19 @@
"additionalProperties": false,
"description": "Identity and technical facts for the source media asset.",
"properties": {
"bit_rate": {
"anyOf": [
{
"minimum": 0,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"title": "Bit Rate"
},
"checksum": {
"anyOf": [
{
Expand Down Expand Up @@ -1914,6 +1940,19 @@
"additionalProperties": false,
"description": "Technical metadata for one video stream.",
"properties": {
"bit_rate": {
"anyOf": [
{
"minimum": 0,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"title": "Bit Rate"
},
"codec": {
"minLength": 1,
"title": "Codec",
Expand Down Expand Up @@ -2002,14 +2041,10 @@
},
"rotation_degrees": {
"default": 0,
"enum": [
0,
90,
180,
270
],
"exclusiveMaximum": 360,
"minimum": 0,
"title": "Rotation Degrees",
"type": "integer"
"type": "number"
},
"sample_aspect_ratio": {
"anyOf": [
Expand Down
3 changes: 3 additions & 0 deletions src/semanticvideo/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""Public API for the SemanticVideo reference implementation."""

from semanticvideo.media import inspect_media, parse_ffprobe_json
from semanticvideo.schema import (
AnalysisRun,
Annotation,
Expand Down Expand Up @@ -74,6 +75,8 @@
"SubtitleStream",
"TimeRange",
"VideoStream",
"inspect_media",
"parse_ffprobe_json",
]

__version__ = "0.1.0"
1 change: 1 addition & 0 deletions src/semanticvideo/cli/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Command-line interface package."""
Loading
Loading