feat(analysis): add shot content understanding

CHANGELOG.md

@@ -14,3 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 - Example Japan trip semantic manifest.
 - Deterministic `ffprobe` media inspection and `semanticvideo inspect` CLI.
 - Fixture-driven parser coverage and a synthetic video integration test.
+- Single-file `semanticvideo analyze` pipeline with FFmpeg shot detection,
+  representative frames, structured shot content, and traceable provenance.
+- Optional OpenAI vision and reviewed JSON description providers.
+- Opt-in technical, embedded metadata, SHA-256, and raw FFprobe information.

README.md

@@ -44,7 +44,8 @@ came from.
 
 ## Current scope
 
-Milestones 0 through 2 establish the schema foundation and technical inspection:
+Milestones 0 through 3 establish the schema foundation and the first complete
+video-understanding path:
 
 - Pydantic models for media, streams, exact time, segments, annotations,
   entities, evidence, and provenance
@@ -53,10 +54,14 @@ Milestones 0 through 2 establish the schema foundation and technical inspection:
 - an example `.semantic.json` manifest
 - deterministic `ffprobe` inspection for real video files
 - a scriptable `semanticvideo inspect` command with JSON output
+- FFmpeg scene-change detection and representative-frame extraction
+- provider-neutral shot descriptions with an OpenAI adapter and reviewed JSON import
+- one editing-oriented `.semantic.json` containing media, shots, descriptions,
+  provenance, and analysis parameters
 - tests, linting, typing, CI, documentation, and architectural decisions
 
-Shot detection, semantic AI providers, search, EditPlan, OpenTimelineIO, and
-FFmpeg rendering are intentionally scheduled for later milestones.
+Search, EditPlan, OpenTimelineIO, and FFmpeg editing/rendering are scheduled for
+later milestones.
 
 ## Quick start
 
@@ -82,6 +87,33 @@ The command reports source identity, exact duration, container, bitrate, video/a
 subtitle streams, codecs, dimensions, frame rate, time base, rotation, color
 metadata, audio layout, language, timestamps, and filesystem facts as JSON.
 
+Generate the required editing information in one file:
+
+```bash
+uv sync --extra openai
+set OPENAI_API_KEY=your_key
+uv run semanticvideo analyze GX010231.MP4 --language zh-CN
+```
+
+The default `GX010231.semantic.json` always contains core media facts, contiguous
+shot ranges, one representative time per shot, and a structured scene description.
+The command fails instead of silently writing an incomplete manifest if description
+generation is unavailable.
+
+Optional information is opt-in and remains in that same JSON:
+
+```bash
+uv run semanticvideo analyze GX010231.MP4 --include technical --include metadata
+uv run semanticvideo analyze GX010231.MP4 --include checksum --include raw
+```
+
+Descriptions produced elsewhere or reviewed by a person can be imported from an
+object keyed by shot ID:
+
+```bash
+uv run semanticvideo analyze GX010231.MP4 --descriptions descriptions.json
+```
+
 Load and validate a manifest:
 
 ```python
@@ -95,7 +127,7 @@ document = SemanticVideoDocument.model_validate_json(
 print(document.media.duration.seconds)
 ```
 
-See [media inspection](docs/media-inspection.md),
+See [video analysis](docs/video-analysis.md), [media inspection](docs/media-inspection.md),
 [the semantic format](docs/semantic-format.md), [architecture](docs/architecture.md),
 and [roadmap](ROADMAP.md) for details.
 

ROADMAP.md

@@ -13,18 +13,19 @@ must not compromise the provider-neutral schema foundation.
 - **Milestone 2 — Media inspection:** safe `ffprobe` execution, pure JSON
   parsing, filesystem identity, technical stream metadata, scriptable CLI,
   fixtures, and a synthetic video integration test.
+- **Milestone 3 — Shot content analysis:** FFmpeg shot detection, representative
+  frames, provider-neutral structured descriptions, optional OpenAI and reviewed
+  JSON providers, and a complete single-file analysis command.
 
 ## Next milestones
 
-1. **Vertical editing slice:** manually authored manifests to a minimal
+1. **Vertical editing slice:** generated manifests to a minimal
    EditPlan and deterministic FFmpeg cut/concatenate renderer.
-2. **Frame extraction and shot detection:** modular sampling and boundaries.
-3. **Representative frames and signal quality:** local deterministic metrics.
-4. **Timed transcription:** provider interface and one reference adapter.
-5. **Structured visual semantics:** provider-neutral VLM adapter contracts.
-6. **Semantic retrieval:** embeddings behind a replaceable local index.
-7. **Editorial interchange:** validated EditPlan and OpenTimelineIO export.
-8. **Japan trip demo:** semantic selection and a human-reviewable rough cut.
+2. **Representative-frame quality:** local deterministic image/audio metrics.
+3. **Timed transcription:** provider interface and one reference adapter.
+4. **Semantic retrieval:** embeddings behind a replaceable local index.
+5. **Editorial interchange:** validated EditPlan and OpenTimelineIO export.
+6. **Japan trip demo:** semantic selection and a human-reviewable rough cut.
 
 ## Future exploration
 

docs/video-analysis.md

@@ -0,0 +1,92 @@
+# Video analysis
+
+`semanticvideo analyze` is the first end-to-end editing-oriented analysis path. It
+uses FFprobe for source facts, FFmpeg for shot boundaries and representative JPEGs,
+and a replaceable visual-description provider. The final deliverable is one
+`.semantic.json`; temporary frames are removed after analysis.
+
+## Required output
+
+The command treats the following as required rather than optional:
+
+- source URI, exact duration, file size, container, codecs, dimensions, frame rate,
+  audio sample rate, and channel count
+- contiguous shot ranges covering the complete source duration
+- one representative timestamp inside every shot
+- one structured scene annotation for every shot
+- provenance, evidence timestamp, provider/model identity, and analysis parameters
+
+If any representative frame or description cannot be produced, the command exits
+with an error and does not pretend that the analysis is complete.
+
+Scene annotations include a concise description plus optional environment, subjects,
+actions, objects, visible text, location hint, shot type, camera movement, editorial
+role, and confidence. Unknown values remain empty instead of being guessed.
+
+## OpenAI provider
+
+Install the optional dependency and configure the API key in the environment:
+
+```powershell
+uv sync --extra openai
+$env:OPENAI_API_KEY = "..."
+uv run semanticvideo analyze input.mp4 --language zh-CN
+```
+
+The adapter submits representative images through the Responses API and requests a
+strict JSON Schema result. The core pipeline only depends on the `ShotDescriber`
+contract, so local or other hosted models can be added without changing the format.
+
+The default model is `gpt-5.6`; use `--model` to choose another compatible model.
+The key is read from `OPENAI_API_KEY` and is never stored in the output manifest.
+
+## Reviewed JSON provider
+
+For offline runs, human review, or another vision system, provide a JSON object whose
+keys match generated shot IDs:
+
+```json
+{
+  "shot.0001": {
+    "description": "A traveler walks through a railway station.",
+    "environment": ["indoor station"],
+    "subjects": ["traveler"],
+    "actions": ["walking"],
+    "objects": ["luggage"],
+    "shot_type": "wide shot"
+  }
+}
+```
+
+Run it with:
+
+```powershell
+uv run semanticvideo analyze input.mp4 --descriptions descriptions.json
+```
+
+## Optional information
+
+Core editing facts are always present. Repeat `--include` to add information:
+
+| Value | Additional content |
+| --- | --- |
+| `technical` | bitrate, pixel format, time bases, aspect ratio, color and VFR hints |
+| `metadata` | embedded tags and filesystem/embedded timestamps |
+| `checksum` | SHA-256 of the complete source file |
+| `raw` | namespaced raw FFprobe response under `extensions` |
+
+For example:
+
+```powershell
+uv run semanticvideo analyze input.mp4 `
+  --include technical `
+  --include metadata `
+  --output input.semantic.json
+```
+
+## Shot controls
+
+`--scene-threshold` is FFmpeg's scene-change threshold and must be between zero and
+one. Lower values detect more cuts. `--minimum-shot-duration` removes very short
+detections and defaults to 0.5 seconds. Both values are persisted in `analysis_runs`
+so the result can be reproduced and compared.

pyproject.toml

@@ -26,6 +26,11 @@ dependencies = [
   "pydantic>=2.11,<3",
 ]
 
+[project.optional-dependencies]
+openai = [
+  "openai>=1.0",
+]
+
 [project.urls]
 Homepage = "https://github.com/TristinOrg/SemanticVideo"
 Documentation = "https://github.com/TristinOrg/SemanticVideo/tree/main/docs"

semanticvideo.schema.json

@@ -1368,6 +1368,14 @@
       "additionalProperties": false,
       "description": "Perceptual and editorial description of a visual scene.",
       "properties": {
+        "actions": {
+          "default": [],
+          "items": {
+            "type": "string"
+          },
+          "title": "Actions",
+          "type": "array"
+        },
         "camera_movement": {
           "anyOf": [
             {
@@ -1405,6 +1413,26 @@
           "title": "Environment",
           "type": "array"
         },
+        "location_hint": {
+          "anyOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Location Hint"
+        },
+        "objects": {
+          "default": [],
+          "items": {
+            "type": "string"
+          },
+          "title": "Objects",
+          "type": "array"
+        },
         "shot_type": {
           "anyOf": [
             {
@@ -1416,6 +1444,22 @@
           ],
           "default": null,
           "title": "Shot Type"
+        },
+        "subjects": {
+          "default": [],
+          "items": {
+            "type": "string"
+          },
+          "title": "Subjects",
+          "type": "array"
+        },
+        "visible_text": {
+          "default": [],
+          "items": {
+            "type": "string"
+          },
+          "title": "Visible Text",
+          "type": "array"
         }
       },
       "required": [

src/semanticvideo/analysis/__init__.py

@@ -0,0 +1,20 @@
+"""High-level video analysis pipeline."""
+
+from semanticvideo.analysis.pipeline import analyze_video
+from semanticvideo.analysis.shots import (
+    build_shot_ranges,
+    detect_shot_boundaries,
+    extract_frame,
+    representative_time,
+)
+from semanticvideo.analysis.types import ShotDescriber, ShotDescription
+
+__all__ = [
+    "ShotDescriber",
+    "ShotDescription",
+    "analyze_video",
+    "build_shot_ranges",
+    "detect_shot_boundaries",
+    "extract_frame",
+    "representative_time",
+]