feat: Advanced video content analysis (beyond text extraction)

[aj1126]

Summary

Elevate the video ingestion pipeline from text-only extraction (OCR + Whisper transcription) to actual visual content analysis. The current implementation in worker.js extracts keyframes for OCR and audio for transcription, but does not perform any analysis of the visual content itself.

Motivation

UAP/UAV video recordings contain critical visual data — object shapes, movement trajectories, lighting anomalies, environmental context, and temporal behavior patterns — that are completely invisible to the current text-extraction-only pipeline. A text transcript of cockpit audio tells you what was said; visual analysis tells you what was seen.

Proposed Scope

Tier 1: Keyframe Visual Feature Extraction

• Object Detection — Identify and classify visible objects in keyframes (aircraft, lights, clouds, terrain, structures) using a lightweight pre-trained model (e.g., YOLO, MobileNet via ONNX runtime or TensorFlow Lite).
• Scene Classification — Classify the environment of each keyframe (sky, ocean, urban, rural, night, day, infrared/FLIR).
• Motion Estimation — Compare consecutive keyframes to estimate object displacement vectors and apparent velocity.

Tier 2: Temporal Behavior Analysis

• Track Continuity — Link detected objects across sequential keyframes to build persistent tracks.
• Anomaly Flagging — Flag tracks exhibiting non-standard kinematics (sudden acceleration, impossible direction changes, stationary hover, transmedium transitions) aligned with the kinematic vector analysis directives in .gemini/GEMINI.md.
• Timestamp Correlation — Cross-reference visual events with audio transcription timestamps for synchronized event timelines.

Tier 3: Structured Output Integration

• Visual Metadata Schema — Extend the SIR schema to include visual analysis results:

{
  "name": "DOD_111689133.mp4",
  "content": "<transcribed audio text>",
  "metadata": {
    "type": "video",
    "duration_seconds": 34,
    "keyframes_analyzed": 12,
    "resolution": "1920x1080",
    "fps": 30
  },
  "visual_analysis": {
    "detected_objects": [
      { "label": "unidentified_aerial", "confidence": 0.87, "keyframe": 3, "bbox": [120, 340, 80, 60] },
      { "label": "aircraft", "confidence": 0.92, "keyframe": 7, "bbox": [400, 200, 120, 90] }
    ],
    "scene_classifications": [
      { "keyframe": 1, "scene": "sky_day", "confidence": 0.95 },
      { "keyframe": 8, "scene": "infrared_flir", "confidence": 0.88 }
    ],
    "tracks": [
      { "track_id": 1, "keyframes": [3, 4, 5, 6], "displacement_px": [45, 62, 78], "anomaly_flags": ["acceleration_spike"] }
    ]
  },
  "dates": [...],
  "locations": [...]
}

• Analytics Tier Integration — Feed visual detections into the diagnostic (correlation matrix), predictive (trend forecasting), and prescriptive (recommendation) tiers.

Technical Constraints

• Zero-Cost / FOSS Mandate — All models must be open-source and run locally (no cloud vision APIs). Candidates: YOLOv8-nano (AGPL-3.0), MobileNet-SSD (Apache 2.0), or OpenCV DNN module.
• Python Offloading — Follow the established pattern of offloading compute-heavy work to Python scripts (like scripts/video_ingestion.py) to avoid nested WebAssembly/worker thread issues.
• Resource Budgeting — Keyframe sampling rate should be configurable (e.g., --keyframe-interval=5 for every 5th second) to avoid processing every frame of long recordings.

Acceptance Criteria

• At least one pre-trained object detection model runs locally on extracted keyframes
• Visual detections are serialized into the SIR schema visual_analysis field
• Scene classification labels are attached per keyframe
• Motion estimation produces displacement vectors between consecutive keyframes
• Anomaly flagging identifies at least one kinematic anomaly class
• Existing test suite passes with no regressions
• New test cases cover the visual analysis pipeline with mock keyframe data
• docs/architecture.md updated to reflect the visual analysis layer

Related Files

• src/ingestion/worker.js — current video processing (keyframe + Whisper only)
• scripts/video_ingestion.py — Python offloading for ffmpeg/Whisper/OCR
• src/pipeline.js — analytics pipeline orchestrator
• src/analytics/analyzer.js — analytics tier entry point
• .gemini/GEMINI.md — kinematic vector analysis directives
• docs/ROADMAP.md — feature tracking

Dependencies

• Depends on or complements: Audio ingestion pipeline (standalone audio files)
• Python packages: ultralytics (YOLOv8) or opencv-python with DNN module
• Keyframe extraction: existing ffmpeg integration in video_ingestion.py