────────────────────────────────────────
- 30-second overview ──────────────────────────────────────── We will run a real-time pipeline:
audio → speaker diarization → “who is talking now?”
video → face detection & tracking → “where is each person?”
cropper → keep the active speaker in 16:9 best-fit, animate the crop smoothly.
Everything is done in Python with PyAV for decoding/encoding, pyannote.audio for diarization, face_recognition (dlib) or mediapipe for faces, and OpenCV for the final crop & encode.
──────────────────────────────────────── 2. High-level architecture ──────────────────────────────────────── ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ │ Audio │→ │ Diarization │→ │ Active-ID │→ │ Smooth │ │ stream │ │ (pyannote) │ │ + timing │ │ switcher │ └─────────────┘ └──────────────┘ └──────────────┘ └──────┬──────┘ │ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ Video │→ │ Face detect │→ │ Person boxes │───────────┘ │ stream │ │ + track │ │ per frame │ └─────────────┘ └──────────────┘ └──────────────┘
Crop engine: every frame, given (active-ID → bounding box), compute the 16:9 crop window, apply an exponential smoothing & motion limiter so the window glides instead of jumping.
──────────────────────────────────────── 3. Detailed design ──────────────────────────────────────── 3.1 Offline preparation (once) • pip install pyannote.audio==3.1 opencv-python mediapipe av numpy scipy • Download pyannote speaker-diarization model: huggingface-cli login # get token from pyannote.audio import Pipeline pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=HF_TOKEN) • Calibrate face ↔ speaker association (see 3.4).
3.2 Input handling • Open input with PyAV: container = av.open("input.mp4") vstream = container.streams.video[0] astream = container.streams.audio[0] fps = float(vstream.average_rate)
3.3 Audio diarization (streaming style) • Read audio in 0.5-s chunks → 16 kHz mono. • Run pyannote on a rolling 5-s buffer; emit “speaker_0 active since t=12.3 s”. • Keep a deque of the last N predictions to avoid flip-flop (median filter).
3.4 Link faces to voices (only at start, or once per 30 s) • Whenever we have ≥2 faces, collect 1-s windows of audio whose diarization is “pure” (only one speaker). • Compute voice embeddings and face embeddings → simple cosine similarity clustering. • Store mapping: speaker_id → face_id (an int index).
3.5 Face detection & tracking per frame • Use MediaPipe FaceMesh → 468 3D landmarks. • Convert to tight bounding box: expand 20 % around landmarks. • Sort boxes left→right to maintain stable ids even if they swap seats. • Kalman filter on each box centroid for temporal smoothness.
3.6 Crop logic • Inputs: active_speaker_id, list_of_boxes. • Look up which box belongs to that speaker (from 3.4). • Compute tight 16:9 crop: desired_w = h_crop * 16/9 center_x = (x1+x2)/2 center_y = (y1+y2)/2 scale = max(desired_w, w_box) / w_frame crop_box = [cx-w/2, cy-h/2, cx+w/2, cy+h/2] • Clamp to frame edges. • Smooth with exponential moving average on center and scale (α = 0.15). • Limit max velocity so window never moves > 5 % of frame per frame.
3.7 Encode & write • Create PyAV output stream (H.264). • For each frame, crop with cv2.warpAffine / np slicing, then encode. • Audio: passthrough untouched.
3.8 Real-time vs file • For real-time webcam: run diarization on 1-s chunks and faces on every frame (GPU). Latency ~300 ms is acceptable. • For file processing: run diarization once on entire audio, then merge with per-frame boxes.
────────────────────────────────────────
7. Deliverables
────────────────────────────────────────
crop_to_speaker.py (CLI)
realtime_crop.py (webcam demo)
requirements.txt
README.md with usage examples
That’s the end-to-end plan; you can start with the snippets above and incrementally add the bells & whistles.