diff --git a/CLAUDE.md b/CLAUDE.md index b4cede8..633dc4d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,11 @@ vertical video out via headless Chrome + ffmpeg + ElevenLabs). - `crates/videoeditor-chrome` — CDP driver (long-lived headless Chrome; NEVER single-shot `--screenshot`, it hangs on macOS). - `crates/videoeditor-media` — all ffmpeg/ffprobe invocations + assembly. -- `crates/videoeditor-voice` — ElevenLabs TTS/STT (`ELEVENLABS_API_KEY`). +- `crates/videoeditor-voice` — TTS/STT with two backends each: local + default (piper voice via sherpa-onnx; whisper.cpp) and ElevenLabs + (`ELEVENLABS_API_KEY`; `tts:` frontmatter / `VIDEOEDITOR_TTS` / + `VIDEOEDITOR_STT` select). Local models come from the nix closure + (`WHISPER_MODEL`, `PIPER_VOICE`). - `crates/videoeditor-genai` — typed image-generation clients: xAI Grok Imagine (`XAI_API_KEY`, reference images) + Google Imagen (`AI_STUDIO`); Veo/Grok video is the planned next tenant. diff --git a/README.md b/README.md index 95eb558..36765be 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,9 @@ from markdown + SVG placeholders. This strip is actual output.* - **Rust** orchestrates everything (`videoeditor`, one binary). - **Web tech** does the animation: scenes are HTML templates rendered frame-by-frame by headless Chrome as pure functions of `(data, t)`. -- **ffmpeg** does the heavy lifting; **ElevenLabs** voices the narration. +- **ffmpeg** does the heavy lifting; **local speech models** voice the + narration and transcribe (piper + whisper.cpp, no API key), with + ElevenLabs as an opt-in upgrade. - **Built to be driven by [Claude Code](https://claude.com/claude-code)** — scaffolds wire the session up automatically. @@ -26,8 +28,9 @@ from markdown + SVG placeholders. This strip is actual output.* ## Install With [nix](https://install.determinate.systems) (preferred — pins the binary, -ffmpeg, and the render browser from the committed lockfile; details and every -run variant in [docs/nix.md](docs/nix.md)): +ffmpeg, the render browser, and the local speech stack (whisper.cpp STT + +a piper voice) from the committed lockfile; details and every run variant +in [docs/nix.md](docs/nix.md)): ```bash curl -fsSL https://install.determinate.systems/nix | sh -s -- install # once @@ -38,9 +41,12 @@ Without nix: `cargo install videoeditor`, then bring ffmpeg (`brew/apt/dnf install ffmpeg`) and Chrome (system install is auto-detected; `CHROME_BIN` overrides). macOS and Linux; on Windows use WSL. -Voicing needs an `ELEVENLABS_API_KEY` +Everything runs keyless by default: narration uses a bundled local piper +voice, transcription uses bundled whisper.cpp. Prefer ElevenLabs voices? +Set `tts: elevenlabs` in the script frontmatter with an `ELEVENLABS_API_KEY` ([elevenlabs.io](https://elevenlabs.io) → profile → API keys; free tier is -plenty). Everything except `tts`/`analyze` runs keyless. +plenty). Non-nix installs bring their own whisper.cpp/sherpa-onnx for the +local stack (see `videoeditor guide`, Env section). ## Your first video @@ -72,7 +78,7 @@ anything costs money. Or skip the wizard and ask in your own words: ``` script.md ──parse──► timeline plan │ - ├─ videoeditor tts [CLIP:] → ElevenLabs → audio/clips/ + audio/clips.json + ├─ videoeditor tts [CLIP:] → piper (or ElevenLabs) → audio/clips/ + audio/clips.json ├─ videoeditor render [SCENE:] → Chrome frames → ffmpeg → build/scenes/ └─ videoeditor assemble concat + narration@offsets + music → build/final.mp4 ``` diff --git a/crates/videoeditor-media/src/lib.rs b/crates/videoeditor-media/src/lib.rs index 645dd13..44ffc09 100644 --- a/crates/videoeditor-media/src/lib.rs +++ b/crates/videoeditor-media/src/lib.rs @@ -147,6 +147,36 @@ pub fn extract_audio(video: &Path, out: &Path) -> Result<()> { ]) } +/// Downmix any audio to 16 kHz mono PCM wav — the input whisper.cpp expects. +pub fn to_whisper_wav(audio: &Path, out: &Path) -> Result<()> { + ffmpeg(&[ + "-i", + audio.to_str().unwrap(), + "-ar", + "16000", + "-ac", + "1", + "-c:a", + "pcm_s16le", + out.to_str().unwrap(), + ]) +} + +/// Encode a wav (e.g. piper TTS output) to the pipeline's mp3 format. +pub fn wav_to_mp3(wav: &Path, out: &Path) -> Result<()> { + ffmpeg(&[ + "-i", + wav.to_str().unwrap(), + "-ar", + "44100", + "-codec:a", + "libmp3lame", + "-b:a", + "128k", + out.to_str().unwrap(), + ]) +} + /// Detect scene cuts with ffmpeg's `select=gt(scene,threshold)` filter; /// returns cut timestamps in seconds. pub fn scene_cuts(video: &Path, threshold: f32) -> Result> { diff --git a/crates/videoeditor-record/src/lib.rs b/crates/videoeditor-record/src/lib.rs index fa8a22c..2af5f18 100644 --- a/crates/videoeditor-record/src/lib.rs +++ b/crates/videoeditor-record/src/lib.rs @@ -393,7 +393,8 @@ struct Review { mean_db: f64, max_db: f64, clipped: bool, - /// None = no ELEVENLABS_API_KEY; level/timing coaching still runs. + /// None = no STT backend available (whisper binary+model, or + /// ELEVENLABS_API_KEY); level/timing coaching still runs. transcript: Option, accuracy_pct: Option, missing: Vec, @@ -407,8 +408,8 @@ struct Review { /// Archive and analyze a pending take WITHOUT keeping it: the take lands /// in `audio/takes//` permanently (data safety — retakes lose /// nothing), then local level metrics via ffmpeg always; script-accuracy -/// / pacing / dead-air / background-noise coaching via ElevenLabs Scribe -/// when a key is present. +/// / pacing / dead-air / background-noise coaching via STT (whisper by +/// default, ElevenLabs Scribe via VIDEOEDITOR_STT) when available. fn review_take(ep: &Episode, id: &str, body: &[u8], mime: &str) -> Result { let (scene, clip) = find_clip(ep, id)?; let n = store_take(ep, id, body, mime)?; @@ -420,9 +421,9 @@ fn review_take(ep: &Episode, id: &str, body: &[u8], mime: &str) -> Result -0.2; - // Scribe is optional — no key, no transcript coaching. - let stt = if videoeditor_voice::api_key().is_ok() { - Some(videoeditor_voice::stt(&mp3).context("ElevenLabs STT")?) + // STT is optional — no backend, no transcript coaching. + let stt = if videoeditor_voice::stt_available() { + Some(videoeditor_voice::stt(&mp3).context("speech-to-text")?) } else { None }; diff --git a/crates/videoeditor-timeline/src/lib.rs b/crates/videoeditor-timeline/src/lib.rs index a77385a..2bd0fcf 100644 --- a/crates/videoeditor-timeline/src/lib.rs +++ b/crates/videoeditor-timeline/src/lib.rs @@ -9,7 +9,8 @@ //! --- //! title: My Episode //! fps: 30 -//! voice_id: pNInz6obpgDQGcFmaJgB # "Adam" — an ElevenLabs public preset +//! tts: piper # narration backend: piper (local, default) | elevenlabs +//! voice_id: pNInz6obpgDQGcFmaJgB # elevenlabs only — "Adam", a public preset //! music: assets/music/bed.mp3 //! --- //! @@ -52,6 +53,9 @@ pub struct Meta { /// Template packs this episode uses (frontmatter `packs:`, /// comma-separated paths relative to the episode dir). pub packs: Vec, + /// Narration backend (frontmatter `tts:`): "piper" | "elevenlabs". + /// None = unset; the voice crate resolves the default (piper). + pub tts: Option, pub voice_id: Option, pub model_id: String, /// ElevenLabs voice_settings — low stability reads livelier, less robotic. @@ -285,6 +289,7 @@ fn parse_meta(front: &str) -> Result { let mut width = 1080u32; let mut height = 1920u32; let mut packs = Vec::new(); + let mut tts = None; let mut voice_id = None; let mut model_id = "eleven_multilingual_v2".to_string(); let mut voice_stability = 0.4; @@ -310,6 +315,7 @@ fn parse_meta(front: &str) -> Result { .filter(|p| !p.is_empty()) .collect() } + "tts" => tts = Some(v), "voice_id" => voice_id = Some(v), "model_id" => model_id = v, "voice_stability" => voice_stability = v.parse().context("voice_stability")?, @@ -326,6 +332,7 @@ fn parse_meta(front: &str) -> Result { width, height, packs, + tts, voice_id, model_id, voice_stability, @@ -510,6 +517,7 @@ mod tests { const SCRIPT: &str = r#"--- title: Demo fps: 30 +tts: elevenlabs voice_id: pNInz6obpgDQGcFmaJgB music: assets/music/bed.mp3 --- @@ -544,6 +552,7 @@ Second line joins the same clip. assert_eq!(meta.title, "Demo"); assert_eq!(meta.fps, 30); assert_eq!(meta.width, 1080); + assert_eq!(meta.tts.as_deref(), Some("elevenlabs")); assert_eq!(meta.voice_id.as_deref(), Some("pNInz6obpgDQGcFmaJgB")); assert_eq!(meta.music.as_deref(), Some("assets/music/bed.mp3")); } diff --git a/crates/videoeditor-voice/Cargo.toml b/crates/videoeditor-voice/Cargo.toml index decbe8e..b7420a1 100644 --- a/crates/videoeditor-voice/Cargo.toml +++ b/crates/videoeditor-voice/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "videoeditor-voice" -description = "ElevenLabs TTS and STT client for videoeditor: narration synthesis and reference transcription" -keywords = ["tts", "stt", "elevenlabs", "narration", "video"] +description = "Voice I/O for videoeditor: local piper TTS + whisper.cpp STT (default) or ElevenLabs, narration synthesis and reference transcription" +keywords = ["tts", "stt", "whisper", "narration", "video"] categories = ["multimedia::audio", "api-bindings"] version.workspace = true edition.workspace = true diff --git a/crates/videoeditor-voice/examples/transcribe.rs b/crates/videoeditor-voice/examples/transcribe.rs new file mode 100644 index 0000000..4c7cd12 --- /dev/null +++ b/crates/videoeditor-voice/examples/transcribe.rs @@ -0,0 +1,20 @@ +//! Transcribe one audio file with the resolved STT backend and print the +//! normalized transcript JSON: +//! +//! ```sh +//! cargo run -p videoeditor-voice --example transcribe -- take.mp3 +//! ``` + +use anyhow::{Context, Result}; +use std::path::PathBuf; + +fn main() -> Result<()> { + let audio: PathBuf = std::env::args_os() + .nth(1) + .context("usage: transcribe ")? + .into(); + eprintln!("stt backend: {}", videoeditor_voice::stt_name()); + let transcript = videoeditor_voice::stt(&audio)?; + println!("{}", serde_json::to_string_pretty(&transcript)?); + Ok(()) +} diff --git a/crates/videoeditor-voice/src/elevenlabs.rs b/crates/videoeditor-voice/src/elevenlabs.rs new file mode 100644 index 0000000..4646e65 --- /dev/null +++ b/crates/videoeditor-voice/src/elevenlabs.rs @@ -0,0 +1,100 @@ +//! ElevenLabs cloud backend: TTS (voice_id presets, mp3 out) and Scribe STT. + +use anyhow::{Context, Result, bail}; +use serde_json::Value; +use std::env; +use std::fs; +use std::io::Read; +use std::path::Path; +use videoeditor_timeline::Meta; + +pub fn api_key() -> Result { + env::var("ELEVENLABS_API_KEY") + .or_else(|_| env::var("ELEVENLAB")) + .context("set ELEVENLABS_API_KEY (your ElevenLabs API key)") +} + +pub fn synth(meta: &Meta, text: &str, out: &Path) -> Result<()> { + let key = api_key()?; + let voice = meta + .voice_id + .as_deref() + .context("frontmatter needs voice_id: for ElevenLabs TTS")?; + let url = + format!("https://api.elevenlabs.io/v1/text-to-speech/{voice}?output_format=mp3_44100_128"); + let resp = ureq::post(&url) + .set("xi-api-key", &key) + .send_json(serde_json::json!({ + "text": text, + "model_id": meta.model_id, + "voice_settings": { + "stability": meta.voice_stability, + "similarity_boost": meta.voice_similarity, + "style": meta.voice_style, + "use_speaker_boost": true + } + })); + let resp = match resp { + Ok(r) => r, + Err(ureq::Error::Status(code, r)) => { + bail!( + "ElevenLabs TTS {code}: {}", + r.into_string().unwrap_or_default() + ) + } + Err(e) => return Err(e.into()), + }; + let mut bytes = Vec::new(); + resp.into_reader().read_to_end(&mut bytes)?; + fs::write(out, bytes)?; + Ok(()) +} + +/// Transcribe with ElevenLabs Scribe (word-level timestamps). Returns the +/// Scribe response as-is — it is already the crate's transcript shape. +pub fn stt(audio: &Path) -> Result { + let key = api_key()?; + let bytes = fs::read(audio)?; + let boundary = "----videoeditorboundary7d1c9a2f"; + let mut body = Vec::new(); + for (name, value) in [ + ("model_id", "scribe_v1"), + ("timestamps_granularity", "word"), + ] { + body.extend_from_slice( + format!( + "--{boundary}\r\nContent-Disposition: form-data; name=\"{name}\"\r\n\r\n{value}\r\n" + ) + .as_bytes(), + ); + } + body.extend_from_slice( + format!( + "--{boundary}\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n" + ) + .as_bytes(), + ); + body.extend_from_slice(&bytes); + body.extend_from_slice(format!("\r\n--{boundary}--\r\n").as_bytes()); + + let resp = ureq::post("https://api.elevenlabs.io/v1/speech-to-text") + .set("xi-api-key", &key) + .set( + "content-type", + &format!("multipart/form-data; boundary={boundary}"), + ) + .send_bytes(&body); + let resp = match resp { + Ok(r) => r, + Err(ureq::Error::Status(code, r)) => { + bail!( + "ElevenLabs STT {code}: {}", + r.into_string().unwrap_or_default() + ) + } + Err(e) => return Err(e.into()), + }; + let mut s = String::new(); + resp.into_reader().read_to_string(&mut s)?; + Ok(serde_json::from_str(&s)?) +} diff --git a/crates/videoeditor-voice/src/lib.rs b/crates/videoeditor-voice/src/lib.rs index 5bd477e..74b3105 100644 --- a/crates/videoeditor-voice/src/lib.rs +++ b/crates/videoeditor-voice/src/lib.rs @@ -1,19 +1,89 @@ -//! ElevenLabs voice I/O: text-to-speech (one MP3 per `[CLIP:]`, name-keyed, -//! plus a `clips.json` manifest with probed durations) and Scribe -//! speech-to-text for reference-video transcription. +//! Voice I/O: text-to-speech (one MP3 per `[CLIP:]`, name-keyed, plus a +//! `clips.json` manifest with probed durations) and speech-to-text for +//! reference-video transcription and take coaching. +//! +//! Two backends each, local by default: +//! +//! - TTS: **piper** (default — a piper voice via sherpa-onnx, no API key) or +//! **elevenlabs**. Picked by frontmatter `tts:`, then `VIDEOEDITOR_TTS`. +//! - STT: **whisper** (default — whisper.cpp, no API key) or **elevenlabs** +//! (Scribe). Picked by `VIDEOEDITOR_STT`. +//! +//! Both STT backends return the same transcript shape: +//! `{"text": …, "words": [{"type":"word","text","start","end"}, …]}`. -use anyhow::{Context, Result, bail}; +mod elevenlabs; +mod piper; +mod whisper; + +pub use elevenlabs::api_key; + +use anyhow::{Result, bail}; use serde_json::Value; use std::env; use std::fs; -use std::io::Read; use std::path::Path; use videoeditor_timeline::{ClipInfo, Episode, Meta}; -pub fn api_key() -> Result { - env::var("ELEVENLABS_API_KEY") - .or_else(|_| env::var("ELEVENLAB")) - .context("set ELEVENLABS_API_KEY (your ElevenLabs API key)") +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum TtsBackend { + Piper, + ElevenLabs, +} + +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum SttBackend { + Whisper, + ElevenLabs, +} + +/// Resolve the TTS backend: frontmatter `tts:` → `VIDEOEDITOR_TTS` → piper. +pub fn tts_backend(meta: &Meta) -> Result { + let name = meta + .tts + .clone() + .or_else(|| env::var("VIDEOEDITOR_TTS").ok()) + .unwrap_or_else(|| "piper".to_string()); + match name.as_str() { + "piper" => Ok(TtsBackend::Piper), + "elevenlabs" => Ok(TtsBackend::ElevenLabs), + other => bail!("unknown TTS backend {other:?} — use \"piper\" or \"elevenlabs\""), + } +} + +/// Resolve the STT backend: `VIDEOEDITOR_STT` → whisper. +pub fn stt_backend() -> Result { + match env::var("VIDEOEDITOR_STT").as_deref() { + Err(_) | Ok("whisper") => Ok(SttBackend::Whisper), + Ok("elevenlabs") => Ok(SttBackend::ElevenLabs), + Ok(other) => bail!("unknown STT backend {other:?} — use \"whisper\" or \"elevenlabs\""), + } +} + +/// Human-readable name of the resolved STT backend (for progress lines). +pub fn stt_name() -> &'static str { + match stt_backend() { + Ok(SttBackend::Whisper) => "whisper", + Ok(SttBackend::ElevenLabs) => "elevenlabs", + Err(_) => "unknown", + } +} + +/// Can `stt()` run right now (binary + model present, or API key set)? +pub fn stt_available() -> bool { + match stt_backend() { + Ok(SttBackend::Whisper) => whisper::available(), + Ok(SttBackend::ElevenLabs) => api_key().is_ok(), + Err(_) => false, + } +} + +/// Transcribe an audio file (word-level timestamps) with the resolved backend. +pub fn stt(audio: &Path) -> Result { + match stt_backend()? { + SttBackend::Whisper => whisper::stt(audio), + SttBackend::ElevenLabs => elevenlabs::stt(audio), + } } /// Generate narration clips for every `[CLIP:]` (skips existing files unless @@ -28,12 +98,14 @@ pub fn run(ep: &Episode, only_clip: Option<&str>, force: bool) -> Result<()> { return Ok(()); } - let voice = ep - .meta - .voice_id - .as_deref() - .context("frontmatter needs voice_id: for TTS")?; - let key = api_key()?; + let backend = tts_backend(&ep.meta)?; + println!( + "tts: backend {}", + match backend { + TtsBackend::Piper => "piper (local)", + TtsBackend::ElevenLabs => "elevenlabs", + } + ); let mut manifest: Vec = Vec::new(); for scene in &ep.scenes { @@ -46,7 +118,10 @@ pub fn run(ep: &Episode, only_clip: Option<&str>, force: bool) -> Result<()> { bail!("clip {id} has no narration text"); } println!("tts: {id} ({} chars)", clip.text.len()); - synth(&key, voice, &ep.meta, &clip.text, &path)?; + match backend { + TtsBackend::Piper => piper::synth(&clip.text, &path)?, + TtsBackend::ElevenLabs => elevenlabs::synth(&ep.meta, &clip.text, &path)?, + } } if path.exists() { manifest.push(ClipInfo { @@ -72,81 +147,13 @@ pub fn run(ep: &Episode, only_clip: Option<&str>, force: bool) -> Result<()> { Ok(()) } -fn synth(key: &str, voice: &str, meta: &Meta, text: &str, out: &Path) -> Result<()> { - let url = - format!("https://api.elevenlabs.io/v1/text-to-speech/{voice}?output_format=mp3_44100_128"); - let resp = ureq::post(&url) - .set("xi-api-key", key) - .send_json(serde_json::json!({ - "text": text, - "model_id": meta.model_id, - "voice_settings": { - "stability": meta.voice_stability, - "similarity_boost": meta.voice_similarity, - "style": meta.voice_style, - "use_speaker_boost": true - } - })); - let resp = match resp { - Ok(r) => r, - Err(ureq::Error::Status(code, r)) => { - bail!( - "ElevenLabs TTS {code}: {}", - r.into_string().unwrap_or_default() - ) - } - Err(e) => return Err(e.into()), - }; - let mut bytes = Vec::new(); - resp.into_reader().read_to_end(&mut bytes)?; - fs::write(out, bytes)?; - Ok(()) -} - -/// Transcribe an audio file with ElevenLabs Scribe (word-level timestamps). -pub fn stt(audio: &Path) -> Result { - let key = api_key()?; - let bytes = fs::read(audio)?; - let boundary = "----videoeditorboundary7d1c9a2f"; - let mut body = Vec::new(); - for (name, value) in [ - ("model_id", "scribe_v1"), - ("timestamps_granularity", "word"), - ] { - body.extend_from_slice( - format!( - "--{boundary}\r\nContent-Disposition: form-data; name=\"{name}\"\r\n\r\n{value}\r\n" - ) - .as_bytes(), - ); +/// Is `bin` runnable — an existing path, or a name found on `$PATH`? +pub(crate) fn find_in_path(bin: &str) -> bool { + let p = Path::new(bin); + if p.components().count() > 1 { + return p.exists(); } - body.extend_from_slice( - format!( - "--{boundary}\r\nContent-Disposition: form-data; name=\"file\"; filename=\"audio.mp3\"\r\nContent-Type: audio/mpeg\r\n\r\n" - ) - .as_bytes(), - ); - body.extend_from_slice(&bytes); - body.extend_from_slice(format!("\r\n--{boundary}--\r\n").as_bytes()); - - let resp = ureq::post("https://api.elevenlabs.io/v1/speech-to-text") - .set("xi-api-key", &key) - .set( - "content-type", - &format!("multipart/form-data; boundary={boundary}"), - ) - .send_bytes(&body); - let resp = match resp { - Ok(r) => r, - Err(ureq::Error::Status(code, r)) => { - bail!( - "ElevenLabs STT {code}: {}", - r.into_string().unwrap_or_default() - ) - } - Err(e) => return Err(e.into()), - }; - let mut s = String::new(); - resp.into_reader().read_to_string(&mut s)?; - Ok(serde_json::from_str(&s)?) + env::var_os("PATH") + .map(|paths| env::split_paths(&paths).any(|dir| dir.join(bin).exists())) + .unwrap_or(false) } diff --git a/crates/videoeditor-voice/src/piper.rs b/crates/videoeditor-voice/src/piper.rs new file mode 100644 index 0000000..9f82181 --- /dev/null +++ b/crates/videoeditor-voice/src/piper.rs @@ -0,0 +1,72 @@ +//! Piper local TTS backend: a piper voice (vits onnx) run through +//! `sherpa-onnx-offline-tts`. `PIPER_VOICE` points at the voice directory +//! (model.onnx + tokens.txt + espeak-ng-data); the nix package/dev shell pin +//! one (en_US-lessac-medium). More voices: +//! . + +use anyhow::{Context, Result, bail}; +use std::env; +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; + +pub fn bin() -> String { + env::var("SHERPA_TTS_BIN").unwrap_or_else(|_| "sherpa-onnx-offline-tts".to_string()) +} + +pub fn voice_dir() -> Result { + let dir = env::var("PIPER_VOICE").map(PathBuf::from).context( + "set PIPER_VOICE to a piper voice directory (the nix install pins one; \ + otherwise unpack a vits-piper-* release from \ + github.com/k2-fsa/sherpa-onnx, or use tts: elevenlabs)", + )?; + if !dir.is_dir() { + bail!( + "PIPER_VOICE points at {} which is not a directory", + dir.display() + ); + } + Ok(dir) +} + +/// Synthesize `text` to mp3: sherpa-onnx renders a wav, ffmpeg encodes it to +/// the pipeline's mp3 format. +pub fn synth(text: &str, out: &Path) -> Result<()> { + let voice = voice_dir()?; + let onnx = fs::read_dir(&voice)? + .filter_map(|e| e.ok().map(|e| e.path())) + .find(|p| p.extension().is_some_and(|x| x == "onnx")) + .with_context(|| format!("no .onnx voice model in {}", voice.display()))?; + + let wav = out.with_extension("wav"); + let run = Command::new(bin()) + .arg(format!("--vits-model={}", onnx.display())) + .arg(format!( + "--vits-tokens={}", + voice.join("tokens.txt").display() + )) + .arg(format!( + "--vits-data-dir={}", + voice.join("espeak-ng-data").display() + )) + .arg(format!("--output-filename={}", wav.display())) + .arg(text) + .output() + .with_context(|| { + format!( + "{} not found — install sherpa-onnx (the nix install bundles it) \ + or set SHERPA_TTS_BIN / tts: elevenlabs", + bin() + ) + })?; + if !run.status.success() || !wav.exists() { + bail!( + "piper TTS failed: {}{}", + String::from_utf8_lossy(&run.stderr), + String::from_utf8_lossy(&run.stdout) + ); + } + let encode = videoeditor_media::wav_to_mp3(&wav, out); + let _ = fs::remove_file(&wav); + encode +} diff --git a/crates/videoeditor-voice/src/whisper.rs b/crates/videoeditor-voice/src/whisper.rs new file mode 100644 index 0000000..9112bc5 --- /dev/null +++ b/crates/videoeditor-voice/src/whisper.rs @@ -0,0 +1,132 @@ +//! whisper.cpp local STT backend: `whisper-cli` over a pinned ggml model. +//! The nix package/dev shell put both on PATH/env; bare-cargo installs set +//! `WHISPER_BIN` / `WHISPER_MODEL` themselves. + +use anyhow::{Context, Result, bail}; +use serde_json::{Value, json}; +use std::env; +use std::fs; +use std::path::PathBuf; +use std::process::Command; + +pub fn bin() -> String { + env::var("WHISPER_BIN").unwrap_or_else(|_| "whisper-cli".to_string()) +} + +pub fn model() -> Result { + let m = env::var("WHISPER_MODEL").map(PathBuf::from).context( + "set WHISPER_MODEL to a ggml whisper model (the nix install pins one; \ + otherwise download one with whisper-cpp-download-ggml-model, \ + or set VIDEOEDITOR_STT=elevenlabs)", + )?; + if !m.exists() { + bail!( + "WHISPER_MODEL points at {} which does not exist", + m.display() + ); + } + Ok(m) +} + +pub fn available() -> bool { + model().is_ok() && crate::find_in_path(&bin()) +} + +/// Transcribe an audio file locally. Any input format — ffmpeg downmixes to +/// the 16 kHz mono wav whisper.cpp expects, then `whisper-cli` runs with +/// word-level segmentation (`--max-len 1 --split-on-word`). +pub fn stt(audio: &std::path::Path) -> Result { + let model = model()?; + let work = audio + .parent() + .unwrap_or_else(|| std::path::Path::new(".")) + .join(format!(".whisper-{}", std::process::id())); + fs::create_dir_all(&work)?; + let result = stt_in(audio, &model, &work); + let _ = fs::remove_dir_all(&work); + result +} + +fn stt_in( + audio: &std::path::Path, + model: &std::path::Path, + work: &std::path::Path, +) -> Result { + let wav = work.join("audio16k.wav"); + videoeditor_media::to_whisper_wav(audio, &wav)?; + + let out_base = work.join("transcript"); + let out = Command::new(bin()) + .arg("-m") + .arg(model) + .arg("-f") + .arg(&wav) + .args(["--max-len", "1", "--split-on-word", "-oj", "-np", "-of"]) + .arg(&out_base) + .output() + .with_context(|| { + format!( + "{} not found — install whisper.cpp (the nix install bundles it) \ + or set WHISPER_BIN / VIDEOEDITOR_STT=elevenlabs", + bin() + ) + })?; + if !out.status.success() { + bail!( + "whisper STT failed on {}: {}", + audio.display(), + String::from_utf8_lossy(&out.stderr) + ); + } + let raw: Value = serde_json::from_str(&fs::read_to_string(out_base.with_extension("json"))?)?; + Ok(normalize(&raw)) +} + +/// whisper.cpp JSON → the crate's transcript shape (ElevenLabs-Scribe-like): +/// `{"text": ..., "words": [{"type":"word","text","start","end"}]}` with +/// seconds instead of whisper's millisecond offsets. +pub fn normalize(raw: &Value) -> Value { + let segments = raw["transcription"].as_array(); + let mut text = String::new(); + let mut words = Vec::new(); + for seg in segments.into_iter().flatten() { + let seg_text = seg["text"].as_str().unwrap_or(""); + text.push_str(seg_text); + let word = seg_text.trim(); + if word.is_empty() { + continue; + } + words.push(json!({ + "type": "word", + "text": word, + "start": seg["offsets"]["from"].as_f64().unwrap_or(0.0) / 1000.0, + "end": seg["offsets"]["to"].as_f64().unwrap_or(0.0) / 1000.0, + })); + } + json!({ "text": text.trim(), "words": words }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn normalizes_whisper_json_to_transcript_shape() { + let raw = json!({ + "systeminfo": "…", + "transcription": [ + { "offsets": {"from": 0, "to": 60}, "text": "" }, + { "offsets": {"from": 60, "to": 220}, "text": " The" }, + { "offsets": {"from": 220, "to": 560}, "text": " quick" } + ] + }); + let t = normalize(&raw); + assert_eq!(t["text"], "The quick"); + let words = t["words"].as_array().unwrap(); + assert_eq!(words.len(), 2); // empty segment dropped + assert_eq!(words[0]["type"], "word"); + assert_eq!(words[0]["text"], "The"); + assert_eq!(words[0]["start"], 0.06); + assert_eq!(words[1]["end"], 0.56); + } +} diff --git a/crates/videoeditor/guide.md b/crates/videoeditor/guide.md index 2f1dda6..124af75 100644 --- a/crates/videoeditor/guide.md +++ b/crates/videoeditor/guide.md @@ -10,7 +10,7 @@ READMEs) points here; if another doc disagrees with this one, this one wins. ``` videoeditor new [--format meme-benchmark|blank] scaffold (renders as-is) videoeditor parse resolved plan as JSON (scene starts, clips) -videoeditor tts narration via ElevenLabs (needs ELEVENLABS_API_KEY) +videoeditor tts narration: local piper voice (default, offline) or ElevenLabs videoeditor record record narration YOURSELF: local web teleprompter + mic videoeditor render [--scene name] headless-Chrome frames → scene mp4s videoeditor assemble concat + narration@offsets + music → build/final.mp4 @@ -30,7 +30,8 @@ each scene resolves to (episode dir → `packs:` frontmatter → ```markdown --- title: My Short -voice_id: pNInz6obpgDQGcFmaJgB # "Adam", an ElevenLabs public preset +tts: piper # narration backend: piper (local, default) | elevenlabs +voice_id: pNInz6obpgDQGcFmaJgB # elevenlabs only — "Adam", a public preset packs: ../my-brand-pack # optional shared template layers music: assets/music/bed.mp3 # optional; skipped with a note if missing --- @@ -57,9 +58,10 @@ episode-relative. `` and unknown `[MARKERS:]` are ignored. takes to `audio/clips/__.mp3` (previous audio is archived in `audio/takes/`), and fit-checks each take against its scene window. After every take a COACH panel reviews it before you keep it: level / - clipping checks always; with `ELEVENLABS_API_KEY` set it also - transcribes the take (Scribe) and flags dropped script words, ad-libs, - pace, dead air, and background noises. Then READ the ⚠ fit-check + clipping checks always; it also transcribes the take (whisper, local + and default — ElevenLabs Scribe via `VIDEOEDITOR_STT=elevenlabs`) and + flags dropped script words, ad-libs, pace, dead air, and (Scribe only) + background noises. Then READ the ⚠ fit-check warnings. Narration overlap is a real defect (two voices at once). Recompute: scene duration = clip `at` + measured clip length ÷ tempo + hold; re-place downstream `at`s. Re-run @@ -95,9 +97,9 @@ Display snippets are minimal but honest versions of the real bench code. stiff constructions. Simple English; speak at most ONE rounded number per beat and never read stat strings aloud — the screen holds the digits. Add human stakes. Don't rush: let beats breathe; duration follows pace. Dunk the -loser explicitly — longest rant, shortest mercy. TTS: low stability reads -livelier; never `atempo` ≥ 1.2 (squeezed pauses sound robotic); write -flowing sentences — fragments read staccato. +loser explicitly — longest rant, shortest mercy. TTS: on elevenlabs, low +stability reads livelier; never `atempo` ≥ 1.2 (squeezed pauses sound +robotic); write flowing sentences — fragments read staccato. **Visuals**: one moving element per beat — sequential motion reads clean. Anything the viewer must read holds ≥1.5s; tables ~3s; scoreboards ~4–5s. @@ -136,13 +138,19 @@ compose the blocks. inserting a scene shifts them; re-render or rename. - Moving a clip between scenes renames its audio key (`__.mp3`) — `mv` the file to keep a good take. -- TTS takes vary run-to-run at low stability — re-roll a slow take - (`tts --clip --force`) before rewriting script text. +- ElevenLabs takes vary run-to-run at low stability — re-roll a slow take + (`tts --clip --force`) before rewriting script text. Piper + is deterministic: same text, same take. - Iterate visuals on ONE scene (`render --scene X`); full renders come last. ## Env -- `ELEVENLABS_API_KEY` — required for `tts`/`analyze` only. +- Speech is local by default — no key needed. `tts:` frontmatter (or + `VIDEOEDITOR_TTS`) picks the narration backend, `VIDEOEDITOR_STT` the + transcription one (`whisper` default, `elevenlabs` opt-in for both). +- `WHISPER_MODEL` / `PIPER_VOICE` — the local models; the nix install and + dev shell pin them, override to swap voice or model size. +- `ELEVENLABS_API_KEY` — only for the elevenlabs backends. - `XAI_API_KEY` — `image --provider grok` (the default; takes `--ref` images). - `AI_STUDIO` (or `GEMINI_API_KEY`) — `image --provider imagen` (safety-filtered, no references; rejections say so — reroute those prompts to grok). diff --git a/crates/videoeditor/src/analyze.rs b/crates/videoeditor/src/analyze.rs index 75ec647..237ca31 100644 --- a/crates/videoeditor/src/analyze.rs +++ b/crates/videoeditor/src/analyze.rs @@ -1,6 +1,6 @@ //! Reference-video analysis: the "understand the viral" half of the tool. -//! Extracts audio → ElevenLabs Scribe STT (word timestamps) → ffmpeg scene-cut -//! detection → analysis.json + a human-readable timing table. +//! Extracts audio → STT (whisper by default; word timestamps) → ffmpeg +//! scene-cut detection → analysis.json + a human-readable timing table. use anyhow::{Context, Result}; use serde_json::json; @@ -20,7 +20,7 @@ pub fn run(video: &Path, out: Option<&Path>, threshold: f32) -> Result<()> { let audio = out_dir.join("audio.mp3"); videoeditor_media::extract_audio(&video, &audio)?; - println!("analyze: transcribing (ElevenLabs Scribe)…"); + println!("analyze: transcribing ({})…", videoeditor_voice::stt_name()); let transcript = videoeditor_voice::stt(&audio)?; fs::write( out_dir.join("transcript.json"), diff --git a/crates/videoeditor/src/assets.rs b/crates/videoeditor/src/assets.rs index 9b0c73c..cd9badf 100644 --- a/crates/videoeditor/src/assets.rs +++ b/crates/videoeditor/src/assets.rs @@ -121,7 +121,8 @@ should never have to remember the pipeline. - Assets: logos/memes/music on hand, or keep the placeholder SVGs? Custom look? (`videoeditor templates` to browse; `videoeditor pack init .` + templates/CLAUDE.md to author.) - - Voice: keep the default preset or their ElevenLabs voice_id? + - Voice: keep the local piper default (free, offline), or ElevenLabs + (`tts: elevenlabs` + a voice_id + ELEVENLABS_API_KEY)? 4. SCRIPT — write `script.md` per the guide's craft rules. SHOW the user the narration beats and get approval BEFORE running tts (it costs API credits). diff --git a/crates/videoeditor/src/main.rs b/crates/videoeditor/src/main.rs index 652b25e..d08a5f1 100644 --- a/crates/videoeditor/src/main.rs +++ b/crates/videoeditor/src/main.rs @@ -27,7 +27,8 @@ struct Cli { enum Cmd { /// Parse an episode's script.md and print the resolved plan as JSON Parse { episode: PathBuf }, - /// Generate narration clips via ElevenLabs (skips existing clips) + /// Generate narration clips — local piper voice by default, or + /// ElevenLabs via frontmatter `tts:` (skips existing clips) Tts { episode: PathBuf, /// Regenerate only this clip (scene/clip name) @@ -48,7 +49,8 @@ enum Cmd { Assemble { episode: PathBuf }, /// Full pipeline: tts + render + assemble Build { episode: PathBuf }, - /// Analyze a reference video: transcript (ElevenLabs STT) + scene cuts + /// Analyze a reference video: transcript (whisper STT by default) + + /// scene cuts Analyze { video: PathBuf, /// Output directory (default: