diff --git a/astro.config.mjs b/astro.config.mjs index ce0a225..cf0d999 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -111,6 +111,8 @@ export default defineConfig({ '**/DbDiagram*.js', '**/ppu-paddle-ocr*.js', '**/ort-*.wasm', + '**/transformers*.js', + '**/*huggingface*.js', ], runtimeCaching: [ { diff --git a/docs/superpowers/plans/2026-08-01-voice-to-text.md b/docs/superpowers/plans/2026-08-01-voice-to-text.md new file mode 100644 index 0000000..5ad0114 --- /dev/null +++ b/docs/superpowers/plans/2026-08-01-voice-to-text.md @@ -0,0 +1,34 @@ +# Voice to Text — Implementation Plan + +Spec: `docs/superpowers/specs/2026-08-01-voice-to-text-design.md`. TDD per task: +failing test → confirm fail → implement → confirm pass. Pure libs first, island last. + +## Task 0 — Dependency +- `npm i @huggingface/transformers` (`.npmrc` handles peer deps). +- Add `@huggingface/transformers` chunk glob to `astro.config.mjs` `workbox.globIgnores`. + +## Task 1 — `stt.lib.ts` (pure) + tests +- `TranscriptSegment`, `mixToMono`, `segmentsToText`, `formatClock`, + `formatSrtTime`, `formatVttTime`, `segmentsToSrt`, `segmentsToVtt`. +- Tests cover formatting edge cases + null-end fallback + channel averaging. + +## Task 2 — `stt-audio.lib.ts` +- `decodeToMono16k(blob)` via `AudioContext({ sampleRate: 16000 })`, reuse `mixToMono`. +- Browser API → build + manual smoke (no unit test). + +## Task 3 — `stt.engine.ts` (SDK boundary) +- `createTranscriber(model, onProgress)`; WebGPU feature-detect; map chunks→segments. +- Dynamically `import('@huggingface/transformers')`. Build + manual smoke. + +## Task 4 — `useAudioRecorder.ts` + tests +- MediaRecorder capture → Blob; typed errors; track cleanup. Tests mock + `getUserMedia` + a fake `MediaRecorder`. + +## Task 5 — Island `VoiceToText.tsx` +- Record/upload input, model selector, progress bar, transcribe, tabbed output + (Text / Timestamped / Subtitles), copy + download (.txt/.srt/.vtt). + +## Task 6 — Register + verify loop +- Registry entry (`voice-to-text`, Media, `Mic`, beta). +- `vitest run` + `lint` + `build` green; resolve any transformers.js/Vite SSR issue + (optimizeDeps.exclude / ssr.external); no precache warning. diff --git a/docs/superpowers/specs/2026-08-01-voice-to-text-design.md b/docs/superpowers/specs/2026-08-01-voice-to-text-design.md new file mode 100644 index 0000000..06e590a --- /dev/null +++ b/docs/superpowers/specs/2026-08-01-voice-to-text-design.md @@ -0,0 +1,166 @@ +# Voice to Text Tool — Design + +**Date:** 2026-08-01 +**Tool:** Media → Voice to Text (`/tools/voice-to-text`) — NEW +**Type:** New tool (on-device ML) +**Icon:** `Mic` (lucide-react) +**Category:** Media + +## Problem + +Users want to transcribe speech to text — from a microphone recording or an uploaded +audio/video file — without sending their audio to a server. + +## Goal + +A privacy-first, **fully on-device** transcription tool. Audio never leaves the browser; +only the Whisper model weights are fetched from a CDN at runtime (same trade-off the OCR +tool already makes). Output as editable plain text, subtitle files (SRT/VTT), and an +inline timestamped view. + +## Approach + +On-device Whisper via **transformers.js** (`@huggingface/transformers`) — it wraps +onnxruntime-web (already a project dep), provides the `automatic-speech-recognition` +pipeline with a WebGPU→WASM device option, a model-download `progress_callback`, and +segment timestamps via `return_timestamps: true`. Dynamically imported inside the engine +boundary so the island chunk stays small; its built chunk + ORT wasm are added to +`workbox.globIgnores`. + +**Scope (YAGNI):** batch "record/upload → transcribe," **not** live streaming +transcription. Streaming (WhisperTextStreamer) is a possible follow-up. + +## Verified API (transformers.js v3, confirmed via docs) + +```js +import { pipeline } from '@huggingface/transformers'; +const transcriber = await pipeline('automatic-speech-recognition', modelId, { + device: 'webgpu' | 'wasm', + dtype, // e.g. 'q8' on wasm to shrink the download + progress_callback, // { status, file, progress, loaded, total } +}); +const out = await transcriber(float32AudioAt16k, { + return_timestamps: true, + chunk_length_s: 30, + stride_length_s: 5, +}); +// out = { text: string, chunks: [{ timestamp: [start, end], text }] } +``` + +Models (`onnx-community/*`, ONNX-ready): +- **English · Fast** → `onnx-community/whisper-tiny.en` +- **English · Accurate** → `onnx-community/whisper-base.en` +- **Multilingual** → `onnx-community/whisper-base` (auto-detects language) + +## Files + +- `src/tools/media/stt.lib.ts` — pure transcript formatting (segments→text/SRT/VTT, time + formatters, `mixToMono`). Unit-tested. +- `src/tools/media/stt.engine.ts` — the ONLY file touching transformers.js. `createTranscriber`. +- `src/tools/media/stt-audio.lib.ts` — `decodeToMono16k(blob)` (Web Audio API; reuses `mixToMono`). +- `src/hooks/useAudioRecorder.ts` — mic capture via MediaRecorder → Blob. Unit-tested (mocks). +- `src/islands/media/VoiceToText.tsx` — thin island (default export). +- `src/registry/tools.ts` — register `voice-to-text` (Media, `Mic`, `status: 'beta'`). +- `astro.config.mjs` — add `@huggingface/transformers` chunk glob to `workbox.globIgnores`. + +## Library API + +### `stt.lib.ts` (pure) + +```ts +export interface TranscriptSegment { start: number; end: number; text: string } + +export function mixToMono(channels: Float32Array[]): Float32Array; // avg channels +export function segmentsToText(segments: TranscriptSegment[]): string; // trimmed, space-joined +export function formatClock(seconds: number): string; // 'm:ss' for the inline view +export function formatSrtTime(seconds: number): string; // 'HH:MM:SS,mmm' +export function formatVttTime(seconds: number): string; // 'HH:MM:SS.mmm' +export function segmentsToSrt(segments: TranscriptSegment[]): string; +export function segmentsToVtt(segments: TranscriptSegment[]): string; // 'WEBVTT\n\n' + cues +``` + +Robustness: a segment with a null/undefined `end` (Whisper can emit an open final +timestamp) falls back to `start`; negatives clamp to 0. + +### `stt.engine.ts` (SDK boundary) + +```ts +export type SttBackend = 'webgpu' | 'wasm'; +export type SttModelId = + | 'onnx-community/whisper-tiny.en' + | 'onnx-community/whisper-base.en' + | 'onnx-community/whisper-base'; + +export interface Transcriber { + backend: SttBackend; + transcribe(audio: Float32Array): Promise; +} + +export function createTranscriber( + model: SttModelId, + onProgress?: (ratio: number) => void, // 0..1 during model download +): Promise; +``` + +- Picks `webgpu` when available (feature-detect `navigator.gpu`), else `wasm` (with a + quantized `dtype` to shrink the download). +- `transcribe` calls the pipeline with `return_timestamps: true`, maps `out.chunks` → + `TranscriptSegment[]` (falling back to a single segment from `out.text` if no chunks). + +### `stt-audio.lib.ts` + +```ts +export function decodeToMono16k(blob: Blob): Promise; +``` + +Uses `new AudioContext({ sampleRate: 16000 })` → `decodeAudioData` (resamples to 16k) → +`mixToMono(channels)`; closes the context in a `finally`. + +### `useAudioRecorder.ts` + +Returns `{ recording, seconds, error, blob, start, stop, reset }`. +`start()` → `getUserMedia({ audio: true })` → `MediaRecorder` collecting chunks; `stop()` +finalizes a `Blob` and releases tracks. Errors map to typed reasons +(`unsupported` / `denied` / `unknown`), mirroring `useCamera`. + +## Island (`VoiceToText.tsx`) + +1. **Input** — two ways: + - **Record**: mic button (via `useAudioRecorder`) with a running timer + Stop. + - **Upload**: `Dropzone` accepting `audio/*,video/*`. + Either produces a `Blob` shown with a small `