feat: add word-timestamp auto-caption example - #53
Merged
ronak-create merged 1 commit intoAug 15, 2026
Conversation
📝 WalkthroughWalkthroughAdded an auto-captions CLI example. It loads word-timestamped JSON or transcribes media with faster-whisper, groups words into timed captions, creates karaoke text clips on track ChangesAuto-captions workflow
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant AutoCaptions as auto_captions.py
participant Transcript as Transcript JSON or faster-whisper
participant Output as Caption JSON
Operator->>AutoCaptions: provide transcript or media and options
AutoCaptions->>Transcript: load or transcribe word timestamps
Transcript-->>AutoCaptions: return normalized words
AutoCaptions->>AutoCaptions: group words and create karaoke clips
AutoCaptions->>Output: write timed clips on track V3
Output-->>Operator: report generated caption JSON
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/auto-captions/auto_captions.py`:
- Around line 79-81: Validate the parsed max_words and max_seconds options
immediately after args.parse_args() in the argument-handling flow: require
max_words to be a positive integer and max_seconds to be finite and greater than
zero, rejecting invalid values before caption grouping begins.
- Line 82: Extract the sorting and interval-validation logic from load_words
into a shared word-normalization helper, then apply that helper to both
load_words transcript results and transcribe(args.audio, args.model) output
before caption grouping. Preserve the existing normalized behavior for
transcript input and ensure audio-derived words follow the same ordering and
validation.
- Around line 27-31: Update the timestamp validation in the word-cleaning
function before appending to clean: reject any start or end value that is not
finite, in addition to the existing end <= start check. Use the standard
finite-number validation and preserve the current ValueError behavior for
invalid intervals.
- Around line 63-66: Update the caption timing fields in the line-building
logic: compute duration from the latest word end across the complete line rather
than relying on line[-1]["end"], and derive wordRate from the full caption span
divided by len(line), preserving the existing minimum duration/rate safeguards
while allowing gaps and overlaps to influence karaoke timing.
In `@examples/auto-captions/README.md`:
- Around line 29-33: The README example should explain how the top-level clips
array maps to MCP operations: each generated clip must become its own addClip
operation in ops[].clip when using fablecut_patch_project. Update the example or
surrounding text to show or clearly state this one-to-one transformation while
preserving the existing project update guidance.
- Around line 31-35: Update the example addClip payload in the README so its
wordRate matches the transcript timing: use the generated value or approximately
0.4 seconds per word for three words spanning 1.2 seconds. Leave the surrounding
caption-grouping documentation unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1774224f-601f-4b8d-aa5a-db8db8768337
📒 Files selected for processing (2)
examples/auto-captions/README.mdexamples/auto-captions/auto_captions.py
Comment on lines
+27
to
+31
| start, end = float(item["start"]), float(item["end"]) | ||
| if end <= start: | ||
| raise ValueError(f"word has invalid interval: {item!r}") | ||
| clean.append({"word": str(item["word"]).strip(), "start": start, "end": end}) | ||
| return sorted(clean, key=lambda item: item["start"]) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Reject non-finite timestamps.
float("nan") and float("inf") pass the current interval check. The generated artifact can then contain NaN or Infinity, which project JSON consumers can reject. Validate that both timestamps are finite before adding the word.
Proposed fix
+import math
+
...
for item in words:
if not isinstance(item, dict) or not item.get("word"):
continue
start, end = float(item["start"]), float(item["end"])
- if end <= start:
+ if not math.isfinite(start) or not math.isfinite(end) or end <= start:
raise ValueError(f"word has invalid interval: {item!r}")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| start, end = float(item["start"]), float(item["end"]) | |
| if end <= start: | |
| raise ValueError(f"word has invalid interval: {item!r}") | |
| clean.append({"word": str(item["word"]).strip(), "start": start, "end": end}) | |
| return sorted(clean, key=lambda item: item["start"]) | |
| start, end = float(item["start"]), float(item["end"]) | |
| if not math.isfinite(start) or not math.isfinite(end) or end <= start: | |
| raise ValueError(f"word has invalid interval: {item!r}") | |
| clean.append({"word": str(item["word"]).strip(), "start": start, "end": end}) | |
| return sorted(clean, key=lambda item: item["start"]) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/auto-captions/auto_captions.py` around lines 27 - 31, Update the
timestamp validation in the word-cleaning function before appending to clean:
reject any start or end value that is not finite, in addition to the existing
end <= start check. Use the standard finite-number validation and preserve the
current ValueError behavior for invalid intervals.
Comment on lines
+63
to
+66
| "text": " ".join(word["word"] for word in line), | ||
| "start": line[0]["start"], | ||
| "duration": max(line[-1]["end"] - line[0]["start"], 0.01), | ||
| "wordRate": max((word["end"] - word["start"] for word in line), default=0.15), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the karaoke renderer and tests to confirm the `wordRate` unit.
rg -n -C 5 \
-g '*.ts' -g '*.tsx' -g '*.js' -g '*.jsx' -g '*.json' \
'\bwordRate\b|["'\'']karaoke["'\'']' .Repository: ronak-create/FableCut
Length of output: 7120
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- auto_captions.py structure ---'
ast-grep outline examples/auto-captions/auto_captions.py || true
printf '%s\n' '--- caption construction ---'
cat -n examples/auto-captions/auto_captions.py | sed -n '1,120p'
printf '%s\n' '--- renderer timing logic ---'
cat -n app.js | sed -n '4750,4770p;4890,4905p'
rg -n -C 8 'wordRate|textAnim === "karaoke"|textAnim == "karaoke"|karaoke' app.js mcp-server.jsRepository: ronak-create/FableCut
Length of output: 18181
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- auto_captions.py structure ---'
ast-grep outline examples/auto-captions/auto_captions.py || true
printf '%s\n' '--- caption construction ---'
cat -n examples/auto-captions/auto_captions.py | sed -n '1,120p'
printf '%s\n' '--- renderer timing logic ---'
cat -n app.js | sed -n '4750,4770p;4890,4905p'
rg -n -C 8 'wordRate|textAnim === "karaoke"|textAnim == "karaoke"|karaoke' app.js mcp-server.jsRepository: ronak-create/FableCut
Length of output: 18181
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- per-word animation calculation ---'
cat -n app.js | sed -n '5040,5130p'
printf '%s\n' '--- deterministic grouping probe ---'
python3 - <<'PY'
import ast
from pathlib import Path
source = Path("examples/auto-captions/auto_captions.py").read_text()
tree = ast.parse(source)
ns = {}
exec(compile(ast.Module(body=[n for n in tree.body if isinstance(n, (ast.FunctionDef, ast.Import, ast.ImportFrom))], type_ignores=[]), "<probe>", "exec"), ns)
cases = {
"overlap": [
{"word": "one", "start": 0.0, "end": 0.4},
{"word": "two", "start": 0.2, "end": 1.4},
{"word": "three", "start": 0.3, "end": 0.8},
],
"gaps": [
{"word": "one", "start": 0.0, "end": 0.1},
{"word": "two", "start": 0.9, "end": 1.0},
{"word": "three", "start": 1.1, "end": 1.2},
],
}
for name, words in cases.items():
result = ns["group_words"](words, max_words=10, max_seconds=10.0)
print(name, result)
line = words
span_end = max(w["end"] for w in line)
span = span_end - line[0]["start"]
print("expected_span_end", span_end, "expected_span", span,
"expected_rate_span_per_word", span / len(line))
PYRepository: ronak-create/FableCut
Length of output: 4889
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- per-word animation calculation ---'
cat -n app.js | sed -n '5040,5130p'
printf '%s\n' '--- deterministic grouping probe ---'
python3 - <<'PY'
import ast
from pathlib import Path
source = Path("examples/auto-captions/auto_captions.py").read_text()
tree = ast.parse(source)
ns = {}
exec(compile(ast.Module(
body=[n for n in tree.body if isinstance(n, (ast.FunctionDef, ast.Import, ast.ImportFrom))],
type_ignores=[],
), "<probe>", "exec"), ns)
cases = {
"overlap": [
{"word": "one", "start": 0.0, "end": 0.4},
{"word": "two", "start": 0.2, "end": 1.4},
{"word": "three", "start": 0.3, "end": 0.8},
],
"gaps": [
{"word": "one", "start": 0.0, "end": 0.1},
{"word": "two", "start": 0.9, "end": 1.0},
{"word": "three", "start": 1.1, "end": 1.2},
],
}
for name, words in cases.items():
result = ns["group_words"](words, max_words=10, max_seconds=10.0)
print(name, result)
span_end = max(w["end"] for w in words)
span = span_end - words[0]["start"]
print(
"expected_span_end", span_end,
"expected_span", span,
"expected_rate_span_per_word", span / len(words),
)
PYRepository: ronak-create/FableCut
Length of output: 4889
Derive timing from the complete caption span.
Use the latest word end for duration, not line[-1]["end"]. The renderer treats wordRate as the seconds between word entrances. Derive it from the full caption span and word count, such as span / len(line), so overlaps and gaps affect karaoke timing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/auto-captions/auto_captions.py` around lines 63 - 66, Update the
caption timing fields in the line-building logic: compute duration from the
latest word end across the complete line rather than relying on line[-1]["end"],
and derive wordRate from the full caption span divided by len(line), preserving
the existing minimum duration/rate safeguards while allowing gaps and overlaps
to influence karaoke timing.
Comment on lines
+79
to
+81
| parser.add_argument("--max-words", type=int, default=4) | ||
| parser.add_argument("--max-seconds", type=float, default=1.8) | ||
| args = parser.parse_args() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Reject invalid grouping limits.
--max-words 0 or a negative value still creates one-word captions. Non-positive, non-finite --max-seconds values also produce misleading grouping behavior. Validate both options after parsing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/auto-captions/auto_captions.py` around lines 79 - 81, Validate the
parsed max_words and max_seconds options immediately after args.parse_args() in
the argument-handling flow: require max_words to be a positive integer and
max_seconds to be finite and greater than zero, rejecting invalid values before
caption grouping begins.
| parser.add_argument("--max-words", type=int, default=4) | ||
| parser.add_argument("--max-seconds", type=float, default=1.8) | ||
| args = parser.parse_args() | ||
| words = load_words(args.transcript) if args.transcript else transcribe(args.audio, args.model) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Apply the same normalization to --audio input.
The --audio path bypasses the sorting and interval validation in load_words(). Extract a shared word-normalization helper and apply it to both transcript and faster-whisper output before grouping captions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/auto-captions/auto_captions.py` at line 82, Extract the sorting and
interval-validation logic from load_words into a shared word-normalization
helper, then apply that helper to both load_words transcript results and
transcribe(args.audio, args.model) output before caption grouping. Preserve the
existing normalized behavior for transcript input and ensure audio-derived words
follow the same ordering and validation.
Comment on lines
+29
to
+33
| The output contains a `clips` array. Add those clips to an existing `project.json` with the MCP `fablecut_patch_project` tool, or merge them into the project document before using `PUT /api/project`. | ||
|
|
||
| ```json | ||
| {"ops":[{"op":"addClip","clip":{"kind":"text","track":"V3","start":0,"duration":1.2,"props":{"text":"Welcome to FableCut","textAnim":"karaoke","wordRate":0.2}}}]} | ||
| ``` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Document the clips to ops conversion for MCP.
The generated artifact contains a top-level clips array, but mcp-server.js applies one addClip operation at a time through ops[].clip. The example shows only one hand-written operation and does not explain how to apply all generated clips. Add a short transformation example or state that each generated clip must become one addClip operation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/auto-captions/README.md` around lines 29 - 33, The README example
should explain how the top-level clips array maps to MCP operations: each
generated clip must become its own addClip operation in ops[].clip when using
fablecut_patch_project. Update the example or surrounding text to show or
clearly state this one-to-one transformation while preserving the existing
project update guidance.
Comment on lines
+31
to
+35
| ```json | ||
| {"ops":[{"op":"addClip","clip":{"kind":"text","track":"V3","start":0,"duration":1.2,"props":{"text":"Welcome to FableCut","textAnim":"karaoke","wordRate":0.2}}}]} | ||
| ``` | ||
|
|
||
| The generated clips default to `V3`, use four words per line, cap a line at about 1.8 seconds, and derive `wordRate` from the observed word durations. Tune the grouping with `--max-words` and `--max-seconds`. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use a timing-consistent wordRate example.
The sample transcript spans 1.2 seconds across three words, but wordRate: 0.2 does not match the documented seconds-per-word timing. Use the generated value, or set this illustrative example to approximately 0.4 seconds per word.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/auto-captions/README.md` around lines 31 - 35, Update the example
addClip payload in the README so its wordRate matches the transcript timing: use
the generated value or approximately 0.4 seconds per word for three words
spanning 1.2 seconds. Leave the surrounding caption-grouping documentation
unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4
What changed
This PR adds the requested
examples/auto-captionsexample without changing the editor itself. It converts engine-agnostic word timestamps into FableCutkind:"text"clips using the built-inkaraoketext animation.Implementation
auto_captions.pyaccepts either a JSON transcript containing awordsarray or a bare word list.wordRatefrom observed speech timing.faster-whisper, while keeping the default path independent of any STT engine.README.mddocuments the transcript format, CLI usage, MCP patch workflow, REST workflow, and optional local transcription.Verification
textAnim: "karaoke", and the expected grouped text.git diff --checksuccessfully.Scope
The example deliberately emits a reviewable
captions.jsonartifact instead of silently overwritingproject.json. This keeps it safe for live editor sessions and lets users inspect or transform captions before applying them.Known limitations
The sample does not bundle an STT model or audio dependency. Users may provide timestamps from any engine, or install
faster-whisperfor the optional local path.Summary by CodeRabbit
New Features
Documentation