Gobbler

CLI-first content conversion to Markdown for humans and AI agents

Gobbler converts YouTube transcripts, audio/video files, documents, web pages, and intentionally selected browser tabs into structured Markdown. The public automation contract is the gobbler CLI; the Skills in this repository teach agents to use that same CLI.

Project status

Gobbler is an active beta. Current supported surfaces are:

• gobbler for conversion, diagnostics, batches, jobs, providers, and browser automation.
• Local Markdown Skills for CLI-capable AI agents.
• An optional Chromium extension and local relay for tabs deliberately placed in a Gobbler tab group.

Websites and AI chat products can change their DOM without notice, so browser integrations are more fragile than direct file and URL conversion.

Install from source

Requirements: Python 3.11+ and uv. Docker is optional for document/webpage conversion; ffmpeg is required for audio/video transcription and YouTube frame extraction.

git clone https://github.com/Enablement-Engineering/gobbler.git
cd gobbler

# Reproducible source-checkout environment
uv sync
uv run gobbler --version
uv run gobbler doctor --json

Run commands from the checkout with uv run gobbler ..., or install an isolated global CLI:

uv tool install .
gobbler --version

For local development, use uv sync --extra dev. See the installation guide for Docker and browser setup.

Convert content

# YouTube transcript; URL can also come from stdin
uv run gobbler youtube "https://youtube.com/watch?v=VIDEO_ID" -o transcript.md

# Deterministic visual overview; no transcript provider is called
uv run gobbler youtube "https://youtube.com/watch?v=VIDEO_ID" \
  --frames-only --frames 8 -o overview.md

# Repeat exact timestamps or refine an inclusive range
uv run gobbler youtube "https://youtube.com/watch?v=VIDEO_ID" \
  --frames-only --frame-at 24:16.500 --frame-at 24:18.200 -o exact.md
uv run gobbler youtube "https://youtube.com/watch?v=VIDEO_ID" \
  --frames-only --frame-range 24:12-24:24 --range-frames 5 -o refinement.md

# Local faster-whisper transcription
uv run gobbler audio meeting.mp3 --model small -o meeting.md

# Docling service required
uv run gobbler document report.pdf --no-ocr -o report.md

# Crawl4AI service required
uv run gobbler webpage "https://example.com" --no-proxy -o page.md

Without -o, normal Markdown output is written to stdout. --format json produces a single JSON object for individual conversions. Batch commands use --json for newline-delimited JSON events.

Frame requests require durable storage through --output or --frames-dir. Gobbler writes deterministic JPEG files under <output-stem>.assets/frames/ by default and returns timestamped Markdown/JSON manifests. Gobbler extracts the requested frames; the calling human or agent interprets them. This release does not add frame extraction for playlists or local video files. See the YouTube frame workflow.

Service readiness

YouTube and local audio conversion do not need Docker. Document and webpage conversion use the services in docker-compose.yml:

make start-docker
uv run gobbler doctor --json

• Crawl4AI: http://localhost:11235
• Docling: http://localhost:5001

gobbler doctor --json is the broad agent-friendly diagnostic. gobbler status --json focuses on conversion-provider readiness and exits nonzero when overall status is degraded while still emitting JSON.

Batch and queued work

# Playlist output directory is required
uv run gobbler batch youtube-playlist \
  "https://youtube.com/playlist?list=PLAYLIST_ID" \
  -o ./transcripts --dry-run

# Local files
uv run gobbler batch directory ./documents -o ./markdown --pattern "*.pdf"

# URL list; one URL per line
uv run gobbler batch webpages urls.txt -o ./pages --dry-run

# Explicitly queue a webpage batch in the SQLite job store
uv run gobbler batch webpages urls.txt -o ./pages --queue --json
uv run gobbler jobs worker start
uv run gobbler jobs list

Queued work is opt-in through --queue. The current queue is SQLite-backed; Redis/RQ is not required.

Providers and configuration

Inspect the runtime instead of relying on a static provider list:

uv run gobbler providers list --format json
uv run gobbler config path
uv run gobbler config show
uv run gobbler config init

The default config file is ~/.config/gobbler/config.yml. Configuration is deep-merged over built-in defaults. CLI flags take precedence for the commands that expose them. Environment variables are used by specific integrations, not as a general override system:

• TRANSCRIPTAPI_KEY: paid YouTube transcript fallback.
• WEBSHARE_USER / WEBSHARE_PASS or YOUTUBE_PROXY: YouTube proxy configuration.
• CRAWL4AI_PROXY: webpage proxy fallback.
• OPENAI_API_KEY: openai-whisper transcription provider.
• CRAWL4AI_API_TOKEN: Docker Compose API token.
• GOBBLER_MODELS_PATH: Docling model-cache mount.

See the configuration guide for the current schema.

Browser extension

The extension communicates with a local relay on port 4625. Most browser operations auto-start it; browser status is intentionally read-only and does not.

uv run gobbler relay start
uv run gobbler browser status
uv run gobbler browser list
uv run gobbler browser extract -o page.md
uv run gobbler notebooklm query "Summarize the current notebook"

Browser command guards target tabs whose group ID matches the extension's stored Gobbler group. Use Allow & Add/Add Tab to create or populate it and grant origin access for extraction and page APIs. A different group with the same title does not match, while manually moving a tab into the existing managed group makes it eligible for commands; debugger-based browser exec does not require the separate origin permission. The extension does not bypass authentication, access controls, rate limits, or bot detection. See the browser extension guide.

AI-agent Skills

# Inspect available Skills
npx skills@latest add Enablement-Engineering/gobbler --list

# Interactive install
npx skills@latest add Enablement-Engineering/gobbler

# Install the main conversion Skill globally
npx skills@latest add Enablement-Engineering/gobbler \
  --skill gobbler --global --yes

This installs Skill files only; it does not install the Gobbler CLI. The repository currently provides:

• gobbler: conversion and batch workflows.
• gobbler-browser: authenticated-tab and supported AI-chat workflows.
• gobbler-setup: installation, diagnostics, and service troubleshooting.

Output contract

Successful Markdown conversions include YAML frontmatter when the selected converter provides metadata. For automation:

• Individual --format json commands emit one JSON object.
• Batch --json commands emit JSON Lines.
• Versioned conversion, diagnostic, batch, and job JSON contracts include schema_version: 1.
• Configuration, provider-registry, and browser JSON outputs are command-specific and currently unversioned.
• Error payloads use stable error_code values where documented.
• Credential-bearing URL components are sanitized in diagnostic payloads.

Development

uv sync --extra dev
uv run pytest tests/unit/ -q --tb=short
uv run ruff format --check src/ tests/
uv run ruff check src/ tests/
uv run --extra docs mkdocs build --strict

Project layout:

src/gobbler_cli/       Typer CLI
src/gobbler_core/      converters, providers, configuration, utilities
src/gobbler_queue/     SQLite job queue and worker
src/gobbler_relay/     browser-extension HTTP/WebSocket relay
browser-extension/     Chromium extension
skills/                CLI instruction Skills

See CONTRIBUTING.md, SECURITY.md, and the full documentation.

License

MIT License. See LICENSE.