Durable job queue with parallel downloads; contain handler errors; workflow gates

[miridius]

Problem

1. A latent crash-loop in the polling loop — real but possibly never fired (not observed in production; identified from the code).

• handlerTimeout defaults to 90 s, but a download is allowed 300 s, so a slow-enough video makes handleUpdate reject mid-flight.
• With no bot.catch registered, telegraf's default error handler rethrows, killing the polling loop.
• bot.launch()'s promise is never awaited, so that becomes an unhandled rejection, which in Bun crashes the process.

If it fired, docker's restart: always would revive the bot and Telegram would redeliver unacked updates, so it could pass for a blip — but a deterministic per-update error (e.g. a corrupt cache entry, also fixed here) would become a crash loop pinned on that update, which the bot.catch below now contains.

2. Serial processing, no durability. telegraf awaits each update batch before polling again, so one slow download stalls every chat until it finishes — and the bot's only durability is Telegram's redelivery of unacked updates, which stops working the moment you stop blocking on handlers.

3. A stale yt-dlp breaks the bot for a day. Extractors rot as sites change out from under them; a broken one makes every download fail until someone notices and manually updates yt-dlp.

4. Failures are all-or-nothing. A transient network blip drops the video outright, while a genuinely-unsupported URL gets retried for nothing — and a user watching a retry sees a fresh "Downloading…" thread spawn each attempt.

5. The filesystem was being used as a database — and it raced. Job queue, parked confirmations, the URL-info cache, and video blobs were all coordinated by raw files under /storage (_jobs/*.json, _pending-downloads/*.json, _video-info/* + symlink aliases, title-templated blob files with sibling .id files). Every multi-step state change was a hand-rolled dance of atomic renames, read-then-unlink, and in-place rewrites — and each one kept producing a fresh race or durability gap (the take/adopt race, the non-atomic retry-persist, the shared-filename unlink race, the stale-memo-after-unlink bug) that we patched one at a time.

6. No automated guardrails on changes. The repo had no enforced review/QA workflow, and its tests were fully mocked — so real filesystem/process/restart bugs (like #1 and #2) could ship undetected.

Fix

Error handling — bot.catch contains handler errors; a fatal polling crash exits logged and deliberate (docker restarts either way); LogMessage's debounce timer catches its own flush errors and failed replies retry; original errors are logged before user notification can eat them; corrupt info-cache entries are discarded and re-scraped; cache-write failures don't fail a finished scrape.

One embedded SQLite database is the durable coordination store (src/db.ts) — the job queue (jobs), parked confirmations (pending), the content-addressed blob index (blobs), and the URL-info cache (video_info) all live as tables in a single bun:sqlite connection (WAL, busy_timeout). bun:sqlite is built into Bun — no new dependency, no separate daemon (the server is weak), no network hop — and synchronous, so store operations are plain function calls. Every former multi-step FS dance is now one transaction, which is the race fix at the root: the take/adopt race, the non-atomic retry-persist, the shared-filename unlink race, and the stale-memo class all disappear instead of being patched. Schema is versioned by PRAGMA user_version (append-only migrations, each applied once in a transaction).

Content-addressed blob store (src/blob-store.ts) — video bytes can't live in SQLite, so they move to /storage/blobs/<sha>.<ext> keyed by sha256(extractor:id:format_id) (filename-hash fallback). The key is known from --dump-json before download, so the "we already have this" short-circuit survives; yt-dlp writes to its template path and the bytes are then renamed to their content-addressed home. The blobs row owns the path, size, and the Telegram file_id (replacing the old .id sibling file) and is the refcount target: a pending row referencing a blob keeps its bytes alive, and releaseBlob deletes the bytes only when no row references the key and file_id IS NULL. So "do we have it?" = the row has a file_id or the bytes are still on disk; a GC'd blob has neither and is re-downloaded. That invariant makes recovery safe and structurally kills the stale-memo bug class. Two concurrent releases can't both unlink — the second sees the row already gone.

Durable, parallel job queue (src/job-queue.ts) — message handlers persist a job row and return; an in-memory worker pool processes up to 3 jobs in parallel and rebuilds its queue from the jobs table on boot, in FIFO order (ORDER BY id). yt-dlp concurrency is capped at 3 inside execYtdlp, so inline queries count against the same budget. Semantics:

• At-least-once delivery. A job row persists until its work fully completes; recovery re-runs anything left over, so a restart never loses a video. Re-runs are cheap — a finished download is served from the blob store (file_id or bytes on disk).
• Duplicates minimized, not eliminated. The only way at-least-once can duplicate is a crash in the window between a send completing and the queue row being deleted (or the file_id being recorded) — with a 30 s compose stop_grace_period so a planned restart lets in-flight sends finish; a duplicate needs a hard crash (OOM/power) landing in those milliseconds.
• Confirmation is one atomic claim. A parked confirmation is a pending row; confirming it is a single tx() that DELETE … RETURNINGs the pending row and INSERTs the job (adoptJob). Confirm and cancel both DELETE the same row, so exactly one wins — the old confirm→enqueue crash window is closed, and the loser cleanly reports the confirmation gone.
• Retry is persisted before re-queue. A transient failure bumps attempts (and re-serializes the job, so a mutated logMessageId survives) before scheduling the backoff timer; if that write fails the job is dropped rather than orphaned with a stale count. Backoff stays in-memory (a restart re-runs immediately, harmless under at-least-once).
• SIGTERM stops new jobs from starting; in-flight ones finish (the process stays alive until they do); queued rows survive for the next boot.
• handlerTimeout is set to 5 min and only matters for inline queries (which still download in-handler); an inline timeout is logged and left to finish detached, while a timeout on the enqueue-only handlers is logged as a real error.

yt-dlp self-update (updateYtdlp) — a background worker keeps yt-dlp fresh: it copies the binary, runs --update on the copy, then atomically renames it into place. The copy is required because the zipapp's --update rewrites the binary in place, non-atomically — a download exec'ing it mid-rewrite would get a corrupt file; the rename means running execs keep the old inode and new execs see old-or-new, never a partial. It's single-flight and polls every 5 min so a broken extractor is fixed in minutes, not a day.

Failure classification + per-attempt reporting — the queue retries transient download failures (network, 5xx, 408/429, a transient fragment 404) — up to 3 attempts (2 retries) — but fails fast on permanent ones (unsupported URL, private/removed/age-gated video, a webpage 404/410). Because the updater keeps yt-dlp current, an extractor that still can't handle a URL means it's genuinely unsupported, not stale — so retrying is pointless. Each attempt edits one message — ⚠️ "retrying (attempt X of Y)…" → a terminal 💥 — instead of spawning a fresh thread. LogMessage became dest-agnostic for this: url jobs log progress in private chats (a NoLog in groups, which would otherwise be spammed), and confirmed jobs report failures through the same group-capable LogMessage, so there's one send/edit/gone-message path instead of a hand-rolled copy.

Tests mock only unowned boundaries — SQLite and the filesystem are ours, so they run for real: download-video.test.ts and the queue/blob suites use a real DB (resetDb() between tests) and real child processes (test/bin/ stubs on PATH, driven by control files because Bun.spawn snapshots the env at startup); only the Telegram API stays mocked, at the fetch boundary via MockBotApi. The restart/recovery seam — the one testing.md names as exactly what mocks miss — gets a real e2e: a bot boots, recovers a job persisted before a crash, and delivers the video (plus a failure variant that reports ⚠️→⚠️→💥 in one edited message across the 3 attempts) through real SQLite persistence and the blob store.

Workflow skills (.claude/skills/) — /commit, /pr, /merge gate every change through review, manual QA against the live bot, and a final merge check, so the green merge gate always sits on the exact commit being merged. The gate is enforced deterministically: a review-attestation step mints an approval bound (by git write-tree + HEAD) to the exact content reviewed, and a pre-commit / pre-merge hook fails closed when that content-bound approval is absent — so the gate can't be inherited from an earlier commit or skipped.

Decisions (settled)

• bun:sqlite over the FS-as-DB layer. Embedded, ACID, durable, zero new dependency, no daemon, in-process — the transaction is the race fix. Bytes that can't go in the DB move to a content-addressed on-disk layout the DB refcounts.
• Delivery is at-least-once, duplicates minimized as above (a crash in the send→row-delete / send→file_id window). A rare duplicate beats silently losing a video.
• A blob's bytes are GC'd only when file_id IS NULL and nothing references it. That single invariant is what makes recovery re-download a dropped blob instead of replaying a stale memo.
• Keep yt-dlp fresh with a background worker, not per-call. A 5-min poll never blocks the hot path and is one unauthenticated GitHub call per tick (well under the 60/hr/IP limit); updating a copy + atomic rename (rather than --update in place) is what makes a concurrent download safe during an update.
• Retry transient failures only; classify off yt-dlp's ERROR: prose. Assumes the updater keeps yt-dlp current, so unsupported/extraction errors are permanent. (Group url-job failures are still silent — that's pre-existing and tracked for a separate PR.)
• bot.catch no longer sets process.exitCode on contained errors. The bot keeps polling, so a stale per-update error shouldn't make a later graceful shutdown report failure. Fatal polling crashes still exit(1).
• Inline queries deliberately bypass the queue — an inline answer expires in seconds, so durability is pointless; they download in-handler under the 5 min timeout.
• Tunables: JOB_CONCURRENCY = 3, YTDLP_CONCURRENCY = 3, YTDLP_UPDATE_INTERVAL = 5 min, stop_grace_period = 30 s, MAX_ATTEMPTS = 3.

🤖 Generated with Claude Code