Chunked media uploads to bypass Vercel's 4.5 MB request limit

[carlosjdelgado]

Problem

Vercel caps request bodies at 4.5 MB. Any media upload above that fails outright — including images embedded from the rich-text editor and direct uploads from the media browser.

Solution

Split uploads into chunks ≤ 4 MB, stage them server-side, and reassemble at finalize time. The first chunk rides inline in the finalize request, so files ≤ 4 MB still complete in a single round-trip with zero DB writes.

Wire surface

All media writes — file uploads and folder markers — share one endpoint prefix: /api/[owner]/[repo]/[branch]/media/[name]/[path].

Method	Path	Purpose
POST	/media/[name]/[path]/chunk	Stage one chunk (uploadId, idx, chunk blob). idx must be in [1, MAX_CHUNK_IDX] — chunk 0 rides inline in finalize.
POST	/media/[name]/[path]	Finalize: reassemble inline firstChunk + DB-staged chunks, commit to GitHub. Accepts a 0-byte firstChunk when path ends in .gitkeep (folder marker).

Chunks are scoped to (uploadId, userId); one user cannot read another's staging buffer.

Limits

Constant	Value	Why
CHUNK_BYTES	4 MB	Fits in multipart body under the 4.5 MB Vercel cap (overhead < 1 KB)
MAX_TOTAL_BYTES	15 MB	Per-upload ceiling, enforced server-side after reassembly
MAX_CHUNK_IDX	derived (ceil(MAX_TOTAL_BYTES / CHUNK_BYTES) - 1)	Tight cap at the chunk-staging endpoint; prevents DB and memory DoS
CHUNK_CONCURRENCY	4	Parallel chunk uploads (batched)
STALE_CHUNK_AGE_MS	10 min	Stale-chunk TTL; cleanup runs piggyback on every finalize

CHUNK_BYTES and MAX_TOTAL_BYTES live in lib/utils/upload-media.ts and are imported by both the client helper and the server routes. MAX_CHUNK_IDX is derived locally in the chunk handler.

Storage

New table upload_chunk (db/migrations/0013_upload_chunks.sql):

• Composite PK: (upload_id, chunk_idx)
• data stored as bytea (not base64) for size + perf
• user_id and created_at for ownership and TTL cleanup

Client surface

lib/utils/upload-media.ts exports uploadMediaChunked — a single source of truth used by every callsite that writes to media:

uploadMediaChunked({
  file,                // a File (including 0-byte for .gitkeep markers)
  owner, repo, branch,
  mediaName,           // media schema name
  targetPath,          // path inside schema.input
  onConflict,          // "error" | "rename" (default rename)
});

Callers wired to it in this PR:

• components/media/media-upload.tsx — media browser uploader (drag-drop, click)
• fields/core/rich-text/edit-component.tsx — inline image upload from the rich-text editor
• components/folder-create.tsx (media branch) — .gitkeep folder marker creation

Refactors bundled

1. githubSaveFile extracted to lib/utils/github-save-file.ts. Previously inline in files/[path]/route.ts; now shared between that route and the media POST.

2. Rich-text image upload migrated from the JSON POST /files/[path] route (type: "media", base64 in body) to the chunked endpoint. Per-upload cap rises from ~4.5 MB (Vercel JSON body limit) to 15 MB. Drops the local FileReader/base64 conversion.

3. .gitkeep for media folders moved from POST /files/[path] (type: "media", JSON content: "") to the chunked endpoint with a 0-byte inline firstChunk. The server special-cases .gitkeep: skips the extension check and allows an empty firstChunk.

4. POST /files/[path] case "media" removed entirely. No live callers remain after points 2 and 3. The files endpoint now handles only content and settings.

What did NOT change

• POST /files/[path] for content and settings (collection/file folder markers and the .pages.yml config) — unchanged.
• DELETE / rename surfaces for media — unchanged.

Test plan

• Upload media file under 4 MB → single request, zero rows in upload_chunk
• Upload media file between 4 MB and 15 MB → multi-chunk path, staged then cleaned
• Upload > 15 MB → client-side cap rejects before any network call
• Upload via rich-text editor → uses the same flow; image renders in document
• Create media folder via the media browser → 0-byte .gitkeep round-trips through the chunked endpoint
• Create content folder via the collection view → still goes through POST /files/[path]
• Migration 0013 applies cleanly on a fresh DB and on a DB already at 0012