Self-hosted v0.0.3: documents over ~64 KiB permanently wedge the ingest queue (128 KiB storage-value limit) — never done/failed, undeletable (409 loop), no cancel

[benediktbluemelhuber]

Summary

On the self-hosted binary (server-v0.0.3), any document whose content exceeds roughly the storage engine's 128 KiB value limit permanently wedges in the ingest queue: it never reaches done or failed, the retry cron re-queues it forever, and it can never be deleted (DELETE returns 409 "Document is still processing" indefinitely, across restarts). There is no ingest-cancel, so the only cleanup is wiping the data dir — which also regenerates the org id and API key.

Fully synthetic, deterministic repro below (no real data involved).

Related (not a duplicate)

• Distinct from Self-hosted v0.0.3: encrypted snapshot OOMs (RangeError in node:crypto) past ~150 MB DB, orphaning docs into a permanent retry-cron loop (plus lossy deletion and a runaway memory agent) #1177 (snapshot path OOM past ~150 MB DB). That one is node:crypto/whole-DB-string; this one is a per-document 128 KiB storage-value limit hit by the embeddings-batch write during ingest of a single large doc, on a tiny store (< 1 MB). The failure signature differs (value is too large (max 128 KiB) + Rollback traversal halted vs RangeError: Out of memory).
• The cloud pipeline ingests the same content fine, so it's specific to the local engine.

Environment

• server-v0.0.3 release binary (supermemory-server-linux-arm64), run in Docker (debian:bookworm-slim, arm64) on Apple M4 / macOS host
• Default local embeddings (Xenova/bge-base-en-v1.5), SUPERMEMORY_EMBEDDING_RAM_LIMIT=2gb, SUPERMEMORY_INGEST_CONCURRENCY=2, extraction via OpenAI-compatible endpoint
• Fresh data dir (only the two repro docs in the store)

Repro (synthetic)

python3 - <<'PY'
import json
s = "Fact %d: the quick brown fox jumps over the lazy dog. "
open("/tmp/doc50k.json","w").write(json.dumps({"content": "".join(s % i for i in range(940))[:50000], "containerTag": "repro"}))
open("/tmp/doc120k.json","w").write(json.dumps({"content": "".join(s % i for i in range(2300))[:120000], "containerTag": "repro"}))
PY
curl -sS -X POST $BASE/v3/documents --oauth2-bearer "$KEY" -H 'Content-Type: application/json' --data @/tmp/doc50k.json    # control
curl -sS -X POST $BASE/v3/documents --oauth2-bearer "$KEY" -H 'Content-Type: application/json' --data @/tmp/doc120k.json  # repro
# poll both ids

Observed

• 50k-char control: queued → embedding → done in ~30 s. ✅
• 120k-char doc: stuck at status:"queued" (>13 min on the otherwise-empty fresh instance; indefinitely — hours, across restarts — on a longer-running instance), never done, never failed.
• On the longer-running instance carrying several such documents (60k–5M chars, mirrored agent-session logs), the ingest/retry cycle logs:

  [workflow] embeddings-batch-1 ✗ value is too large (max 128 KiB)
  [workflow] embeddings-batch-1 ✗ Rollback traversal halted
  [retry-queued] Retried <id> (attempt 1)
  [Cron] Done: N retried, 0 failed
  

  The retry counter never advances past "attempt 1"; the same ids are re-queued every cron cycle, forever.
• DELETE /v3/documents/<id> → 409 {"error":"Document is still processing"} on every attempt, including immediately after a server restart. The doc is effectively immortal.
• Behavior persists across container restarts (state is in the store).

On a real corpus (auto-captured agent session logs), we measured the cliff empirically: every document ≥ ~70k chars wedged; every document ≤ ~63k chars completed. That's consistent with an internal serialized value (chunk/embedding batch) crossing the 128 KiB limit somewhere between those sizes.

Expected

Any of:

1. Chunk the embeddings-batch/storage writes so large documents ingest (the cloud pipeline handles the same content), or
2. Fail fast: mark the document failed with a clear error when a value exceeds the storage limit and stop retrying, and
3. Allow DELETE (or an ingest-cancel) for documents stuck in processing, so operators can clean up without wiping the data dir.

Right now the combination (silent wedge + infinite retry + undeletable + no cancel) turns one oversized document into permanent queue pollution and makes "queue drains" impossible to achieve.

Workaround (for other users)

Split documents client-side to ≤ ~60k chars before POSTing. If a doc is already wedged: stop the server, wipe the data dir, re-ingest — and note the wipe regenerates the org id + API key (they live inside the store).

[linear-code]

ENG-988