Self-hosted v0.0.3: encrypted snapshot OOMs (RangeError in node:crypto) past ~150 MB DB, orphaning docs into a permanent retry-cron loop (plus lossy deletion and a runaway memory agent)

[spdnitian]

Summary

On the self-hosted binary (server-v0.0.3, macOS arm64), once the local DB (~/.supermemory/data/data) reaches about 150 MB, the periodic encrypted snapshot fails every time with RangeError: Out of memory thrown inside node:crypto. The process throws at under 2 GB RSS on a machine with plenty of free RAM, so this is not RAM exhaustion. The snapshot serializes the entire DB into a single base64 JS string that exceeds the engine's maximum string length (about 512 MB, 0x1fffffe8). From there it cascades into a permanent CPU loop, and the only cleanup available is lossy.

Related (not a duplicate)

• Distinct from Self-hosted server: unbounded memory growth in local embedding workers under continuous ingestion (reaches 22–50 GB) #1093 (embedding-worker WASM-heap OOM, fixed). That OOM is in the embedding pool (ort-web WASM heap). This one is in the storage snapshot path (node:crypto), which base64-encodes and encrypts the whole DB as one blob. Different subsystem, different stack trace.
• The ~4 h runaway memory agent (Observed New UI #4) is separate from the existing memory-agent-failure reports (Ollama item_reference, reasoning_content). The request here is specifically a timeout or iteration cap.
• The docs (self-hosting/local-vs-enterprise) say "local is bounded by one machine," but this is a hard failure at ~150 MB on a single machine, not a throughput or scaling limit, and no DB-size limit or snapshot setting is documented.

Environment

• server-v0.0.3 (bun build --compile binary), macOS arm64
• Local embeddings (Xenova/bge-base-en-v1.5), default config
• DB about 156 MB when the failures begin

Observed

1. Snapshot OOM (136 times in one log):

[storage] Snapshot failed
RangeError: Out of memory
    at AR (/$bunfs/root/supermemory-server:42:2067)
    at update3 (node:crypto:278:35)

data/data is a single SMD1-prefixed encrypted blob (the whole DB). Strings in the binary show the snapshot path uses BToA/base64 plus JSON.stringify, and the binary carries the constant 536870888 (0x1FFFFFE8, V8's String::kMaxLength) next to ERR_STRING_TOO_LONG and "Cannot create a string longer than…". So it builds one giant string of the whole DB and hits the max-string ceiling.

2. Orphan retry loop (consequence):

[Cron] Found N stuck/failed documents
[Cron] Marked <id> as failed: Permanently failed (no retry params key)
[Cron] Marked <id> as terminally failed: No retry params on disk (pre-existing doc or store cleared)

When the snapshot fails, per-doc retry-params don't persist, so docs stay at status=failed. The retry cron re-finds and re-marks the same ids every cycle, indefinitely (one id appeared 243 times in a single log), alongside constant [NODE-CRON] missed execution … high CPU warnings. The retry counter (maxRetries/retryCount, capped at 3) appears to live only in the on-disk retry-params file, so once that file is gone the give-up check never fires and the doc is never excluded from the scan.

3. Deletion is lossy (consequence): deleting 4 failed docs with DELETE /v3/documents/{id} dropped the store's unique extracted-memory count by exactly the sum of those docs' .memories (196). So deleting a document also deletes its memories. DELETE also rejects in-flight docs ({"error":"Document is still processing"}), so a stuck queued doc cannot be removed at all.

4. Runaway memory agent (separate):

[Workflow] Document <id> memory agent failed (14644306ms)
WARN [Workflow] Self-hosted memory agent failed for document, skipping memory generation

About 4 hours of CPU on a single document before it gave up.

Likely root cause

The snapshot is monolithic: it serializes and base64-encodes the whole DB into one JS string before encrypting, so peak allocation grows with total DB size and eventually exceeds the engine's maximum single-string length. This is a hard ceiling, independent of free RAM, so it cannot be tuned around and recurs deterministically once the DB is large enough.

Suggested fixes

1. Stream or chunk the snapshot (encrypt in chunks, or use incremental persistence) so it never builds one giant string. This is the core fix.
2. Persist retryCount in the DB row, not only in the on-disk file, so the give-up check survives a lost file. Also exclude "no retry params" and terminally-failed docs from the stuck/failed scan so they cannot loop forever.
3. Add a timeout or iteration cap to the memory agent so a single doc cannot run for hours.
4. Deletion safety: document that DELETE removes the doc's memories, and consider a keepMemories/detach option. Allow deleting stuck queued docs.

Workarounds (for other self-hosters)

• Back up through the API (POST /v3/documents/list, then GET /v3/documents/{id} per doc). The internal snapshot is unreliable once the DB is large.
• Don't purge failed docs to stop the loop, because it deletes their memories.
• A restart sheds the CPU and RAM creep, but the snapshot keeps failing once the DB is large.

[linear-code]

ENG-929