Skip to content

Unbounded memory on large transactions: peek cap escalates until the whole txn fits in RAM #6

Description

@balegas

Found during the production-readiness investigation.

Symptom / failure scenario: decode_batch buffers a whole transaction in tx_envs to stamp commit lsn/xid/seq, and when a transaction doesn't fit in PEEK_CAP (5000 rows) the cap escalates ×4 repeatedly until it does (apps/engine/src/replication.rs:76-166). Under REPLICA IDENTITY FULL each change carries old+new tuples, and the raw peek strings + decoded envelopes + the tailer's per-shape staging (engine.rs:1252-1276) coexist. A 1M-row UPDATE is a multi-hundred-MB-to-GB transient spike — the engine OOMs instead of degrading.

Fix direction (staged):

  1. Guard now: a max-txn-size limit that fails loudly (invalidate affected shapes → client refetch) instead of OOM. Upstream uses 250 MiB (message_converter.ex:96-102).
  2. Spill: past a threshold, spill tx_envs to a temp file, stream through stamping on COMMIT.
  3. Real fix: streaming logical replication with pgoutput proto v2+ (PG 14+), which streams in-progress large transactions — eliminates whole-txn buffering at the source (upstream hasn't done this either; their spill is still a TODO).

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions