Found during the production-readiness investigation.
Symptom / failure scenario: decode_batch buffers a whole transaction in tx_envs to stamp commit lsn/xid/seq, and when a transaction doesn't fit in PEEK_CAP (5000 rows) the cap escalates ×4 repeatedly until it does (apps/engine/src/replication.rs:76-166). Under REPLICA IDENTITY FULL each change carries old+new tuples, and the raw peek strings + decoded envelopes + the tailer's per-shape staging (engine.rs:1252-1276) coexist. A 1M-row UPDATE is a multi-hundred-MB-to-GB transient spike — the engine OOMs instead of degrading.
Fix direction (staged):
- Guard now: a max-txn-size limit that fails loudly (invalidate affected shapes → client refetch) instead of OOM. Upstream uses 250 MiB (
message_converter.ex:96-102).
- Spill: past a threshold, spill
tx_envs to a temp file, stream through stamping on COMMIT.
- Real fix: streaming logical replication with pgoutput proto v2+ (PG 14+), which streams in-progress large transactions — eliminates whole-txn buffering at the source (upstream hasn't done this either; their spill is still a TODO).
🤖 Generated with Claude Code
Found during the production-readiness investigation.
Symptom / failure scenario:
decode_batchbuffers a whole transaction intx_envsto stamp commit lsn/xid/seq, and when a transaction doesn't fit inPEEK_CAP(5000 rows) the cap escalates ×4 repeatedly until it does (apps/engine/src/replication.rs:76-166). UnderREPLICA IDENTITY FULLeach change carries old+new tuples, and the raw peek strings + decoded envelopes + the tailer's per-shape staging (engine.rs:1252-1276) coexist. A 1M-row UPDATE is a multi-hundred-MB-to-GB transient spike — the engine OOMs instead of degrading.Fix direction (staged):
message_converter.ex:96-102).tx_envsto a temp file, stream through stamping on COMMIT.🤖 Generated with Claude Code