test: replication receive-side stress regressions (catch-up memory + blob save)

[kriszyp]

Summary

Two cluster integration tests targeting the receive-side failure modes that knocked a prod node off the cluster earlier this month. Both are guards against regression of fixes already on main / pending review.

Test	Guards	Currently in main?
receiveBacklogMemory.test.mjs	PR #147 (RECEIVE_EVENT_HIGH_WATER_MARK backpressure)	yes — should pass
blobSaveRejectionContainment.test.mjs	PR #149 (outstandingBlobsToFinish stores catch-handled promise)	no — passes only after #149 merges

Why

In early May a Harper Pro 5.0.16 node OOM'd repeatedly during peer catch-up. Two distinct receive-side defects compounded:

1. onWSMessage decoded every audit record in a WS message synchronously inside a tight do { ... } while (...) loop. A single message with thousands of records ballooned heap past the 2 GB old-gen ceiling — ERR_WORKER_OUT_OF_MEMORY every ~25 s. Fixed by PR fix: bound replication receive memory to stop worker OOM crash loops #147, which yields when the consumer queue exceeds RECEIVE_EVENT_HIGH_WATER_MARK = 100.
2. When saveBlob rejected (the same node was producing Blob save failed for X from peer Y at ~35/s, ENOENT during catch-up), the raw rejecting promise sat in outstandingBlobsToFinish and await Promise.all(...) inside end_txn's onCommit propagated it out as uncaughtException. Fixed by PR fix(replication): swallow blob save rejection in outstandingBlobsToFinish #149, which stores the catch-handled promise instead.

These tests reproduce both conditions directly so future changes can't quietly reintroduce either.

What each test does

receiveBacklogMemory.test.mjs

• Brings up a 2-node cluster.
• killHarper(B), then bursts 40 transactions × 500 records each on A — each transaction is a single WS message with 500 audit entries (well past the HWM of 100). Total backlog: 20 k records.
• Restarts B; samples system_information.memory.rss every 500 ms while B catches up.
• Polls describe_table.record_count for an unambiguous catch-up signal (no dependency on cluster_status.lastReceivedVersion shape).
• Asserts: catch-up completes, no ERR_WORKER_OUT_OF_MEMORY in log, peak RSS < 1.5 GB.

blobSaveRejectionContainment.test.mjs

• Brings up a 2-node cluster with the existing Location blob-bearing fixture deployed to both.
• Pre-installs a new fault-injection component on B only (via setupHarperWithFixture) that monkey-patches fs.createWriteStream to emit ENOENT for every 7th call targeting /blobs/. The injector arms only when HARPER_TEST_BLOB_FAIL_INTERVAL is set, so it's inert if the fixture is ever picked up elsewhere.
• Drives 400 /Location/{n} requests on A → each creates a blob via sourcedFrom → replicates to B → every 7th blob save on B trips the injector.
• Asserts:
  1. The injector actually fired (test is non-vacuous).
  2. [error] [replication]: Blob save failed for <id> from <peer> appears (the .catch ran).
  3. No uncaughtException lines mentioning Blob/ENOENT.*blobs — this is the regression.
  4. B still reports A connected in cluster_status.
  5. A fresh write on A still propagates to B after the failures (liveness).

Where to look

• fixture-blob-fail-injector/resources.js — the monkey-patch uses createRequire to get the CJS fs module (ESM namespace objects are frozen) and replaces createWriteStream. Harper's dist code uses require('node:fs').createWriteStream(...) at call time, so the patch is picked up live. Confirmed via cross-model review.
• The BATCH_SIZE = 500 and FAIL_INTERVAL = 7 are tuned to comfortably exceed the relevant thresholds without blowing test time. Could be made more aggressive if CI runners turn out to be faster than expected; the bounds in the assertions stay valid either way.
• The 1.5 GB peak-RSS bound is intentionally generous. The bug burst past 2 GB; anything well under that means the HWM-driven pause is taking effect. Tuning down later is easy and orthogonal.

Risk / known-flaky areas

• Local validation hit a pre-existing harness/env issue ("Maximum call stack size exceeded" inside replicationTopology.test.mjs setup) that also affects main. The new tests are structurally identical to replicationLoad.test.mjs and should run cleanly in CI even though local was noisy.
• blobSaveRejectionContainment will fail on main today because PR fix(replication): swallow blob save rejection in outstandingBlobsToFinish #149 isn't merged yet. Either land that first, or this PR is the trailing half of the same change and lands after.

Testing

• npx oxlint --quiet on all new files → 0 errors, only the pre-existing new Array(concurrency) warning in clusterShared.mjs that lint:required already tolerates on main
• npx prettier --check → clean
• node --check on all new mjs/js → clean
• Cross-model review (Gemini): positive; no required changes

Test plan

• CI passes for receiveBacklogMemory (it should — PR fix: bound replication receive memory to stop worker OOM crash loops #147 is in main)
• Once fix(replication): swallow blob save rejection in outstandingBlobsToFinish #149 merges, blobSaveRejectionContainment passes
• Spot-check on a CI run that the injector banner appears in B's log (grep '[blob-fail-injector] installed' ...) — assertion 1 should fail loudly if not

🤖 Generated with Claude Code

[claude]

Reviewed; no blockers found.