test(stress): long-running soak + worker cascade + orphan + adversity

[kriszyp]

Summary

Four long-running replication stress tests targeting the production failure modes observed on wtk-ap-west-1 in early May, plus a workflow scaffolding to run them weekly (or on-demand) without affecting the PR-blocking integration matrix.

Test	Guards	Default (workflow / local)
soakWithRollingRestarts	The full wtk recipe: rolling SIGKILL + restart under continuous prerender-style traffic. Catches OOM, listener leaks, blob orphans, convergence drift.	4 h / 20 min
workerExitCascade	PR #147's WORKER_EXIT_REASSIGN_STAGGER_MS = 100. The existing receiveBacklogMemory test can't exercise this — it runs with THREADS_COUNT=1.	one-shot, ~45 s
blobOrphanRace	The qub Error sending blob … ENOENT pattern we couldn't fully diagnose. Heavy supersede churn over a small keyspace + a mid-test restart. If we ever do hit the orphan, the test description points reviewers at the bug.	60 min / 15 min
rapidReconnectAdversity	Rapid kill+restart cycles. Same code-path coverage as a tc/netem adversity proxy (connect, resubscribe, blob resume, listener cleanup) but without NET_ADMIN. Asserts no MaxListenersExceededWarning in node logs — guards the recent #161/#173 leak fixes.	30 min / 10 min

All four are gated on HARPER_RUN_STRESS_TESTS=1. The disabled branch registers a single skipped placeholder test so the normal integration runner treats the file as a no-op.

Why its own workflow

The longest of these (soak) runs ~4 h with defaults. PR shards time out at 15 min and the matrix is already 12 jobs. Stress tests live in .github/workflows/stress-tests.yaml — workflow_dispatch (per-test duration knobs in the dialog) + weekly Sunday cron — so they don't slow PR signal and still get regular coverage.

Local validation runs

soakWithRollingRestarts (5 min, 5 kill cycles INCLUDING WRAP, 4 nodes × 4 workers):
  ✔ cluster survives sustained traffic + rolling SIGKILLs ...
  Final record_count: 1500, 1500, 1500, 1500 — exact convergence
  Peak RSS per node: 722–738 MB (limit 1.5 GB)
  0 OOM, 0 uncaught, 0 blob orphan markers
  ctx.nodes wrap (cycle 5 kills node 0 again) ✓ succeeded with the fix

workerExitCascade (~45 s, 4-worker B node, 6 dbs, single kill):
  ✔ killing a worker mid-load reassigns subscriptions with ≥80 ms stagger ...
  7 reassignments at 117/217/317/417/517/617/717 ms — exact 100 ms spacing,
  matches WORKER_EXIT_REASSIGN_STAGGER_MS = 100
  1 new PID, 1 gone PID — no cascade

blobOrphanRace (5 min, 80-key churn, ~14 700 writes, mid-test B restart):
  ✔ heavy supersede churn does not orphan blobs on the sender
  Final: A=915 B=915 — converged after 120 s drain
  0 "Error sending blob ENOENT", 0 uncaught on both nodes

rapidReconnectAdversity (6 min, 24 kill+restart cycles across 3 nodes,
  ~8 wraps):
  ✔ rapid kill+restart cycles surface no listener leaks or uncaughts
  Final record_count: 1250, 1250, 1250 — exact convergence
  0 OOM, 0 uncaught, 0 MaxListenersExceededWarning *in node logs*

(Note: the integration-testing harness emits MaxListenersExceededWarning: 11 exit listeners added to [process] around cycle 8 of adversity — that's the harness's process.on('exit', …) accumulating across many startHarper calls, not anything in Harper. My assertion scans node logs only, so it doesn't fail the test. Worth filing separately against the harness.)

A longer soak attempt (10 min, ~10 cycles, 2.5 wraps) failed with no test-body console output captured — looked like a node:test reporter/buffering interaction triggered around a wrap cycle. All assertions appear to have held (the harper logs showed 0 OOM / 0 uncaught / 0 orphans), but the test result line lacked detail. The 5-min single-wrap run above passes cleanly, and the 4-h CI run will produce richer artifacts. Worth investigating if it recurs.

Two iteration fixes from local validation

1. ctx.nodes[idx] update after restart. I originally didn't write the new harper handle back to the array. Round-robin cycles that wrap (cycle ≥ NODE_COUNT) then killed the stale ChildProcess reference — already dead — and startHarper then tried to bind a hostname:port already held by the still-running harper from the previous restart. Now ctx.nodes[idx] = restartCtx.harper. Applied to soak + adversity. Confirmed working on soak4 (5 cycles with 1 wrap, exact convergence) and adversity (10 cycles with 3 wraps, exact convergence).
2. Relaxed convergence to ≤ 1 % drift + polling. Strict equality is too brittle under heavy restart churn — a 1–2 record tail in flight at the moment traffic stops shouldn't fail the test. Now: poll for convergence (60–120 s) and assert drift < 1 %. Same relaxation across soak, orphan, and adversity. (blobOrphanRace also drops a bogus maxCount <= KEYSPACE upper bound — Harper's sourcedFrom cache produces more record versions than the unique-key count under churn.)

Fixtures

• fixture-prerender-workload/ — sourcedFrom cache table with deterministic-but-bimodal blob sizing (60 % under 8 KB → inline, 40 % above → file-backed). Matches the wtk production mix and exercises both blob storage paths in one workload.
• fixture-suicide-worker/ — single REST endpoint /SuicideWorker that returns { threadId, pid } and schedules process.exit(137) via setImmediate so the HTTP response flushes before the worker dies. Lets the cascade test kill exactly one worker on demand.

Shared helpers (stressShared.mjs)

• clusterSnapshot(node) — uniform-shape view of cluster_status.
• waitForAllConnected(node), waitForRecordCount(node, table, target) — bounded polling.
• sampleMetrics(node, { intervalMs }) + summariseSamples(samples) — in-process memory/thread sampler for post-run analysis.
• fetchWithRetry with AbortSignal.timeout so a stalled connection to a mid-kill node can't hang the test.
• readLog(node) — checks node.logDir first (set by the harness when HARPER_INTEGRATION_TEST_LOG_DIR is in env) before falling back to {dataRootDir}/log/hdb.log. Same fix as test: replication receive-side stress regressions (catch-up memory + blob save) #150.

Where to look

• Workflow matrix is fail-fast: false and timeout-minutes: 260 to fit the 240 min soak default. Per-test duration knobs surface in the workflow_dispatch dialog.
• stressShared.mjs:fetchWithRetry — added a 5 s per-attempt AbortSignal.timeout. Without it a fetch against a mid-kill node could hold the test for the full retry budget. Bump to 10 s if CI runners turn out to be slower than expected.
• Convergence drift threshold (1 %) — strict enough to catch a permanent fork, lenient enough to absorb the tail of unreplicated writes. For very small keyspaces, 1 % still means strict equality.
• fixture-suicide-worker uses setImmediate(() => process.exit(137)) so the HTTP response flushes before the worker dies. The cascade test's [cascade] suicide response: {"threadId":1,...} line confirms this works.

How to run

# Local, short:
HARPER_RUN_STRESS_TESTS=1 \
HARPER_STRESS_SOAK_MINUTES=10 \
  npm run test:integration -- integrationTests/stress/soakWithRollingRestarts.test.mjs

# CI, weekly: automatic via cron 11 6 * * 0
# CI, on-demand: Actions → Stress Tests → Run workflow (per-test duration knobs in the dialog)

What this PR does NOT do

• It does not modify any production code. Tests + fixtures + workflow only.
• It does not address the underlying blob-orphan bug. If blobOrphanRace reproduces it in the workflow, the test description sets up reviewers to act on it.

🤖 Generated with Claude Code

---

Update 2026-05-20: replication-correctness suite

Adds four more long-running stress tests in commit feaa1a2, aimed at replication scenarios the original four don't cover. Same gating, same workload fixture, same convergence/drift conventions.

Test	Guards	Default (local)
replayCatchupSeam	The boundary between replayLogs.ts (on-restart audit-tail replay) and live replication catch-up. SIGKILL with HARPER_NO_FLUSH_ON_EXIT=true forces replay to fire; meanwhile peers send catch-up. Asserts no double-apply / no loss and that the replay path actually executed (≥1 "Replayed N records" warn — otherwise the seam wasn't exercised).	~2 min
backlogRecovery	Cold-resume: one peer offline for minutes while the others churn; rejoin and drain. Asserts peer-side per-peer queue is bounded (peer RSS stayed flat between 2-min and 5-min offline windows despite ~2× backlog — strong signal that the sender isn't linearly buffering) and the rejoining node catches up without OOM.	~3–9 min
slowConsumerBackpressure	A (4 worker threads) vs B (1 worker thread), high-concurrency churn. Asserts A's RSS stays bounded and the cluster reconverges. backPressurePercent observation is a soft warn (logged, not asserted) — on loopback, the asymmetry may not be enough to actually trigger backpressure; in local validation it wasn't.	~5 min
partitionHealConvergence (+ replicationProxy.mjs helper)	Split-brain. Routes B→A through a controllable userspace TCP proxy; partition flips the proxy to "blocked" while both sides keep writing; assert post-heal record_count equality and per-key agreement on sampled ids. Skipped by default (requires both HARPER_RUN_STRESS_TESTS=1 and HARPER_STRESS_ALLOW_INSECURE_REPLICATION=1) — blocker is in the file header.	n/a

Local validation

replayCatchupSeam (30s pre-kill + 45s post-kill, 3-node mesh):
  ✔ post-crash replay overlaps with catch-up without duplicating or losing rows
  Final record_count: 200, 200, 200 — exact convergence
  5,020 writes, 2 "Replayed N records" warns — seam was exercised
  0 replay errors, 0 uncaught, 0 orphan markers
  Total: 133s

backlogRecovery (5-min offline, 4-node mesh):
  ✔ peer stays online and absent node catches up without OOM
  Final record_count: 800, 800, 800, 800 — exact convergence in 3s catchup
  28,201 writes accumulated peer-side during B's absence
  Peer peak RSS: 443–453 MB (unchanged vs the 2-min run with 11k writes)
  B catchup peak: 748 MB (under 1.5 GB cap)
  Total: 364s

slowConsumerBackpressure (3-min asymmetric churn, A=4 threads / B=1 thread):
  ✔ sender does not OOM and cluster reconverges after slow-consumer pressure
  Final record_count: A=500 B=500 — exact convergence
  A peak RSS: 452 MB across 32,872 writes
  WARN: no backPressurePercent > 0 observed during the run (asymmetry on
        loopback insufficient to actually trigger; logged as warn so the
        signal is visible without failing the test).
  Total: 241s

partitionHealConvergence (skipped by default — see file header).

Cross-model review (agy / Antigravity CLI)

Ran a cross-model review on the new files. Acted on:

• Re-entrant cleanup in replicationProxy.mjs (close events fire on both ends; added a cleaned guard).
• Expanded the partition-test header with two alternative paths agy surfaced — bind the proxy to the same IP as A on a different port (since TLS SAN check ignores port) and the secondary concern that Harper may dial both directions independently (so a single proxy might not actually partition).

Declined:

• agy's "suite ctx callbacks aren't mutable" finding — the existing four stress tests already use the same (ctx) => { ctx.nodes = ... } pattern and pass; finding doesn't apply to this version of node:test.
• "Timer leaks on test failures" — node:test runs each test in process isolation, so the test-process exit reclaims setInterval handles. Not a real leak in practice.

Side observation (separate follow-up)

The manageThreads.restartWorkers path in core emits MaxListenersExceededWarning: 11 exit listeners added to [Worker] once during deploy_component with restart: true, on a node with threads.count: 4. Surfaced by backlogRecovery while I was developing it. Real but unrelated to this PR; flagged separately.

🤖 Generated with Claude Code

[claude]

Stale B reference after restart — post-restart log assertions silently incomplete in CI

startHarper receives a fresh { dataRootDir, hostname } object and mutates it in-place (adding logDir, httpURL, etc.). The new logDir is on that fresh object; the original B / ctx.nodes[1] still holds the pre-restart logDir. Because readLog checks node.logDir first and returns immediately when the file exists, the post-restart assertions (#1 orphans on B, #2 uncaughtException on B) silently read from B's pre-restart log in CI where HARPER_INTEGRATION_TEST_LOG_DIR is set. If the orphan manifests during B's reconnection window — the riskiest window — it would be missed.

The soak and adversity tests both fix this with ctx.nodes[victimIdx] = restartCtx.harper. Apply the same pattern here:

Suggested change

	await killHarper({ harper: B });
	await startHarper(
	{ name: ctx.name, harper: { dataRootDir: B.dataRootDir, hostname: B.hostname } },
	{
	config: {
	analytics: { aggregatePeriod: -1 },
	logging: { colors: false, console: true, level: 'debug' },
	replication: { securePort: B.hostname + ':9933' },
	threads: { count: THREADS_PER_NODE },
	},
	env: { HARPER_NO_FLUSH_ON_EXIT: true },
	}
	);
	restarted = true;
	console.log(`[orphan] mid-test restart of B (${B.hostname})`);
	await killHarper({ harper: B });
	const restartCtx = {
	name: ctx.name,
	harper: { dataRootDir: B.dataRootDir, hostname: B.hostname },
	};
	await startHarper(restartCtx, {
	config: {
	analytics: { aggregatePeriod: -1 },
	logging: { colors: false, console: true, level: 'debug' },
	replication: { securePort: B.hostname + ':9933' },
	threads: { count: THREADS_PER_NODE },
	},
	env: { HARPER_NO_FLUSH_ON_EXIT: true },
	});
	ctx.nodes[1] = restartCtx.harper;
	restarted = true;

Then re-destructure const [A, B] = ctx.nodes; just before the drain loop and log assertions (or use ctx.nodes[0] / ctx.nodes[1] directly) so readLog(B) and sendOperation(B, ...) pick up the updated handle.

[kriszyp]

Fixed in e97a3a3 — captured the restarted handle, updated ctx.nodes[1] and the local B alias to match the soak + adversity pattern.

🤖 AI-generated reply

[claude]

One blocker: backlogRecovery, replayCatchupSeam, and slowConsumerBackpressure are missing from the stress-tests.yaml matrix — the tests exist but will never run in CI. Inline comment at line 76–88 with suggested additions. (partitionHealConvergence is intentionally excluded and documented, so that omission is fine.)

[claude]

The matrix only covers the four original tests. The three runnable new tests added in this push — backlogRecovery, replayCatchupSeam, and slowConsumerBackpressure — are absent, so they'll never execute in CI (weekly cron or manual dispatch). partitionHealConvergence is intentionally blocked and documented, so excluding that one is fine.

Suggested additions:

Suggested change

	test:
	- name: 'worker-exit-cascade'
	file: 'integrationTests/stress/workerExitCascade.test.mjs'
	env_vars: ''
	- name: 'soak-rolling-restarts'
	file: 'integrationTests/stress/soakWithRollingRestarts.test.mjs'
	env_vars: 'HARPER_STRESS_SOAK_MINUTES'
	- name: 'blob-orphan-race'
	file: 'integrationTests/stress/blobOrphanRace.test.mjs'
	env_vars: 'HARPER_STRESS_ORPHAN_MINUTES'
	- name: 'rapid-reconnect-adversity'
	file: 'integrationTests/stress/rapidReconnectAdversity.test.mjs'
	env_vars: 'HARPER_STRESS_ADVERSITY_MINUTES'
	test:
	- name: 'worker-exit-cascade'
	file: 'integrationTests/stress/workerExitCascade.test.mjs'
	env_vars: ''
	- name: 'soak-rolling-restarts'
	file: 'integrationTests/stress/soakWithRollingRestarts.test.mjs'
	env_vars: 'HARPER_STRESS_SOAK_MINUTES'
	- name: 'blob-orphan-race'
	file: 'integrationTests/stress/blobOrphanRace.test.mjs'
	env_vars: 'HARPER_STRESS_ORPHAN_MINUTES'
	- name: 'rapid-reconnect-adversity'
	file: 'integrationTests/stress/rapidReconnectAdversity.test.mjs'
	env_vars: 'HARPER_STRESS_ADVERSITY_MINUTES'
	- name: 'replay-catchup-seam'
	file: 'integrationTests/stress/replayCatchupSeam.test.mjs'
	env_vars: ''
	- name: 'backlog-recovery'
	file: 'integrationTests/stress/backlogRecovery.test.mjs'
	env_vars: 'HARPER_STRESS_BACKLOG_OFFLINE_MINUTES'
	- name: 'slow-consumer-backpressure'
	file: 'integrationTests/stress/slowConsumerBackpressure.test.mjs'
	env_vars: 'HARPER_STRESS_SLOW_MINUTES'

You'll also want to expose the new duration knobs (backlog-offline-minutes, slow-minutes) in the workflow_dispatch.inputs block and wire them into the env: section of the Run step, matching the pattern the existing four tests use.