Problem
tests/integration/scenarios/12-pubsub-fanout.ts timed out after 15 s in one run of the docker-compose integration suite and passed in 704 ms in the very next run of the same suite on the same tree. A scenario whose normal runtime is under a second failing at a 15 s deadline is a 20x margin, so this is resource starvation rather than a slow path.
The failing run was the last of seventeen suites executed back to back — sixteen broker suites first, each standing up and tearing down its own compose project, then the five-node controller suite. The clean run started against an idle daemon. That ordering is the most plausible trigger and is worth stating, because the packaged test:integration:brokers chains every suite with && and a maintainer running "everything" hits exactly this shape.
Failure mode is partial, not total: the first fan-out phase succeeded on all five subscribers, and only the second phase stalled, on one node.
[12] baseline received: node-a=0, node-b=0, node-c=0, node-d=0, node-e=0
[12] publishing 10 events from node-a...
[12] all 5 subscribers received +10
[12] publishing 5 more events from node-b...
[controller] FAIL 12-pubsub-fanout (15494ms)
error: waitFor: "node-c received +15" did not become true within 15000ms
at waitFor (/app/tests/integration/scenarios/types.ts:50:13)
Evidence
Both runs were against develop at 3934273 (the 50-oldest bug/security wave merged), on Docker 29.6.2, Windows host.
- Run 1 —
12-pubsub-fanout FAIL (15494 ms). 14 of 15 scenarios passed.
- Run 2 —
12-pubsub-fanout PASS (704 ms). All 15 passed.
Three hypotheses were checked against the run-1 log and ruled out, so this is recorded as environmental rather than a product defect:
- Not the new inbound handshake deadline. The wave added one to
TcpTransport's accept path (an accepted socket that has not sent hello within HANDSHAKE_TIMEOUT_MS is closed). Grepping the full run-1 log for it returns zero occurrences — it never fired on any node.
- Not a lost connection. No disconnect, no connection-lost and no unreachable-member line anywhere in the run.
node-c's only warnings are the ordinary merge: refusing …'s claim that we are "up" gossip lines that every node emits continuously.
- Not a source change. The wave touched no pub/sub or topic file under
src/ (git diff --name-only f4e2bc7c develop -- src/ has no match for pubsub/topic/distributed).
Evidence strength is deliberately stated as weak: n = 2, one failure. This is a report of an observation, not a measured flake rate.
Proposal
Roughly in order of cost:
- Quantify it. Run the controller suite N times in a loop — once against an idle daemon and once immediately after the broker suites — and record the rate for each. Without that split the trigger stays a hypothesis.
- Give the fan-out assertion a deadline proportional to what it is waiting for. 15 s is generous for a scenario that normally needs 0.7 s, which suggests the limit was chosen for the happy path rather than for a loaded host. If a second phase legitimately needs longer than the first, saying so explicitly is better than one shared constant.
- Consider whether the runner should pause between compose projects. Sixteen teardowns in sequence leave the daemon reclaiming resources while the next project starts; a short settle, or a disk/daemon readiness probe like the one this repository already uses for database readiness, would remove the coupling between suites.
Related and deliberately kept separate: #290 is the parallel-execution flake catalogue, scoped to bun test (tests/multi-node/tcp-*, udp-*, websocket-*, tests/unit/persistence/filesystem-*) with port collisions and filesystem races as its named causes. This is a different runner — a controller container driving five node containers over a real network — and a different cause class, which is why it is filed separately rather than appended there. #418 (shared awaitCondition helper replacing sleep-based waits) is the closest structural neighbour, since the failure surfaced through waitFor. #538 and #818 cover nightly runs of the quarantined and soak suites and would be the natural home for the loop in proposal 1.
Verification status
CONFIRMED-BY-RUNNING — two consecutive full runs of bun run test:integration on 2026-08-14, logs retained. The remaining sixteen broker suites passed in both sessions (S3, MQTT, Kafka, AMQP, NATS, Redis Streams, gRPC, Kubernetes, Postgres, MariaDB, libSQL, MSSQL, CockroachDB, YugabyteDB, MongoDB, DynamoDB), so nothing else in the integration tier is implicated.
No CHANGELOG entry needed.
Problem
tests/integration/scenarios/12-pubsub-fanout.tstimed out after 15 s in one run of the docker-compose integration suite and passed in 704 ms in the very next run of the same suite on the same tree. A scenario whose normal runtime is under a second failing at a 15 s deadline is a 20x margin, so this is resource starvation rather than a slow path.The failing run was the last of seventeen suites executed back to back — sixteen broker suites first, each standing up and tearing down its own compose project, then the five-node controller suite. The clean run started against an idle daemon. That ordering is the most plausible trigger and is worth stating, because the packaged
test:integration:brokerschains every suite with&&and a maintainer running "everything" hits exactly this shape.Failure mode is partial, not total: the first fan-out phase succeeded on all five subscribers, and only the second phase stalled, on one node.
Evidence
Both runs were against
developat 3934273 (the 50-oldest bug/security wave merged), on Docker 29.6.2, Windows host.12-pubsub-fanoutFAIL (15494 ms). 14 of 15 scenarios passed.12-pubsub-fanoutPASS (704 ms). All 15 passed.Three hypotheses were checked against the run-1 log and ruled out, so this is recorded as environmental rather than a product defect:
TcpTransport's accept path (an accepted socket that has not senthellowithinHANDSHAKE_TIMEOUT_MSis closed). Grepping the full run-1 log for it returns zero occurrences — it never fired on any node.node-c's only warnings are the ordinarymerge: refusing …'s claim that we are "up"gossip lines that every node emits continuously.src/(git diff --name-only f4e2bc7c develop -- src/has no match for pubsub/topic/distributed).Evidence strength is deliberately stated as weak: n = 2, one failure. This is a report of an observation, not a measured flake rate.
Proposal
Roughly in order of cost:
Related and deliberately kept separate: #290 is the parallel-execution flake catalogue, scoped to
bun test(tests/multi-node/tcp-*,udp-*,websocket-*,tests/unit/persistence/filesystem-*) with port collisions and filesystem races as its named causes. This is a different runner — a controller container driving five node containers over a real network — and a different cause class, which is why it is filed separately rather than appended there. #418 (sharedawaitConditionhelper replacing sleep-based waits) is the closest structural neighbour, since the failure surfaced throughwaitFor. #538 and #818 cover nightly runs of the quarantined and soak suites and would be the natural home for the loop in proposal 1.Verification status
CONFIRMED-BY-RUNNING — two consecutive full runs of
bun run test:integrationon 2026-08-14, logs retained. The remaining sixteen broker suites passed in both sessions (S3, MQTT, Kafka, AMQP, NATS, Redis Streams, gRPC, Kubernetes, Postgres, MariaDB, libSQL, MSSQL, CockroachDB, YugabyteDB, MongoDB, DynamoDB), so nothing else in the integration tier is implicated.No CHANGELOG entry needed.