Skip to content

perf(mysql): coalesce result-set response into one write to kill 40ms per-query stall - #7

Merged
luthermonson merged 1 commit into
mainfrom
fix/nodelay-result-set-flush
Jul 11, 2026
Merged

perf(mysql): coalesce result-set response into one write to kill 40ms per-query stall#7
luthermonson merged 1 commit into
mainfrom
fix/nodelay-result-set-flush

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Root cause

The MySQL frontend stalled ~44ms on every query that returns a result set while single-OK-packet responses were unaffected. Measured via ePHPm single-node SQLite (PHP PDO, loopback):

op before
point SELECT 44.01ms p50
INSERT 1.36ms p50

The asymmetry is the tell: an INSERT reply is a single OK packet; a SELECT reply is a multi-packet result set (column-count, one column-def per column, EOF, one packet per row, EOF).

opensrv-mysql writes each of those packets to the socket individually (PacketWriter::end_packet → one write_vectored per packet) and calls flush() only once, after the whole response. On a raw OwnedWriteHalf each packet becomes its own TCP segment. PHP's mysqlnd client does not set TCP_NODELAY, so its Nagle withholds the ACK of the first segment while it waits to coalesce more data to send, and Linux delayed-ACK holds that ACK for ~40ms — deadlocking every result-set round trip.

The server-side set_nodelay(true) added in 54b5fce does not fix this, because the stall is driven by the client's Nagle, not the server's. That's also why it only reproduced through PHP: a client that sets nodelay itself (mysql_async, the mysql CLI) never sees it.

Evidence

  • Same litewire frontend, mysql_async client (nodelay on): SELECT 0.42ms, INSERT 0.33ms — no stall, no asymmetry.
  • ePHPm PHP PDO client (nodelay off): SELECT 44.0ms, INSERT 1.3ms.
  • Protocol-differential probe through ePHPm: both server-side-prepared (binary) and emulated (text COM_QUERY) SELECTs stalled at 44.0ms; only the shape of the response (multi-packet vs single OK) mattered — confirming a response-path Nagle/delayed-ACK deadlock, not a query-send issue.
  • 44ms ≈ 40ms Linux delayed-ACK + query time.

(Packet capture inside the WSL2/rootless-podman test host is non-functional for intra-netns loopback, so the root cause was isolated via differential client/protocol experiments rather than tcpdump.)

Fix

Wrap the write half in a BufWriter. opensrv's single trailing flush() now emits the entire result set as one write → one TCP segment, so there is no intermediate packet for the client to sit on. The handshake greeting and every command response are already flushed at their protocol boundaries, so this changes only how many segments a response spans, not protocol semantics. Responses larger than the buffer still flush in chunks (correct, and not subject to the deadlock since the client is actively draining).

Postgres uses pgwire's buffered Framed writer and TDS is a placeholder, so no change is needed there. The existing set_nodelay stays as a correct complementary optimization.

Measured result (rebuilt ePHPm image, same bench)

op before after
point SELECT (select_hot) 44.01ms p50 0.23ms p50
select_spread ~44ms p50 0.25ms p50
INSERT 1.36ms p50 0.30ms p50

~177x faster on the point-SELECT hot path; sub-millisecond as required. All 23 mysql_e2e tests pass; cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings are clean.

… stall

The MySQL frontend stalled ~44ms on every query that returns a result set
(point SELECT: 44.0ms p50 via PHP PDO) while single-OK-packet responses
(INSERT: 1.3ms) were unaffected. Root cause is a Nagle + delayed-ACK
deadlock on the multi-packet response path:

opensrv-mysql emits a result set as several separate packets (column
count, one column-def per column, EOF, one packet per row, EOF) and
writes each to the socket individually, calling flush() only once after
the whole response. On a raw socket each packet is its own TCP segment.
PHP's mysqlnd client does NOT set TCP_NODELAY, so its Nagle withholds the
ACK of the first segment while it waits to coalesce more data to send,
and Linux delayed-ACK holds that ACK ~40ms. The server-side set_nodelay
added in 54b5fce does not help because the stall is driven by the
*client's* Nagle, not the server's; a client that sets nodelay
(mysql_async, the mysql CLI) never sees it, which is why it reproduced
only through PHP.

Wrapping the write half in a BufWriter makes opensrv's single trailing
flush() emit the entire result set as one write -> one TCP segment, so
there is no intermediate packet for the client to sit on. The handshake
greeting and every command response are already flushed at their
protocol boundaries, so buffering changes only how many segments a
response spans, not protocol semantics. Larger-than-buffer responses
still flush in chunks, which remains correct.

Measured via ePHPm single-node SQLite (PHP PDO point SELECT): 44.0ms ->
sub-millisecond p50.

Postgres uses pgwire's Framed writer (already buffered) and TDS is a
placeholder, so no change is needed there.
@luthermonson
luthermonson merged commit df605f4 into main Jul 11, 2026
3 checks passed
@luthermonson
luthermonson deleted the fix/nodelay-result-set-flush branch July 11, 2026 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant