perf(mysql): coalesce result-set response into one write to kill 40ms per-query stall - #7
Merged
Merged
Conversation
… stall The MySQL frontend stalled ~44ms on every query that returns a result set (point SELECT: 44.0ms p50 via PHP PDO) while single-OK-packet responses (INSERT: 1.3ms) were unaffected. Root cause is a Nagle + delayed-ACK deadlock on the multi-packet response path: opensrv-mysql emits a result set as several separate packets (column count, one column-def per column, EOF, one packet per row, EOF) and writes each to the socket individually, calling flush() only once after the whole response. On a raw socket each packet is its own TCP segment. PHP's mysqlnd client does NOT set TCP_NODELAY, so its Nagle withholds the ACK of the first segment while it waits to coalesce more data to send, and Linux delayed-ACK holds that ACK ~40ms. The server-side set_nodelay added in 54b5fce does not help because the stall is driven by the *client's* Nagle, not the server's; a client that sets nodelay (mysql_async, the mysql CLI) never sees it, which is why it reproduced only through PHP. Wrapping the write half in a BufWriter makes opensrv's single trailing flush() emit the entire result set as one write -> one TCP segment, so there is no intermediate packet for the client to sit on. The handshake greeting and every command response are already flushed at their protocol boundaries, so buffering changes only how many segments a response spans, not protocol semantics. Larger-than-buffer responses still flush in chunks, which remains correct. Measured via ePHPm single-node SQLite (PHP PDO point SELECT): 44.0ms -> sub-millisecond p50. Postgres uses pgwire's Framed writer (already buffered) and TDS is a placeholder, so no change is needed there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
The MySQL frontend stalled ~44ms on every query that returns a result set while single-OK-packet responses were unaffected. Measured via ePHPm single-node SQLite (PHP PDO, loopback):
SELECTINSERTThe asymmetry is the tell: an
INSERTreply is a single OK packet; aSELECTreply is a multi-packet result set (column-count, one column-def per column, EOF, one packet per row, EOF).opensrv-mysqlwrites each of those packets to the socket individually (PacketWriter::end_packet→ onewrite_vectoredper packet) and callsflush()only once, after the whole response. On a rawOwnedWriteHalfeach packet becomes its own TCP segment. PHP's mysqlnd client does not setTCP_NODELAY, so its Nagle withholds the ACK of the first segment while it waits to coalesce more data to send, and Linux delayed-ACK holds that ACK for ~40ms — deadlocking every result-set round trip.The server-side
set_nodelay(true)added in 54b5fce does not fix this, because the stall is driven by the client's Nagle, not the server's. That's also why it only reproduced through PHP: a client that sets nodelay itself (mysql_async, themysqlCLI) never sees it.Evidence
mysql_asyncclient (nodelay on): SELECT 0.42ms, INSERT 0.33ms — no stall, no asymmetry.COM_QUERY) SELECTs stalled at 44.0ms; only the shape of the response (multi-packet vs single OK) mattered — confirming a response-path Nagle/delayed-ACK deadlock, not a query-send issue.(Packet capture inside the WSL2/rootless-podman test host is non-functional for intra-netns loopback, so the root cause was isolated via differential client/protocol experiments rather than tcpdump.)
Fix
Wrap the write half in a
BufWriter. opensrv's single trailingflush()now emits the entire result set as one write → one TCP segment, so there is no intermediate packet for the client to sit on. The handshake greeting and every command response are already flushed at their protocol boundaries, so this changes only how many segments a response spans, not protocol semantics. Responses larger than the buffer still flush in chunks (correct, and not subject to the deadlock since the client is actively draining).Postgres uses pgwire's buffered
Framedwriter and TDS is a placeholder, so no change is needed there. The existingset_nodelaystays as a correct complementary optimization.Measured result (rebuilt ePHPm image, same bench)
SELECT(select_hot)select_spreadINSERT~177x faster on the point-SELECT hot path; sub-millisecond as required. All 23
mysql_e2etests pass;cargo fmt --all --checkandcargo clippy --workspace --all-targets -- -D warningsare clean.