NAS-141771 / 26.0.0-RC.1 / fix NTB TCP 0 sized (permanent) window clamp (by yocalebo)#318
Merged
Conversation
stall with the peer advertising a zero receive window that never reopens. File copies hang silently, DLM drops its connection for roughly 20 seconds at a time, and long lived websocket sessions see unexplained disconnects. This change fixes the root cause in the ntb_netdev receive path. ntb_netdev posts full MTU sized receive buffers into the transport ring and delivers every received frame to the network stack in that buffer no matter how small the frame actually is. At the default 64KB transport MTU each buffer rounds up to a 128KB allocation, so a 54 byte pure TCP ACK reaches the socket with a truesize near 131KB, a charge of roughly 2400 times the real data size against the socket receive buffer. TCP sizes its advertised window from truesize based accounting. A burst of small control messages fills the receive buffer after a few dozen packets, the advertised window collapses to zero, and the window regrowth paths are gated on incoming packets having a sane length to truesize ratio, which never happens on this device. Bidirectional ssh traffic can then deadlock permanently because application level flow control wedges on both sides while both receive windows are shut. Copy received frames of rx_copybreak bytes or less (default 3646, runtime tunable) into a right sized skb allocated from the page frag cache and recycle the jumbo ring buffer immediately. Small control frames then reach the stack with honest truesize and receive windows stay open. Bulk frames keep the existing zero copy buffer swap. The receive hot path also stops performing a high order atomic allocation for every small frame. Measured on H20 and F60 HA pairs at 64KB MTU using a reproduction harness that drives bidirectional bulk copies, small message chatter and CPU load, comparing builds that differ only by this patch. - truesize per small packet drops from 131328 bytes to 960 bytes - receive buffer overcommit drops from about 1651x to about 12x - TCPWantZeroWindowAdv over a four minute load window drops from 24392 to about 1000 - the unpatched build parks multiple sockets in the TCP persist state within three seconds of load while the patched build shows zero wedged sockets across repeated full runs - a production configured F60 accumulated over 600k wanted zero window events in ten hours of ordinary operation before the fix and near zero after - on the 6.12 stable train the same defect surfaces as receive queue collapse storms of roughly 170k TCPRcvCollapsed per four minute load window, driven by the same truesize inflation this patch eliminates (cherry picked from commit 0d46f76)
The rx_handler() recycle paths re-enqueue an skb using ndev->mtu + ETH_HLEN as the buffer length, but change_mtu() cannot drain an skb an in-flight handler is holding. After an MTU increase such a recycled buffer can be smaller than that length and be overflowed by a full-size frame. Post skb_tailroom(nskb), the buffer's true capacity, so the transport's length check rejects any frame that would overflow it. When the buffer is too small for the current MTU, also reallocate a right-sized one before re-enqueuing so it does not reject every full-size frame for its ring slot until it drains; if that allocation fails the smaller buffer is re-posted unchanged and replaced on a later pass. In steady state the buffer is already large enough, so the hot path gains no allocation. Reported in PR review by @ixhamza. (cherry picked from commit 5b4e034)
The copybreak path allocates a right-sized skb of len bytes and then eth_type_trans() reads the 14-byte Ethernet header. For a frame with len < ETH_HLEN that reads uninitialized bytes past the copied data and delivers a malformed runt up the stack. len comes from the peer-controlled hdr->len, so a misbehaving peer (including hdr->len == 0) can reach it. Widen the existing len < 0 guard to len < ETH_HLEN so sub-header frames are counted as length errors and their buffer recycled before either delivery path reaches eth_type_trans(). Frames of at least ETH_HLEN bytes are unaffected. (cherry picked from commit b329744)
Author
yocalebo
approved these changes
Jul 13, 2026
Author
|
This PR has been merged and conversations have been locked. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TCP connections crossing the NTB interconnect between HA controllers can
stall with the peer advertising a zero receive window that never reopens.
File copies hang silently, DLM drops its connection for roughly 20 seconds
at a time, and long lived websocket sessions see unexplained disconnects.
This change fixes the root cause in the ntb_netdev receive path.
Root cause
ntb_netdev posts full MTU sized receive buffers into the transport ring and
delivers every received frame to the network stack in that buffer no matter
how small the frame actually is. At the default 64KB transport MTU each
buffer rounds up to a 128KB allocation, so a 54 byte pure TCP ACK reaches
the socket with a truesize near 131KB, a charge of roughly 2400 times the
real data size against the socket receive buffer.
TCP sizes its advertised window from truesize based accounting. A burst of
small control messages fills the receive buffer after a few dozen packets,
the advertised window collapses to zero, and the window regrowth paths are
gated on incoming packets having a sane length to truesize ratio, which
never happens on this device. Bidirectional ssh traffic can then deadlock
permanently because application level flow control wedges on both sides
while both receive windows are shut.
Fix
Copy received frames of rx_copybreak bytes or less (default 2048, runtime
tunable) into a right sized skb allocated from the page frag cache and
recycle the jumbo ring buffer immediately. Small control frames then reach
the stack with honest truesize and receive windows stay open. Bulk frames
keep the existing zero copy buffer swap. The receive hot path also stops
performing a high order atomic allocation for every small frame.
Verification
Measured on H20 and F60 HA pairs at 64KB MTU using a reproduction harness
that drives bidirectional bulk copies, small message chatter and CPU load,
comparing builds that differ only by this patch.
about 1000
within three seconds of load while the patched build shows zero wedged
sockets across repeated full runs
events in ten hours of ordinary operation before the fix and near zero
after
collapse storms of roughly 170k TCPRcvCollapsed per four minute load
window, driven by the same truesize inflation this patch eliminates
Original PR: #312