From 0d46f76d1705933451df16a199cc4c0015d6967c Mon Sep 17 00:00:00 2001 From: caleb Date: Mon, 13 Jul 2026 10:14:11 -0400 Subject: [PATCH 1/3] TCP connections crossing the NTB interconnect between HA controllers can stall with the peer advertising a zero receive window that never reopens. File copies hang silently, DLM drops its connection for roughly 20 seconds at a time, and long lived websocket sessions see unexplained disconnects. This change fixes the root cause in the ntb_netdev receive path. ntb_netdev posts full MTU sized receive buffers into the transport ring and delivers every received frame to the network stack in that buffer no matter how small the frame actually is. At the default 64KB transport MTU each buffer rounds up to a 128KB allocation, so a 54 byte pure TCP ACK reaches the socket with a truesize near 131KB, a charge of roughly 2400 times the real data size against the socket receive buffer. TCP sizes its advertised window from truesize based accounting. A burst of small control messages fills the receive buffer after a few dozen packets, the advertised window collapses to zero, and the window regrowth paths are gated on incoming packets having a sane length to truesize ratio, which never happens on this device. Bidirectional ssh traffic can then deadlock permanently because application level flow control wedges on both sides while both receive windows are shut. Copy received frames of rx_copybreak bytes or less (default 3646, runtime tunable) into a right sized skb allocated from the page frag cache and recycle the jumbo ring buffer immediately. Small control frames then reach the stack with honest truesize and receive windows stay open. Bulk frames keep the existing zero copy buffer swap. The receive hot path also stops performing a high order atomic allocation for every small frame. Measured on H20 and F60 HA pairs at 64KB MTU using a reproduction harness that drives bidirectional bulk copies, small message chatter and CPU load, comparing builds that differ only by this patch. - truesize per small packet drops from 131328 bytes to 960 bytes - receive buffer overcommit drops from about 1651x to about 12x - TCPWantZeroWindowAdv over a four minute load window drops from 24392 to about 1000 - the unpatched build parks multiple sockets in the TCP persist state within three seconds of load while the patched build shows zero wedged sockets across repeated full runs - a production configured F60 accumulated over 600k wanted zero window events in ten hours of ordinary operation before the fix and near zero after - on the 6.12 stable train the same defect surfaces as receive queue collapse storms of roughly 170k TCPRcvCollapsed per four minute load window, driven by the same truesize inflation this patch eliminates --- drivers/net/ntb_netdev.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/drivers/net/ntb_netdev.c b/drivers/net/ntb_netdev.c index 55c630158602..e332660582e2 100644 --- a/drivers/net/ntb_netdev.c +++ b/drivers/net/ntb_netdev.c @@ -71,6 +71,21 @@ static unsigned int tx_start = 10; /* Number of descriptors still available before stop upper layer tx */ static unsigned int tx_stop = 5; +/* Copy received frames of this many bytes or less into a right-sized skb + * and recycle the full-MTU ring buffer. Posted ring buffers are always + * ndev->mtu + ETH_HLEN bytes, so without this a 54-byte pure TCP ACK is + * delivered with the truesize of a full-MTU allocation (~128KB at the + * default 64KB MTU), overcharging socket receive-buffer accounting by + * three orders of magnitude and collapsing TCP receive windows to zero. + * + * Default is SKB_MAX_ORDER(NET_SKB_PAD, 0) - NET_IP_ALIGN (3646 on x86-64), + * the largest frame whose right-sized copy still fits one order-0 page after + * netdev_alloc_skb_ip_align() reserves its NET_IP_ALIGN headroom. + */ +static unsigned int rx_copybreak = 3646; +module_param(rx_copybreak, uint, 0644); +MODULE_PARM_DESC(rx_copybreak, "Copy RX frames of this size or less into right-sized skbs"); + struct ntb_netdev { struct pci_dev *pdev; struct net_device *ndev; @@ -119,6 +134,20 @@ static void ntb_netdev_rx_handler(struct ntb_transport_qp *qp, void *qp_data, goto enqueue_again; } + if (len <= rx_copybreak) { + struct sk_buff *cskb; + + cskb = netdev_alloc_skb_ip_align(ndev, len); + if (cskb) { + skb_put_data(cskb, skb->data, len); + /* Recycle the ring buffer, deliver the copy */ + nskb = skb; + skb = cskb; + goto deliver; + } + /* Fall back to handing off the ring buffer itself */ + } + nskb = netdev_alloc_skb(ndev, ndev->mtu + ETH_HLEN); if (!nskb) { ndev->stats.rx_errors++; @@ -128,6 +157,7 @@ static void ntb_netdev_rx_handler(struct ntb_transport_qp *qp, void *qp_data, } skb_put(skb, len); +deliver: skb->protocol = eth_type_trans(skb, ndev); skb->ip_summed = CHECKSUM_NONE; From 5b4e0348f132c048f5b182a2126e1bccf3e24974 Mon Sep 17 00:00:00 2001 From: caleb Date: Mon, 13 Jul 2026 10:40:09 -0400 Subject: [PATCH 2/3] ntb_netdev: size reposted RX buffers to the current MTU The rx_handler() recycle paths re-enqueue an skb using ndev->mtu + ETH_HLEN as the buffer length, but change_mtu() cannot drain an skb an in-flight handler is holding. After an MTU increase such a recycled buffer can be smaller than that length and be overflowed by a full-size frame. Post skb_tailroom(nskb), the buffer's true capacity, so the transport's length check rejects any frame that would overflow it. When the buffer is too small for the current MTU, also reallocate a right-sized one before re-enqueuing so it does not reject every full-size frame for its ring slot until it drains; if that allocation fails the smaller buffer is re-posted unchanged and replaced on a later pass. In steady state the buffer is already large enough, so the hot path gains no allocation. Reported in PR review by @ixhamza. --- drivers/net/ntb_netdev.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/net/ntb_netdev.c b/drivers/net/ntb_netdev.c index e332660582e2..86fc00bddfa1 100644 --- a/drivers/net/ntb_netdev.c +++ b/drivers/net/ntb_netdev.c @@ -170,8 +170,26 @@ static void ntb_netdev_rx_handler(struct ntb_transport_qp *qp, void *qp_data, } enqueue_again: + /* A recycled buffer may predate an MTU increase that change_mtu() + * could not drain because this handler was holding it. It is then too + * small for the current MTU and, once we post its true capacity below, + * would reject every full-size frame for its ring slot until it drains. + * Swap in a right-sized buffer; on allocation failure keep the smaller + * one, which stays safe and is replaced on a later pass. Steady state + * takes neither branch, so the hot path adds no allocation. + */ + if (skb_tailroom(nskb) < ndev->mtu + ETH_HLEN) { + struct sk_buff *rskb; + + rskb = netdev_alloc_skb(ndev, ndev->mtu + ETH_HLEN); + if (rskb) { + dev_kfree_skb_any(nskb); + nskb = rskb; + } + } + rc = ntb_transport_rx_enqueue(qp, nskb, nskb->data, - ndev->mtu + ETH_HLEN); + skb_tailroom(nskb)); if (rc) { dev_kfree_skb_any(nskb); ndev->stats.rx_errors++; From b329744c65b9024ad0e153b1a7b7fd637b8cd086 Mon Sep 17 00:00:00 2001 From: caleb Date: Mon, 13 Jul 2026 10:40:36 -0400 Subject: [PATCH 3/3] ntb_netdev: drop frames shorter than an Ethernet header The copybreak path allocates a right-sized skb of len bytes and then eth_type_trans() reads the 14-byte Ethernet header. For a frame with len < ETH_HLEN that reads uninitialized bytes past the copied data and delivers a malformed runt up the stack. len comes from the peer-controlled hdr->len, so a misbehaving peer (including hdr->len == 0) can reach it. Widen the existing len < 0 guard to len < ETH_HLEN so sub-header frames are counted as length errors and their buffer recycled before either delivery path reaches eth_type_trans(). Frames of at least ETH_HLEN bytes are unaffected. --- drivers/net/ntb_netdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ntb_netdev.c b/drivers/net/ntb_netdev.c index 86fc00bddfa1..ca0f46044dbd 100644 --- a/drivers/net/ntb_netdev.c +++ b/drivers/net/ntb_netdev.c @@ -127,7 +127,7 @@ static void ntb_netdev_rx_handler(struct ntb_transport_qp *qp, void *qp_data, netdev_dbg(ndev, "%s: %d byte payload received\n", __func__, len); - if (len < 0) { + if (len < ETH_HLEN) { ndev->stats.rx_errors++; ndev->stats.rx_length_errors++; nskb = skb;