Skip to content

Fix path validation livelock on high-RTT paths - #526

Open
aswanthk777 wants to merge 1 commit into
Tencent:developfrom
aswanthk777:bonding/path-validation-livelock
Open

Fix path validation livelock on high-RTT paths#526
aswanthk777 wants to merge 1 commit into
Tencent:developfrom
aswanthk777:bonding/path-validation-livelock

Conversation

@aswanthk777

Copy link
Copy Markdown

Problem

Path validation can never complete on a path whose RTT exceeds the challenge retry window, due to two interacting behaviors in src/connection/path.rs:

  1. The PATH_CHALLENGE retry timer starts at INITIAL_CHAL_TIMEOUT (25ms), and on_path_chal_timeout pops the expired challenge data from sent_chals. On a path with RTT above the current retry window (inter-continental, satellite, loaded cellular…), the PATH_RESPONSE always arrives after its challenge was discarded — it matches nothing and cannot validate the path.
  2. on_path_resp_received reset lost_chal unconditionally, even for a response that matched no outstanding challenge. Every stale response restarted the exponential backoff at INITIAL_CHAL_TIMEOUT, so the retry window could never grow past the path RTT.

The combination is a livelock: challenge → retry timer expires → late response resets the backoff → fresh 25ms challenge → forever. The path never leaves Validating/ValidatingMTU and keeps emitting a PATH_CHALLENGE roughly every RTT. In practice this breaks any path whose RTT is above roughly 2×INITIAL_CHAL_TIMEOUT (~50ms).

Impact in MPQUIC mode

ACK frames for a path's packet number space are only written when that path is active() (try_write_ack_frame), so neither endpoint ever acknowledges anything sent on the affected path. The unacknowledged (and never loss-reclaimed) challenge packets accumulate as bytes_in_flight until they exceed the congestion window, permanently blocking the path while it still looks healthy (srtt set from validation timing, zero PTOs). We observed this live as a multipath connection whose added path silently carries no traffic whenever its RTT exceeds ~50ms, with the datagram send queue backing up behind the dead path.

Fix

Following RFC 9000 §8.2.2 (a PATH_RESPONSE matching any previously sent PATH_CHALLENGE validates the path):

  • Keep the data of expired challenges in a bounded stale_chals deque and match responses against it as a fallback — a late response now validates the path in one round trip at any RTT.
  • Reset the retry backoff only when a response actually matched a challenge — even without a stale match, the retry window can now grow past the path RTT and validation converges.

Testing

  • New path_chal_high_rtt_late_response unit test pins the livelock sequence: challenge expires before the response arrives, a garbage response must not reset the backoff, and the late matching response must validate the path.
  • Full suite on this branch: 559 passed / 0 failed.
  • Verified end-to-end on an emulated 2×100ms-one-way multipath link (netns + netem): before the fix, the added path never validates and carries nothing (pings through a datagram tunnel degrade from 200ms to seconds as the send queue backs up); after it, validation completes in ~1 RTT and both paths carry traffic.

Related: #523 fixed a different validation deadlock (anti-amplification) — this one is independent and applies to any high-RTT path, client- or server-initiated.

Path validation can never complete on a path whose RTT exceeds the
challenge retry window, because two behaviors interact:

1. The PATH_CHALLENGE retry timer starts at INITIAL_CHAL_TIMEOUT (25ms)
   and on_path_chal_timeout POPS the expired challenge data from
   sent_chals. On a path with RTT above the current window (e.g. any
   inter-continental, satellite, or loaded-cellular path), the
   PATH_RESPONSE always arrives after its challenge was discarded, so
   it matches nothing and cannot validate the path.

2. on_path_resp_received reset lost_chal unconditionally — even for a
   response that matched no outstanding challenge. Every stale response
   therefore restarted the exponential backoff at INITIAL_CHAL_TIMEOUT,
   so the retry window could never grow past the path RTT.

The combination is a livelock: challenge -> retry timer expires ->
late response resets the backoff -> fresh 25ms challenge -> forever.
The path stays in Validating/ValidatingMTU, never becomes active, and
keeps emitting a PATH_CHALLENGE roughly every RTT.

In MPQUIC mode the consequences go beyond the unusable path: ACK
frames for a path's packet number space are only written when that
path is active, so neither endpoint ever acknowledges anything sent on
the affected path. The unacknowledged (and never loss-reclaimed)
challenge packets accumulate as bytes_in_flight until they exceed the
congestion window, permanently blocking the path while it still looks
healthy (srtt set from validation timing, zero PTOs). Observed in
practice as a multipath connection whose added path carries no traffic
whenever its RTT is above ~50ms, with the datagram send queue backing
up behind the dead path.

Fix, following RFC 9000 Section 8.2.2 (a PATH_RESPONSE matching any
previously sent PATH_CHALLENGE validates the path):

- Keep the data of expired challenges in a bounded stale_chals deque
  and match responses against it as a fallback, so a late response
  validates the path in one round trip at any RTT.
- Reset the retry backoff only when a response actually matched a
  challenge, so even without a stale match the retry window grows past
  the path RTT and validation converges.

The new path_chal_high_rtt_late_response test pins the livelock
sequence: challenge expires before the response arrives, a garbage
response must not reset the backoff, and the late matching response
must validate the path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant