Fix path validation livelock on high-RTT paths - #526
Open
aswanthk777 wants to merge 1 commit into
Open
Conversation
Path validation can never complete on a path whose RTT exceeds the challenge retry window, because two behaviors interact: 1. The PATH_CHALLENGE retry timer starts at INITIAL_CHAL_TIMEOUT (25ms) and on_path_chal_timeout POPS the expired challenge data from sent_chals. On a path with RTT above the current window (e.g. any inter-continental, satellite, or loaded-cellular path), the PATH_RESPONSE always arrives after its challenge was discarded, so it matches nothing and cannot validate the path. 2. on_path_resp_received reset lost_chal unconditionally — even for a response that matched no outstanding challenge. Every stale response therefore restarted the exponential backoff at INITIAL_CHAL_TIMEOUT, so the retry window could never grow past the path RTT. The combination is a livelock: challenge -> retry timer expires -> late response resets the backoff -> fresh 25ms challenge -> forever. The path stays in Validating/ValidatingMTU, never becomes active, and keeps emitting a PATH_CHALLENGE roughly every RTT. In MPQUIC mode the consequences go beyond the unusable path: ACK frames for a path's packet number space are only written when that path is active, so neither endpoint ever acknowledges anything sent on the affected path. The unacknowledged (and never loss-reclaimed) challenge packets accumulate as bytes_in_flight until they exceed the congestion window, permanently blocking the path while it still looks healthy (srtt set from validation timing, zero PTOs). Observed in practice as a multipath connection whose added path carries no traffic whenever its RTT is above ~50ms, with the datagram send queue backing up behind the dead path. Fix, following RFC 9000 Section 8.2.2 (a PATH_RESPONSE matching any previously sent PATH_CHALLENGE validates the path): - Keep the data of expired challenges in a bounded stale_chals deque and match responses against it as a fallback, so a late response validates the path in one round trip at any RTT. - Reset the retry backoff only when a response actually matched a challenge, so even without a stale match the retry window grows past the path RTT and validation converges. The new path_chal_high_rtt_late_response test pins the livelock sequence: challenge expires before the response arrives, a garbage response must not reset the backoff, and the late matching response must validate the path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Path validation can never complete on a path whose RTT exceeds the challenge retry window, due to two interacting behaviors in
src/connection/path.rs:INITIAL_CHAL_TIMEOUT(25ms), andon_path_chal_timeoutpops the expired challenge data fromsent_chals. On a path with RTT above the current retry window (inter-continental, satellite, loaded cellular…), the PATH_RESPONSE always arrives after its challenge was discarded — it matches nothing and cannot validate the path.on_path_resp_receivedresetlost_chalunconditionally, even for a response that matched no outstanding challenge. Every stale response restarted the exponential backoff atINITIAL_CHAL_TIMEOUT, so the retry window could never grow past the path RTT.The combination is a livelock: challenge → retry timer expires → late response resets the backoff → fresh 25ms challenge → forever. The path never leaves Validating/ValidatingMTU and keeps emitting a PATH_CHALLENGE roughly every RTT. In practice this breaks any path whose RTT is above roughly 2×
INITIAL_CHAL_TIMEOUT(~50ms).Impact in MPQUIC mode
ACK frames for a path's packet number space are only written when that path is
active()(try_write_ack_frame), so neither endpoint ever acknowledges anything sent on the affected path. The unacknowledged (and never loss-reclaimed) challenge packets accumulate asbytes_in_flightuntil they exceed the congestion window, permanently blocking the path while it still looks healthy (srtt set from validation timing, zero PTOs). We observed this live as a multipath connection whose added path silently carries no traffic whenever its RTT exceeds ~50ms, with the datagram send queue backing up behind the dead path.Fix
Following RFC 9000 §8.2.2 (a PATH_RESPONSE matching any previously sent PATH_CHALLENGE validates the path):
stale_chalsdeque and match responses against it as a fallback — a late response now validates the path in one round trip at any RTT.Testing
path_chal_high_rtt_late_responseunit test pins the livelock sequence: challenge expires before the response arrives, a garbage response must not reset the backoff, and the late matching response must validate the path.Related: #523 fixed a different validation deadlock (anti-amplification) — this one is independent and applies to any high-RTT path, client- or server-initiated.