Add support for multi-process port sharing with CIBIR.

[ProjectsByJackHe]

Description

Fixes #5795

The XDP datapath can be configured to intercept packets based on QUIC Connection ID instead of local port.
This behavior existed in MsQuic but was not heavily exercised until recently.
One issue was that MsQuic always attempted to reserve UDP / TCP sockets for each application server process.
But for multiple server processes that may want to share a single port, we would run into port collision errors.
This PR adds support for CIBIR across multiple processes on the same port and document the behavior

Potential options to allow for multi-process port sharing:

• Option 1: MsQuic delegates port protection to applications and provides best practice recommendations.

Analysis: Deferring the responsibility of port protection and isolation to the application has the upside of enabling the most potential scenarios but could also be a footgun.

• Option 2: MsQuic makes sure some persistent reservation exists at a port.

Analysis: Note that LookUpPersistentReservation does not require admin privileges, but CreatePersistentReservation does require admin. This is useful in that if any reservation exists on a port, we can reasonably trust that an admin knew what they were doing when they created it. safety, and ensure the consumers of CIBIR must know what they are doing.

• Option 3: MsQuic creates per-proc sockets with SIO_CPU_AFFINITY, but does not reserve the port.

Analysis: If another unrelated app creates a socket with SIO_CPU_AFFINITY, then they can bind to the CIBIR shared port. But for all other apps, trying to bind a socket to a CIBIR port will result in a collision.

Option chosen: 1

MsQuic's stance is that the application takes responsibility for book-keeping and protecting sharing shared local ports when using XDP + CIBIR.

• Multiple MsQuic processes in Cibir+XDP mode can share a local port for server sockets only.
• Applications should also not assume the shared port is safe from other non-Msquic processes binding to it.

MsQuic will NOT make an OS port reservation for server sockets when CIBIR+XDP is enabled. Clients on the other hand, MsQuic will always make OS port reservations.

• Applications using server sockets + CIBIR/XDP must specify a well-known local port.

What changed

• Server sockets with XDP+CIBIR both enabled/available will skip OS port reservation and OS socket creation to rely on XDP.

any failures plumbing xdp rules will bubble up as a socket creation error to the app. Can't fall back to OS sockets.

• Client sockets with XDP+CIBIR both enabled/available will still do OS port reservation and socket creation but rely on XDP.

any failures plumbing xdp rules will silently fall back to using OS sockets. CIBIR transport negotiation can still work without XDP.

• Server sockets with CIBIR enabled but XDP not available/enabled will do OS port reservation and fall back to OS sockets
• Client sockets with CIBIR enabled but XDP not available/enabled will do OS port reservation and fall back to OS sockets

Port protection options

• Windows has the https://learn.microsoft.com/en-us/windows/win32/api/iphlpapi/nf-iphlpapi-createpersistentudpportreservation API, to allow sysadmins to pre-allocate a block of ports and disallow other applications from binding to it.
• A well known CIBIR registry key can be used to detail shared ports, and sysadmins can coordinate their system such that other apps will not bind to those ports.
• ALE policies; applications can configure WFP to block certain ports from being binded to by other apps.

Testing

A new DataPathTest was added.

Documentation

Settings.md

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.08%. Comparing base (37d6673) to head (3b550f8).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5798      +/-   ##
==========================================
+ Coverage   84.80%   85.08%   +0.27%     
==========================================
  Files          60       60              
  Lines       18731    18731              
==========================================
+ Hits        15885    15937      +52     
+ Misses       2846     2794      -52     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[guhetier]

In general, I am concerned that we keep adding incremental exceptions to the port reservation logic to solve the next issue, but without having a clear design goal.
This code will be hard to maintain and confusing for apps as there is no simple rule about what can be done with ports.

I think we need to take the time soon to come up with a clear story about when can port be shared and when they can't + document it + check we implement it.

[ProjectsByJackHe]

In general, I am concerned that we keep adding incremental exceptions to the port reservation logic to solve the next issue, but without having a clear design goal. This code will be hard to maintain and confusing for apps as there is no simple rule about what can be done with ports.

I think we need to take the time soon to come up with a clear story about when can port be shared and when they can't + document it + check we implement it.

I agree. I added a CIBIR.md.

[ProjectsByJackHe]

In general, I am concerned that we keep adding incremental exceptions to the port reservation logic to solve the next issue, but without having a clear design goal. This code will be hard to maintain and confusing for apps as there is no simple rule about what can be done with ports.

I think we need to take the time soon to come up with a clear story about when can port be shared and when they can't + document it + check we implement it.

please see the updated XDP.md and CIBIR.md

[nibanks]

Maybe we should actually go through the rest of the standardization process for https://datatracker.ietf.org/doc/draft-banks-quic-cibir/ since you're still using it....

[guhetier]

Maybe we should actually go through the rest of the standardization process for https://datatracker.ietf.org/doc/draft-banks-quic-cibir/ since you're still using it....

Do you have more context about it? I am interested in understanding whether we should work on standardizing CIBIR or if another solution was adopted by other deployments that we should align with.

[nibanks]

Maybe we should actually go through the rest of the standardization process for https://datatracker.ietf.org/doc/draft-banks-quic-cibir/ since you're still using it....

Do you have more context about it? I am interested in understanding whether we should work on standardizing CIBIR or if another solution was adopted by other deployments that we should align with.

I know of no other solution/proposal that achieves what CIBIR does. We can chat more on Discord about the process.

[mtfriesen]

This is looking close, aside from the refinements requested by various comments.

No longer blocking merging.

[guhetier]

The PR description says:

any failures plumbing xdp rules will bubble up as a socket creation error to the app. Can't fall back to OS sockets.

But errors when setting up XDP rules seems to be silently eaten:

   RawSocketCreateUdp                     → returns QUIC_STATUS
     └─ CxPlatTryAddSocket(...)           → returns QUIC_STATUS (checked ✅)
     └─ CxPlatDpRawPlumbRulesOnSocket(..) → void (NOT checked ❌)
          └─ CxPlatDpRawInterfaceAddRules → void, silently fails on alloc failure or rule overflow
               └─ CxPlatDpRawInterfaceUpdateRules → void, has TODO: "Figure out how to better handle failure"
                    └─ XdpCreateProgram → returns QUIC_STATUS (failure logged, then `continue`)

When XDP is not backed up with OS sockets, we would silently ignore all traffic. Probably applies to QTIP too.

[ProjectsByJackHe]

The PR description says:

any failures plumbing xdp rules will bubble up as a socket creation error to the app. Can't fall back to OS sockets.

But errors when setting up XDP rules seems to be silently eaten:

   RawSocketCreateUdp                     → returns QUIC_STATUS
     └─ CxPlatTryAddSocket(...)           → returns QUIC_STATUS (checked ✅)
     └─ CxPlatDpRawPlumbRulesOnSocket(..) → void (NOT checked ❌)
          └─ CxPlatDpRawInterfaceAddRules → void, silently fails on alloc failure or rule overflow
               └─ CxPlatDpRawInterfaceUpdateRules → void, has TODO: "Figure out how to better handle failure"
                    └─ XdpCreateProgram → returns QUIC_STATUS (failure logged, then `continue`)

When XDP is not backed up with OS sockets, we would silently ignore all traffic. Probably applies to QTIP too.

Yeah, looking through the code, xdp failures in rule plumbing get silently ignored. This is a latent bug impacting QTIP (now CIBIR as well). Let's create a separate issue for this and address it in a follow up PR. This PR is starting to balloon into a very hard to review size.

[ProjectsByJackHe]

The PR description says:

any failures plumbing xdp rules will bubble up as a socket creation error to the app. Can't fall back to OS sockets.

But errors when setting up XDP rules seems to be silently eaten:

   RawSocketCreateUdp                     → returns QUIC_STATUS
     └─ CxPlatTryAddSocket(...)           → returns QUIC_STATUS (checked ✅)
     └─ CxPlatDpRawPlumbRulesOnSocket(..) → void (NOT checked ❌)
          └─ CxPlatDpRawInterfaceAddRules → void, silently fails on alloc failure or rule overflow
               └─ CxPlatDpRawInterfaceUpdateRules → void, has TODO: "Figure out how to better handle failure"
                    └─ XdpCreateProgram → returns QUIC_STATUS (failure logged, then `continue`)

When XDP is not backed up with OS sockets, we would silently ignore all traffic. Probably applies to QTIP too.

#5935

[guhetier]

Looks good to me, two minor nits and it is ready to go.