[Question]: Is efa-dp-direct integration planned as a GPU-initiated transport for libfabric/EFA?

[quanta42]

Question

In issue #4, @a-szegel pointed to amzn/efa-dp-direct as the EFA equivalent of IBGDA, a CUDA library that lets GPU kernels post work requests and poll completions directly on EFA queue pairs without CPU proxy involvement.

Currently, NVSHMEM's libfabric transport on EFA routes all device-initiated operations through a CPU proxy thread. The performance docs note that IBGDA "differs from other transports that rely on passing messages to a CPU proxy thread to initiate the transfer." For fine-grained P2P workloads this proxy path is a severe bottleneck.

Our use case: Ring attention for distributed video diffusion training on AWS P5en (H200, 4× EFA NICs/node). The workload performs nvshmemx_putmem_nbi_on_stream + nvshmemx_signal_op across nodes (32 layers × 7 ring hops × forward+backward).

Observed:
NVSHMEM cross-node ring: ~248 s/step, 1% GPU utilization
Same topology with NCCL P2P: ~65 s/step — 2.6× faster
GPU shows burst-then-idle pattern: 100% during compute, then 0% while proxy drains transfers
This matches the Perseus paper (arXiv:2605.00686) analysis of proxy fence serialization reducing signaled transfer throughput to ~2% of unsignaled at high concurrency on non-IBGDA fabrics.

Questions:
Is there a plan to integrate efa-dp-direct as a GPU-initiated transport in NVSHMEM (similar to how IBGDA provides GPU-direct NIC submission for Mellanox)?

If not planned, are there alternative API patterns or tuning parameters that can reduce the proxy serialization overhead for fine-grained signaled P2P on EFA?

Environment:
NVSHMEM v3.6.5-0 (with patches #35, #76)
CUDA 12.8.1, NCCL v2.29.7, aws-ofi-nccl v1.19.0, GDRCopy v2.5.1
4× H200 nodes (P5en), 8 GPUs/node, 4× EFA NICs/node

[amitrad-aws]

Thanks for the detailed report and the reference to the Perseus paper.

To answer your questions,
There is no current plan to integrate efa-dp-direct to the libfabric transport.
The libfabric transport is designed to be fabric-agnostic through the OFI abstraction. Integrating efa-dp-direct would mean bypassing libfabric entirely and adding a hardware-specific GPU-initiated path for a single provider, which defeats the purpose of the transport's portability.

As for the alternative, the proxy isn't actually the problem here.
The Perseus paper you reference actually arrives at this conclusion — the performance collapse comes from fence-induced serialization in the ordering mechanism between put and signal, not from the proxy submission model itself.
This is addressed in the upcoming NVSHMEM 3.7 release.
We've made significant optimizations to both the libfabric transport and the proxy in general, including removing the need for a fence between put and signal.

I'd recommend re-benchmarking your ring attention workload on 3.7 when it's available.

[quanta42]

Thanks for the detailed report and the reference to the Perseus paper.

To answer your questions, There is no current plan to integrate efa-dp-direct to the libfabric transport. The libfabric transport is designed to be fabric-agnostic through the OFI abstraction. Integrating efa-dp-direct would mean bypassing libfabric entirely and adding a hardware-specific GPU-initiated path for a single provider, which defeats the purpose of the transport's portability.

As for the alternative, the proxy isn't actually the problem here. The Perseus paper you reference actually arrives at this conclusion — the performance collapse comes from fence-induced serialization in the ordering mechanism between put and signal, not from the proxy submission model itself. This is addressed in the upcoming NVSHMEM 3.7 release. We've made significant optimizations to both the libfabric transport and the proxy in general, including removing the need for a fence between put and signal.

I'd recommend re-benchmarking your ring attention workload on 3.7 when it's available.

Thank you, the fence removal in 3.7 is great news and should address the dominant bottleneck we observed. We'll definitely re-benchmark once it's available.

One clarification on "the proxy isn't actually the problem here." We agree the fence serialization is the primary issue for our current workload. But architecturally, the proxy path is still fundamentally different from IBGDA: a single CPU thread submitting to the NIC vs hundreds of GPU threads posting work queue entries in parallel. IBGDA has no proxy at all. Even with the fence removed, the proxy's single-threaded submission rate will be a ceiling that doesn't exist on IBGDA — whether that ceiling matters in practice depends on message rates and transfer sizes, which is why benchmarking on 3.7 is the right next step.

Two follow-ups:
Is there a timeline or early access path for 3.7? We'd like to test ring attention and fused FSDP (AllGather+GEMM overlap with per-tile signaling) as soon as possible.

Does the optimization apply to putmem_signal_nbi specifically, or to the broader put + signal pattern across all libfabric providers?

Best,

[amitrad-aws]

I understand your concern about the message rate with the proxy, there are limitations with creating a GDA path for libfabric that we need to see the real advantage to see if we should tackle it.
The optimization is broader and applies for put + signal/amo across all libfabric providers.