Add new NVSHMEM transpose communication backend with SM-based P2P copies.

[romerojosh]

This PR introduces a new transpose communication backend, CUDECOMP_TRANSPOSE_COMM_NVSHMEM_SM, that uses SM-driven transfers to perform intra-group (NVLink-connected) P2P transfers instead of using copy-engine (CE) driven transfers.

The existing CUDECOMP_TRANSPOSE_COMM_NVSHMEM backend issues all intra-group puts via the NVSHMEM host API nvshmemx_putmem_on_stream, which dispatches through the CEs. For large transfers, it is often the case that CE-driven transfers achieve higher bandwidth than SM-driven ones, which is why the existing backend was designed that way. However, as per-peer transfer sizes reduce, like in strong scaling or extreme weak scaling scenarios, SM-driven transfers become more competitive. In addition to this, the GPUs on systems equipped with NVSwitch-connectivity between GPUs (e.g., DGX systems, MNNVL systems) can typically only run a single CE-driven P2P transfer at a time, which can be a problem if individual transfer sizes become to small to saturate NVLink on their own. SM-driven transfers can more effectively run NVLink transfers to multiple peers concurrently, providing better total NVLink utilization in these scenarios.

Historically, the NCCL backend of cuDecomp has served the role of the "SM-driven" alltoall backend to fall back on when the CE-driven NVSHMEM backends were suboptimal. This PR provides a new alternative SM-driven backend for cuDecomp to make use of.

[romerojosh]

/build

[github-actions]

🚀 Build workflow triggered! View run

[github-actions]

✅ Build workflow passed! View run