fix misleading benchmarking for fp8 gemm

[shunting314]

Stacked PRs:

• ->fix misleading benchmarking for fp8 gemm #1980
• warning for process group name not found conditionally #1973
• fix _clone_args when there are non-tensor input #1963
• autotune fp8 matmul reduce scatter in unit test #1962
• only check accuracy of mutated args #1961
• consolidate accuracy check APIs #1910

---

fix misleading benchmarking for fp8 gemm

The fp8 gemm benchmarking is very misleading. Before the fix it shows helion get 5.23x speedup on h100 for 1024/1024/1024 shape. But that's too good to be true. The baseline latency is much slower than expected. It turns out that 2 things makes the baseline slow

1. we allocate the scale tensor inside the kernel. They should be passed in
2. we transpose matrix B in the kernel. It should be pre-transposed outside of the kernel.

The baseline latency now change from 0.0814 ms to 0.0097 ms (8.4x difference..)

[jansel]

What happens if you remove this line? This patch is changing the layout used by the non-reference versions as well, so it is not apples-to-apples with the prior version.

[shunting314]

Removing this line will cause torch._scaled_mm fail. The kernel requires matrix B to be column major.

I think the transpose call should usually being fused with proceeding ops in practice? But I can check how it looks like in vllm

[jansel]

There are two different kernels:

1. fp8 gemm with both args contiguous
2. fp8 gemm with second arg transposed

The issue is eager mode has a kernel for 2, but no kernel for 1. If we are measuring 1, then you don't get to pre-compute anything -- it requires two kernels to do in eager and we should measure both.