Does InfiniStore support Intel NICs for RDMA?

[HonkitMaiStorage]

Issue encountered

When I try to run infinistore/example/client.py, to connect to a InfiniStore server set up using infinistore ... command, I got Unexpected opcode: 129 at first.

Solution

So, to solve this, I change the L139 in src/libinfinistore.cpp from

else if (wc[i].opcode == IBV_WC_RECV) {

to

else if (wc[i].opcode == IBV_WC_RECV || wc[i].opcode == IBV_WC_RECV_RDMA_WITH_IMM) {

Bad address Issue

Afterwards, I can run the client.py code if the src and dst devices are both CPU. For GPU usage, it raises another error, which is Exception: Failed to register memory regions (this error is raised when doing GPU <-> GPU, GPU <-> CPU). The errono displayed is errono 14: Bad address so I assume that InfiniStore cannot find the memory pointer which is in GPU (initialized via torch.zeros(4096, device='cuda:0')).

After a little bit of research, I found out that this function called in libinfinistore.cpp, ibv_reg_mr is the one that raises this issue. The reason why it raises this issue is because nvidia-peermem package is not installed in my machine so the function cannot find the memory address given (which is in GPU). To use nvidia-peermem, I would need ConnectX-3 or newer adapters.

Does this means Intel NIC (like E810) is actually not supported by InfiniStore by default?