[GPU][BSE-5357] Make sure GPU buffers are ready before calling MPI

[ehsantn]

Changes included in this PR

Resolves segfault inside UCX for multi-node / multi GPU TPCH Q5.

Reference: https://github.com/rapidsai/rapidsmpf/blob/a1580b7f67619e7dcafd32d4a1b933f1fce61a84/cpp/include/rapidsmpf/memory/buffer.hpp#L250

Testing strategy

Multi GPU TPCH where 2 ranks are assigned to the same GPU reliably reproduces the issue. Though this can occur with multiple GPUs and one rank per GPU. Both cases now should pass.

User facing changes

Checklist

• PR title contains "[GPU]" if changes target Bodo DataFrames GPU acceleration.
• Pipelines passed before requesting review. To run CI you must include [run CI] in your commit message.
• I am familiar with the Contributing Guide
• I have installed + ran pre-commit hooks.

[DrTodd13]

thanks.

[DrTodd13]

Can we just check if the stream is done and not do this function or say that we have nothing to send rather than just waiting here? Is there a reason that waiting is required here?

[scott-routledge2]

I don't think there's a reason we can't. I created a followup issue for that (BSE-5359) which can be done next.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.13%. Comparing base (9c48231) to head (b03532e).
⚠️ Report is 27 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1075      +/-   ##
==========================================
- Coverage   68.18%   62.13%   -6.06%     
==========================================
  Files         195      195              
  Lines       68080    68104      +24     
  Branches     9708     9713       +5     
==========================================
- Hits        46423    42318    -4105     
- Misses      18841    22860    +4019     
- Partials     2816     2926     +110