[Bug]: Running multiple instances of genesis maxes out GPU compute

[vybhav-ibr]

Bug Description

I would like to run two/more instances of genesis on one GPU. But when i tried to run the rigid benchmarks examples the GPU runs out of compute.

If i run the anymal benchmark with 40 envs, split across 4 instances of genesis, the GPU compute stays at 90% percent.Running 40 envs in one instance of genesis does not lead to this behavior

Running multiple genesis instances on the same GPU helps with trying out different reward strategies in parallel. IsaacLab(physX) supports this

If this is the expected behavior, I would like to know if the same behavior scales with compute. Am i just observing the overhead ?

Steps to Reproduce

monitor the GPU compute and memory while running the anymal-c benchmark with the 3000 envs split across 2-4 instances of genesis. Compare the GPU usage vs running 3000 in a single instance of genesis.

Expected Behavior

The GPU compute stays consistent in both the scenarios

Environment

• OS: Ubuntu 22.04
• GPU/CPU: RTX 3050
• GPU-driver version: 550.163.01
• CUDA / CUDA-toolkit version:12.4

Release version or Commit ID

latest main

[duburcqa]

If i run the anymal benchmark with 40 envs, split across 4 instances of genesis, the GPU compute stays at 90% percent.Running 40 envs in one instance of genesis does not lead to this behavior

I'm not entirely surprised. This may be due to how Quadrants (aka Taichi) reserves memory. There is some lower-level argument device_memory_GB that can be passed to Quadrants when initialising it. Maybe you could try and see? if you do not want to modify the source code, you can just set one of these environments variables: QD_DEVICE_MEMORY_FRACTION=0.2 or QD_DEVICE_MEMORY_GB=4. It would suggest the first one for your use-case.

[vybhav-ibr]

I have tried to set the QD_DEVICE_MEMORY_FRACTION from 0.2 to 0.5, the behavior doesn't change when running two instances of genesis. The overall VRAM usage is less than 1GB(total VRAM is 4GB) but compute is >90%

can you address this, If this is true i will rent a VM with better GPU like a 4090

If this is the expected behavior, I would like to know if the same behavior scales with compute. Am i just observing the overhead ?

might be relevant, if i have two instances with 0.5 reserved for each instance GPU compute averages at 95%(ignoring some small fluctuations). If i reserve 0.3 per 3 instances, the compute hits 99%. So It seems like there is an additional overhead per instance when running multiple instances ?

the results here were obtained with show_viewer=False

[duburcqa]

Ah ok I see what you mean. Yes, it is necessary to pack all the environments running in parallel in a single Genesis instance to maximise GPU utilisation. Unless each Genesis instance is already batching around 8192 envs, beyond this point performance improvement should be more or less flat, and running multiple instances may even speedup simulation but I'm not sure about this. The secrete sauce for fast simulation is basically batching on GPU, if you run on CPU, then small batch sizes around 64 should be already enough to max out compute, but GPUs are different beasts. You need batch size 8192 to max out compute, which is restricting, but the advantage is much higher FLOPS in such a case. If you are interested in running around 40 envs, you are better off running on parallel CPU.

[vybhav-ibr]

The 40 envs split across multiple instances is purely for testing, I would ideally like to run 2-4 instances of genesis each with 4096-8192 envs, and each of these instances train a separate rl policy , so i can experiment with different reward weights in parallel.

So my question is, If i have a really powerful GPU like an RTX 5090(or server grade cards like H100) and i am able to run 32768 envs in parallel, Would i be able to instead run 4 genesis instances with 8192 envs each?, what compute overhead can be expected per instance?. RL experiments are better/faster with even 4096 envs across 4 instances instead of 32768 in a single instance

Currently i can run 8192 envs of my task on my RTX 3050 4GB and here is there nvtop result. The memory usage and the compute dont even approach 80%. But if i try to run the same task with two instances of genesis with 20 envs each, the compute hits 95%

when you say the 'performace improvement should be flat' after 8192 envs, do you mean the FPS per env saturates but the total FPS continues to scale with the number of envs?, or does the total FPS saturate as well?