how much GPU memory was used

[wkris9527]

Hi, thanks for your excellent work!
Could you please share how much GPU memory was used?
It would be very helpful for reproducing the setup.
Thanks!

[RayYoh]

Thanks for your interest. As we described in the paper, our main experiments are run on 4xH20.
But we also test the performance on 8x4090 for the 3B model.
Simply, you can use -g to set the number of devices; the script can automatically calculate GRAD_ACCUMULATION_STEPS.

bash scripts/train.sh -d PIADv1 -p Seen -g 4 -b 64 -l 0.0001 -e 30 -n exp1

[wkris9527]

Thank you for your response. I am currently running the training process with a single GPU, yet the training efficiency is unacceptably low due to the slow speed. Given this, I intend to scale up the batch size, and I wonder if you could advise me on how to adjust the learning rate (lr) in a corresponding manner to ensure training stability and performance?

[RayYoh]

Actually, I am not sure about the stability of training on a single GPU. If you have a large GPU, it might still be stable.
Usually, scaling up the batch size should also increase the lr. You can try it.

[wkris9527]

I get it, thank you for your answer. Wish you a nice day.