RAM question

[LauSaku]

How much RAM do you use? I encountered a problem of insufficient RAM during training. I used 64G RAM and 2 4090

[LauSaku]

Hope you can reply me

[swimmiing]

In a 2-GPU setup, when only the VGGSound dataloaders (train and validation) are activated, system RAM usage reaches approximately 14GB, and each GPU uses around 5GB of memory. Since multiple dataloaders are used for validation, disabling the unnecessary ones can help prevent memory issues during training.

[LauSaku]

Thank you very much for your reply, but after I tried disabling those redundant data loaders, the RAM memory still overflowed before the second epoch. In fact, I have not prepared other datasets, only the vggsound dataset. And using vggsound10k occupies 14G RAM. I really don’t know where else the problem may be.

[LauSaku]

Maybe there is something wrong with the vggsound dataset I am using? Is the vggsound dataset you are using the center frame of the video and the corresponding 10 seconds of audio?

[swimmiing]

Whether you're using the 10k or 144k dataset, the batch size remains the same, so if approximately 14GB of RAM is used during training, that usage is expected and should not cause any issues. However, it seems that in your environment, memory may be accumulating unexpectedly. You should check whether this accumulation occurs during training itself or during the validation phase.

[LauSaku]

I set some checkpoints and found that the memory is constantly accumulating during training. Now I am only 34% of the first epoch, but the RAM is already used a lot.