Execution keeps breaking even when running on A100 GPU

[hannahhb]

Using the starter kit, the LLAMA baseline isn't working for me still and keeps breaking due to OOM on the evaluation stage despite using a A100 GPU. Is there anything that can be done to stop this from happening?

[hannahhb]

I've tried os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'max_split_size_mb:64' and 128 and 'expandable_segments:True'

[hannahhb]

update: I tried running the script on A6000 rented on cudo compute but it still breaks :(( any suggestions?