Process keeps getting killed

[ZackBradshaw]

Ive tried this on a 3070 and a A30 both threw

root@18135a5a8205:/LandMark# python3 -m app.trainer --config confs/matrixcity/matrixcity_2block_multi.txt
Training with a single process on 1 GPUs.
 31%|████████████████████████████████████████████████████▉                                                                                                                     | 331/1063 [01:06<03:59,  3.06it/s]Killed
root@18135a5a8205:/LandMark# python3 -m app.trainer --config confs/matrixcity/matrixcity_2block_multi.txt
Training with a single process on 1 GPUs.
 33%|███████████████████████████████████████████████████████▋                                                                                                                  | 348/1063 [01:08<02:57,  4.03it/s]Killed

(venv) zack@clownputer:~/code/LandMark$ python3 -m app.trainer --config confs/matrixcity/matrixcity_2block_multi.txt
Training with a single process on 1 GPUs.
20%|████                | 216/1063 [00:27<02:17,  6.17it/s]Killed

[Ga1ahad]

"Insufficient memory can cause this issue. I completed the training using 256GB of memory, and at least 215GB of memory was occupied during the training process."

[Ga1ahad]

Ive tried this on a 3070 and a A30 both threw

root@18135a5a8205:/LandMark# python3 -m app.trainer --config confs/matrixcity/matrixcity_2block_multi.txt
Training with a single process on 1 GPUs.
 31%|████████████████████████████████████████████████████▉                                                                                                                     | 331/1063 [01:06<03:59,  3.06it/s]Killed
root@18135a5a8205:/LandMark# python3 -m app.trainer --config confs/matrixcity/matrixcity_2block_multi.txt
Training with a single process on 1 GPUs.
 33%|███████████████████████████████████████████████████████▋                                                                                                                  | 348/1063 [01:08<02:57,  4.03it/s]Killed

(venv) zack@clownputer:~/code/LandMark$ python3 -m app.trainer --config confs/matrixcity/matrixcity_2block_multi.txt
Training with a single process on 1 GPUs.
20%|████                | 216/1063 [00:27<02:17,  6.17it/s]Killed

"Insufficient memory can cause this issue. I completed the training using 256GB of memory, and at least 215GB of memory was occupied during the training process."