Optimization engine did not run and no 3D structure converged.

[shalemeghana]

I am working on Auto3D on Linux based workstation to optimize and to generate the 3D structures. I have given a 500 compounds in .smi format. Here i have provided the parameter file which i have used in my case.
parameter.txt
I have used this command to run Auto3D engine.

python3 auto3D.py parameters.yaml

After runing the above command it is showing this error.
RuntimeError: CUDA out of memory. Tried to allocate 4.27 GiB. GPU 1 has a total capacty of 15.74 GiB of which 11.88 MiB is free. Including non-PyTorch memory, this process has 15.71 GiB memory in use. Of the allocated memory 11.76 GiB is allocated by PyTorch, and 3.81 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
A clear and concise description of what the bug is.

And the complete output of the command is given below-

/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ '
""")

     _              _             _____   ____  
    / \     _   _  | |_    ___   |___ /  |  _ \ 
   / _ \   | | | | | __|  / _ \    |_ \  | | | |
  / ___ \  | |_| | | |_  | (_) |  ___) | | |_| |
 /_/   \_\  \__,_|  \__|  \___/  |____/  |____/  2.2.8
    // Automatic generation of the low-energy 3D structures

/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ '
""")
/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ '
""")
Checking input file...
/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ '
""")
There are 499 SMILES in the input file /home/sylab02/Auto3D_pkg/standard_smiles_2.smi.
All SMILES and IDs are valid.
Suggestions for choosing isomer_engine and optimizing_engine:
Isomer engine options: RDKit and Omega.
Optimizing engine options: AIMNET.
The available memory is 60 GB.
The task will be divided into 1 jobs.
Job1, number of inputs: 499
/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ '
""")
/home/sylab02/Auto3D_pkg/auto3D.py:166: SyntaxWarning: invalid escape sequence '\ '
""")

Isomer generation for job1
Enumerating cis/tran isomers for unspecified double bonds...
Enumerating R/S isomers for unspecified atomic centers...
Removing enantiomers...
Stereo centers for 320 are not fully enumerated.
Stereo centers for 382 are not fully enumerated.
Enantiomers not removed for 54
Enumerating conformers/rotamers, removing duplicates...
100%|██████████████████████████████████| 13720/13720 [14:45:56<00:00, 3.87s/it]

Optimizing on job1
Preparing for parallel optimizing... (Max optimization steps: 10000)
Total 3D conformers: 7502
0%| | 0/10000 [00:00<?, ?it/s]
Process Process-5:
Traceback (most recent call last):
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/auto3D.py", line 142, in optim_rank_wrapper
optimizer.run()
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 426, in run
optdict = ensemble_opt(ani, coord_padded, numbers_padded, charges,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 323, in ensemble_opt
n_steps(state, param['opt_steps'], param['opttol'], param['patience'])
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 244, in n_steps
e, f = state['nn'].forward_batched(coord, numbers,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 185, in forward_batched
_e, _f = self(coord[batch], numbers[batch], charges[batch])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/Auto3D/batch_opt/batchopt.py", line 146, in forward
d = self.ani(
^^^^^^^^^
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sylab02/miniconda3/envs/auto3D/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/torch/ensemble/___torch_mangle_4.py", line 27, in forward
else:
pass
_out = (_0).forward(_in, )
~~~~~~~~~~~ <--- HERE
_r = annotate(Dict[str, Tensor], {})
_6 = torch.keys(_out)
File "code/torch/aimnet/modules.py", line 20, in forward
1 = torch.requires_grad(data[x])
module = self.module
data0 = (module).forward(data, )
~~~~~~~~~~~~~~~ <--- HERE
multipass_module = self.multipass_module
if multipass_module:
File "code/torch/aimnet/models/aimnet2.py", line 30, in forward
_out0 = (self)._zero_padded(data1, _out, )
data2 = (self)._update_q(data1, _out0, False, )
_3 = [(self)._prepare_in_a(data2, ), (self)._prepare_in_q(data2, )]
~~~~~~~~~~~~~~~~~~~ <--- HERE
_in0 = torch.cat(_3, -1)
_out1 = (_1).forward(_in0, )
File "code/torch/aimnet/models/aimnet2.py", line 108, in _prepare_in_a
a_i0 = a_i
conv_a = self.conv_a
avf_a = (conv_a).forward(a_j, data["gs"], data["gv"], )
~~~~~~~~~~~~~~~ <--- HERE
return torch.cat([a_i0, avf_a], -1)
def _prepare_in_q(self: torch.aimnet.models.aimnet2.AIMNet2,
File "code/torch/aimnet/aev.py", line 103, in forward
d2features0 = self.d2features
if d2features0:
avf_v0 = torch.einsum("...nmgd,...mag,agh->...nahd", [gv2, a, agh])
~~~~~~~~~~~~ <--- HERE
avf_v = avf_v0
else:

Traceback of TorchScript, original code (most recent call last):
File "/data/roman/AIMNet2Paper/models/ensemble.py", line 22, in forward
if k in self.x:
_in[k] = data[k]
_out = model(_in)
~~~~~ <--- HERE
_r = dict()
for k in out:
File "/home/roman/repo/aimnet2/aimnet/modules.py", line 252, in forward
torch.set_grad_enabled(True)
data[self.x].requires_grad(True)
data = self.module(data)
~~~~~~~~~~~ <--- HERE
if self.multipass_module:
y = data[self.y][self.ipass]
File "/home/roman/repo/aimnet2/aimnet/models/aimnet2.py", line 130, in forward
_in = self._prepare_in_a(data)
else:
_in = torch.cat([self._prepare_in_a(data), self._prepare_in_q(data)], dim=-1)
~~~~~~~~~~~~~~~~~~ <--- HERE

        _out = mlp(_in)

File "/home/roman/repo/aimnet2/aimnet/models/aimnet2.py", line 87, in _prepare_in_a
if self.d2features:
a_i = a_i.flatten(-2, -1)
avf_a = self.conv_a(a_j, data['gs'], data['gv'])
~~~~~~~~~~~ <--- HERE
_in = torch.cat([a_i, avf_a], dim=-1)
return _in
File "/home/roman/repo/aimnet2/aimnet/aev.py", line 131, in forward
agh = self.agh
if self.d2features:
avf_v = torch.einsum('...nmgd,...mag,agh->...nahd', gv, a, agh)
~~~~~~~~~~~~ <--- HERE
else:
avf_v = torch.einsum('...nmgd,...ma,agh->...nahd', gv, a, agh)
RuntimeError: CUDA out of memory. Tried to allocate 4.27 GiB. GPU 1 has a total capacty of 15.74 GiB of which 11.88 MiB is free. Including non-PyTorch memory, this process has 15.71 GiB memory in use. Of the allocated memory 11.76 GiB is allocated by PyTorch, and 3.81 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The optimization engine did not run, or no 3D structure converged.
The reason might be one of the following:
1. Allocated memory is not enough;
2. The input SMILES encodes invalid chemical structures;
3. Patience is too small
Process Process-3:

System information:

• Operating System: Linux ubuntu20.04 64-bit.
• Auto3D version: 2.2.8
• Python version: 3.12.1
• RDKit version: 2023.09.5
• PyTorch version: 2.1.2

[LiuCMU]

Hello, thanks for the detailed bug report. The bug is likely due to GPU 1 out of memory.

I would suggest keep the memory to the actual memory GPU1 has and set the capacity arguments to its default value 42. This will probably solve the issue.

Also, the value for max_confs was 1 in the original parameter file. Was there a specific reason for that? In this case, Auto3D will only generate 1 conformer for each molecule and optimize that one.