Applying FermionOperator to wavefunction gives memory error

[kevinsung]

This example has 4 electrons in 8 orbitals. Why is it trying to allocate a 64 GB array?

import fqe
from openfermion import FermionOperator, hermitian_conjugated

op = FermionOperator("5^ 3 2 12^") + FermionOperator("2 4^ 11^ 15 12 14^ 4 8^")
op += hermitian_conjugated(op)
wfn = fqe.get_wavefunction(4, 0, 8)
wfn.set_wfn(strategy="hartree-fock")
new_wfn = wfn.apply(op)

Traceback (most recent call last):
  File "/home/kjs/projects/scratch/fqe-repro.py", line 8, in <module>
    new_wfn = wfn.apply(op)
              ^^^^^^^^^^^^^
  File "/home/kjs/.pyenv/versions/scratch/lib/python3.12/site-packages/fqe/fqe_decorators.py", line 365, in convert
    hamil = build_hamiltonian(ops,
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kjs/.pyenv/versions/scratch/lib/python3.12/site-packages/fqe/fqe_decorators.py", line 104, in build_hamiltonian
    ops_mat[index] = fermionops_tomatrix(term, norb)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kjs/.pyenv/versions/scratch/lib/python3.12/site-packages/fqe/fqe_decorators.py", line 225, in fermionops_tomatrix
    tensor = numpy.zeros(tensor_dim, dtype=numpy.complex128)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 64.0 GiB for an array with shape (16, 16, 16, 16, 16, 16, 16, 16) and data type complex128

[gkc1000]

It looks like it is building a Hamiltonian with a general 8 fermion interaction (but here defined only with one non zero element). Sent from Gmail Mobile

…

On Sun, Nov 17, 2024 at 12:54 PM Kevin J. Sung ***@***.***> wrote: This example has 4 electrons in 8 orbitals. Why is it trying to allocate a 64 GB array? import fqefrom openfermion import FermionOperator, hermitian_conjugated op = FermionOperator("5^ 3 2 12^") + FermionOperator("2 4^ 11^ 15 12 14^ 4 8^")op += hermitian_conjugated(op)wfn = fqe.get_wavefunction(4, 0, 8)wfn.set_wfn(strategy="hartree-fock")new_wfn = wfn.apply(op) Traceback (most recent call last): File "/home/kjs/projects/scratch/fqe-repro.py", line 8, in <module> new_wfn = wfn.apply(op) ^^^^^^^^^^^^^ File "/home/kjs/.pyenv/versions/scratch/lib/python3.12/site-packages/fqe/fqe_decorators.py", line 365, in convert hamil = build_hamiltonian(ops, ^^^^^^^^^^^^^^^^^^^^^^ File "/home/kjs/.pyenv/versions/scratch/lib/python3.12/site-packages/fqe/fqe_decorators.py", line 104, in build_hamiltonian ops_mat[index] = fermionops_tomatrix(term, norb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kjs/.pyenv/versions/scratch/lib/python3.12/site-packages/fqe/fqe_decorators.py", line 225, in fermionops_tomatrix tensor = numpy.zeros(tensor_dim, dtype=numpy.complex128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ numpy.core._exceptions._ArrayMemoryError: Unable to allocate 64.0 GiB for an array with shape (16, 16, 16, 16, 16, 16, 16, 16) and data type complex128 — Reply to this email directly, view it on GitHub <#136>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN5NRF5PWYICGXYVETGMLT2BD7B3AVCNFSM6AAAAABR6KAV5WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGY3DMNBZHA2TQMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[shiozaki]

If I remember correctly if you pass FermonOperator directly to apply, it would create a dense operator, because FQE does not know if your operator is better handled as a sparse operator or not. Can you first create a SparseHamiltonian object explicitly and pass that to apply?

https://github.com/quantumlib/OpenFermion-FQE/blob/master/src/fqe/hamiltonians/sparse_hamiltonian.py

@awhite862 please follow up when you get a chance.

[kevinsung]

Ah okay this works:

new_wfn = wfn.apply(fqe.get_sparse_hamiltonian(op))

In this case it would be nice to automatically choose to use sparse instead of dense. I'll leave this issue open in case you want to track that, otherwise feel free to close.

[awhite862]

We could potentially add some more sophisticated logic to try to guess the most efficient representation of the operator when it is passed as a FermionOperator. Currently it will only choose to treat the operator a SparseHamiltonian if there are two or fewer terms (in this example there are 4 including the hermitian conjugates). This is a reasonable choice if those terms are 1-body terms, but for many-body terms (like the 4-body terms in this example) it may not be an efficient choice. However, making the most efficient choice in general will be difficult, so it might be reasonable to leave this optimization up to the user.