Fix multiple gpu support

[joshkamm]

I'm currently meeting with Paul and we are looking into why this isn't working. Paul removed USE_ACC preprocessor statements to ensure that ACC pragma statements are immediately preceding loops.

Paul thinks if you try to compute the 3 center integrals on multiple GPUs it will fail. He modified compute_ints in hartree.cpp to only use 1 GPU for the 3 center integrals. Search for "CPMZ kludge"

We should finish SlaterGPU/#16 and then come back to this.

[joshkamm]

Note from meeting 1/2/25: I think Paul took care of #16 in his Zsgpu branch. He is still having difficulty compiling for multiple gpus though. His experience is that PGI compiler version 20.7 is able to compile this repo and successfully utilize multiple gpus, but nvidia hpc versions 22.9, 24.9, and 24.11 all fail on the same code. It seems like there is some change in how the compilers handle a combination of openmp and openacc directives and it isn't clear whether this change in behavior was intentional or unintentional.

Paul's recommendations:

1. Try to get support from nvidia on their forum
2. If that is unable to resolve the problem, see if we can get this repo to run on multiple gpus using the gnu compilers
3. If that doesn't work, try MPI instead of OpenMP

[joshkamm]

Alex wrote a draft support forum post and we just refined it together.

[joshkamm]

This is being discussed on the nvidia forum at https://forums.developer.nvidia.com/t/unable-to-get-code-to-run-on-multiple-gpus/320032

[joshkamm]

It looks like there may be an actual issue with the compiler but also a workaround

[joshkamm]

Meeting notes with Alex:

He's looking into Paul making similar changes in zsgpu that he made in his branch
Hartree fock is only single gpu
When Alex compiles FancyElectrons with his branch of SlaterGPU, it gets stuck and hangs at some point during runtime
Maybe the gpus are in an invalid state when SlaterGPU finishes
Alex is planning to run SlaterGPU separately to see if FancyElectrons can use the integrals successfully when run in a separate process

[joshkamm]

Meeting notes with Alex:

Alex ran some tests and we just submitted an update to the nvidia forum post that we've been working with (above).

[joshkamm]

Meeting notes with Paul:

• Check in with Alex, and come in together to talk with Paul if we're not sure how to proceed
• We could try using ai to help generate a minimal reproducing example in its own repository to help with debugging
  • You have OMP for loop and within you have OpenACC do a bunch of operations and then repeat that a bunch of times
• Could also just try seeing if ai suggests anything reasonable when feeding it the error that we're currently experiencing
• Compiling with gcc is probably a no go

Meeting notes with Paul 3/19/25:

• Vaibhav could work on the minimal reproducing example
• Try to check in with Alex, Vaibhav, and Nate and encourage them to talk to each other

[joshkamm]

Paul has been running his own tests and it seems like the double** to double* conversion may only partially overcome the nvidia compiler regression that was found

[joshkamm]

Nvidia fixed the compiler bug in hpc sdk 25.5 and this should now be resolved in the zsgpu branch