KHARMA is an implementation of the HARM scheme for general relativistic magnetohydrodynamics (GRMHD) in C++. It is based on the Parthenon AMR framework, using Kokkos for parallelism and GPU support. It is composed of modular "packages," which in theory make it easy to add or swap components representing different algorithmic components or physics processes.
KHARMA is capable of closely matching other HARM implementations, e.g. iharm3d. However, it also updates the scheme to support static and adaptive mesh refinement, new methods for primitive variable recovery, new boundary conditions, and new stability features for running difficult simulations at high resolutions reliably.
There is a bunch of documentation on the wiki. If you have a basic question, it might be answered there! There is also a Slack workspace for users of KHARMA and the associated imaging and analysis codes -- message or email @c-prather on GitHub for the link.
KHARMA requires that the system have a C++17-compliant compiler, MPI, and parallel HDF5. All other dependencies are included as submodules, and can be checked out with git by running
git submodule update --init --recursiveWhen updating the KHARMA source code, you may also have to update the submodules with
git submodule update --recursiveOld submodules are a common cause of compile errors!
On directly supported systems, or systems with standard install locations, you may be able to run:
./make.sh cleanor if you're compiling for Nvidia GPUs,
./make.sh clean cudaIf you're on Mac, you will have to use zsh because the MacOS version of bash is too old:
zsh ./make.sh <arguments>If your system does not have HDF5, KHARMA can attempt to compile it for you -- just add hdf5 when you run make.sh. If you want to omit MPI, you can add nompi (the resulting binary can still be run on multiple CPU cores! Just not multiple GPUs, or multiple nodes of a cluster).
After a successful configuration (after you see -- Generating done (X.Ys)), subsequent invocations can omit clean. If ./make.sh is not working on a supported machine (those with a file in machines/), please open an issue. Broken builds aren't uncommon, as HPC machines change software all the time.
There are many more options for make.sh! You can find them on the wiki page describing the build system.
Run a particular problem with e.g.
./run.sh -i pars/tests/orszag_tang.parnote that all options are runtime. The single KHARMA binary can run any of the parameter files in pars/, and indeed this is checked as a part of the regression tests. Note you can still disable some sub-systems manually at compile time, and of course in that case the accompanying problems will crash.
As a broad and capable code, KHARMA has quite a lot of options. Most are documented here, with specific problem setups described here.
If you need more control, you can use kharma.host or kharma.cuda directly. The script is provided mostly to load any modules or environment variables a machine needs (again, soruced from the file in machines/), regardless of whether you're running interactively or as part of a batch script.
Further information can be found on the wiki page.
KHARMA has some documentation for developers on the wiki. The docs cover some quirks of coding in C++, in particular with Kokkos/GPU programming, and in particular with Parthenon.
KHARMA is made available under the BSD 3-clause license included in each file and in the file LICENSE at the root of this repository.
This repository also carries a substantial portion of the Kokkos Kernels, in the directory external/kokkos-kernels, which is provided under the license included in that directory.
Submodules of this repository are subject to their own licenses.