Skip to content

added ROCM mpi support #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ScSteffen wants to merge 5 commits into
base: main
Choose a base branch
from
Open

added ROCM mpi support #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e8aee37
added cuda mpi support
ScSteffen Feb 13, 2026
96bfeea
streamline installer
ScSteffen Feb 18, 2026
e47d7d8
compatibility with apptainer
ScSteffen Feb 18, 2026
e554175
gmsh now always uses container
ScSteffen Feb 19, 2026
e401efa
update sim utils
ScSteffen Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,10 @@ Precedence for hyperparameters is:

Uses image/executable:
`kitrt_code/tools/singularity/kit_rt_MPI_cuda.sif` and `./kitrt_code/build_singularity_cuda/KiT-RT`.
CUDA runs are dispatched as:
`singularity exec --nv ... mpirun -np <gpu_count> ./kitrt_code/build_singularity_cuda/KiT-RT ...`.
`<gpu_count>` is auto-detected from `CUDA_VISIBLE_DEVICES` or `nvidia-smi`.
Override rank count with `KITRT_CUDA_MPI_RANKS=<N>`.

4. **SLURM mode, raw (no Singularity)**

Expand Down
202 changes: 0 additions & 202 deletions explore_exploit_hohlraum.py
View file
Open in desktop

This file was deleted.

44 changes: 41 additions & 3 deletions install_kitrt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,29 @@ has_cuda_gpu() {
nvidia-smi -L >/dev/null 2>&1
}

# clone KiT-RT
git clone "${KITRT_REPO_URL}" kitrt_code
has_rocm_gpu() {
if command -v rocm-smi >/dev/null 2>&1; then
rocm-smi --showid 2>/dev/null | grep -q "^GPU\\["
return
fi

if command -v rocminfo >/dev/null 2>&1; then
rocminfo 2>/dev/null | grep -qi "gfx"
return
fi

return 1
}

# clone KiT-RT once and reuse the existing checkout on subsequent runs
if [ -d "kitrt_code/.git" ]; then
echo "Existing kitrt_code checkout detected. Skipping clone + checkout."
elif [ -d "kitrt_code" ]; then
echo "Directory kitrt_code exists but is not a git repository. Please remove or rename it first." >&2
exit 1
else
git clone "${KITRT_REPO_URL}" kitrt_code
fi

# go to kitrt_code directory
cd kitrt_code
Expand All @@ -22,7 +43,11 @@ git submodule update --init --recursive

# navigate to directory where the singularity scripts are located
cd tools/singularity
chmod +x build_container.sh install_kitrt_singularity.sh install_kitrt_singularity_cuda.sh
chmod +x \
build_container.sh \
install_kitrt_singularity.sh \
install_kitrt_singularity_cuda.sh \
install_kitrt_singularity_rocm.sh

# build CPU singularity container. This requires root privileges.
echo "Building CPU singularity container (sudo required)."
Expand All @@ -40,5 +65,18 @@ else
echo "No CUDA GPU detected. Skipping CUDA singularity container and CUDA build."
fi

# optionally build and compile ROCm KiT-RT if a ROCm GPU is present
if has_rocm_gpu; then
if [ -f "kit_rt_MPI_rocm72.def" ]; then
echo "ROCm GPU detected. Building ROCm singularity container and ROCm KiT-RT binary."
sudo ./build_container.sh rocm
singularity exec --rocm kit_rt_MPI_rocm72.sif ./install_kitrt_singularity_rocm.sh
else
echo "ROCm GPU detected, but kit_rt_MPI_rocm72.def was not found. Skipping ROCm build."
fi
else
echo "No ROCm GPU detected. Skipping ROCm singularity container and ROCm build."
fi

# go back to CharmKiT repo root
cd ../../../
Loading