Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 37 additions & 12 deletions docs/software/apptainer.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,57 @@
# Apptainer

Apptainer and other container runtimes have many uses; Mainly, but not limited to, isolated, portable, and reproducible software execution and development. For running containers on HPC systems we offer and use Apptainer ( formerly Singularity ). Other container runtimes like Docker require users to have escalated system privileges, and for most that's typically not an issue, for shared multi-user systems letting any and all users have escalated privileges can pose a security risk. Luckily, apptainer allows users to run unprivileged containers in user-space including Docker containers and any OCI compatible containers.
For running containers on HPC systems we offer and use Apptainer ( formerly Singularity ). Other container runtimes like Docker require users to have escalated system priveleges, and for most that's typically not an issue, but for shared multi-user systems letting any and all users have escalated privileges can pose a security risk. Luckily, apptainer allows users to run unprivileged containers in user-space including Docker containers and any OCI compatible containers.

## Module System
## Quickstart

HPC@UCD provides apptainer as an environment module. Every time you want to use apptainer you must first run: `module load apptainer`.
The quick and dirty way to get started on exploring and testing your software in an apptainer ASAP:

## Quickstart
`srun -A adamgrp -p high -n 1 -c 8 --mem 32G --time 60:00 --pty bash -c "module load apptainer; apptainer shell docker://tensorflow/tensorflow"`

Feel free to curtail the srun parameters for your own needs.

## Singularity Image Format

For the absolute fastest way to get started on testing your software in an apptainer you can use something like this: `srun -A adamgrp -p high -n 1 -c 8 --mem 32G --time 30:00 --pty bash -c "module load apptainer; apptainer shell docker://tensorflow/tensorflow"`. But, you may need to curtail some `srun` parameters for your own needs.
A ".sif" is just a container image that you've already downloaded and stored locally and the result of commands like:

## Pulling Apptainer Images
`apptainer build tflo.sif docker://tensorflow/tensorflow`

Container images are like filesystems containing an Operating System (OS) with the relevant pre-installed software that is then booted like a Virtual Machine (VM). We recommend downloading your container image(s) locally once before your slurm job by running: `apptainer build tflo.sif docker://tensorflow/tensorflow`. After, you should find the `tflo.sif` apptainer image in your current directory. You can run this image in any number of future containers, which is why you only need to do it once and shouldn't run it in every slurm job.
Mostly for if you want the image in a specific location, not really necessary. By default all apptainer images are stored in the apptainer cache located in: `~/.apptainer/cache`.

Or with: `apptainer cache --help`

## Running Apptainer Images

There are a few ways to interact with your apptainer image. Using `apptainer shell tflo.sif` will open an interactive shell within the container allowing you to interact with the installed software directly as though it were a typical VM.Using `apptainer exec tflo.sif <command(s)>...` is non-interactive and will execute the commands you specify within the container and return the output. There's also `apptainer run` if you or your apptainer have provided runscripts which you can learn more about *[here](https://apptainer.org/docs/user/latest/quick_start.html#running-a-container)*.
There are a few ways to interact with your apptainers.

`apptainer shell tflo.sif`
as seen in the quickstart section, this will open an interactive shell within the container allowing you to interact with the installed software manually as though it were a typical VM.

`apptainer exec docker://tensorflow/tensorflow /path/to/script.sh`
is non-interactive and will execute the commands you specify within the container and return the output. There's also:

These commands that run, or otherwise execute containers (shell, exec) can take an --nv option, which will setup the container’s environment to use an NVIDIA GPU and the basic CUDA libraries to run a CUDA enabled application.
`apptainer run`
is also non-interactive and similar to exec or for if you have provided runscripts.

You can learn more about these *[here](https://apptainer.org/docs/user/latest/quick_start.html#running-a-container)*.

## Definition Files

We won't be delving into def files in this tutorial but something you should absolutely know and consider can be found *[here](https://apptainer.org/docs/user/main/quick_start.html#apptainer-definition-files)*

## Bind Mounts

If you need, you can mount directories from your host machine into the container. We use the `APPTAINER_BIND` environment variable to specify what directories to mount in the container. Use a comma-delimited string of bind path specifications in the format `src[:dest[:opts]]`, where `src` and `dest` are paths outside and inside of the container respectively. If `dest` is not given, it is set equal to `src`. Mount options (opts) may be specified as `ro` (read-only) or `rw` (read/write, which is the default). For example, if you need to mount your group or lab directory in the container you can: `export APPTAINER_BIND=/mnt,/group/subdir:container/dir:rw`.
If you need, you can mount directories from your host machine into the container. We can use the `APPTAINER_BIND` environment variable in our apptainer definition files to specify what directories to mount in the container. Use a comma-delimited string of bind path specifications in the format `src[:dest[:opts]]`, where `src` and `dest` are paths outside and inside of the container respectively. If dest is not given, it is set equal to src. Mount options (opts) may be specified as `ro` (read-only) or `rw` (read/write, which is the default). For example, if you need to mount your group or lab directory in the container you can add this line to your definition file:

`export APPTAINER_BIND=/mnt,/group/subdir:container/dir:rw`

If not using a definition file:

`apptainer shell --path /mnt,/group/subdir:container/dir:rw`

## GPUs

If you want to make use of GPUs within your apptainer you can use the `--nv` flag with `apptainer (shell | exec) --nv ...`.
If you want to make use of Nvidia GPUs within your apptainer you can use the `--nv` flag with `apptainer (shell | exec) --nv ...`.

## Slurm Batch Jobs

Expand Down Expand Up @@ -98,6 +123,6 @@ For ease of use we recommend writing a wrapper script like such:
apptainer build tflo.sif docker://tensorflow/tensorflow:latest-gpu
fi

apptainer shell --nv tflo.sif $@ # "$@" adds optional support for additional parameters that can be passed when executing this wrapper script that are passed to the apptainer
apptainer shell --nv tflo.sif $@ # "$@" for additional parameters that can be passed when executing this wrapper script that are passed to the apptainer
```
This wrapper starts a shell inside whatever apptainer you specify
Loading