!> Please note that we are effectively treating the design and implementation of SDF as a green field deployment. As such certain technologies and tools may not be available. Please contact us if you wish for anything to be added or removed.
It is not recommended to store your conda environments in your $HOME due to 1) quota limits, and 2) an inability to share conda environments across groups. We generally recommend that you install software into your $GROUP space (eg /sdf/group/<group_name>/sw - please see $GROUP storage).
Download the latest version of Miniconda from the conda website and follow the Instructions. Change the installion prefix to point to an appropriate $GROUP directory:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /tmp/Miniconda3-latest-Linux-x86_64.sh
bash /tmp/Miniconda3-latest-Linux-x86_64.sh -p /sdf/group/<group_name>/sw/conda/Replacing <group_name> appropriately.
We can also modify our ~/.condarc file as follows;
channels:
- defaults
- anaconda
- conda-forge
- pytorch
envs_dirs:
- /sdf/group/<group_name/sw/conda/envs
pkgs_dirs:
- /sdf/group/<group_name/sw/conda/pkgs
auto_activate_base: falseConda environments are a nice way of switching between different software versions/packages without multiple conda installs.
There is no unique way to create a conda environment, we illustrate here how to do so from a .yaml file (see the conda documentation for more details).
In order to create an environment called mytest with python=3.6 and numpy and pandas in it, create mytest-environment.yaml:
name: mytest
dependencies:
- python=3.6
- numpy
- pandasThen run the following command: conda env create -f mytest-environment.yaml.
If successful, you should see mytest when listing your environments: conda env list.
You can now activate your environment and use it: conda activate test. To double-check that you have the right packages, you can type conda list once in the environment and check that you see numpy and pandas.
For a detailed explanation of how to use Singularity on SDF, you can go through Yee-Ting Li's presentation here where you can find both the slides and the zoom recording.
As pulling and building a new image can use quite a lot of disk space, we recommend that you set the appropriate cache paths for singularity to not use your $HOME directory. Therefore, before pulling or building an image, define the following environment variables:
export DIR=/scratch/${USER}/.singularity
mkdir $DIR -p
export SINGULARITY_LOCALCACHEDIR=$DIR
export SINGULARITY_CACHEDIR=$DIR
export SINGULARITY_TMPDIR=$DIRTo pull an image from DockerHub, do:
singularity pull docker://<user or organization>/<repository>:<tag>We offer various of GCC through (Redhat Software Collections](https://developers.redhat.com/products/softwarecollections/overview). These are wrapped up in Environment Modules and are available with a module load devtoolset/<version>.
We also provide symlinks to the appropriate GCC version to the above devtoolset packages with module load gcc/<version>.
We provide tunnel-less access to jupyter instances running on SDF via an easy to use web portal where you can 'bring-your-own' jupyter environments. See Interactive Jupyter for more information.