Skip to content

DEVCONTAINERS.md

Latest commit

 

History

History
82 lines (61 loc) · 2.9 KB

DEVCONTAINERS.md

File metadata and controls

82 lines (61 loc) · 2.9 KB

Instructions to use devcontainers

The first step is to create a Python environment with a few required packages.

python3 -m venv ams-env && . ams-env/bin/activate
pip install numpy torch h5py cffi pyyaml
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$(python -c 'import torch;print(f"{torch.__path__[0]}.libs")') # For libopenblasp etc
export TORCH_DIR=$(python -c 'import torch;print(f"{torch.utils.cmake_prefix_path}/Torch")')
export PYTHONPATH="/usr/lib/flux/python3.11" #To use Flux python
#flux env | grep PYTHONPATH

Next, we create the RabbitMQ configuration file. Note that the order of operations matters here, the variable AMS_RMQ_CONFIG will be used by CMake to create the final AMS configuration.

echo """{
    \"rabbitmq-user\": \"${RABBITMQ_USER}\",
    \"rabbitmq-password\": \"${RABBITMQ_PASS}\",
    \"service-port\": ${RABBITMQ_PORT},
    \"service-host\": \"${RABBITMQ_HOST}\",
    \"rabbitmq-vhost\": \"/\",
    \"rabbitmq-queue-physics\": \"test-ci\",
    \"rabbitmq-exchange-training\": \"ams-fanout\",
    \"rabbitmq-key-training\": \"training\"
    }""" > rmq.json
export AMS_RMQ_CONFIG=$(cat ./rmq.json)

Next, we compile AMS.

Important: as of now MPI and CUDA will not work in a devcontainers.

cd /workspace && mkdir -p build && cd build
cmake -DBUILD_SHARED_LIBS=On -DWITH_CALIPER=On -DWITH_HDF5=On -DCMAKE_BUILD_TYPE=Release -DTorch_DIR=$TORCH_DIR -DWITH_CUDA=Off -DWITH_MPI=Off -DWITH_TESTS=On -DWITH_AMS_DEBUG=On -DWITH_WORKFLOW=On -DWITH_RMQ=On .. && make -j4

You can find the test configuration generated by AMS here: /workspace/build/tests/AMSlib/ams_interface/rmq.json.

Then, you can run the tests with make test.

Note that, if you want to manually inject the content of AMS_RMQ_CONFIG into the JSON configuration:

jq --argjson value "$AMS_RMQ_CONFIG" '.db.rmq_config = $value' /workspace/build/tests/AMSlib/ams_interface/rmq.json > tmp.json
mv tmp.json /workspace/build/tests/AMSlib/ams_interface/rmq.json

Test MariaDB

If you want to only test the database part of AMS with MariaDB you can run:

export AMS_DB_URL="mysql+mysqlconnector://${MARIADB_USER}:${MARIADB_PASSWORD}@${MARIADB_HOST}:${MARIADB_PORT}/${MARIADB_DATABASE}"
python -m unittest discover -s /workspace/tests/AMSWorkflow -p "test_*.py" -v

Please make that AMS_DB_URL is correct.

Troubleshooting

credsStore problem on macOS

On macOS, there is a common issue with Docker Desktop, if you get the following error messages:

failed to solve: error getting credentials - err: exec: "docker-credential-desktop": executable file not found in $PATH, out: ``

The problem lies in the file ~/.docker/config.json where credsStore should be spelled credStore. To fix it, you can simply run:

sed -i '' 's/credsStore/credStore/' ~/.docker/config.json

Note that this issue will re-appear each time you restart Docker.