Consider implementing a better way of testing deployment changes with Dokku

[StevenMaude]

Why are we doing this?

While we have a reasonable setup for local development and running the OpenCodelists server, this does not entirely replicate how it runs in the production.

We don't have a good way of testing out changes that affect the Dokku deployment.

In certain cases, there may be changes that:

• either behave differently in the Dokku deployment compared with a setup without Dokku
• are not testable without having Dokku running (for example, changes to health checks)

How will we know when it's done?

When we have a working easy-to-reproduce setup that incorporates the OpenCodelists server running within Dokku, where we can rebuild the production Docker image, and see the deployment update.

What are we doing?

Possibly adding configuration that allows for local testing with Dokku?

---

Defining delivery tasks guidance

[StevenMaude]

Getting a working Dokku setup in Codespaces

On using the docker-compose.yaml installation route

I found it much trickier to get Dokku running outside of a Docker Compose setup in Codespaces.

While I got this working once:

• devcontainers don't typically have systemd installed, which makes Docker and maybe Dokku trickier to install
  • I found a configuration that was supposed to get this working, which I think did, but also still didn't fix the glitches with running Docker and Dokku
• I was able to run dockerd manually, but that kept breaking on failed Dokku deployments or for some other reason, and I had to keep restarting it on failure

Getting quick feedback

The health checks we have configured for Dokku don't really facilitate quick feedback on failed deployment, when working out this kind of setup. I resorted to reducing the number of checks, their delay and their timeouts, and building the Docker image with this modified configuration.

Another way I was debugging things was running commands via the entrypoint scripts to see what was happening.

Dokku does a way of getting logs of failed deploys too.

In the repository

1. Add a .devcontainer/devcontainer.json config:

   {
       "name": "OpenCodelists",
       "features": {
           "ghcr.io/devcontainers/features/docker-in-docker:2": {}
       }
   }

2. Start a GitHub codespace using that configuration.

On the codespace host

1. Install just (we could do apt install just if a devcontainer image with Ubuntu 24.04 is used; the default uses 22.04)

   cd /tmp && wget https://github.com/casey/just/releases/download/1.40.0/just-1.40.0-x86_64-unknown-linux-musl.tar.gz && tar -xzf just-1.40.0-x86_64-unknown-linux-musl.tar.gz && chmod 555 just
   sudo mv just /usr/bin/
   cd -

2. Build the example databases:

   cd /workspaces/opencodelists
   cp dotenv-sample .env
   touch db.sqlite3 # workaround because build-dbs-for-local-development expects a db.sqlite3
   just build-dbs-for-local-development nuclear

3. Add docker-compose.yaml for Dokku 0.32.4 as on our servers (possibly only the port 7000:7000 is needed?)

   services:
     dokku:
       image: dokku/dokku:0.32.4
       container_name: dokku
       network_mode: bridge
       ports:
         - "3022:22"
         - "8080:80"
         - "8443:443"
         - "7000:7000"
       volumes:
         - "/var/lib/dokku:/mnt/dokku"
         - "/var/run/docker.sock:/var/run/docker.sock"
       environment:
         DOKKU_HOSTNAME: dokku.me
         DOKKU_HOST_ROOT: /var/lib/dokku/home/dokku
         DOKKU_LIB_HOST_ROOT: /var/lib/dokku/var/lib/dokku
       restart: unless-stopped

4. Run docker-compose up and wait for Dokku to start.

In the container as root

On the host, docker exec -it dokku bash to enter the container and run:

dokku apps:create opencodelists
dokku config:set opencodelists BASE_URLS="http://localhost:7000,http://127.0.0.1:7000" DATABASE_DIR="/storage" DATABASE_URL="sqlite:////storage/db.sqlite3" DJANGO_SETTINGS_MODULE="opencodelists.settings" SECRET_KEY="thisisatestsecretkeyfortestingonly" TRUD_API_KEY="thisisatesttrudkeyfortestingonly"

On the host

cd /workspaces/opencodelists

sudo mkdir -p /var/lib/dokku/data/storage/opencodelists
sudo mkdir -p /var/lib/dokku/data/storage/opencodelists/coding_systems/bnf/
sudo mkdir -p /var/lib/dokku/data/storage/opencodelists/coding_systems/dmd/
sudo mkdir -p /var/lib/dokku/data/storage/opencodelists/coding_systems/icd10/
sudo mkdir -p /var/lib/dokku/data/storage/opencodelists/coding_systems/snomedct/

sudo cp db.sqlite3 /var/lib/dokku/data/storage/opencodelists/
sudo cp coding_systems/bnf/bnf_test_20200101.sqlite3 /var/lib/dokku/data/storage/opencodelists/coding_systems/bnf/
sudo cp coding_systems/dmd/dmd_test_20200101.sqlite3 /var/lib/dokku/data/storage/opencodelists/coding_systems/dmd/
sudo cp coding_systems/icd10/icd10_test_20200101.sqlite3 /var/lib/dokku/data/storage/opencodelists/coding_systems/icd10/
sudo cp coding_systems/snomedct/snomedct_test_20200101.sqlite3 /var/lib/dokku/data/storage/opencodelists/coding_systems/snomedct/

sudo chown -R 10003:10003 /var/lib/dokku/data/storage/opencodelists

cd /workspaces/opencodelists/docker
just build prod

In the container as root

dokku storage:mount opencodelists /var/lib/dokku/data/storage/opencodelists/:/storage
dokku git:from-image opencodelists opencodelists:latest

That should allow you to connect to the URL on port 7000 in the codespace Ports tab, and see a running server with the test data.

On starting the codespace again after it is stopped

On the host

cd /workspaces/opencodelists
docker-compose up

In the container

dokku ps:rebuild opencodelists opencodelists:latest

[StevenMaude]

I sketched out a working Codespaces configuration in #2487.