Skip to content

docs(production): document service-management permissions #357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wrigjl merged 1 commit into from
Jul 6, 2026
Merged

docs(production): document service-management permissions #357

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 90 additions & 92 deletions Documentation/production.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,127 +2,125 @@

Running redux on the production server is documented here.

## Initial configuration
## Common operations

We're going to assume the system is using systemd (the current
system process manager used by almost all Linux distributions).
### How to restart the services

First, install the systemd service file. The file below should
be installed to `/etc/systemd/system/redux.service`, owned by root/root and
not writeable by anyone else.
All of the services are run as a single docker composition and it is
started and stopped via systemd. Members of the `redux` group are granted
permission to start/stop/restart the service via sudo (see
[How it is configured](#how-it-is-configured) below), so no password is
required.

To start/restart:
```
sudo systemctl restart redux
```
[Unit]
Description=redux
Documentation=Redux backend
After=network-online.target
After=remote-fs.target
Wants=network-online.target
Wants=remote-fs.target
ConditionPathExists=/home/USER/Redux

[Service]
Type=simple
User=USER
ExecStart=/home/USER/.dotnet/dotnet run
WorkingDirectory=/home/USER/Redux
ExecReload=/bin/kill $MAINPID
Restart=on-failure
To stop:
```
sudo systemctl stop redux
```

[Install]
WantedBy=multi-user.target
If you want to disable the service (or re-enable it), systemctl's
enable/disable commands can be used. Note that the sudo grant only
covers start/stop/restart; enable/disable require a password.

#
# There are a few variables you might want to edit:
# - ExecStart
# - WorkingDirectory
# - User
# - ConditionPathExists
#
# ExecStart should be the location of the dotnet command
# which can be found by doing "which dotnet". It should be
# followed by the "run" command
#
# WorkingDirectory is the location of the API.csproj file
# for Redux (the base directory for the repository)
#
# User is the username of the user whose permissions should be
# used to run the backend.
#
# ConditionPathExists should be the same as WorkingDirectory
# (if you delete the directory, this service will not try to start)
```
### Viewing the logs

To ensure the file ownership and mode:
systemd services use journalctl to manage their logs. To view, use the command below.
Note that this log has entries for all of the redux services.

```
chown root:root /etc/systemd/system/redux.service
chmod 644 /etc/systemd/system/redux.service
sudo journalctl -xeu redux
```
### Updating

The docker containers are NOT upgraded regularly and must be manually upgraded on
the production machine (portneuf). They are regularly updated on `redux.thought.net`,
though.

Make sure you kill off any currently running Redux server, and then
let's get systemd to do the rest of the work:
To update the packages on the production server:

```
# systemd will notice/read the new service file
systemctl daemon-reload
# systemd will start the service on boot
systemctl enable redux.service
# go ahead and start it now
systemctl start redux.service
$ sudo docker compose pull
[+] Pulling 4/4
✔ redux-gui Pulled 0.8s
✔ mcpredux Pulled 0.8s
✔ redux Pulled 0.9s
✔ quantumsolver Pulled 0.8s
$ sudo systemctl restart redux
```

That's it to get things into normal operation!
The restart completes without a password prompt because the sudoers rule
grants the `redux` group NOPASSWD access to manage the service.

## What about updates?
## How it is configured

From time to time, you'll want to update the code base and
restart the service.
All of the redux services (Redux_GUI, Redux, mcpredux, quantumsolver) are
composed into a set of docker containers. Each container is built by github actions
and the composition file uses those containers. There is no source code on the
production server.

```
# get to the right directory
cd [working directory with API.csproj in it]
The compose file can be found in `/home/redux/docker-compose.yml`. It currently
exposes the following ports to the internet at large:
- Redux (http on 27000)
- Redux_GUI (http on 3000)
- mcpredux (http on 27200)
- quantumsolver (http on 27100)

# pull changes into working directory
git pull origin
Internally, Redux_GUI and mcpredux connect to Redux on the docker private network.
Likewise, Redux connects to quantumsolver on the docker private network. The public
exposure of the other ports is a legacy behavior.

# restart the service
sudo systemctl restart redux.service
```
In front of the composed container is an nginx server which handles TLS and connects
to redux as a reverse proxy.

## What about errors?
### service restarts (sudo)

Errors logged via `Console.WriteLine` are sent to the standard output
of dotnet programs. Luckily, systemd logs these in a circular buffer
by default. To view it:
Members of the `redux` group are allowed to start/stop/restart the service
without a password via the following line in /etc/sudoers (edit with
`visudo`):

```
# show the console log for redux
journalctl -xeu redux
```
```
%redux ALL=(root) NOPASSWD: /usr/bin/systemctl restart redux, \
/usr/bin/systemctl stop redux, \
/usr/bin/systemctl start redux
```

## Stop the service
This covers start/stop/restart only; enabling/disabling the unit is not
granted and still prompts for a password. The command match is exact, so
only these three invocations (no extra arguments) are permitted.

If you want to stop the service, easy enough:
### docker container updates

```
sudo systemctl stop redux.service
```
This requires the following line in /etc/sudoers:

Stopping it stops it for now. It will still restart on boot. To
disable it (make sure it never starts again):
```
%redux ALL=(root) CWD=/home/redux NOPASSWD: /usr/bin/docker compose pull
```

```
sudo systemctl disable redux.service
```
### systemd

The service file can be found in `/etc/systemd/system/redux.service` and
looks something like:

## Why use systemd?
```
[Unit]
Description=Redux composition
After=docker.service
Requires=docker.service

Running services as root is dangerous. If there's a vulnerability, the
attacker can run code as the user running the process. Using an
unpriviledged user means that the potential damage from a vulnerability
is limited drastically.
[Service]
Type=simple
User=redux
Group=redux
WorkingDirectory=/home/redux
ExecStart=docker compose up
ExecStop=docker compose down
Restart=on-failure
RestartSec=5

Further, we get logging infrastructure for free: systemd logs the
standard output in such a way that we don't have to worry about filling
the disk on really bad looping errors, etc.
[Install]
WantedBy=multi-user.target
```