better-lyrics · boidushya · May 3, 2026 · May 3, 2026
diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml
@@ -73,6 +73,8 @@ jobs:
 
       - name: Comment on PR
         if: always()
+        env:
+          PREVIEW_DOMAIN: ${{ vars.PREVIEW_DOMAIN }}
         uses: actions/github-script@v7
         with:
           github-token: ${{ steps.app-token.outputs.token || github.token }}
@@ -83,7 +85,7 @@ jobs:
             const sha = context.payload.pull_request.head.sha;
             const shortSha = sha.slice(0, 7);
             const branch = context.payload.pull_request.head.ref;
-            const url = `https://pr-${pr}.preview.lyrics-api.boidu.dev`;
+            const url = `https://pr-${pr}.${process.env.PREVIEW_DOMAIN}`;
             const runUrl = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`;
             const commitUrl = `${context.serverUrl}/${owner}/${repo}/commit/${sha}`;
             const now = new Date().toISOString().replace('T', ' ').slice(0, 19) + ' UTC';

diff --git a/README.md b/README.md
@@ -3,7 +3,6 @@
 ![GitHub top language](https://img.shields.io/github/languages/top/better-lyrics/api)
 ![GitHub License](https://img.shields.io/github/license/better-lyrics/api)
 ![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/better-lyrics/api/go.yml)
-![Railway](https://img.shields.io/badge/deployment-railway-javascript?logo=railway&logoColor=fff&color=851AE6)
 
 This repository contains the source code for the official Better Lyrics API - primarily serving as the backend for [Better Lyrics](https://better-lyrics.boidu.dev).
 
@@ -12,69 +11,49 @@ This repository contains the source code for the official Better Lyrics API - pr
 
 ## Table of Contents
 
-- [Better Lyrics API](#better-lyrics-api)
-  - [Table of Contents](#table-of-contents)
-  - [Installation](#installation)
-  - [Usage](#usage)
-  - [API Endpoints](#api-endpoints)
-  - [Deployment](#deployment)
-    - [Railway](#railway)
-  - [Contributing](#contributing)
-  - [License](#license)
+- [Quickstart](#quickstart)
+- [API Endpoints](#api-endpoints)
+- [Deployment](#deployment)
+- [Contributing](#contributing)
+- [License](#license)
 
-## Installation
+## Quickstart
 
-To install and run the Lyrics API Go, follow these steps:
+If you just want to run it locally, you need a Go toolchain (1.22+) and a populated `.env`.
 
-1. Clone the repository: `git clone https://github.com/better-lyrics/api.git`
-2. Navigate to the project directory: `cd api`
-3. Install the dependencies: `go mod tidy`
-4. Copy the `.env.example` file to `.env` and update the environment variables as needed: `cp .env.example .env`
-5. Start the server: `go run main.go`
-
-## Usage
+```bash
+git clone https://github.com/better-lyrics/api.git && cd api
+go mod tidy
+cp .env.example .env       # fill in upstream API endpoints + credentials
+go run main.go             # serves on :8080
+```
 
-Once the server is running, you can access the API endpoints to retrieve lyrics for songs.
+The server logs request lines as it boots; once you see the listener line, hit `http://localhost:8080/health`. For hot reload during development, `./scripts/run.sh` watches the source via `nodemon`.
 
 ## API Endpoints
 
-- `GET /getLyrics?a={artist}&s={song}`: Retrieves the lyrics for the specified artist and song.
-
-## Deployment
-
-### Railway
-
-This project uses Railway's persistent volumes to maintain the cache database across deployments.
+Public:
 
-**Setup Steps:**
+- `GET /getLyrics?a={artist}&s={song}` - Retrieves synchronized lyrics for the specified artist and song
+- `GET /artwork?s={song}&a={artist}` - Returns animated album artwork
+- `GET /health` - Health check
+- `GET /stats` - API statistics (requires `Authorization` header)
 
-1. Create a new project on [Railway](https://railway.app) and connect your GitHub repository
+A full list of admin/cache endpoints (`/cache/*`, `/revalidate`, `/override`, `/health/mut`, etc.) is documented in [`CLAUDE.md`](./CLAUDE.md).
 
-2. **Create a Volume (CRITICAL):**
-   - Go to your service in Railway dashboard
-   - Click **Settings** → **Volumes** tab
-   - Click **+ New Volume**
-   - **Mount Path:** `/data`
-   - Click **Add**
+## Deployment
 
-3. **Set Environment Variables:**
-   - Go to **Variables** tab
-   - Add: `CACHE_DB_PATH=/data/cache.db`
-   - Configure all other required variables from `.env.example`
+Production runs on a single Hetzner CAX21 (ARM64, Helsinki). The whole server stack (Caddy, the API, Infisical agent for secrets sync, Beszel agent for metrics, Logdy for log streaming, B2 backups, UFW, fail2ban) lives in [`infra/`](./infra/README.md) as code.
 
-4. **Deploy!**
+To rebuild from scratch on any Ubuntu 24.04 host:
 
-**Verification:**
-After deployment, check your logs for:
-```
-[Cache] Loaded X entries from disk to memory
+```bash
+cp infra/secrets.env.example infra/secrets.env
+$EDITOR infra/secrets.env                    # fill in every value from your password manager
+sudo ./infra/bootstrap.sh                    # about 10 minutes, idempotent
 ```
-If you see `Loaded 0 entries` on subsequent deploys (after caching data), the volume isn't persisting.
 
-**Troubleshooting:**
-- Ensure the volume mount path is exactly `/data`
-- Verify `CACHE_DB_PATH=/data/cache.db` is set in Railway variables
-- The volume must be created BEFORE deploying with the updated env var
+See [`infra/README.md`](./infra/README.md) for the prerequisites and the manual steps that stay manual (DNS, provisioning, `cache.db` restore).
 
 ## Contributing
 

diff --git a/infra/.gitignore b/infra/.gitignore
@@ -0,0 +1,2 @@
+secrets.env
+*.bak
diff --git a/infra/README.md b/infra/README.md
@@ -0,0 +1,127 @@
+# infra/
+
+Everything needed to stand up the server lives here. If the Hetzner box dies or you want to move providers, `sudo ./bootstrap.sh` on a fresh Ubuntu 24.04 host rebuilds it in about ten minutes.
+
+## What runs on the box
+
+| Component | Purpose | Reachable at |
+|---|---|---|
+| `caddy` | Reverse proxy with TLS via Cloudflare DNS-01 | 80/443 |
+| `lyrics-api` | The Go API | localhost:8080 (proxied) |
+| `lyrics-api@.service` | Per-PR preview environments | localhost:9000+PR |
+| `infisical-agent` | Syncs prod secrets from Infisical Cloud into `/etc/lyrics-api.env` and restarts the API on change | n/a |
+| `beszel-agent` | System metrics, reports to a hub | localhost:45876 |
+| `logdy` | Browser log viewer for `lyrics-api` journal | localhost:8888 (proxied) |
+| Backup scripts | Daily `cache.db` dump, off-site upload to Backblaze B2 | cron |
+| `ufw` + `fail2ban` | Firewall + sshd brute-force protection | n/a |
+
+## Prerequisites
+
+On the box:
+- Fresh Ubuntu 24.04 (or compatible) with sudo and outbound network
+- SSH access for the account that will run `sudo`
+
+Off the box, before you run bootstrap:
+- DNS records pre-pointed at the box (see "Manual steps" below)
+- A Cloudflare API token scoped to Zone:DNS:Edit on the parent zone of your domains
+- A Backblaze B2 application key with read+write on the backups bucket
+- An Infisical Cloud machine identity (Universal Auth) with read access to the `prod` env
+- A Beszel hub somewhere reachable, with an agent KEY/TOKEN pair generated for this host
+- `secrets.env` populated next to `bootstrap.sh` (template: `secrets.env.example`)
+
+The compiled `lyrics-api-go` binary is optional at bootstrap time. Phase 04 installs the systemd unit either way and waits to start it until both the binary and `/etc/lyrics-api.env` exist.
+
+## Running
+
+```bash
+# from the repo root on the target box
+cp infra/secrets.env.example infra/secrets.env
+$EDITOR infra/secrets.env                    # fill every blank
+sudo ./infra/bootstrap.sh                    # full bootstrap
+sudo ./infra/bootstrap.sh --phase 03         # re-run a single phase (here: Caddy)
+```
+
+Logs go to `/var/log/bli-bootstrap.log`. Phases are idempotent: re-running reconciles drift (rewriting systemd units to match repo state, refreshing config files) without breaking anything already in place.
+
+## Manual steps (not automated)
+
+These happen outside the box and stay manual:
+
+- **Provision the Hetzner instance** with `hcloud server create --type cax21 --image ubuntu-24.04 --location hel1 ...`
+- **DNS records in Cloudflare** for the four hostnames in `secrets.env` plus the preview wildcard, all proxied (orange cloud)
+- **Infisical secrets** in the project's `prod` env. The agent only syncs; it does not create.
+- **Beszel hub** running somewhere reachable, with an agent slot for this host. The hub UI hands you the KEY/TOKEN pair for `secrets.env`.
+- **`cache.db` restore** from B2 if you're rebuilding after a loss. Separate process: `rclone copy b2:lyrics-api-backups/daily/<file> /var/lib/lyrics-api/data/cache.db`.
+
+## Deploying the lyrics-api binary
+
+The IaC installs the systemd unit but does not ship the Go binary. Two ways to get it on the box:
+
+1. **From CI** (the path used in prod): GitHub Actions builds, `scp`s to `/opt/lyrics-api/lyrics-api-go`, then `systemctl restart lyrics-api`.
+2. **From source on the box**: `git clone`, `go build -o /opt/lyrics-api/lyrics-api-go .`, `chown deploy:deploy`, `systemctl restart lyrics-api`.
+
+Either way, `infisical-agent` writes `/etc/lyrics-api.env` once it starts, which is what unblocks the first `lyrics-api` start.
+
+## Security model
+
+Most secrets live in Infisical Cloud and sync read-only to the box. The exceptions, all kept off the world-readable systemd config:
+
+- `CF_API_TOKEN` is in `/etc/caddy.env` (mode 600, root:caddy)
+- `B2_*` is in `/home/deploy/.config/rclone/rclone.conf` (mode 600, deploy:deploy)
+- `LOGDY_UI_PASS` is in `/etc/logdy.env` (mode 640, root:deploy)
+- The Infisical `client-secret` is at `/etc/infisical-agent/client-secret` (mode 600, root)
+
+`BESZEL_AGENT_TOKEN` is the one wart: it sits in `Environment=` lines on a mode-644 unit, since that's how the upstream installer ships it. The blast radius if leaked is impersonating the agent to the hub, which sends fake metrics but does not grant credentials back. The same EnvironmentFile pattern Caddy uses would close it; it is not done yet.
+
+`lyrics-api.service` itself runs as `deploy` with `ProtectSystem=strict`, `ProtectHome=true`, `PrivateTmp=true`, `NoNewPrivileges=true`. UFW restricts inbound traffic to 22/80/443. `fail2ban` watches `sshd`.
+
+## Verification after bootstrap
+
+Substitute your own hostnames from `secrets.env` for `$PRIMARY_DOMAIN` and `$LOGS_DOMAIN`.
+
+```bash
+systemctl is-active caddy lyrics-api infisical-agent beszel-agent logdy fail2ban
+ls -l /etc/caddy.env                                # -rw------- root caddy
+sudo -u nobody cat /etc/caddy.env                   # permission denied
+curl -sI https://$PRIMARY_DOMAIN/health             # 200
+curl -sI https://$LOGS_DOMAIN/                      # 200 (then 401 on actual UI without auth)
+journalctl -u infisical-agent -n 20                 # successful auth + sync
+```
+
+## Selective rebuild scenarios
+
+| Scenario | Command |
+|---|---|
+| Caddy config changed | `sudo ./bootstrap.sh --phase 03` |
+| Memory drop-in needs adjusting | edit `files/lyrics-api/memory.conf`, then `sudo ./bootstrap.sh --phase 04` |
+| Backup schedule changed | edit `files/backups/crontab.fragment`, then `sudo ./bootstrap.sh --phase 08` |
+| Logdy version bump | bump `LOGDY_VERSION` in `secrets.env`, then `sudo ./bootstrap.sh --phase 07` |
+| Beszel agent token rotated | update `BESZEL_AGENT_TOKEN` in `secrets.env`, then `sudo ./bootstrap.sh --phase 06` |
+
+## What's intentionally not here
+
+- **Provisioning** (`hcloud server create`). One command, varies per provider, not worth scripting.
+- **DNS records**. Lives in Cloudflare; the UI is fine.
+- **The `lyrics-api-go` binary**. Shipped from CI, not infra.
+- **A self-hosted Infisical instance**. 600MB resident is more than the project warrants.
+- **`cache.db` restoration**. Separate runbook, depends on which B2 snapshot you want.
+
+## GitHub Actions configuration
+
+Two workflows talk to this box: `.github/workflows/deploy-hetzner.yml` (prod deploys on push to `master`) and `.github/workflows/preview.yml` (per-PR previews). Both need configuration set in the GitHub repo:
+
+Repository **secrets** (Settings > Secrets and variables > Actions > Secrets):
+- `APP_ID`, `APP_PRIVATE_KEY` for the GitHub App that posts PR comments
+- `SSH_KEY` private key authorised on the box for the deploy user
+- `SSH_HOST` the box's IP or DNS name
+- `SSH_USER` the SSH login (e.g. `deploy`)
+
+Repository **variables** (Settings > Secrets and variables > Actions > Variables):
+- `PREVIEW_DOMAIN` the wildcard zone for PR previews, e.g. `preview.api.example.com`. Must match `PREVIEW_WILDCARD` in `secrets.env` on the box (same value, minus the leading `*.`).
+
+## When something breaks
+
+1. Read `/var/log/bli-bootstrap.log` for the failing phase
+2. Re-run just that phase with `--phase NN`
+3. If the failure is upstream (apt repo down, a GitHub release moved), the phase script is the source of truth. Open it, fix it, re-run.
+4. For infisical-agent issues, `journalctl -t infisical-agent -n 50` shows the reload script's logger output.
diff --git a/infra/bootstrap.sh b/infra/bootstrap.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+# Better Lyrics API - bootstrap a fresh Ubuntu 24.04 box from zero to production.
+# Usage: sudo ./bootstrap.sh [--phase NN]
+#   With --phase NN, runs only that single phase (e.g. --phase 03 to reconfigure Caddy).
+# Source: secrets.env (must exist next to this script).
+# Logs: /var/log/bli-bootstrap.log
+
+set -euo pipefail
+
+INFRA_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+LOG=/var/log/bli-bootstrap.log
+SECRETS="${INFRA_DIR}/secrets.env"
+
+if [ "$EUID" -ne 0 ]; then
+    echo "ERROR: bootstrap.sh must run as root (use sudo)" >&2
+    exit 1
+fi
+
+if [ ! -f "$SECRETS" ]; then
+    echo "ERROR: secrets.env not found at $SECRETS" >&2
+    echo "Copy secrets.env.example to secrets.env and fill in values." >&2
+    exit 1
+fi
+
+# shellcheck disable=SC1090
+set -a; source "$SECRETS"; set +a
+
+REQUIRED_VARS=(
+    CF_API_TOKEN ACME_EMAIL
+    INFISICAL_CLIENT_ID INFISICAL_CLIENT_SECRET INFISICAL_PROJECT_ID INFISICAL_ENV
+    B2_KEY_ID B2_APP_KEY B2_BUCKET
+    BESZEL_HUB_URL BESZEL_AGENT_KEY BESZEL_AGENT_TOKEN BESZEL_AGENT_PORT
+    LOGDY_UI_PASS LOGDY_VERSION
+    PUBLIC_IP PRIMARY_DOMAIN STAGING_DOMAIN LOGS_DOMAIN METRICS_DOMAIN PREVIEW_WILDCARD
+)
+missing=()
+for v in "${REQUIRED_VARS[@]}"; do
+    if [ -z "${!v:-}" ]; then missing+=("$v"); fi
+done
+if [ ${#missing[@]} -gt 0 ]; then
+    echo "ERROR: missing required secrets: ${missing[*]}" >&2
+    exit 1
+fi
+
+export INFRA_DIR
+
+mkdir -p "$(dirname "$LOG")"
+exec > >(tee -a "$LOG") 2>&1
+
+ONLY_PHASE=""
+if [ "${1:-}" = "--phase" ]; then
+    ONLY_PHASE="${2:?--phase requires a number}"
+fi
+
+run_phase() {
+    local script="$1"
+    local name
+    name=$(basename "$script")
+    echo
+    echo "===> $name $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+    bash "$script"
+    echo "<=== $name OK"
+}
+
+echo "=== Better Lyrics API bootstrap ==="
+echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+echo "Host: $(hostname)"
+echo "Infra dir: $INFRA_DIR"
+echo
+
+for script in "$INFRA_DIR"/phases/[0-9][0-9]-*.sh; do
+    if [ -n "$ONLY_PHASE" ]; then
+        case "$(basename "$script")" in
+            "${ONLY_PHASE}-"*) run_phase "$script" ;;
+        esac
+    else
+        run_phase "$script"
+    fi
+done
+
+echo
+echo "=== bootstrap complete ==="
+echo "Verify with:"
+echo "  systemctl is-active caddy lyrics-api infisical-agent beszel-agent logdy fail2ban"
+echo "  curl -sI https://${PRIMARY_DOMAIN}/health"
diff --git a/infra/files/backups/backup-cache.sh b/infra/files/backups/backup-cache.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+set -euo pipefail
+source /etc/lyrics-api.env
+curl -s -H "Authorization: $CACHE_ACCESS_TOKEN" http://localhost:8080/cache/backup > /dev/null
diff --git a/infra/files/backups/crontab.fragment b/infra/files/backups/crontab.fragment
@@ -0,0 +1,5 @@
+# deploy user crontab - managed by infra/phases/08-backups.sh
+# Backup local cache.db at 03:00 UTC, upload to B2 at 03:30 UTC, prune local at 04:00 UTC
+0 3 * * * /usr/local/bin/backup-cache.sh
+30 3 * * * /usr/local/bin/upload-backup.sh
+0 4 * * * find /var/lib/lyrics-api/backups -name 'cache_backup_*.db' -mtime +1 -delete
diff --git a/infra/files/backups/upload-backup.sh b/infra/files/backups/upload-backup.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+set -euo pipefail
+
+LATEST=$(ls -1t /var/lib/lyrics-api/backups/cache_backup_*.db 2>/dev/null | head -1)
+[ -z "$LATEST" ] && { echo "No backup file found"; exit 1; }
+
+# Always upload to daily/
+/usr/bin/rclone copy "$LATEST" b2:lyrics-api-backups/daily/ --transfers 4
+
+# On Sundays only, also upload to weekly/
+if [ "$(date -u +%u)" = "7" ]; then
+    /usr/bin/rclone copy "$LATEST" b2:lyrics-api-backups/weekly/ --transfers 4
+fi