WeOwnNetwork · ncimino · Jul 15, 2026 · Jul 15, 2026 · Jul 15, 2026 · Jul 15, 2026
@@ -32,6 +32,7 @@ Changes in this section will be promoted to a dated release entry on merge to `m
 
 ### Security
 
+- **anythingllm-docker — client-side encrypted backups (GPG) + ZDR LLM routing posture (2026-07-15)** — closes two compliance gaps for customer instances holding financial/personal records (GLBA-service-provider posture). **Backups:** `backup.sh` now encrypts the tarball with OpenPGP/GPG (per-customer ed25519/cv25519 public key, `BACKUP_GPG_PUBLIC_KEY` injected via `infisical run`; keyring-free `--recipient-file` + ephemeral `GNUPGHOME`) before the Spaces upload — the object at rest is ciphertext to DigitalOcean; the private key lives ONLY in the operator store (`BACKUP_GPG_PRIVATE_KEY_<project>`, auto-generated by `deploy-new-site.sh` in an ephemeral keyring). Fail-loud when encryption is configured but `gpg` is missing; loud warning on unencrypted uploads. `restore.sh` fetches `.tar.gz.gpg`, pulls the private key **in-process** from operator-tools via the logged-in infisical CLI (paste fallback), stages it into droplet **tmpfs** (0600, ephemeral GNUPGHOME, shredded on exit via trap), decrypts, and proceeds; legacy plaintext objects still restore. GPG chosen over `age` for auditor familiarity (owner call); `gnupg` was already in cloud-init packages. **LLM routing:** per-customer OpenRouter keys operate under the account-level **Zero-Data-Retention guardrail** (ZDR-only endpoints — no provider retention or training; OpenRouter itself retains nothing); `provision-openrouter-key.sh` prints the posture requirement, and the compliance facts table in [`docs/CUSTOMER_INSTANCE_PROVISIONING.md`](docs/CUSTOMER_INSTANCE_PROVISIONING.md) documents both. **Compliance**: FTC Safeguards (GLBA) encryption-at-rest expectation; NIST CSF 2.0 PR.DS-1/PR.DS-2; CIS v8 3.11; supports IRC §7216 posture for tax data. **Verification**: 6-check GPG round-trip (encrypt → ciphertext-opaque → decrypt → byte-identical → wrong-key-rejected) passes with the exact script logic; rendered-shell `bash -n` clean.
 - **M2 fleet-wide — Infisical Machine-Identity secrets & DO Spaces creds moved off command-line argv (2026-07-14)** — closes the AGENTS.md S2 finding that `infisical login --client-secret="$INFISICAL_CLIENT_SECRET"` and `tofu init -backend-config="secret_key=…"` exposed secrets on process argv (readable via `ps` / `/proc/<pid>/cmdline` on the droplet). **Infisical login** now uses env-var auth — `export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID/_SECRET` before a flagless `infisical login --method=universal-auth` (the CLI reads them natively), matching the pattern openclaw's entrypoint already used. **Tofu/OpenTofu** now exports `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` (the S3 backend reads them from env) and drops the two `-backend-config` access/secret lines; `sse_customer_key` stays on `-backend-config` (no env equivalent). Applied across every `*-docker` template and rendered site (anythingllm, keycloak, owncloud, wordpress, searxng, signoz, sandbox, openclaw, devbox) plus `otel-agent/` and the `scripts/*otel*` helpers, and the doc examples that showed the old form. Only `--plain`/`--silent` and non-secret flags remain on argv. **Compliance**: NIST CSF 2.0 PR.DS-1/PR.AC-4; CIS v8 3.11. **Verification**: `bash -n` + `shellcheck -S error` clean on all changed scripts; rendered ansible YAML parses; 0 remaining `--client-secret=`/`-backend-config="secret_key="` in tracked code. **Note**: needs a live deploy test (copier render → ansible/tofu path) before promotion — the human pre-merge gate.
 
 ### Added

@@ -97,6 +97,41 @@ rm -rf "$WORK_DIR"
 FINAL_SIZE=$(ls -lh "$BACKUP_DIR/${BACKUP_NAME}.tar.gz" | awk '{print $5}')
 echo "==> Local backup complete: $BACKUP_DIR/${BACKUP_NAME}.tar.gz ($FINAL_SIZE)"
 
+# --- Client-side encryption (compliance: financial/personal records) ---
+# When BACKUP_GPG_PUBLIC_KEY (an ASCII-armored OpenPGP public key, injected via
+# `infisical run`) is set, the tarball is encrypted BEFORE upload, so the
+# object in Spaces is ciphertext to DigitalOcean and to anyone without the
+# private key. The private key never touches this droplet — it lives in the
+# operator secret store (pushed there by deploy-new-site.sh at provisioning).
+# Encryption is keyring-free (--recipient-file + ephemeral GNUPGHOME), so no
+# state accumulates on the box. The LOCAL plaintext tarball is kept for fast
+# local restores: it sits on the same disk as the live volumes, so it adds no
+# new exposure class.
+UPLOAD_FILE="$BACKUP_DIR/${BACKUP_NAME}.tar.gz"
+if [[ -n "${BACKUP_GPG_PUBLIC_KEY:-}" ]]; then
+  if ! command -v gpg >/dev/null 2>&1; then
+    echo "ERROR: BACKUP_GPG_PUBLIC_KEY is set but 'gpg' is not installed — refusing to" >&2
+    echo "       upload an UNENCRYPTED backup when encryption was configured." >&2
+    echo "       Fix: apt-get install -y gnupg (cloud-init installs it on new droplets)." >&2
+    exit 1
+  fi
+  echo "==> Encrypting backup (GPG, client-side)..."
+  GPG_TMP="$(mktemp -d)"
+  chmod 700 "$GPG_TMP"
+  printf '%s\n' "$BACKUP_GPG_PUBLIC_KEY" > "$GPG_TMP/pubkey.asc"
+  GNUPGHOME="$GPG_TMP" gpg --batch --yes --trust-model always \
+    --recipient-file "$GPG_TMP/pubkey.asc" \
+    -o "$BACKUP_DIR/${BACKUP_NAME}.tar.gz.gpg" \
+    --encrypt "$BACKUP_DIR/${BACKUP_NAME}.tar.gz"
+  rm -rf "$GPG_TMP"
+  UPLOAD_FILE="$BACKUP_DIR/${BACKUP_NAME}.tar.gz.gpg"
+else
+  echo "WARNING: BACKUP_GPG_PUBLIC_KEY not set — the remote backup will be UNENCRYPTED" >&2
+  echo "         (DO's server-side encryption only). For customer instances holding" >&2
+  echo "         financial/personal records this is a compliance gap: set" >&2
+  echo "         BACKUP_GPG_PUBLIC_KEY in the site's Infisical project." >&2
+fi
+
 # --- Remote upload (DO Spaces) ---
 if [[ "$REMOTE_STORAGE" == "do-spaces" ]]; then
   if [[ -z "${SPACES_ACCESS_KEY:-}" ]] || [[ -z "${SPACES_SECRET_KEY:-}" ]]; then
@@ -105,11 +140,13 @@ if [[ "$REMOTE_STORAGE" == "do-spaces" ]]; then
     echo "==> Uploading to DO Spaces (s3://${SPACES_BUCKET}/{{ project_name }}/)..."
     AWS_ACCESS_KEY_ID="$SPACES_ACCESS_KEY" \
     AWS_SECRET_ACCESS_KEY="$SPACES_SECRET_KEY" \
-    aws s3 cp "$BACKUP_DIR/${BACKUP_NAME}.tar.gz" \
+    aws s3 cp "$UPLOAD_FILE" \
       "s3://${SPACES_BUCKET}/{{ project_name }}/" \
       --endpoint-url "https://${SPACES_REGION}.digitaloceanspaces.com" \
       --quiet
-    echo "==> Remote backup uploaded successfully"
+    echo "==> Remote backup uploaded successfully ($(basename "$UPLOAD_FILE"))"
+    # The encrypted copy exists in Spaces now; don't leave a second local one.
+    [[ "$UPLOAD_FILE" == *.gpg ]] && rm -f "$UPLOAD_FILE"
   fi
 fi
 

@@ -96,6 +96,9 @@ SPACES_BUCKET="{{ backup_do_spaces_bucket }}"
 SPACES_REGION="{{ backup_do_spaces_region }}"
 
 # --- Fetch backup from DO Spaces if not present locally ---
+# Remote objects may be GPG-ENCRYPTED (.tar.gz.gpg — the compliance default when
+# BACKUP_GPG_PUBLIC_KEY is set at backup time). Try plaintext first (legacy),
+# then the encrypted object.
 if [[ ! -f "\$BACKUP_FILE" && "$REMOTE_STORAGE" == "do-spaces" && -n "\${SPACES_ACCESS_KEY:-}" && -n "\${SPACES_SECRET_KEY:-}" ]]; then
   echo "==> Backup not found locally, fetching from DO Spaces..."
   mkdir -p "\$BACKUP_DIR"
@@ -104,8 +107,39 @@ if [[ ! -f "\$BACKUP_FILE" && "$REMOTE_STORAGE" == "do-spaces" && -n "\${SPACES_
   aws s3 cp "s3://${SPACES_BUCKET}/{{ project_name }}/${BACKUP_NAME}.tar.gz" \
     "\$BACKUP_FILE" \
     --endpoint-url "https://${SPACES_REGION}.digitaloceanspaces.com" \
-    --quiet
-  echo "==> Downloaded from DO Spaces"
+    --quiet 2>/dev/null || true
+  if [[ ! -f "\$BACKUP_FILE" ]]; then
+    AWS_ACCESS_KEY_ID="\$SPACES_ACCESS_KEY" \
+    AWS_SECRET_ACCESS_KEY="\$SPACES_SECRET_KEY" \
+    aws s3 cp "s3://${SPACES_BUCKET}/{{ project_name }}/${BACKUP_NAME}.tar.gz.gpg" \
+      "\${BACKUP_FILE}.gpg" \
+      --endpoint-url "https://${SPACES_REGION}.digitaloceanspaces.com" \
+      --quiet 2>/dev/null || true
+  fi
+  echo "==> Fetch from DO Spaces complete"
+fi
+
+# --- Decrypt if the backup is GPG-encrypted ---
+# The GPG PRIVATE key never lives on this droplet: the operator wrapper stages
+# it into tmpfs (BACKUP_GPG_KEY_FILE under /dev/shm, 0600) for the duration of
+# the restore and shreds it after. Import + decrypt happen in an ephemeral
+# GNUPGHOME in tmpfs — no keyring state survives.
+if [[ ! -f "\$BACKUP_FILE" && -f "\${BACKUP_FILE}.gpg" ]]; then
+  if [[ -z "\${BACKUP_GPG_KEY_FILE:-}" || ! -f "\${BACKUP_GPG_KEY_FILE:-/nonexistent}" ]]; then
+    echo "ERROR: backup is ENCRYPTED (\${BACKUP_NAME}.tar.gz.gpg) but no GPG private key was staged." >&2
+    echo "       Re-run ./scripts/restore.sh from the operator machine (it fetches the key" >&2
+    echo "       from the operator secret store: BACKUP_GPG_PRIVATE_KEY_${PROJECT_NAME}), or" >&2
+    echo "       stage the armored key to a tmpfs file and set BACKUP_GPG_KEY_FILE." >&2
+    exit 1
+  fi
+  command -v gpg >/dev/null 2>&1 || { echo "ERROR: 'gpg' not installed on this host (apt-get install -y gnupg)." >&2; exit 1;  }
+  echo "==> Decrypting backup (GPG, ephemeral keyring in tmpfs)..."
+  GPG_TMP="\$(mktemp -d /dev/shm/restore-gnupg.XXXXXX)"
+  chmod 700 "\$GPG_TMP"
+  GNUPGHOME="\$GPG_TMP" gpg --batch --import "\$BACKUP_GPG_KEY_FILE" 2>/dev/null
+  GNUPGHOME="\$GPG_TMP" gpg --batch --yes -o "\$BACKUP_FILE" --decrypt "\${BACKUP_FILE}.gpg"
+  rm -rf "\$GPG_TMP"
+  rm -f "\${BACKUP_FILE}.gpg"
 fi
 
 if [[ ! -f "\$BACKUP_FILE" ]]; then
@@ -178,6 +212,30 @@ SCRIPT
 
   if [[ -n "$host" ]]; then
     echo "==> Running restore on remote: ${host}"
+    # --- Stage the GPG private key (encrypted backups) ---
+    # If the backup was taken with BACKUP_GPG_PUBLIC_KEY set, the object in
+    # Spaces is .tar.gz.gpg and the droplet needs the PRIVATE key to decrypt.
+    # The key lives in the operator secret store as
+    # BACKUP_GPG_PRIVATE_KEY_<project> (operator-tools project, pushed at
+    # provisioning). It is fetched IN-PROCESS via the logged-in infisical CLI
+    # (never printed), staged into tmpfs on the droplet (never its disk), and
+    # shredded after the restore. Fallback: paste the armored key (Ctrl-D to
+    # end; Ctrl-D immediately for legacy plaintext backups).
+    GPG_KEY_FILE=""
+    GPG_PRIVATE_KEY="$(infisical secrets get "BACKUP_GPG_PRIVATE_KEY_{{ project_name | replace('-', '_') }}" \
+      --projectId=operator-tools --env=prod --plain 2>/dev/null || true)"
+    if [[ -z "${GPG_PRIVATE_KEY:-}" ]]; then
+      echo "Key not found in operator-tools — paste the ASCII-armored GPG private key"
+      echo "(the whole BEGIN/END PGP PRIVATE KEY BLOCK), then Ctrl-D. Ctrl-D alone = unencrypted backup:"
+      GPG_PRIVATE_KEY="$(cat)"
+    fi
+    if [[ -n "$(printf '%s' "${GPG_PRIVATE_KEY:-}" | tr -d '[:space:]')" ]]; then
+      GPG_KEY_FILE="/dev/shm/{{ project_name | replace('-', '_') }}-restore-gpg.asc"
+      printf '%s\n' "$GPG_PRIVATE_KEY" | ssh "$host" "umask 077; cat > '$GPG_KEY_FILE'"
+      unset GPG_PRIVATE_KEY
+      # Always shred the staged key when this function exits, success or not.
+      trap "ssh '$host' 'shred -u \"$GPG_KEY_FILE\" 2>/dev/null || rm -f \"$GPG_KEY_FILE\"' 2>/dev/null || true" EXIT
+    fi
     # Reuse the auth helper that cloud-init wrote on the droplet (contains the
     # Machine Identity Client ID + Secret, 0700 root). That `infisical login`
     # call seeds the local CLI session; then `infisical run` does the actual
@@ -189,7 +247,7 @@ SCRIPT
     # by cloud-init; we source it + run `infisical login` here, then exec
     # the droplet's restore.sh with the backup name as positional arg.
     ssh "$host" \
-      "INFISICAL_PROJECT_ID='$INFISICAL_PROJECT_ID' INFISICAL_ENV='$INFISICAL_ENV' PROJECT_NAME='{{ project_name | replace('-', '_') }}' BACKUP_NAME='$BACKUP_NAME' bash -s" <<'EOF'
+      "INFISICAL_PROJECT_ID='$INFISICAL_PROJECT_ID' INFISICAL_ENV='$INFISICAL_ENV' PROJECT_NAME='{{ project_name | replace('-', '_') }}' BACKUP_NAME='$BACKUP_NAME' BACKUP_GPG_KEY_FILE='$GPG_KEY_FILE' bash -s" <<'EOF'
 set -euo pipefail
 source "/opt/$PROJECT_NAME/.infisical-auth.env"
 export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID="$INFISICAL_CLIENT_ID"

@@ -145,10 +145,10 @@ themselves — ToS / AUP / DPA — are maintained outside this repo):
 |---|---|
 | **Tenancy** | Single-tenant: dedicated droplet, dedicated Infisical project + Machine Identity, dedicated LLM key. No shared compute, storage, or credentials between customers. |
 | **Data locality** | Documents, embeddings (LanceDB), and chat history live on the customer's droplet volume; backups in DO Spaces under a per-project prefix. Region chosen at provisioning. |
-| **Encryption** | TLS 1.3 in transit (Caddy/Let's Encrypt); backups offloaded to DO Spaces (SSE); tofu state SSE-C encrypted. Secrets never on disk (Infisical runtime injection). |
-| **LLM processing** | Chat prompts + retrieved document context are sent to the configured LLM provider (OpenRouter) at question time — per-customer key, hard monthly cap. Model choice is pinned by WeOwn (`admin`-held config). |
+| **Encryption** | TLS 1.3 in transit (Caddy/Let's Encrypt). **Backups are client-side encrypted** (OpenPGP/GPG, per-customer ed25519/cv25519 keypair, keyring-free `--recipient-file` encrypt) before upload — objects in Spaces are ciphertext to DigitalOcean; the private key lives only in the operator secret store, never on the droplet. Tofu state SSE-C encrypted. Secrets never on disk (Infisical runtime injection). |
+| **LLM processing** | Chat prompts + retrieved document context are sent to the configured LLM provider (OpenRouter) at question time — per-customer key, hard monthly cap, under the account-level **Zero-Data-Retention guardrail** (routing restricted to ZDR-only endpoints: providers that neither retain nor train on the data; OpenRouter itself retains nothing). Model choice is pinned by WeOwn (`admin`-held config). |
 | **Access** | Customer holds a `manager` account (their data/team). WeOwn holds `admin` + SSH (ops keys via `OPS_AUTHORIZED_KEYS`, auditable + revocable per deploy). |
-| **Backups & retention** | Daily GFS backups (30d/12mo/yearly). On cancellation: final backup retained **60 days**, then deleted from Spaces; droplet, reserved IP, Infisical project, and the customer's OpenRouter key destroyed/revoked at deprovision. |
+| **Backups & retention** | Daily GFS backups (30d/12mo/yearly), client-side encrypted per the Encryption row (`BACKUP_GPG_PUBLIC_KEY` in the site project; private key in operator-tools as `BACKUP_GPG_PRIVATE_KEY_<project>` — generated automatically by `deploy-new-site.sh`). Restores fetch the private key in-process from the operator store, stage it into droplet **tmpfs** for the run (ephemeral GNUPGHOME), and shred it. On cancellation: final backup retained **60 days**, then deleted from Spaces; droplet, reserved IP, Infisical project, and the customer's OpenRouter key destroyed/revoked at deprovision. |
 | **Monitoring** | Host metrics + reverse-proxy access logs → SigNoz (operational telemetry only; document contents are not shipped). DO resource alerts. |
 | **Subprocessors** | DigitalOcean (compute/storage/backups), Infisical (secret management), OpenRouter → underlying model providers (LLM inference), SigNoz (telemetry), Let's Encrypt (TLS issuance). |
 | **Framework mapping** | Controls map to NIST CSF 2.0 (PR.DS, PR.AC, DE.CM), CIS v8 IG1 (3.11, 4.1), ISO 27001-ready (A.5.17, A.8.24) — see [`docs/COMPLIANCE_ROADMAP.md`](COMPLIANCE_ROADMAP.md) for the phased program. |

@@ -295,6 +295,41 @@ else
       success "Pushed EMBEDDING_MODEL_PREF"
     fi
 
+    # BACKUP_GPG_PUBLIC_KEY — per-customer client-side backup encryption (GPG,
+    # ed25519/cv25519, no passphrase — protection at rest is the secret store).
+    # The backup script encrypts tarballs with this PUBLIC key before upload,
+    # so objects in Spaces are ciphertext to DO. The PRIVATE key goes to
+    # operator-tools — it must never live in the site project the droplet's
+    # Machine Identity can read, or the box could decrypt its own offsite
+    # backups and the key-off-box guarantee is gone. Keys are generated in an
+    # ephemeral GNUPGHOME so nothing lands in the operator's real keyring.
+    if command -v gpg &>/dev/null; then
+      log "Generating per-customer backup encryption keypair (GPG)..."
+      GPG_KEYHOME="$(mktemp -d)"
+      chmod 700 "$GPG_KEYHOME"
+      BACKUP_KEY_UID="backups+${PROJECT_NAME}@weown.invalid"
+      cat > "$GPG_KEYHOME/genkey" <<GENKEY
+%no-protection
+Key-Type: eddsa
+Key-Curve: ed25519
+Subkey-Type: ecdh
+Subkey-Curve: cv25519
+Name-Real: ${PROJECT_NAME} backup
+Name-Email: ${BACKUP_KEY_UID}
+Expire-Date: 0
+%commit
+GENKEY
+      GNUPGHOME="$GPG_KEYHOME" gpg --batch --gen-key "$GPG_KEYHOME/genkey" 2>/dev/null
+      GPG_PUB="$(GNUPGHOME="$GPG_KEYHOME" gpg --batch --armor --export "$BACKUP_KEY_UID")"
+      GPG_PRIV_SECRET_NAME="BACKUP_GPG_PRIVATE_KEY_${PROJECT_NAME//[^a-zA-Z0-9]/_}"
+      infisical secrets set BACKUP_GPG_PUBLIC_KEY="$GPG_PUB" --projectId="$PROJECT_ID" --env=prod --silent
+      infisical secrets set "$GPG_PRIV_SECRET_NAME=$(GNUPGHOME="$GPG_KEYHOME" gpg --batch --armor --export-secret-keys "$BACKUP_KEY_UID")" --projectId=operator-tools --env=prod --silent
+      rm -rf "$GPG_KEYHOME"
+      success "Backup encryption keypair provisioned (public key → site project; private key → operator-tools/$GPG_PRIV_SECRET_NAME)"
+    else
+      warn "gpg not found (brew install gnupg) — remote backups will be UNENCRYPTED until BACKUP_GPG_PUBLIC_KEY is set in the site project"
+    fi
+
     # OPENROUTER_API_KEY — mint a per-customer, budget-capped key via the
     # provisioning helper (reads OPENROUTER_PROVISIONING_KEY from the
     # operator-tools Infisical project; cap defaults to \$50/mo, override with

@@ -174,5 +174,11 @@ fi
 echo
 echo "Done — '$KEY_NAME' minted (\$$LIMIT_USD/mo cap) and stored as OPENROUTER_API_KEY."
 echo "No key value touched disk, history, or this terminal."
+echo
+echo "ZDR posture: keys inherit the OpenRouter ACCOUNT-level Zero-Data-Retention"
+echo "guardrail (Settings → Privacy: restrict routing to ZDR-only endpoints). For"
+echo "customer instances handling financial/personal records that guardrail MUST be"
+echo "on — verify it once per account; every per-customer key is then covered."
+echo
 echo "Next: deploy/redeploy the instance so the container picks up the key (see"
 echo "anythingllm-docker/DEPLOYMENT_GUIDE.md §6.5). Verify a real chat completes."