diff --git a/CHANGELOG.md b/CHANGELOG.md index 021f35d..bc2a1bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,7 @@ Changes in this section will be promoted to a dated release entry on merge to `m ### Security +- **anythingllm-docker — client-side encrypted backups (GPG) + ZDR LLM routing posture (2026-07-15)** — closes two compliance gaps for customer instances holding financial/personal records (GLBA-service-provider posture). **Backups:** `backup.sh` now encrypts the tarball with OpenPGP/GPG (per-customer ed25519/cv25519 public key, `BACKUP_GPG_PUBLIC_KEY` injected via `infisical run`; keyring-free `--recipient-file` + ephemeral `GNUPGHOME`) before the Spaces upload — the object at rest is ciphertext to DigitalOcean; the private key lives ONLY in the operator store (`BACKUP_GPG_PRIVATE_KEY_`, auto-generated by `deploy-new-site.sh` in an ephemeral keyring). Fail-loud when encryption is configured but `gpg` is missing; loud warning on unencrypted uploads. `restore.sh` fetches `.tar.gz.gpg`, pulls the private key **in-process** from operator-tools via the logged-in infisical CLI (paste fallback), stages it into droplet **tmpfs** (0600, ephemeral GNUPGHOME, shredded on exit via trap), decrypts, and proceeds; legacy plaintext objects still restore. GPG chosen over `age` for auditor familiarity (owner call); `gnupg` was already in cloud-init packages. **LLM routing:** per-customer OpenRouter keys operate under the account-level **Zero-Data-Retention guardrail** (ZDR-only endpoints — no provider retention or training; OpenRouter itself retains nothing); `provision-openrouter-key.sh` prints the posture requirement, and the compliance facts table in [`docs/CUSTOMER_INSTANCE_PROVISIONING.md`](docs/CUSTOMER_INSTANCE_PROVISIONING.md) documents both. **Compliance**: FTC Safeguards (GLBA) encryption-at-rest expectation; NIST CSF 2.0 PR.DS-1/PR.DS-2; CIS v8 3.11; supports IRC §7216 posture for tax data. **Verification**: 6-check GPG round-trip (encrypt → ciphertext-opaque → decrypt → byte-identical → wrong-key-rejected) passes with the exact script logic; rendered-shell `bash -n` clean. - **M2 fleet-wide — Infisical Machine-Identity secrets & DO Spaces creds moved off command-line argv (2026-07-14)** — closes the AGENTS.md S2 finding that `infisical login --client-secret="$INFISICAL_CLIENT_SECRET"` and `tofu init -backend-config="secret_key=…"` exposed secrets on process argv (readable via `ps` / `/proc//cmdline` on the droplet). **Infisical login** now uses env-var auth — `export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID/_SECRET` before a flagless `infisical login --method=universal-auth` (the CLI reads them natively), matching the pattern openclaw's entrypoint already used. **Tofu/OpenTofu** now exports `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` (the S3 backend reads them from env) and drops the two `-backend-config` access/secret lines; `sse_customer_key` stays on `-backend-config` (no env equivalent). Applied across every `*-docker` template and rendered site (anythingllm, keycloak, owncloud, wordpress, searxng, signoz, sandbox, openclaw, devbox) plus `otel-agent/` and the `scripts/*otel*` helpers, and the doc examples that showed the old form. Only `--plain`/`--silent` and non-secret flags remain on argv. **Compliance**: NIST CSF 2.0 PR.DS-1/PR.AC-4; CIS v8 3.11. **Verification**: `bash -n` + `shellcheck -S error` clean on all changed scripts; rendered ansible YAML parses; 0 remaining `--client-secret=`/`-backend-config="secret_key="` in tracked code. **Note**: needs a live deploy test (copier render → ansible/tofu path) before promotion — the human pre-merge gate. ### Added diff --git a/anythingllm-docker/template/scripts/backup.sh.jinja b/anythingllm-docker/template/scripts/backup.sh.jinja index 5b4b117..80cf3fe 100644 --- a/anythingllm-docker/template/scripts/backup.sh.jinja +++ b/anythingllm-docker/template/scripts/backup.sh.jinja @@ -97,6 +97,41 @@ rm -rf "$WORK_DIR" FINAL_SIZE=$(ls -lh "$BACKUP_DIR/${BACKUP_NAME}.tar.gz" | awk '{print $5}') echo "==> Local backup complete: $BACKUP_DIR/${BACKUP_NAME}.tar.gz ($FINAL_SIZE)" +# --- Client-side encryption (compliance: financial/personal records) --- +# When BACKUP_GPG_PUBLIC_KEY (an ASCII-armored OpenPGP public key, injected via +# `infisical run`) is set, the tarball is encrypted BEFORE upload, so the +# object in Spaces is ciphertext to DigitalOcean and to anyone without the +# private key. The private key never touches this droplet — it lives in the +# operator secret store (pushed there by deploy-new-site.sh at provisioning). +# Encryption is keyring-free (--recipient-file + ephemeral GNUPGHOME), so no +# state accumulates on the box. The LOCAL plaintext tarball is kept for fast +# local restores: it sits on the same disk as the live volumes, so it adds no +# new exposure class. +UPLOAD_FILE="$BACKUP_DIR/${BACKUP_NAME}.tar.gz" +if [[ -n "${BACKUP_GPG_PUBLIC_KEY:-}" ]]; then + if ! command -v gpg >/dev/null 2>&1; then + echo "ERROR: BACKUP_GPG_PUBLIC_KEY is set but 'gpg' is not installed — refusing to" >&2 + echo " upload an UNENCRYPTED backup when encryption was configured." >&2 + echo " Fix: apt-get install -y gnupg (cloud-init installs it on new droplets)." >&2 + exit 1 + fi + echo "==> Encrypting backup (GPG, client-side)..." + GPG_TMP="$(mktemp -d)" + chmod 700 "$GPG_TMP" + printf '%s\n' "$BACKUP_GPG_PUBLIC_KEY" > "$GPG_TMP/pubkey.asc" + GNUPGHOME="$GPG_TMP" gpg --batch --yes --trust-model always \ + --recipient-file "$GPG_TMP/pubkey.asc" \ + -o "$BACKUP_DIR/${BACKUP_NAME}.tar.gz.gpg" \ + --encrypt "$BACKUP_DIR/${BACKUP_NAME}.tar.gz" + rm -rf "$GPG_TMP" + UPLOAD_FILE="$BACKUP_DIR/${BACKUP_NAME}.tar.gz.gpg" +else + echo "WARNING: BACKUP_GPG_PUBLIC_KEY not set — the remote backup will be UNENCRYPTED" >&2 + echo " (DO's server-side encryption only). For customer instances holding" >&2 + echo " financial/personal records this is a compliance gap: set" >&2 + echo " BACKUP_GPG_PUBLIC_KEY in the site's Infisical project." >&2 +fi + # --- Remote upload (DO Spaces) --- if [[ "$REMOTE_STORAGE" == "do-spaces" ]]; then if [[ -z "${SPACES_ACCESS_KEY:-}" ]] || [[ -z "${SPACES_SECRET_KEY:-}" ]]; then @@ -105,11 +140,13 @@ if [[ "$REMOTE_STORAGE" == "do-spaces" ]]; then echo "==> Uploading to DO Spaces (s3://${SPACES_BUCKET}/{{ project_name }}/)..." AWS_ACCESS_KEY_ID="$SPACES_ACCESS_KEY" \ AWS_SECRET_ACCESS_KEY="$SPACES_SECRET_KEY" \ - aws s3 cp "$BACKUP_DIR/${BACKUP_NAME}.tar.gz" \ + aws s3 cp "$UPLOAD_FILE" \ "s3://${SPACES_BUCKET}/{{ project_name }}/" \ --endpoint-url "https://${SPACES_REGION}.digitaloceanspaces.com" \ --quiet - echo "==> Remote backup uploaded successfully" + echo "==> Remote backup uploaded successfully ($(basename "$UPLOAD_FILE"))" + # The encrypted copy exists in Spaces now; don't leave a second local one. + [[ "$UPLOAD_FILE" == *.gpg ]] && rm -f "$UPLOAD_FILE" fi fi diff --git a/anythingllm-docker/template/scripts/restore.sh.jinja b/anythingllm-docker/template/scripts/restore.sh.jinja index eff6c1d..da00d10 100644 --- a/anythingllm-docker/template/scripts/restore.sh.jinja +++ b/anythingllm-docker/template/scripts/restore.sh.jinja @@ -96,6 +96,9 @@ SPACES_BUCKET="{{ backup_do_spaces_bucket }}" SPACES_REGION="{{ backup_do_spaces_region }}" # --- Fetch backup from DO Spaces if not present locally --- +# Remote objects may be GPG-ENCRYPTED (.tar.gz.gpg — the compliance default when +# BACKUP_GPG_PUBLIC_KEY is set at backup time). Try plaintext first (legacy), +# then the encrypted object. if [[ ! -f "\$BACKUP_FILE" && "$REMOTE_STORAGE" == "do-spaces" && -n "\${SPACES_ACCESS_KEY:-}" && -n "\${SPACES_SECRET_KEY:-}" ]]; then echo "==> Backup not found locally, fetching from DO Spaces..." mkdir -p "\$BACKUP_DIR" @@ -104,8 +107,39 @@ if [[ ! -f "\$BACKUP_FILE" && "$REMOTE_STORAGE" == "do-spaces" && -n "\${SPACES_ aws s3 cp "s3://${SPACES_BUCKET}/{{ project_name }}/${BACKUP_NAME}.tar.gz" \ "\$BACKUP_FILE" \ --endpoint-url "https://${SPACES_REGION}.digitaloceanspaces.com" \ - --quiet - echo "==> Downloaded from DO Spaces" + --quiet 2>/dev/null || true + if [[ ! -f "\$BACKUP_FILE" ]]; then + AWS_ACCESS_KEY_ID="\$SPACES_ACCESS_KEY" \ + AWS_SECRET_ACCESS_KEY="\$SPACES_SECRET_KEY" \ + aws s3 cp "s3://${SPACES_BUCKET}/{{ project_name }}/${BACKUP_NAME}.tar.gz.gpg" \ + "\${BACKUP_FILE}.gpg" \ + --endpoint-url "https://${SPACES_REGION}.digitaloceanspaces.com" \ + --quiet 2>/dev/null || true + fi + echo "==> Fetch from DO Spaces complete" +fi + +# --- Decrypt if the backup is GPG-encrypted --- +# The GPG PRIVATE key never lives on this droplet: the operator wrapper stages +# it into tmpfs (BACKUP_GPG_KEY_FILE under /dev/shm, 0600) for the duration of +# the restore and shreds it after. Import + decrypt happen in an ephemeral +# GNUPGHOME in tmpfs — no keyring state survives. +if [[ ! -f "\$BACKUP_FILE" && -f "\${BACKUP_FILE}.gpg" ]]; then + if [[ -z "\${BACKUP_GPG_KEY_FILE:-}" || ! -f "\${BACKUP_GPG_KEY_FILE:-/nonexistent}" ]]; then + echo "ERROR: backup is ENCRYPTED (\${BACKUP_NAME}.tar.gz.gpg) but no GPG private key was staged." >&2 + echo " Re-run ./scripts/restore.sh from the operator machine (it fetches the key" >&2 + echo " from the operator secret store: BACKUP_GPG_PRIVATE_KEY_${PROJECT_NAME}), or" >&2 + echo " stage the armored key to a tmpfs file and set BACKUP_GPG_KEY_FILE." >&2 + exit 1 + fi + command -v gpg >/dev/null 2>&1 || { echo "ERROR: 'gpg' not installed on this host (apt-get install -y gnupg)." >&2; exit 1; } + echo "==> Decrypting backup (GPG, ephemeral keyring in tmpfs)..." + GPG_TMP="\$(mktemp -d /dev/shm/restore-gnupg.XXXXXX)" + chmod 700 "\$GPG_TMP" + GNUPGHOME="\$GPG_TMP" gpg --batch --import "\$BACKUP_GPG_KEY_FILE" 2>/dev/null + GNUPGHOME="\$GPG_TMP" gpg --batch --yes -o "\$BACKUP_FILE" --decrypt "\${BACKUP_FILE}.gpg" + rm -rf "\$GPG_TMP" + rm -f "\${BACKUP_FILE}.gpg" fi if [[ ! -f "\$BACKUP_FILE" ]]; then @@ -178,6 +212,30 @@ SCRIPT if [[ -n "$host" ]]; then echo "==> Running restore on remote: ${host}" + # --- Stage the GPG private key (encrypted backups) --- + # If the backup was taken with BACKUP_GPG_PUBLIC_KEY set, the object in + # Spaces is .tar.gz.gpg and the droplet needs the PRIVATE key to decrypt. + # The key lives in the operator secret store as + # BACKUP_GPG_PRIVATE_KEY_ (operator-tools project, pushed at + # provisioning). It is fetched IN-PROCESS via the logged-in infisical CLI + # (never printed), staged into tmpfs on the droplet (never its disk), and + # shredded after the restore. Fallback: paste the armored key (Ctrl-D to + # end; Ctrl-D immediately for legacy plaintext backups). + GPG_KEY_FILE="" + GPG_PRIVATE_KEY="$(infisical secrets get "BACKUP_GPG_PRIVATE_KEY_{{ project_name | replace('-', '_') }}" \ + --projectId=operator-tools --env=prod --plain 2>/dev/null || true)" + if [[ -z "${GPG_PRIVATE_KEY:-}" ]]; then + echo "Key not found in operator-tools — paste the ASCII-armored GPG private key" + echo "(the whole BEGIN/END PGP PRIVATE KEY BLOCK), then Ctrl-D. Ctrl-D alone = unencrypted backup:" + GPG_PRIVATE_KEY="$(cat)" + fi + if [[ -n "$(printf '%s' "${GPG_PRIVATE_KEY:-}" | tr -d '[:space:]')" ]]; then + GPG_KEY_FILE="/dev/shm/{{ project_name | replace('-', '_') }}-restore-gpg.asc" + printf '%s\n' "$GPG_PRIVATE_KEY" | ssh "$host" "umask 077; cat > '$GPG_KEY_FILE'" + unset GPG_PRIVATE_KEY + # Always shred the staged key when this function exits, success or not. + trap "ssh '$host' 'shred -u \"$GPG_KEY_FILE\" 2>/dev/null || rm -f \"$GPG_KEY_FILE\"' 2>/dev/null || true" EXIT + fi # Reuse the auth helper that cloud-init wrote on the droplet (contains the # Machine Identity Client ID + Secret, 0700 root). That `infisical login` # call seeds the local CLI session; then `infisical run` does the actual @@ -189,7 +247,7 @@ SCRIPT # by cloud-init; we source it + run `infisical login` here, then exec # the droplet's restore.sh with the backup name as positional arg. ssh "$host" \ - "INFISICAL_PROJECT_ID='$INFISICAL_PROJECT_ID' INFISICAL_ENV='$INFISICAL_ENV' PROJECT_NAME='{{ project_name | replace('-', '_') }}' BACKUP_NAME='$BACKUP_NAME' bash -s" <<'EOF' + "INFISICAL_PROJECT_ID='$INFISICAL_PROJECT_ID' INFISICAL_ENV='$INFISICAL_ENV' PROJECT_NAME='{{ project_name | replace('-', '_') }}' BACKUP_NAME='$BACKUP_NAME' BACKUP_GPG_KEY_FILE='$GPG_KEY_FILE' bash -s" <<'EOF' set -euo pipefail source "/opt/$PROJECT_NAME/.infisical-auth.env" export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID="$INFISICAL_CLIENT_ID" diff --git a/docs/CUSTOMER_INSTANCE_PROVISIONING.md b/docs/CUSTOMER_INSTANCE_PROVISIONING.md index 6f89094..a88de6e 100644 --- a/docs/CUSTOMER_INSTANCE_PROVISIONING.md +++ b/docs/CUSTOMER_INSTANCE_PROVISIONING.md @@ -145,10 +145,10 @@ themselves — ToS / AUP / DPA — are maintained outside this repo): |---|---| | **Tenancy** | Single-tenant: dedicated droplet, dedicated Infisical project + Machine Identity, dedicated LLM key. No shared compute, storage, or credentials between customers. | | **Data locality** | Documents, embeddings (LanceDB), and chat history live on the customer's droplet volume; backups in DO Spaces under a per-project prefix. Region chosen at provisioning. | -| **Encryption** | TLS 1.3 in transit (Caddy/Let's Encrypt); backups offloaded to DO Spaces (SSE); tofu state SSE-C encrypted. Secrets never on disk (Infisical runtime injection). | -| **LLM processing** | Chat prompts + retrieved document context are sent to the configured LLM provider (OpenRouter) at question time — per-customer key, hard monthly cap. Model choice is pinned by WeOwn (`admin`-held config). | +| **Encryption** | TLS 1.3 in transit (Caddy/Let's Encrypt). **Backups are client-side encrypted** (OpenPGP/GPG, per-customer ed25519/cv25519 keypair, keyring-free `--recipient-file` encrypt) before upload — objects in Spaces are ciphertext to DigitalOcean; the private key lives only in the operator secret store, never on the droplet. Tofu state SSE-C encrypted. Secrets never on disk (Infisical runtime injection). | +| **LLM processing** | Chat prompts + retrieved document context are sent to the configured LLM provider (OpenRouter) at question time — per-customer key, hard monthly cap, under the account-level **Zero-Data-Retention guardrail** (routing restricted to ZDR-only endpoints: providers that neither retain nor train on the data; OpenRouter itself retains nothing). Model choice is pinned by WeOwn (`admin`-held config). | | **Access** | Customer holds a `manager` account (their data/team). WeOwn holds `admin` + SSH (ops keys via `OPS_AUTHORIZED_KEYS`, auditable + revocable per deploy). | -| **Backups & retention** | Daily GFS backups (30d/12mo/yearly). On cancellation: final backup retained **60 days**, then deleted from Spaces; droplet, reserved IP, Infisical project, and the customer's OpenRouter key destroyed/revoked at deprovision. | +| **Backups & retention** | Daily GFS backups (30d/12mo/yearly), client-side encrypted per the Encryption row (`BACKUP_GPG_PUBLIC_KEY` in the site project; private key in operator-tools as `BACKUP_GPG_PRIVATE_KEY_` — generated automatically by `deploy-new-site.sh`). Restores fetch the private key in-process from the operator store, stage it into droplet **tmpfs** for the run (ephemeral GNUPGHOME), and shred it. On cancellation: final backup retained **60 days**, then deleted from Spaces; droplet, reserved IP, Infisical project, and the customer's OpenRouter key destroyed/revoked at deprovision. | | **Monitoring** | Host metrics + reverse-proxy access logs → SigNoz (operational telemetry only; document contents are not shipped). DO resource alerts. | | **Subprocessors** | DigitalOcean (compute/storage/backups), Infisical (secret management), OpenRouter → underlying model providers (LLM inference), SigNoz (telemetry), Let's Encrypt (TLS issuance). | | **Framework mapping** | Controls map to NIST CSF 2.0 (PR.DS, PR.AC, DE.CM), CIS v8 IG1 (3.11, 4.1), ISO 27001-ready (A.5.17, A.8.24) — see [`docs/COMPLIANCE_ROADMAP.md`](COMPLIANCE_ROADMAP.md) for the phased program. | diff --git a/scripts/deploy-new-site.sh b/scripts/deploy-new-site.sh index f136b18..f83dd15 100755 --- a/scripts/deploy-new-site.sh +++ b/scripts/deploy-new-site.sh @@ -295,6 +295,41 @@ else success "Pushed EMBEDDING_MODEL_PREF" fi + # BACKUP_GPG_PUBLIC_KEY — per-customer client-side backup encryption (GPG, + # ed25519/cv25519, no passphrase — protection at rest is the secret store). + # The backup script encrypts tarballs with this PUBLIC key before upload, + # so objects in Spaces are ciphertext to DO. The PRIVATE key goes to + # operator-tools — it must never live in the site project the droplet's + # Machine Identity can read, or the box could decrypt its own offsite + # backups and the key-off-box guarantee is gone. Keys are generated in an + # ephemeral GNUPGHOME so nothing lands in the operator's real keyring. + if command -v gpg &>/dev/null; then + log "Generating per-customer backup encryption keypair (GPG)..." + GPG_KEYHOME="$(mktemp -d)" + chmod 700 "$GPG_KEYHOME" + BACKUP_KEY_UID="backups+${PROJECT_NAME}@weown.invalid" + cat > "$GPG_KEYHOME/genkey" </dev/null + GPG_PUB="$(GNUPGHOME="$GPG_KEYHOME" gpg --batch --armor --export "$BACKUP_KEY_UID")" + GPG_PRIV_SECRET_NAME="BACKUP_GPG_PRIVATE_KEY_${PROJECT_NAME//[^a-zA-Z0-9]/_}" + infisical secrets set BACKUP_GPG_PUBLIC_KEY="$GPG_PUB" --projectId="$PROJECT_ID" --env=prod --silent + infisical secrets set "$GPG_PRIV_SECRET_NAME=$(GNUPGHOME="$GPG_KEYHOME" gpg --batch --armor --export-secret-keys "$BACKUP_KEY_UID")" --projectId=operator-tools --env=prod --silent + rm -rf "$GPG_KEYHOME" + success "Backup encryption keypair provisioned (public key → site project; private key → operator-tools/$GPG_PRIV_SECRET_NAME)" + else + warn "gpg not found (brew install gnupg) — remote backups will be UNENCRYPTED until BACKUP_GPG_PUBLIC_KEY is set in the site project" + fi + # OPENROUTER_API_KEY — mint a per-customer, budget-capped key via the # provisioning helper (reads OPENROUTER_PROVISIONING_KEY from the # operator-tools Infisical project; cap defaults to \$50/mo, override with diff --git a/scripts/provision-openrouter-key.sh b/scripts/provision-openrouter-key.sh index a6fe961..c182b6a 100755 --- a/scripts/provision-openrouter-key.sh +++ b/scripts/provision-openrouter-key.sh @@ -174,5 +174,11 @@ fi echo echo "Done — '$KEY_NAME' minted (\$$LIMIT_USD/mo cap) and stored as OPENROUTER_API_KEY." echo "No key value touched disk, history, or this terminal." +echo +echo "ZDR posture: keys inherit the OpenRouter ACCOUNT-level Zero-Data-Retention" +echo "guardrail (Settings → Privacy: restrict routing to ZDR-only endpoints). For" +echo "customer instances handling financial/personal records that guardrail MUST be" +echo "on — verify it once per account; every per-customer key is then covered." +echo echo "Next: deploy/redeploy the instance so the container picks up the key (see" echo "anythingllm-docker/DEPLOYMENT_GUIDE.md §6.5). Verify a real chat completes."