diff --git a/CHANGELOG.md b/CHANGELOG.md index bc2a1bf5..c928339a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,7 @@ Changes in this section will be promoted to a dated release entry on merge to `m ### Security +- **anythingllm-docker — CIS host hardening play + encrypted-at-rest data volume (2026-07-15)** — Phase A of porting the SOP-INFRA-012 baseline to customer instances. **Hardening:** new `template/ansible/harden.yml` applies DevSec `os_hardening` + `ssh_hardening` (≈ CIS Ubuntu L1 — kernel/sysctl, auth policy, SSH key-only + strong KEX/ciphers) with the Docker-safe `ip_forward=1` task-var override (defaults-level overrides silently lose to inner-role defaults — the 2026-07-03 no-egress landmine), then measures with **Lynis** and verifies Docker survived; `ansible/requirements.yml` pins the `devsec.hardening` collection. Phase A deliberately keeps root key-only SSH (all fleet tooling SSHes as `root@`); Phase B (ops-user model, root-off, per SOP-INFRA-012) is gated on fleet-tooling migration and tracked for pre-first-customer. **Data at rest:** new `digitalocean_volume` (default 50 GB, `prevent_destroy`, `data_volume_size_gb=0` to disable) — DO block storage is platform-encrypted at rest, droplet local disk is not; cloud-init `setup-data-volume.sh` waits out the attach race, formats/mounts, points the **Docker data-root** (chats/vectors/uploads) and the **backups dir** at it before Docker installs. **Compliance**: FTC Safeguards/GLBA at-rest expectation; NIST CSF 2.0 PR.DS-1, PR.PS; CIS v8 4.1/3.11. **Verification**: rendered `harden.yml` passes `ansible-playbook --syntax-check`; `setup-data-volume.sh` `bash -n` clean; rendered `main.tf` parses (`tofu fmt`). - **anythingllm-docker — client-side encrypted backups (GPG) + ZDR LLM routing posture (2026-07-15)** — closes two compliance gaps for customer instances holding financial/personal records (GLBA-service-provider posture). **Backups:** `backup.sh` now encrypts the tarball with OpenPGP/GPG (per-customer ed25519/cv25519 public key, `BACKUP_GPG_PUBLIC_KEY` injected via `infisical run`; keyring-free `--recipient-file` + ephemeral `GNUPGHOME`) before the Spaces upload — the object at rest is ciphertext to DigitalOcean; the private key lives ONLY in the operator store (`BACKUP_GPG_PRIVATE_KEY_`, auto-generated by `deploy-new-site.sh` in an ephemeral keyring). Fail-loud when encryption is configured but `gpg` is missing; loud warning on unencrypted uploads. `restore.sh` fetches `.tar.gz.gpg`, pulls the private key **in-process** from operator-tools via the logged-in infisical CLI (paste fallback), stages it into droplet **tmpfs** (0600, ephemeral GNUPGHOME, shredded on exit via trap), decrypts, and proceeds; legacy plaintext objects still restore. GPG chosen over `age` for auditor familiarity (owner call); `gnupg` was already in cloud-init packages. **LLM routing:** per-customer OpenRouter keys operate under the account-level **Zero-Data-Retention guardrail** (ZDR-only endpoints — no provider retention or training; OpenRouter itself retains nothing); `provision-openrouter-key.sh` prints the posture requirement, and the compliance facts table in [`docs/CUSTOMER_INSTANCE_PROVISIONING.md`](docs/CUSTOMER_INSTANCE_PROVISIONING.md) documents both. **Compliance**: FTC Safeguards (GLBA) encryption-at-rest expectation; NIST CSF 2.0 PR.DS-1/PR.DS-2; CIS v8 3.11; supports IRC §7216 posture for tax data. **Verification**: 6-check GPG round-trip (encrypt → ciphertext-opaque → decrypt → byte-identical → wrong-key-rejected) passes with the exact script logic; rendered-shell `bash -n` clean. - **M2 fleet-wide — Infisical Machine-Identity secrets & DO Spaces creds moved off command-line argv (2026-07-14)** — closes the AGENTS.md S2 finding that `infisical login --client-secret="$INFISICAL_CLIENT_SECRET"` and `tofu init -backend-config="secret_key=…"` exposed secrets on process argv (readable via `ps` / `/proc//cmdline` on the droplet). **Infisical login** now uses env-var auth — `export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID/_SECRET` before a flagless `infisical login --method=universal-auth` (the CLI reads them natively), matching the pattern openclaw's entrypoint already used. **Tofu/OpenTofu** now exports `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` (the S3 backend reads them from env) and drops the two `-backend-config` access/secret lines; `sse_customer_key` stays on `-backend-config` (no env equivalent). Applied across every `*-docker` template and rendered site (anythingllm, keycloak, owncloud, wordpress, searxng, signoz, sandbox, openclaw, devbox) plus `otel-agent/` and the `scripts/*otel*` helpers, and the doc examples that showed the old form. Only `--plain`/`--silent` and non-secret flags remain on argv. **Compliance**: NIST CSF 2.0 PR.DS-1/PR.AC-4; CIS v8 3.11. **Verification**: `bash -n` + `shellcheck -S error` clean on all changed scripts; rendered ansible YAML parses; 0 remaining `--client-secret=`/`-backend-config="secret_key="` in tracked code. **Note**: needs a live deploy test (copier render → ansible/tofu path) before promotion — the human pre-merge gate. diff --git a/anythingllm-docker/copier.yaml b/anythingllm-docker/copier.yaml index 65bfc986..da72b570 100644 --- a/anythingllm-docker/copier.yaml +++ b/anythingllm-docker/copier.yaml @@ -62,6 +62,11 @@ anythingllm_memory_limit: default: "2G" validator: "{% if not anythingllm_memory_limit | regex_search('^[0-9]+[MG]$') %}Must be a Docker memory string like 2G or 512M{% endif %}" +data_volume_size_gb: + type: int + help: "Encrypted-at-rest DO block-storage data volume (GB) for Docker data-root + backups — DO encrypts block storage at rest, the droplet's local disk is not. 0 disables (dev only; customer instances holding financial/personal data must keep this on)." + default: 50 + anythingllm_memory_reservation: type: str help: "AnythingLLM container memory reservation (soft guarantee, below the limit)" diff --git a/anythingllm-docker/template/ansible/harden.yml.jinja b/anythingllm-docker/template/ansible/harden.yml.jinja new file mode 100644 index 00000000..8b451c5d --- /dev/null +++ b/anythingllm-docker/template/ansible/harden.yml.jinja @@ -0,0 +1,76 @@ +# {{ project_name }} — host hardening (CIS-aligned, Phase A) +# +# Applies the DevSec hardening baseline (≈ CIS Ubuntu L1) to the droplet: +# kernel/sysctl, auth policy, filesystem perms, SSH crypto + policy — then +# measures the result with Lynis. Idempotent; run any time after bootstrap: +# +# ansible-galaxy collection install -r ansible/requirements.yml # once +# ansible-playbook ansible/harden.yml -i 'root@,' +# +# PHASE A vs PHASE B (deliberate scope — do not "fix" this into Phase B): +# Phase A (this play): CIS L1 OS + SSH hardening with root login KEPT +# (key-only). Every fleet tool (deploy.sh, backup.sh, manage-droplets.sh) +# SSHes as root@; removing root here would break the fleet mid-flight. +# Phase B (tracked, pre-first-customer): ops-user model (root SSH off, +# password-sudo 2FA), mesh/CIDR-locked port 22 — per SOP-INFRA-012. Needs +# the fleet tooling migrated off root@ first. +# +# Docker-safe exceptions (the CIS-L2-vs-container-runtime conflicts): +# - net.ipv4.ip_forward stays 1 — Docker bridge networking requires it. A +# defaults-level override silently LOSES to the inner role's defaults; +# it must be a task-var on the include (hit live 2026-07-03: containers +# got DNS but no TCP egress when this landed at 0). +--- +- name: Harden {{ project_name }} host (CIS L1 via devsec — Phase A) + hosts: all + become: true + vars: + run_lynis: true + + tasks: + - name: "OS hardening (devsec.hardening.os_hardening ≈ CIS Ubuntu L1)" + ansible.builtin.include_role: + name: devsec.hardening.os_hardening + vars: + # MUST be task-vars here (see header) — Docker bridge needs ip_forward. + sysctl_overwrite: + net.ipv4.ip_forward: 1 + + - name: "SSH hardening (devsec.hardening.ssh_hardening — key-only, strong KEX/ciphers)" + ansible.builtin.include_role: + name: devsec.hardening.ssh_hardening + vars: + # Phase A: root stays (fleet tooling), but key-only and CIS crypto. + ssh_permit_root_login: "without-password" + ssh_password_authentication: false + ssh_challenge_response_authentication: false + + - name: "Measure: install + run Lynis (hardening index; non-blocking)" + when: run_lynis | bool + block: + - name: "Install Lynis" + ansible.builtin.apt: + name: lynis + state: present + update_cache: true + + - name: "Run Lynis audit (quick)" + ansible.builtin.command: lynis audit system --quick --no-colors + register: lynis_out + changed_when: false + failed_when: false + + - name: "Lynis summary (record this index in the site CHANGELOG)" + ansible.builtin.debug: + msg: "{{ '{{' }} lynis_out.stdout_lines | select('search', 'Hardening index|Warnings|Suggestions|Tests performed') | list {{ '}}' }}" + + - name: "Verify Docker survived hardening (containers must still run)" + # brace-free format string: Go-template braces here would be re-templated + # by ansible's own Jinja pass and error out. + ansible.builtin.command: docker info --format json + register: docker_ok + changed_when: false + + - name: "Docker status" + ansible.builtin.debug: + msg: "Docker OK (server {{ '{{' }} (docker_ok.stdout | from_json).ServerVersion | default('unknown') {{ '}}' }}) — hardening did not break the runtime" diff --git a/anythingllm-docker/template/ansible/requirements.yml b/anythingllm-docker/template/ansible/requirements.yml new file mode 100644 index 00000000..34c64c4e --- /dev/null +++ b/anythingllm-docker/template/ansible/requirements.yml @@ -0,0 +1,8 @@ +# Ansible Galaxy requirements for the optional hardening play (harden.yml). +# Install once per operator workstation: +# ansible-galaxy collection install -r ansible/requirements.yml +collections: + # DevSec Hardening Framework — CIS/STIG-aligned os_hardening + ssh_hardening. + # https://github.com/dev-sec/ansible-collection-hardening + - name: devsec.hardening + version: ">=8.8.0" diff --git a/anythingllm-docker/template/terraform/main.tf.jinja b/anythingllm-docker/template/terraform/main.tf.jinja index 54918b66..348f7cf4 100644 --- a/anythingllm-docker/template/terraform/main.tf.jinja +++ b/anythingllm-docker/template/terraform/main.tf.jinja @@ -63,6 +63,33 @@ resource "digitalocean_droplet" "anythingllm" { } } +# Encrypted-at-rest data volume (DO encrypts block storage at rest; the +# droplet's LOCAL disk carries no such guarantee). cloud-init formats/mounts it +# and points the Docker data-root + the backups dir at it, so customer data — +# volumes (chats, vectors, uploads) and local backup tarballs — lives on +# platform-encrypted storage. Set data_volume_size_gb = 0 to disable (dev only). +resource "digitalocean_volume" "data" { + count = var.data_volume_size_gb > 0 ? 1 : 0 + region = var.region + name = "{{ project_name }}-data" + size = var.data_volume_size_gb + # Pre-formatted by DO so first boot never races mkfs; cloud-init still + # carries a mkfs fallback for volumes created outside tofu. + initial_filesystem_type = "ext4" + description = "{{ project_name }} encrypted-at-rest data (docker data-root + backups)" + + lifecycle { + # Customer data lives here — same protection as the droplet itself. + prevent_destroy = true + } +} + +resource "digitalocean_volume_attachment" "data" { + count = var.data_volume_size_gb > 0 ? 1 : 0 + droplet_id = digitalocean_droplet.anythingllm.id + volume_id = digitalocean_volume.data[0].id +} + resource "digitalocean_reserved_ip" "anythingllm" { region = var.region } diff --git a/anythingllm-docker/template/terraform/templates/cloud-init.yaml.jinja b/anythingllm-docker/template/terraform/templates/cloud-init.yaml.jinja index 19251bb8..8d3d25c4 100644 --- a/anythingllm-docker/template/terraform/templates/cloud-init.yaml.jinja +++ b/anythingllm-docker/template/terraform/templates/cloud-init.yaml.jinja @@ -92,6 +92,45 @@ write_files: "storage-driver": "overlay2" } + # Encrypted-at-rest data volume setup (runs BEFORE Docker is installed). + # DO block-storage volumes are encrypted at rest; the droplet's local disk is + # not. When the tofu-attached volume is present, mount it and point the + # Docker data-root (app volumes: chats/vectors/uploads) + the backups dir at + # it. Graceful no-op when no volume is attached (data_volume_size_gb = 0). + # The attachment can land seconds AFTER first boot starts, so wait for the + # device before concluding it's absent. + - path: /tmp/setup-data-volume.sh + permissions: "0755" + content: | + #!/bin/bash + set -euo pipefail + DEV="/dev/disk/by-id/scsi-0DO_Volume_{{ project_name }}-data" + MNT="/mnt/${project_name}_data" + for i in $(seq 1 24); do + [ -e "$DEV" ] && break + sleep 5 + done + if [ ! -e "$DEV" ]; then + echo "no data volume attached after 120s - using droplet local disk (dev mode)" + exit 0 + fi + # DO pre-formats tofu-created volumes (initial_filesystem_type); mkfs is + # the fallback for volumes created outside tofu. + blkid "$DEV" >/dev/null 2>&1 || mkfs.ext4 -q "$DEV" + mkdir -p "$MNT" + grep -q "scsi-0DO_Volume_{{ project_name }}-data" /etc/fstab || \ + echo "$DEV $MNT ext4 defaults,noatime,nofail,discard 0 2" >> /etc/fstab + mountpoint -q "$MNT" || mount "$MNT" + mkdir -p "$MNT/docker" "$MNT/backups" + # Point the Docker data-root at the encrypted volume (Docker not yet + # installed/started, so this is race-free). + jq --arg dr "$MNT/docker" '. + {"data-root": $dr}' /etc/docker/daemon.json > /tmp/daemon.json.new + mv /tmp/daemon.json.new /etc/docker/daemon.json + # Backups dir lives on the encrypted volume; the app path is a symlink. + mkdir -p /opt/${project_name} + [ -e /opt/${project_name}/backups ] || ln -s "$MNT/backups" /opt/${project_name}/backups + echo "data volume mounted at $MNT (docker data-root + backups relocated)" + # Infisical Machine Identity auth file (bootstrap secret v1). Rotation runs # below - once v2 is minted, this file is overwritten with v2 and v1 is # revoked in Infisical. The v1 value embedded in terraform state / DO @@ -258,6 +297,9 @@ write_files: log "===== Rotation complete =====" runcmd: + # Data volume FIRST: docker's data-root must point at the encrypted volume + # before docker is installed/started. + - [bash, /tmp/setup-data-volume.sh] - [bash, /tmp/install-docker.sh] - [bash, /tmp/install-infisical.sh] # AWS CLI v2 (DO Spaces is S3-compatible) via the official installer; apt awscli diff --git a/anythingllm-docker/template/terraform/terraform.tfvars.example.jinja b/anythingllm-docker/template/terraform/terraform.tfvars.example.jinja index a15b49e1..cc9fb6fd 100644 --- a/anythingllm-docker/template/terraform/terraform.tfvars.example.jinja +++ b/anythingllm-docker/template/terraform/terraform.tfvars.example.jinja @@ -139,3 +139,8 @@ alert_email = "{{ alert_email }}" cpu_alert_threshold = {{ cpu_alert_threshold }} memory_alert_threshold = {{ memory_alert_threshold }} disk_alert_threshold = {{ disk_alert_threshold }} + +# Encrypted-at-rest data volume (GB) — Docker data-root + backups live here. +# DO block storage is encrypted at rest; the droplet local disk is not. +# 0 disables (dev only). Customer instances: keep enabled. +data_volume_size_gb = {{ data_volume_size_gb }} diff --git a/anythingllm-docker/template/terraform/variables.tf.jinja b/anythingllm-docker/template/terraform/variables.tf.jinja index 8d196c0b..2d3b857b 100644 --- a/anythingllm-docker/template/terraform/variables.tf.jinja +++ b/anythingllm-docker/template/terraform/variables.tf.jinja @@ -207,3 +207,9 @@ variable "disk_alert_threshold" { type = number default = {{ disk_alert_threshold }} } + +variable "data_volume_size_gb" { + description = "Size (GB) of the encrypted-at-rest DO block-storage data volume holding the Docker data-root + backups. 0 disables the volume (data stays on the droplet's local disk — NOT platform-encrypted; unsuitable for customer instances holding financial/personal records)." + type = number + default = {{ data_volume_size_gb }} +} diff --git a/docs/CUSTOMER_INSTANCE_PROVISIONING.md b/docs/CUSTOMER_INSTANCE_PROVISIONING.md index a88de6e4..82caf127 100644 --- a/docs/CUSTOMER_INSTANCE_PROVISIONING.md +++ b/docs/CUSTOMER_INSTANCE_PROVISIONING.md @@ -144,10 +144,11 @@ themselves — ToS / AUP / DPA — are maintained outside this repo): | Property | Fact | |---|---| | **Tenancy** | Single-tenant: dedicated droplet, dedicated Infisical project + Machine Identity, dedicated LLM key. No shared compute, storage, or credentials between customers. | -| **Data locality** | Documents, embeddings (LanceDB), and chat history live on the customer's droplet volume; backups in DO Spaces under a per-project prefix. Region chosen at provisioning. | +| **Data locality** | Documents, embeddings (LanceDB), and chat history live on a dedicated **DO block-storage data volume** attached to the customer's droplet (`data_volume_size_gb`, default 50 GB) — block storage is **platform-encrypted at rest**, unlike droplet local disk. The Docker data-root and local backup tarballs both live on it. Backups offload to DO Spaces under a per-project prefix. Region chosen at provisioning. | | **Encryption** | TLS 1.3 in transit (Caddy/Let's Encrypt). **Backups are client-side encrypted** (OpenPGP/GPG, per-customer ed25519/cv25519 keypair, keyring-free `--recipient-file` encrypt) before upload — objects in Spaces are ciphertext to DigitalOcean; the private key lives only in the operator secret store, never on the droplet. Tofu state SSE-C encrypted. Secrets never on disk (Infisical runtime injection). | | **LLM processing** | Chat prompts + retrieved document context are sent to the configured LLM provider (OpenRouter) at question time — per-customer key, hard monthly cap, under the account-level **Zero-Data-Retention guardrail** (routing restricted to ZDR-only endpoints: providers that neither retain nor train on the data; OpenRouter itself retains nothing). Model choice is pinned by WeOwn (`admin`-held config). | | **Access** | Customer holds a `manager` account (their data/team). WeOwn holds `admin` + SSH (ops keys via `OPS_AUTHORIZED_KEYS`, auditable + revocable per deploy). | +| **Host hardening** | CIS-aligned baseline via `ansible/harden.yml` (DevSec `os_hardening` + `ssh_hardening` ≈ CIS Ubuntu L1: kernel/sysctl, auth policy, SSH key-only + strong crypto), measured with **Lynis** each run. Phase A keeps root key-only SSH (fleet tooling); Phase B (pre-first-customer) moves to the ops-user/root-off model. Run after provisioning: `ansible-galaxy collection install -r ansible/requirements.yml && ansible-playbook ansible/harden.yml -i 'root@,'`. | | **Backups & retention** | Daily GFS backups (30d/12mo/yearly), client-side encrypted per the Encryption row (`BACKUP_GPG_PUBLIC_KEY` in the site project; private key in operator-tools as `BACKUP_GPG_PRIVATE_KEY_` — generated automatically by `deploy-new-site.sh`). Restores fetch the private key in-process from the operator store, stage it into droplet **tmpfs** for the run (ephemeral GNUPGHOME), and shred it. On cancellation: final backup retained **60 days**, then deleted from Spaces; droplet, reserved IP, Infisical project, and the customer's OpenRouter key destroyed/revoked at deprovision. | | **Monitoring** | Host metrics + reverse-proxy access logs → SigNoz (operational telemetry only; document contents are not shipped). DO resource alerts. | | **Subprocessors** | DigitalOcean (compute/storage/backups), Infisical (secret management), OpenRouter → underlying model providers (LLM inference), SigNoz (telemetry), Let's Encrypt (TLS issuance). |