WeOwnNetwork · ncimino · Jul 15, 2026 · Jul 15, 2026
@@ -32,6 +32,7 @@ Changes in this section will be promoted to a dated release entry on merge to `m
 
 ### Security
 
+- **anythingllm-docker — CIS host hardening play + encrypted-at-rest data volume (2026-07-15)** — Phase A of porting the SOP-INFRA-012 baseline to customer instances. **Hardening:** new `template/ansible/harden.yml` applies DevSec `os_hardening` + `ssh_hardening` (≈ CIS Ubuntu L1 — kernel/sysctl, auth policy, SSH key-only + strong KEX/ciphers) with the Docker-safe `ip_forward=1` task-var override (defaults-level overrides silently lose to inner-role defaults — the 2026-07-03 no-egress landmine), then measures with **Lynis** and verifies Docker survived; `ansible/requirements.yml` pins the `devsec.hardening` collection. Phase A deliberately keeps root key-only SSH (all fleet tooling SSHes as `root@`); Phase B (ops-user model, root-off, per SOP-INFRA-012) is gated on fleet-tooling migration and tracked for pre-first-customer. **Data at rest:** new `digitalocean_volume` (default 50 GB, `prevent_destroy`, `data_volume_size_gb=0` to disable) — DO block storage is platform-encrypted at rest, droplet local disk is not; cloud-init `setup-data-volume.sh` waits out the attach race, formats/mounts, points the **Docker data-root** (chats/vectors/uploads) and the **backups dir** at it before Docker installs. **Compliance**: FTC Safeguards/GLBA at-rest expectation; NIST CSF 2.0 PR.DS-1, PR.PS; CIS v8 4.1/3.11. **Verification**: rendered `harden.yml` passes `ansible-playbook --syntax-check`; `setup-data-volume.sh` `bash -n` clean; rendered `main.tf` parses (`tofu fmt`).
 - **anythingllm-docker — client-side encrypted backups (GPG) + ZDR LLM routing posture (2026-07-15)** — closes two compliance gaps for customer instances holding financial/personal records (GLBA-service-provider posture). **Backups:** `backup.sh` now encrypts the tarball with OpenPGP/GPG (per-customer ed25519/cv25519 public key, `BACKUP_GPG_PUBLIC_KEY` injected via `infisical run`; keyring-free `--recipient-file` + ephemeral `GNUPGHOME`) before the Spaces upload — the object at rest is ciphertext to DigitalOcean; the private key lives ONLY in the operator store (`BACKUP_GPG_PRIVATE_KEY_<project>`, auto-generated by `deploy-new-site.sh` in an ephemeral keyring). Fail-loud when encryption is configured but `gpg` is missing; loud warning on unencrypted uploads. `restore.sh` fetches `.tar.gz.gpg`, pulls the private key **in-process** from operator-tools via the logged-in infisical CLI (paste fallback), stages it into droplet **tmpfs** (0600, ephemeral GNUPGHOME, shredded on exit via trap), decrypts, and proceeds; legacy plaintext objects still restore. GPG chosen over `age` for auditor familiarity (owner call); `gnupg` was already in cloud-init packages. **LLM routing:** per-customer OpenRouter keys operate under the account-level **Zero-Data-Retention guardrail** (ZDR-only endpoints — no provider retention or training; OpenRouter itself retains nothing); `provision-openrouter-key.sh` prints the posture requirement, and the compliance facts table in [`docs/CUSTOMER_INSTANCE_PROVISIONING.md`](docs/CUSTOMER_INSTANCE_PROVISIONING.md) documents both. **Compliance**: FTC Safeguards (GLBA) encryption-at-rest expectation; NIST CSF 2.0 PR.DS-1/PR.DS-2; CIS v8 3.11; supports IRC §7216 posture for tax data. **Verification**: 6-check GPG round-trip (encrypt → ciphertext-opaque → decrypt → byte-identical → wrong-key-rejected) passes with the exact script logic; rendered-shell `bash -n` clean.
 - **M2 fleet-wide — Infisical Machine-Identity secrets & DO Spaces creds moved off command-line argv (2026-07-14)** — closes the AGENTS.md S2 finding that `infisical login --client-secret="$INFISICAL_CLIENT_SECRET"` and `tofu init -backend-config="secret_key=…"` exposed secrets on process argv (readable via `ps` / `/proc/<pid>/cmdline` on the droplet). **Infisical login** now uses env-var auth — `export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID/_SECRET` before a flagless `infisical login --method=universal-auth` (the CLI reads them natively), matching the pattern openclaw's entrypoint already used. **Tofu/OpenTofu** now exports `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` (the S3 backend reads them from env) and drops the two `-backend-config` access/secret lines; `sse_customer_key` stays on `-backend-config` (no env equivalent). Applied across every `*-docker` template and rendered site (anythingllm, keycloak, owncloud, wordpress, searxng, signoz, sandbox, openclaw, devbox) plus `otel-agent/` and the `scripts/*otel*` helpers, and the doc examples that showed the old form. Only `--plain`/`--silent` and non-secret flags remain on argv. **Compliance**: NIST CSF 2.0 PR.DS-1/PR.AC-4; CIS v8 3.11. **Verification**: `bash -n` + `shellcheck -S error` clean on all changed scripts; rendered ansible YAML parses; 0 remaining `--client-secret=`/`-backend-config="secret_key="` in tracked code. **Note**: needs a live deploy test (copier render → ansible/tofu path) before promotion — the human pre-merge gate.
 

@@ -62,6 +62,11 @@ anythingllm_memory_limit:
   default: "2G"
   validator: "{% if not anythingllm_memory_limit | regex_search('^[0-9]+[MG]$') %}Must be a Docker memory string like 2G or 512M{% endif %}"
 
+data_volume_size_gb:
+  type: int
+  help: "Encrypted-at-rest DO block-storage data volume (GB) for Docker data-root + backups — DO encrypts block storage at rest, the droplet's local disk is not. 0 disables (dev only; customer instances holding financial/personal data must keep this on)."
+  default: 50
+
 anythingllm_memory_reservation:
   type: str
   help: "AnythingLLM container memory reservation (soft guarantee, below the limit)"

@@ -0,0 +1,76 @@
+# {{ project_name }} — host hardening (CIS-aligned, Phase A)
+#
+# Applies the DevSec hardening baseline (≈ CIS Ubuntu L1) to the droplet:
+# kernel/sysctl, auth policy, filesystem perms, SSH crypto + policy — then
+# measures the result with Lynis. Idempotent; run any time after bootstrap:
+#
+#   ansible-galaxy collection install -r ansible/requirements.yml   # once
+#   ansible-playbook ansible/harden.yml -i 'root@<droplet-ip>,'
+#
+# PHASE A vs PHASE B (deliberate scope — do not "fix" this into Phase B):
+#   Phase A (this play): CIS L1 OS + SSH hardening with root login KEPT
+#     (key-only). Every fleet tool (deploy.sh, backup.sh, manage-droplets.sh)
+#     SSHes as root@; removing root here would break the fleet mid-flight.
+#   Phase B (tracked, pre-first-customer): ops-user model (root SSH off,
+#     password-sudo 2FA), mesh/CIDR-locked port 22 — per SOP-INFRA-012. Needs
+#     the fleet tooling migrated off root@ first.
+#
+# Docker-safe exceptions (the CIS-L2-vs-container-runtime conflicts):
+#   - net.ipv4.ip_forward stays 1 — Docker bridge networking requires it. A
+#     defaults-level override silently LOSES to the inner role's defaults;
+#     it must be a task-var on the include (hit live 2026-07-03: containers
+#     got DNS but no TCP egress when this landed at 0).
+---
+- name: Harden {{ project_name }} host (CIS L1 via devsec — Phase A)
+  hosts: all
+  become: true
+  vars:
+    run_lynis: true
+
+  tasks:
+    - name: "OS hardening (devsec.hardening.os_hardening ≈ CIS Ubuntu L1)"
+      ansible.builtin.include_role:
+        name: devsec.hardening.os_hardening
+      vars:
+        # MUST be task-vars here (see header) — Docker bridge needs ip_forward.
+        sysctl_overwrite:
+          net.ipv4.ip_forward: 1
+
+    - name: "SSH hardening (devsec.hardening.ssh_hardening — key-only, strong KEX/ciphers)"
+      ansible.builtin.include_role:
+        name: devsec.hardening.ssh_hardening
+      vars:
+        # Phase A: root stays (fleet tooling), but key-only and CIS crypto.
+        ssh_permit_root_login: "without-password"
+        ssh_password_authentication: false
+        ssh_challenge_response_authentication: false
+
+    - name: "Measure: install + run Lynis (hardening index; non-blocking)"
+      when: run_lynis | bool
+      block:
+        - name: "Install Lynis"
+          ansible.builtin.apt:
+            name: lynis
+            state: present
+            update_cache: true
+
+        - name: "Run Lynis audit (quick)"
+          ansible.builtin.command: lynis audit system --quick --no-colors
+          register: lynis_out
+          changed_when: false
+          failed_when: false
+
+        - name: "Lynis summary (record this index in the site CHANGELOG)"
+          ansible.builtin.debug:
+            msg: "{{ '{{' }} lynis_out.stdout_lines | select('search', 'Hardening index|Warnings|Suggestions|Tests performed') | list {{ '}}' }}"
+
+    - name: "Verify Docker survived hardening (containers must still run)"
+      # brace-free format string: Go-template braces here would be re-templated
+      # by ansible's own Jinja pass and error out.
+      ansible.builtin.command: docker info --format json
+      register: docker_ok
+      changed_when: false
+
+    - name: "Docker status"
+      ansible.builtin.debug:
+        msg: "Docker OK (server {{ '{{' }} (docker_ok.stdout | from_json).ServerVersion | default('unknown') {{ '}}' }}) — hardening did not break the runtime"
@@ -0,0 +1,8 @@
+# Ansible Galaxy requirements for the optional hardening play (harden.yml).
+# Install once per operator workstation:
+#   ansible-galaxy collection install -r ansible/requirements.yml
+collections:
+  # DevSec Hardening Framework — CIS/STIG-aligned os_hardening + ssh_hardening.
+  # https://github.com/dev-sec/ansible-collection-hardening
+  - name: devsec.hardening
+    version: ">=8.8.0"
@@ -63,6 +63,33 @@ resource "digitalocean_droplet" "anythingllm" {
   }
 }
 
+# Encrypted-at-rest data volume (DO encrypts block storage at rest; the
+# droplet's LOCAL disk carries no such guarantee). cloud-init formats/mounts it
+# and points the Docker data-root + the backups dir at it, so customer data —
+# volumes (chats, vectors, uploads) and local backup tarballs — lives on
+# platform-encrypted storage. Set data_volume_size_gb = 0 to disable (dev only).
+resource "digitalocean_volume" "data" {
+  count  = var.data_volume_size_gb > 0 ? 1 : 0
+  region = var.region
+  name   = "{{ project_name }}-data"
+  size   = var.data_volume_size_gb
+  # Pre-formatted by DO so first boot never races mkfs; cloud-init still
+  # carries a mkfs fallback for volumes created outside tofu.
+  initial_filesystem_type = "ext4"
+  description             = "{{ project_name }} encrypted-at-rest data (docker data-root + backups)"
+
+  lifecycle {
+    # Customer data lives here — same protection as the droplet itself.
+    prevent_destroy = true
+  }
+}
+
+resource "digitalocean_volume_attachment" "data" {
+  count      = var.data_volume_size_gb > 0 ? 1 : 0
+  droplet_id = digitalocean_droplet.anythingllm.id
+  volume_id  = digitalocean_volume.data[0].id
+}
+
 resource "digitalocean_reserved_ip" "anythingllm" {
   region = var.region
 }

@@ -92,6 +92,45 @@ write_files:
         "storage-driver": "overlay2"
       }
 
+  # Encrypted-at-rest data volume setup (runs BEFORE Docker is installed).
+  # DO block-storage volumes are encrypted at rest; the droplet's local disk is
+  # not. When the tofu-attached volume is present, mount it and point the
+  # Docker data-root (app volumes: chats/vectors/uploads) + the backups dir at
+  # it. Graceful no-op when no volume is attached (data_volume_size_gb = 0).
+  # The attachment can land seconds AFTER first boot starts, so wait for the
+  # device before concluding it's absent.
+  - path: /tmp/setup-data-volume.sh
+    permissions: "0755"
+    content: |
+      #!/bin/bash
+      set -euo pipefail
+      DEV="/dev/disk/by-id/scsi-0DO_Volume_{{ project_name }}-data"
+      MNT="/mnt/${project_name}_data"
+      for i in $(seq 1 24); do
+        [ -e "$DEV" ] && break
+        sleep 5
+      done
+      if [ ! -e "$DEV" ]; then
+        echo "no data volume attached after 120s - using droplet local disk (dev mode)"
+        exit 0
+      fi
+      # DO pre-formats tofu-created volumes (initial_filesystem_type); mkfs is
+      # the fallback for volumes created outside tofu.
+      blkid "$DEV" >/dev/null 2>&1 || mkfs.ext4 -q "$DEV"
+      mkdir -p "$MNT"
+      grep -q "scsi-0DO_Volume_{{ project_name }}-data" /etc/fstab || \
+        echo "$DEV $MNT ext4 defaults,noatime,nofail,discard 0 2" >> /etc/fstab
+      mountpoint -q "$MNT" || mount "$MNT"
+      mkdir -p "$MNT/docker" "$MNT/backups"
+      # Point the Docker data-root at the encrypted volume (Docker not yet
+      # installed/started, so this is race-free).
+      jq --arg dr "$MNT/docker" '. + {"data-root": $dr}' /etc/docker/daemon.json > /tmp/daemon.json.new
+      mv /tmp/daemon.json.new /etc/docker/daemon.json
+      # Backups dir lives on the encrypted volume; the app path is a symlink.
+      mkdir -p /opt/${project_name}
+      [ -e /opt/${project_name}/backups ] || ln -s "$MNT/backups" /opt/${project_name}/backups
+      echo "data volume mounted at $MNT (docker data-root + backups relocated)"
+
   # Infisical Machine Identity auth file (bootstrap secret v1). Rotation runs
   # below - once v2 is minted, this file is overwritten with v2 and v1 is
   # revoked in Infisical. The v1 value embedded in terraform state / DO
@@ -258,6 +297,9 @@ write_files:
       log "===== Rotation complete ====="
 
 runcmd:
+  # Data volume FIRST: docker's data-root must point at the encrypted volume
+  # before docker is installed/started.
+  - [bash, /tmp/setup-data-volume.sh]
   - [bash, /tmp/install-docker.sh]
   - [bash, /tmp/install-infisical.sh]
   # AWS CLI v2 (DO Spaces is S3-compatible) via the official installer; apt awscli

@@ -139,3 +139,8 @@ alert_email            = "{{ alert_email }}"
 cpu_alert_threshold    = {{ cpu_alert_threshold }}
 memory_alert_threshold = {{ memory_alert_threshold }}
 disk_alert_threshold   = {{ disk_alert_threshold }}
+
+# Encrypted-at-rest data volume (GB) — Docker data-root + backups live here.
+# DO block storage is encrypted at rest; the droplet local disk is not.
+# 0 disables (dev only). Customer instances: keep enabled.
+data_volume_size_gb = {{ data_volume_size_gb }}
@@ -207,3 +207,9 @@ variable "disk_alert_threshold" {
   type        = number
   default     = {{ disk_alert_threshold }}
 }
+
+variable "data_volume_size_gb" {
+  description = "Size (GB) of the encrypted-at-rest DO block-storage data volume holding the Docker data-root + backups. 0 disables the volume (data stays on the droplet's local disk — NOT platform-encrypted; unsuitable for customer instances holding financial/personal records)."
+  type        = number
+  default     = {{ data_volume_size_gb }}
+}
@@ -144,10 +144,11 @@ themselves — ToS / AUP / DPA — are maintained outside this repo):
 | Property | Fact |
 |---|---|
 | **Tenancy** | Single-tenant: dedicated droplet, dedicated Infisical project + Machine Identity, dedicated LLM key. No shared compute, storage, or credentials between customers. |
-| **Data locality** | Documents, embeddings (LanceDB), and chat history live on the customer's droplet volume; backups in DO Spaces under a per-project prefix. Region chosen at provisioning. |
+| **Data locality** | Documents, embeddings (LanceDB), and chat history live on a dedicated **DO block-storage data volume** attached to the customer's droplet (`data_volume_size_gb`, default 50 GB) — block storage is **platform-encrypted at rest**, unlike droplet local disk. The Docker data-root and local backup tarballs both live on it. Backups offload to DO Spaces under a per-project prefix. Region chosen at provisioning. |
 | **Encryption** | TLS 1.3 in transit (Caddy/Let's Encrypt). **Backups are client-side encrypted** (OpenPGP/GPG, per-customer ed25519/cv25519 keypair, keyring-free `--recipient-file` encrypt) before upload — objects in Spaces are ciphertext to DigitalOcean; the private key lives only in the operator secret store, never on the droplet. Tofu state SSE-C encrypted. Secrets never on disk (Infisical runtime injection). |
 | **LLM processing** | Chat prompts + retrieved document context are sent to the configured LLM provider (OpenRouter) at question time — per-customer key, hard monthly cap, under the account-level **Zero-Data-Retention guardrail** (routing restricted to ZDR-only endpoints: providers that neither retain nor train on the data; OpenRouter itself retains nothing). Model choice is pinned by WeOwn (`admin`-held config). |
 | **Access** | Customer holds a `manager` account (their data/team). WeOwn holds `admin` + SSH (ops keys via `OPS_AUTHORIZED_KEYS`, auditable + revocable per deploy). |
+| **Host hardening** | CIS-aligned baseline via `ansible/harden.yml` (DevSec `os_hardening` + `ssh_hardening` ≈ CIS Ubuntu L1: kernel/sysctl, auth policy, SSH key-only + strong crypto), measured with **Lynis** each run. Phase A keeps root key-only SSH (fleet tooling); Phase B (pre-first-customer) moves to the ops-user/root-off model. Run after provisioning: `ansible-galaxy collection install -r ansible/requirements.yml && ansible-playbook ansible/harden.yml -i 'root@<ip>,'`. |
 | **Backups & retention** | Daily GFS backups (30d/12mo/yearly), client-side encrypted per the Encryption row (`BACKUP_GPG_PUBLIC_KEY` in the site project; private key in operator-tools as `BACKUP_GPG_PRIVATE_KEY_<project>` — generated automatically by `deploy-new-site.sh`). Restores fetch the private key in-process from the operator store, stage it into droplet **tmpfs** for the run (ephemeral GNUPGHOME), and shred it. On cancellation: final backup retained **60 days**, then deleted from Spaces; droplet, reserved IP, Infisical project, and the customer's OpenRouter key destroyed/revoked at deprovision. |
 | **Monitoring** | Host metrics + reverse-proxy access logs → SigNoz (operational telemetry only; document contents are not shipped). DO resource alerts. |
 | **Subprocessors** | DigitalOcean (compute/storage/backups), Infisical (secret management), OpenRouter → underlying model providers (LLM inference), SigNoz (telemetry), Let's Encrypt (TLS issuance). |