diff --git a/CHANGELOG.md b/CHANGELOG.md index c55226a..3064749 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,7 @@ Changes in this section will be promoted to a dated release entry on merge to `m ### Added +- **Per-customer LLM key provisioning + single-tenant instance lifecycle doc (2026-07-15)** — `scripts/provision-openrouter-key.sh` mints a **per-customer, budget-capped** OpenRouter key via the OpenRouter Management API (`POST /api/v1/keys`, `limit` + `limit_reset: monthly`) and writes it into that customer's Infisical app project as `OPENROUTER_API_KEY` — the value never touches the terminal, disk, or shell history; the provisioning key is sourced in-process from the operator Infisical project (`OPENROUTER_PROVISIONING_KEY`, `read -rs` fallback). Refuses to overwrite a live key without `--force` (no silent orphaning); `--dry-run` for shared hosts. This **automates the previously-manual** "create a key in the OpenRouter dashboard and paste it" step in the per-site `bootstrap-*-infisical.sh` scripts, and gives each dedicated instance blast-radius isolation + a hard monthly spend cap. New [`docs/CUSTOMER_INSTANCE_PROVISIONING.md`](docs/CUSTOMER_INSTANCE_PROVISIONING.md) sequences the existing deploy pieces into a per-customer provision→deploy→operate→deprovision lifecycle (technical only; commercial/sales flow kept out of this public repo). **Verification**: `bash -n` + `shellcheck` clean. **Compliance**: NIST CSF 2.0 PR.AC-4 (least privilege per tenant), PR.DS-1; CIS v8 3.11. - **Open-branch consolidation — 10 branches integrated to `main`, 19 retired (2026-07-14)** — a single signed integration of every mergeable open branch, each individually code-reviewed (secrets/public-repo, correctness, template-standard), rebased onto current `main` so the latest template versions (incl. #78 Minimus fail-loud registry login) are preserved. **New:** `supabase-docker/` copier template (v0.1 — Postgres 15 + pgvector); `anythingllm-docker/sites/dev-weown-anythingllm/` (INT-P07/P08 do.weown.tools); `devbox-docker/sites/dev-weown-devbox/` (shared Zed remote-dev droplet, SSH-only, key-only sshd); `keycloak-docker/sites/sso.weown.dev/` migrated to Path-C serving `sso.weown.id`; `openclaw-docker/sites/claw-weown-dev/` + template hardening. **Changed/Fixed:** anythingllm template defaults per Jason (DeepSeek V4 Flash, SearXNG agent-search); searxng bound loopback-only; wordpress core-plugin-bundle fetch task; owncloud oCIS template hardening (backup-name injection guard, `ignore_changes=[user_data,tags]`, corrected security-model claims); smoke-test framework across all 7 templates (fails on unhealthy containers, correct auth-file path). **Security/public-repo scrubs:** real ProtonVPN egress IP → RFC5737; real Infisical project UUIDs → placeholders; offboarded-dev emails → `alerts@example.com`; openclaw ADR-006 auth file `0644`→`0600`/UID-1000; keycloak image pinned `:24.0` + cloud-init `{% raw %}` rotation-breaker fixed. **Retired (skipped + branches deleted):** branches already folded into `e8496d1`, the retired `sites/s004`, obsolete pre-copier flat layouts, and (owner-decided) the `zeroToHundred` webapp and the superseded `security-hardening-ssh` inventory. **Deferred:** M2 Infisical `--client-secret` argv pattern (pre-existing fleet-wide) tracked as a separate hardening pass. - `scripts/deploy-new-site.sh`: Added `--skip-infisical` flag to deploy to existing Infisical projects without creating new ones - **`docs/WEOWN-APP-CLUSTER-RUNBOOK.md` §5.5 — W24 Phase 3 operational findings (2026-06-11)** — net-new subsection consolidating durable lessons from W24 Phase 3 dry-run execution against the live source cluster (Tue–Thu). Seven findings, each with reproduction commands and the evidence that surfaced them: **§5.5.1 Target cluster sizing** (combined-workload sizing, not source-match — 2-node `s-1vcpu-2gb-amd` was insufficient, scaled to 4 then 6, still 92–99 % allocated; recommend ≥7–8 `s-1vcpu-2gb-amd` OR 3–4 `s-2vcpu-4gb-amd`); **§5.5.2 Velero controller memory** (`velero install` default 256Mi OOM-kills on namespaces >50 items; explicit `--velero-pod-mem-limit=768Mi` + `--node-agent-pod-mem-limit=512Mi` now baked into §4.2/§4.3); **§5.5.3 Velero 4 h PVR-wait timeout × capacity wall** (Velero marks restore `PartiallyFailed` after 4 h even when no data has moved — triage via Pending pod describe Events); **§5.5.4 Restore retry pattern — namespace nuke before re-fire** (if pods scheduled on empty PVCs before PVRs ran, a re-fire produces a hollow restore — `Completed` in <30 s with no PVRs; clean redo requires `kubectl delete namespace -staging` first); **§5.5.5 Post-restore CronJob cleanup** (matomo-archive / n8n-backup CronJobs restore into staging and fire jobs against unreachable legacy destinations); **§5.5.6 Interpreting benign warnings** (table of 5 warning patterns that are skip-correct: cert-manager + cilium CRDs already exist, kube-root-ca.crt auto-managed, Helm release annotation drift, CiliumEndpoint cosmetic, 4 h PVR timeout); **§5.5.7 AnythingLLM auth drift** (post-backup admin reconfig invalidates restored password hash; informational, not a failure; soft-smoke via DB file presence). Companion edits to §3.1 (sizing callout) and §4.2 / §4.3 (`--velero-pod-mem-limit=768Mi` + `--node-agent-pod-mem-limit=512Mi` flags). Real cluster identifiers redacted per `.github/copilot-instructions.md` §3.0. **Source**: W24 SOW Day-1 through Day-4 execution; Gate 1 evidence captured in PR comment for `@ncimino` sign-off. diff --git a/docs/CUSTOMER_INSTANCE_PROVISIONING.md b/docs/CUSTOMER_INSTANCE_PROVISIONING.md new file mode 100644 index 0000000..3c78bcf --- /dev/null +++ b/docs/CUSTOMER_INSTANCE_PROVISIONING.md @@ -0,0 +1,144 @@ +# Provisioning a Single-Tenant Customer Instance + +**Audience:** WeOwn operators standing up (and later tearing down) a **dedicated, +single-tenant AnythingLLM instance for one paying customer** — "nobody else on +that server." This is the technical lifecycle only; the commercial/sales flow +(pricing, purchase, onboarding, subscription lifecycle) is tracked separately in +the WeOwn engagement runbooks and intentionally kept out of this public repo. + +> **This doc adds nothing new to the deploy mechanics** — it sequences the +> existing pieces into a per-customer lifecycle and adds the one missing step +> (per-customer LLM key provisioning). The authoritative deploy reference remains +> [`anythingllm-docker/DEPLOYMENT_GUIDE.md`](../anythingllm-docker/DEPLOYMENT_GUIDE.md). + +## Why single-tenant + +A dedicated instance per customer is not just an isolation nicety — it is what +makes the product work at all. AnythingLLM's **shared**-instance multi-tenancy +cannot scope document management to one tenant (`upload` / `update-embeddings` +are `[admin, manager]` and manager is instance-wide; the doc library is global). +On a **dedicated** instance that ceiling disappears — the whole instance is the +customer's, so a `manager`-role account gives them the full native feature set +with zero custom code: + +- **Secure login** — AnythingLLM multi-user mode + password (self-service reset is + via **recovery codes** saved on first login; there is **no email "forgot + password"** flow — resets are an operator action). +- **Private document upload + RAG** — `manager` can upload + manage their own corpus. +- **Embeddable chat widget** — native Embedded Chat Widget; paste the snippet on + their site (see the chat-embed recipe in the engagement SOPs). + +**Managed-service role split (native, no custom layer):** WeOwn holds the `admin` +account (system LLM/embedder/vector settings, the OpenRouter key, infra — never +exposed to the customer); the customer gets a `manager` account (their documents, +workspaces, and team, but cannot change LLM/infra config or break the instance). + +Everything below is the existing hardened stack from +[`anythingllm-docker/`](../anythingllm-docker/): dedicated droplet + reserved IP, +Caddy auto-TLS, app bound to `127.0.0.1`, Infisical runtime secret injection (no +secrets on disk), GFS backups to DO Spaces, and OTel → SigNoz observability. + +--- + +## Lifecycle at a glance + +``` + provision infra ──► mint per-customer LLM key ──► deploy app ──► validate ──► DNS + (itofu.sh) (provision-openrouter-key.sh) (deploy.sh) (real login) (A record) + │ │ + └────────────────────────── suspend / deprovision ◄──────────────────────────────┘ + (final backup + tofu destroy + Infisical cleanup) +``` + +## 1. Provision the instance + +Render a site from the template and provision it per +[`DEPLOYMENT_GUIDE.md` §6.1–6.4](../anythingllm-docker/DEPLOYMENT_GUIDE.md). +One dedicated Infisical **app project** + Machine Identity per customer (least +privilege — a compromise of one box cannot read another customer's project). + +The **automated** path is [`scripts/deploy-new-site.sh`](../scripts/deploy-new-site.sh) +(`--template anythingllm-docker`), which creates the Infisical project + Tier-2 +Machine Identity, renders the site, and runs `tofu` + ansible. See +[`docs/AUTOMATED_DEPLOYMENT.md`](AUTOMATED_DEPLOYMENT.md). + +> **Known gap for the scripted path:** `deploy-new-site.sh` currently pushes +> `JWT_SECRET`, `ADMIN_EMAIL`, and the Spaces backup creds, but **not** +> `OPENROUTER_API_KEY` or `ANYTHINGLLM_IMAGE` — both are `fail-loud`-required or +> the container refuses to boot. Until that script learns them, provision the LLM +> key with step 2 below and set `ANYTHINGLLM_IMAGE` via the per-site +> `bootstrap-*-infisical.sh` (or the Infisical UI). The **manual** +> `DEPLOYMENT_GUIDE.md` flow already handles both. + +## 2. Mint the customer's own LLM key + +Each customer instance gets its **own** OpenRouter key with a **hard monthly +spend cap**, so cost is attributable per customer and a runaway/compromised box +cannot burn the shared account. This replaces the manual "create a key in the +OpenRouter dashboard and paste it" step in the per-site bootstrap scripts. + +```bash +bash scripts/provision-openrouter-key.sh \ + --customer \ + --project-id \ + --limit-usd 50 +``` + +The helper mints a capped, `limit_reset: monthly` key via the OpenRouter +Management API and writes it straight into the customer's Infisical project as +`OPENROUTER_API_KEY` — the value never touches the terminal, disk, or shell +history. It refuses to overwrite an existing key without `--force` (so a re-run +can't silently orphan a live key). The provisioning key it authenticates with +lives in the operator Infisical project as `OPENROUTER_PROVISIONING_KEY` and is +consumed in-process. See the script header for the full security model. + +> There is no OpenTofu resource for this — OpenRouter has no Terraform provider, +> so key provisioning is an API/script step in the deploy flow, not IaC. Per-key +> minting + capping IS the automation; you do **not** hand-create keys. + +## 3. Deploy + validate + +Deploy the app layer and **validate a real login + a real chat**, per +[`DEPLOYMENT_GUIDE.md` §6.5–6.7](../anythingllm-docker/DEPLOYMENT_GUIDE.md) +(`/api/ping` alone is not sufficient — it returns 200 even when auth is broken). +Then point DNS at the reserved IP (§6.8); Caddy issues the cert within ~30–60s. + +Enable multi-user mode and create **two** accounts before handover: the +WeOwn-held **`admin`** (system/LLM/infra — WeOwn only) and the customer's +**`manager`** (their documents/workspaces/team). Hand over only the manager +credentials. Multi-user mode has **no email "forgot password"** flow — the +customer saves **recovery codes** on first login, and operator support handles +resets (there are known reset bugs, so prefer recovery codes). + +## 4. Operate + +Ongoing changes never require Terraform — edit the Infisical value or the +compose/Caddy files and re-run `./scripts/deploy.sh root@` (recreates the +container so injected secrets refresh). Full matrix in +[`DEPLOYMENT_GUIDE.md` §11](../anythingllm-docker/DEPLOYMENT_GUIDE.md). + +## 5. Suspend / deprovision + +When a customer pauses or leaves: + +1. **Final backup** — `./scripts/backup.sh root@` (lands in + `s3://weown-prod-backups//`; retained per GFS). +2. **Suspend** (recoverable) — power the droplet off, or stop the container; keep + the reserved IP + Infisical project so it can resume. +3. **Deprovision** (final) — `cd sites//terraform && ./itofu.sh` destroy + the droplet + reserved IP + firewall, **revoke the customer's OpenRouter key** + in the dashboard (it's per-customer, so revoking is clean), and delete the + customer's Infisical project + Machine Identity. Keep the final backup for the + agreed retention window before deleting it from Spaces. + +See [`docs/AUTOMATED_DEPLOYMENT.md` → Cleaning Up](AUTOMATED_DEPLOYMENT.md) for +the teardown commands. + +--- + +## Related + +- [`anythingllm-docker/DEPLOYMENT_GUIDE.md`](../anythingllm-docker/DEPLOYMENT_GUIDE.md) — authoritative deploy reference. +- [`docs/AUTOMATED_DEPLOYMENT.md`](AUTOMATED_DEPLOYMENT.md) — the `deploy-new-site.sh` tiered-MI automation. +- [`scripts/provision-openrouter-key.sh`](../scripts/provision-openrouter-key.sh) — per-customer LLM key minting. +- [`docs/INFRA_BOOTSTRAP_PATTERN.md`](INFRA_BOOTSTRAP_PATTERN.md) — Path C + Layer 2 bootstrap architecture. diff --git a/scripts/provision-openrouter-key.sh b/scripts/provision-openrouter-key.sh new file mode 100755 index 0000000..a6fe961 --- /dev/null +++ b/scripts/provision-openrouter-key.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# provision-openrouter-key.sh — mint a per-customer, budget-capped OpenRouter API +# key and push it into that customer's Infisical app project as OPENROUTER_API_KEY. +# +# WHY. Each paying customer's private AnythingLLM instance gets its OWN OpenRouter +# key, for two reasons: +# 1. Blast-radius isolation — one customer's leaked/compromised instance can be +# revoked without touching any other customer. +# 2. Clean per-customer cost attribution + a HARD monthly spend cap, so a runaway +# or abused instance can never burn the shared account. The cap is enforced by +# OpenRouter itself (`limit` + `limit_reset: monthly`), not by us. +# This AUTOMATES the previously-manual "create a key in the OpenRouter dashboard +# and paste it" step that the per-site bootstrap-*-infisical.sh scripts prompt for. +# +# HOW. Uses the OpenRouter Management/Provisioning API: +# POST https://openrouter.ai/api/v1/keys {name, limit, limit_reset:"monthly"} +# Auth is an OpenRouter PROVISIONING key (distinct from a runtime inference key), +# sourced IN-PROCESS from the operator Infisical project as OPENROUTER_PROVISIONING_KEY +# — never on argv, never printed, never written to disk (read -rs fallback if absent). +# The minted customer key is written straight into the site Infisical project and is +# NEVER printed to the terminal or written to disk. +# +# There is deliberately no OpenTofu here: OpenRouter has no Terraform provider, so +# key provisioning is a script/API step in the deploy flow, not an IaC resource. +# +# Usage: +# bash scripts/provision-openrouter-key.sh \ +# --customer \ +# --project-id \ +# [--limit-usd 50] [--env prod] [--operator-project operator-tools] [--force] +# +# Prereqs: infisical CLI (logged in), curl, jq. +# +# SECURITY NOTE. Like the per-site bootstrap-*-infisical.sh scripts, the final +# `infisical secrets set OPENROUTER_API_KEY=` passes the value as a process +# argument, so it is briefly visible in `ps`/proc on THIS operator machine only — +# never on disk, in shell history, or on the network. That is acceptable on a +# single-operator laptop. On a shared host, run with --dry-run to mint nothing and +# set the value in the Infisical UI instead, or wrap this in the Infisical API. +set -uo pipefail + +OPENROUTER_KEYS_API="https://openrouter.ai/api/v1/keys" + +# ── defaults ───────────────────────────────────────────────────────────────── +CUSTOMER="" +PROJECT_ID="" +LIMIT_USD="50" +ENV_SLUG="prod" +OPERATOR_PROJECT="operator-tools" +FORCE=0 +DRY_RUN=0 + +usage() { + sed -n '2,45p' "$0" | sed 's/^# \{0,1\}//' + exit "${1:-0}" +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --customer) CUSTOMER="${2:-}"; shift 2 ;; + --project-id) PROJECT_ID="${2:-}"; shift 2 ;; + --limit-usd) LIMIT_USD="${2:-}"; shift 2 ;; + --env) ENV_SLUG="${2:-}"; shift 2 ;; + --operator-project) OPERATOR_PROJECT="${2:-}"; shift 2 ;; + --force) FORCE=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + -h|--help) usage 0 ;; + *) echo "ERROR: unknown option: $1" >&2; usage 1 ;; + esac +done + +# ── validate inputs ────────────────────────────────────────────────────────── +[[ -n "$CUSTOMER" ]] || { echo "ERROR: --customer is required (short slug, e.g. acme-cpa)." >&2; exit 1; } +[[ -n "$PROJECT_ID" ]] || { echo "ERROR: --project-id is required (the customer's site Infisical project id)." >&2; exit 1; } +[[ "$CUSTOMER" =~ ^[a-z0-9-]+$ ]] || { echo "ERROR: --customer must be lowercase alphanumeric with hyphens: $CUSTOMER" >&2; exit 1; } +[[ "$LIMIT_USD" =~ ^[0-9]+$ ]] || { echo "ERROR: --limit-usd must be a whole number of US dollars: $LIMIT_USD" >&2; exit 1; } + +# ── preflight ──────────────────────────────────────────────────────────────── +for bin in infisical curl jq; do + command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found on PATH. Install it first." >&2; exit 1; } +done + +# Ensure an Infisical session up front (the whoami/user probes return non-zero when +# not logged in; they do not hang). This mirrors the per-site bootstrap scripts. +if infisical whoami >/dev/null 2>&1 || infisical user >/dev/null 2>&1; then + echo "Infisical session active." +else + echo "No active Infisical session — launching 'infisical login'..." + infisical login || { echo "ERROR: 'infisical login' did not complete. Log in and re-run." >&2; exit 1; } +fi + +# Clear every secret var no matter how we exit. +trap 'unset PROV_KEY CUSTOMER_KEY 2>/dev/null || true' EXIT + +# ── refuse to clobber an existing key unless --force (avoid orphaning) ──────── +if infisical secrets get OPENROUTER_API_KEY \ + --projectId="$PROJECT_ID" --env="$ENV_SLUG" --path=/ >/dev/null 2>&1; then + if [[ "$FORCE" -ne 1 ]]; then + echo "ERROR: OPENROUTER_API_KEY already set in project $PROJECT_ID (env $ENV_SLUG)." >&2 + echo " Minting a new one would ORPHAN the old key on OpenRouter. Revoke the old" >&2 + echo " key in the OpenRouter dashboard first, then re-run with --force to replace." >&2 + exit 1 + fi + echo "⚠️ OPENROUTER_API_KEY already present — --force given, will overwrite (revoke the OLD key manually)." +fi + +# ── source the OpenRouter PROVISIONING key in-process (never printed) ───────── +# Preferred: from the operator Infisical project. Fallback: hidden prompt. +PROV_KEY="$(infisical secrets get OPENROUTER_PROVISIONING_KEY \ + --projectId="$OPERATOR_PROJECT" --env="$ENV_SLUG" --path=/ --plain 2>/dev/null || true)" +if [[ -z "${PROV_KEY:-}" ]]; then + echo "OPENROUTER_PROVISIONING_KEY not found in Infisical project '$OPERATOR_PROJECT' (env $ENV_SLUG)." + echo "Paste an OpenRouter PROVISIONING key (Settings → Provisioning API Keys). Input is hidden:" + read -rs PROV_KEY; echo +fi +[[ -n "${PROV_KEY:-}" ]] || { echo "ERROR: no provisioning key available; cannot mint." >&2; exit 1; } + +# ── build the key name + request ───────────────────────────────────────────── +CUSTOMER_UPPER="$(printf '%s' "$CUSTOMER" | tr 'a-z-' 'A-Z_')" +KEY_NAME="OPENROUTER_${CUSTOMER_UPPER}_ANYTHINGLLM_MONTHLY_${LIMIT_USD}USD" +REQ_BODY="$(jq -nc --arg name "$KEY_NAME" --argjson limit "$LIMIT_USD" \ + '{name: $name, limit: $limit, limit_reset: "monthly"}')" + +echo +echo "── Minting per-customer OpenRouter key ──" +echo " customer: $CUSTOMER" +echo " key name: $KEY_NAME" +echo " monthly cap: \$$LIMIT_USD (resets 1st of month, UTC)" +echo " site project: $PROJECT_ID (env $ENV_SLUG)" + +if [[ "$DRY_RUN" -eq 1 ]]; then + echo + echo "[DRY RUN] Would POST $OPENROUTER_KEYS_API with: $REQ_BODY" + echo "[DRY RUN] Would set OPENROUTER_API_KEY in project $PROJECT_ID. Nothing minted." + exit 0 +fi + +# ── mint via the OpenRouter Management API ─────────────────────────────────── +HTTP_RESP="$(curl -sS -o - -w $'\n%{http_code}' -X POST "$OPENROUTER_KEYS_API" \ + -H "Authorization: Bearer ${PROV_KEY}" \ + -H "Content-Type: application/json" \ + -d "$REQ_BODY" 2>/dev/null || true)" +HTTP_CODE="${HTTP_RESP##*$'\n'}" +HTTP_BODY="${HTTP_RESP%$'\n'*}" + +if [[ ! "$HTTP_CODE" =~ ^2[0-9][0-9]$ ]]; then + # On a non-2xx there is no minted key in the body — safe to surface for debugging. + echo "ERROR: OpenRouter key creation failed (HTTP ${HTTP_CODE:-none})." >&2 + echo "$HTTP_BODY" | jq . 2>/dev/null >&2 || echo "$HTTP_BODY" >&2 + exit 1 +fi + +# The secret key string is returned once, at the top level (`.key`); some API +# versions nest it under `.data.key`. Parse defensively; never echo the value. +CUSTOMER_KEY="$(printf '%s' "$HTTP_BODY" | jq -r '.key // .data.key // empty')" +if [[ -z "${CUSTOMER_KEY:-}" ]]; then + echo "ERROR: key created but the secret string wasn't found in the response." >&2 + echo " Response top-level fields were: $(printf '%s' "$HTTP_BODY" | jq -rc 'keys? // "unparseable"' 2>/dev/null)" >&2 + echo " Check the OpenRouter dashboard for a dangling key named: $KEY_NAME" >&2 + exit 1 +fi + +# ── push into the site Infisical project (see SECURITY NOTE in header) ─────── +if infisical secrets set "OPENROUTER_API_KEY=${CUSTOMER_KEY}" \ + --projectId="$PROJECT_ID" --env="$ENV_SLUG" --path=/ >/dev/null 2>&1; then + echo " ✓ set OPENROUTER_API_KEY in project $PROJECT_ID" +else + echo "ERROR: minted the key but FAILED to set OPENROUTER_API_KEY in Infisical." >&2 + echo " The key exists on OpenRouter as '$KEY_NAME' — set it in the UI or re-run, then" >&2 + echo " delete any duplicate on OpenRouter to avoid orphans." >&2 + exit 1 +fi + +echo +echo "Done — '$KEY_NAME' minted (\$$LIMIT_USD/mo cap) and stored as OPENROUTER_API_KEY." +echo "No key value touched disk, history, or this terminal." +echo "Next: deploy/redeploy the instance so the container picks up the key (see" +echo "anythingllm-docker/DEPLOYMENT_GUIDE.md §6.5). Verify a real chat completes."