WeOwnNetwork · ncimino · Jul 15, 2026 · Jul 15, 2026 · Jul 15, 2026
@@ -36,6 +36,7 @@ Changes in this section will be promoted to a dated release entry on merge to `m
 
 ### Added
 
+- **Per-customer LLM key provisioning + single-tenant instance lifecycle doc (2026-07-15)** — `scripts/provision-openrouter-key.sh` mints a **per-customer, budget-capped** OpenRouter key via the OpenRouter Management API (`POST /api/v1/keys`, `limit` + `limit_reset: monthly`) and writes it into that customer's Infisical app project as `OPENROUTER_API_KEY` — the value never touches the terminal, disk, or shell history; the provisioning key is sourced in-process from the operator Infisical project (`OPENROUTER_PROVISIONING_KEY`, `read -rs` fallback). Refuses to overwrite a live key without `--force` (no silent orphaning); `--dry-run` for shared hosts. This **automates the previously-manual** "create a key in the OpenRouter dashboard and paste it" step in the per-site `bootstrap-*-infisical.sh` scripts, and gives each dedicated instance blast-radius isolation + a hard monthly spend cap. New [`docs/CUSTOMER_INSTANCE_PROVISIONING.md`](docs/CUSTOMER_INSTANCE_PROVISIONING.md) sequences the existing deploy pieces into a per-customer provision→deploy→operate→deprovision lifecycle (technical only; commercial/sales flow kept out of this public repo). **Verification**: `bash -n` + `shellcheck` clean. **Compliance**: NIST CSF 2.0 PR.AC-4 (least privilege per tenant), PR.DS-1; CIS v8 3.11.
 - **Open-branch consolidation — 10 branches integrated to `main`, 19 retired (2026-07-14)** — a single signed integration of every mergeable open branch, each individually code-reviewed (secrets/public-repo, correctness, template-standard), rebased onto current `main` so the latest template versions (incl. #78 Minimus fail-loud registry login) are preserved. **New:** `supabase-docker/` copier template (v0.1 — Postgres 15 + pgvector); `anythingllm-docker/sites/dev-weown-anythingllm/` (INT-P07/P08 do.weown.tools); `devbox-docker/sites/dev-weown-devbox/` (shared Zed remote-dev droplet, SSH-only, key-only sshd); `keycloak-docker/sites/sso.weown.dev/` migrated to Path-C serving `sso.weown.id`; `openclaw-docker/sites/claw-weown-dev/` + template hardening. **Changed/Fixed:** anythingllm template defaults per Jason (DeepSeek V4 Flash, SearXNG agent-search); searxng bound loopback-only; wordpress core-plugin-bundle fetch task; owncloud oCIS template hardening (backup-name injection guard, `ignore_changes=[user_data,tags]`, corrected security-model claims); smoke-test framework across all 7 templates (fails on unhealthy containers, correct auth-file path). **Security/public-repo scrubs:** real ProtonVPN egress IP → RFC5737; real Infisical project UUIDs → placeholders; offboarded-dev emails → `alerts@example.com`; openclaw ADR-006 auth file `0644`→`0600`/UID-1000; keycloak image pinned `:24.0` + cloud-init `{% raw %}` rotation-breaker fixed. **Retired (skipped + branches deleted):** branches already folded into `e8496d1`, the retired `sites/s004`, obsolete pre-copier flat layouts, and (owner-decided) the `zeroToHundred` webapp and the superseded `security-hardening-ssh` inventory. **Deferred:** M2 Infisical `--client-secret` argv pattern (pre-existing fleet-wide) tracked as a separate hardening pass.
 - `scripts/deploy-new-site.sh`: Added `--skip-infisical` flag to deploy to existing Infisical projects without creating new ones
 - **`docs/WEOWN-APP-CLUSTER-RUNBOOK.md` §5.5 — W24 Phase 3 operational findings (2026-06-11)** — net-new subsection consolidating durable lessons from W24 Phase 3 dry-run execution against the live source cluster (Tue–Thu). Seven findings, each with reproduction commands and the evidence that surfaced them: **§5.5.1 Target cluster sizing** (combined-workload sizing, not source-match — 2-node `s-1vcpu-2gb-amd` was insufficient, scaled to 4 then 6, still 92–99 % allocated; recommend ≥7–8 `s-1vcpu-2gb-amd` OR 3–4 `s-2vcpu-4gb-amd`); **§5.5.2 Velero controller memory** (`velero install` default 256Mi OOM-kills on namespaces >50 items; explicit `--velero-pod-mem-limit=768Mi` + `--node-agent-pod-mem-limit=512Mi` now baked into §4.2/§4.3); **§5.5.3 Velero 4 h PVR-wait timeout × capacity wall** (Velero marks restore `PartiallyFailed` after 4 h even when no data has moved — triage via Pending pod describe Events); **§5.5.4 Restore retry pattern — namespace nuke before re-fire** (if pods scheduled on empty PVCs before PVRs ran, a re-fire produces a hollow restore — `Completed` in <30 s with no PVRs; clean redo requires `kubectl delete namespace <workload>-staging` first); **§5.5.5 Post-restore CronJob cleanup** (matomo-archive / n8n-backup CronJobs restore into staging and fire jobs against unreachable legacy destinations); **§5.5.6 Interpreting benign warnings** (table of 5 warning patterns that are skip-correct: cert-manager + cilium CRDs already exist, kube-root-ca.crt auto-managed, Helm release annotation drift, CiliumEndpoint cosmetic, 4 h PVR timeout); **§5.5.7 AnythingLLM auth drift** (post-backup admin reconfig invalidates restored password hash; informational, not a failure; soft-smoke via DB file presence). Companion edits to §3.1 (sizing callout) and §4.2 / §4.3 (`--velero-pod-mem-limit=768Mi` + `--node-agent-pod-mem-limit=512Mi` flags). Real cluster identifiers redacted per `.github/copilot-instructions.md` §3.0. **Source**: W24 SOW Day-1 through Day-4 execution; Gate 1 evidence captured in PR comment for `@ncimino` sign-off.

@@ -0,0 +1,144 @@
+# Provisioning a Single-Tenant Customer Instance
+
+**Audience:** WeOwn operators standing up (and later tearing down) a **dedicated,
+single-tenant AnythingLLM instance for one paying customer** — "nobody else on
+that server." This is the technical lifecycle only; the commercial/sales flow
+(pricing, purchase, onboarding, subscription lifecycle) is tracked separately in
+the WeOwn engagement runbooks and intentionally kept out of this public repo.
+
+> **This doc adds nothing new to the deploy mechanics** — it sequences the
+> existing pieces into a per-customer lifecycle and adds the one missing step
+> (per-customer LLM key provisioning). The authoritative deploy reference remains
+> [`anythingllm-docker/DEPLOYMENT_GUIDE.md`](../anythingllm-docker/DEPLOYMENT_GUIDE.md).
+
+## Why single-tenant
+
+A dedicated instance per customer is not just an isolation nicety — it is what
+makes the product work at all. AnythingLLM's **shared**-instance multi-tenancy
+cannot scope document management to one tenant (`upload` / `update-embeddings`
+are `[admin, manager]` and manager is instance-wide; the doc library is global).
+On a **dedicated** instance that ceiling disappears — the whole instance is the
+customer's, so a `manager`-role account gives them the full native feature set
+with zero custom code:
+
+- **Secure login** — AnythingLLM multi-user mode + password (self-service reset is
+  via **recovery codes** saved on first login; there is **no email "forgot
+  password"** flow — resets are an operator action).
+- **Private document upload + RAG** — `manager` can upload + manage their own corpus.
+- **Embeddable chat widget** — native Embedded Chat Widget; paste the snippet on
+  their site (see the chat-embed recipe in the engagement SOPs).
+
+**Managed-service role split (native, no custom layer):** WeOwn holds the `admin`
+account (system LLM/embedder/vector settings, the OpenRouter key, infra — never
+exposed to the customer); the customer gets a `manager` account (their documents,
+workspaces, and team, but cannot change LLM/infra config or break the instance).
+
+Everything below is the existing hardened stack from
+[`anythingllm-docker/`](../anythingllm-docker/): dedicated droplet + reserved IP,
+Caddy auto-TLS, app bound to `127.0.0.1`, Infisical runtime secret injection (no
+secrets on disk), GFS backups to DO Spaces, and OTel → SigNoz observability.
+
+---
+
+## Lifecycle at a glance
+
+```
+  provision infra  ──►  mint per-customer LLM key  ──►  deploy app  ──►  validate  ──►  DNS
+  (itofu.sh)            (provision-openrouter-key.sh)   (deploy.sh)      (real login)   (A record)
+        │                                                                                  │
+        └──────────────────────────  suspend / deprovision  ◄──────────────────────────────┘
+                                      (final backup + tofu destroy + Infisical cleanup)
+```
+
+## 1. Provision the instance
+
+Render a site from the template and provision it per
+[`DEPLOYMENT_GUIDE.md` §6.1–6.4](../anythingllm-docker/DEPLOYMENT_GUIDE.md).
+One dedicated Infisical **app project** + Machine Identity per customer (least
+privilege — a compromise of one box cannot read another customer's project).
+
+The **automated** path is [`scripts/deploy-new-site.sh`](../scripts/deploy-new-site.sh)
+(`--template anythingllm-docker`), which creates the Infisical project + Tier-2
+Machine Identity, renders the site, and runs `tofu` + ansible. See
+[`docs/AUTOMATED_DEPLOYMENT.md`](AUTOMATED_DEPLOYMENT.md).
+
+> **Known gap for the scripted path:** `deploy-new-site.sh` currently pushes
+> `JWT_SECRET`, `ADMIN_EMAIL`, and the Spaces backup creds, but **not**
+> `OPENROUTER_API_KEY` or `ANYTHINGLLM_IMAGE` — both are `fail-loud`-required or
+> the container refuses to boot. Until that script learns them, provision the LLM
+> key with step 2 below and set `ANYTHINGLLM_IMAGE` via the per-site
+> `bootstrap-*-infisical.sh` (or the Infisical UI). The **manual**
+> `DEPLOYMENT_GUIDE.md` flow already handles both.
+
+## 2. Mint the customer's own LLM key
+
+Each customer instance gets its **own** OpenRouter key with a **hard monthly
+spend cap**, so cost is attributable per customer and a runaway/compromised box
+cannot burn the shared account. This replaces the manual "create a key in the
+OpenRouter dashboard and paste it" step in the per-site bootstrap scripts.
+
+```bash
+bash scripts/provision-openrouter-key.sh \
+  --customer <slug> \
+  --project-id <customer's site Infisical project id> \
+  --limit-usd 50
+```
+
+The helper mints a capped, `limit_reset: monthly` key via the OpenRouter
+Management API and writes it straight into the customer's Infisical project as
+`OPENROUTER_API_KEY` — the value never touches the terminal, disk, or shell
+history. It refuses to overwrite an existing key without `--force` (so a re-run
+can't silently orphan a live key). The provisioning key it authenticates with
+lives in the operator Infisical project as `OPENROUTER_PROVISIONING_KEY` and is
+consumed in-process. See the script header for the full security model.
+
+> There is no OpenTofu resource for this — OpenRouter has no Terraform provider,
+> so key provisioning is an API/script step in the deploy flow, not IaC. Per-key
+> minting + capping IS the automation; you do **not** hand-create keys.
+
+## 3. Deploy + validate
+
+Deploy the app layer and **validate a real login + a real chat**, per
+[`DEPLOYMENT_GUIDE.md` §6.5–6.7](../anythingllm-docker/DEPLOYMENT_GUIDE.md)
+(`/api/ping` alone is not sufficient — it returns 200 even when auth is broken).
+Then point DNS at the reserved IP (§6.8); Caddy issues the cert within ~30–60s.
+
+Enable multi-user mode and create **two** accounts before handover: the
+WeOwn-held **`admin`** (system/LLM/infra — WeOwn only) and the customer's
+**`manager`** (their documents/workspaces/team). Hand over only the manager
+credentials. Multi-user mode has **no email "forgot password"** flow — the
+customer saves **recovery codes** on first login, and operator support handles
+resets (there are known reset bugs, so prefer recovery codes).
+
+## 4. Operate
+
+Ongoing changes never require Terraform — edit the Infisical value or the
+compose/Caddy files and re-run `./scripts/deploy.sh root@<ip>` (recreates the
+container so injected secrets refresh). Full matrix in
+[`DEPLOYMENT_GUIDE.md` §11](../anythingllm-docker/DEPLOYMENT_GUIDE.md).
+
+## 5. Suspend / deprovision
+
+When a customer pauses or leaves:
+
+1. **Final backup** — `./scripts/backup.sh root@<ip>` (lands in
+   `s3://weown-prod-backups/<project>/`; retained per GFS).
+2. **Suspend** (recoverable) — power the droplet off, or stop the container; keep
+   the reserved IP + Infisical project so it can resume.
+3. **Deprovision** (final) — `cd sites/<domain>/terraform && ./itofu.sh` destroy
+   the droplet + reserved IP + firewall, **revoke the customer's OpenRouter key**
+   in the dashboard (it's per-customer, so revoking is clean), and delete the
+   customer's Infisical project + Machine Identity. Keep the final backup for the
+   agreed retention window before deleting it from Spaces.
+
+See [`docs/AUTOMATED_DEPLOYMENT.md` → Cleaning Up](AUTOMATED_DEPLOYMENT.md) for
+the teardown commands.
+
+---
+
+## Related
+
+- [`anythingllm-docker/DEPLOYMENT_GUIDE.md`](../anythingllm-docker/DEPLOYMENT_GUIDE.md) — authoritative deploy reference.
+- [`docs/AUTOMATED_DEPLOYMENT.md`](AUTOMATED_DEPLOYMENT.md) — the `deploy-new-site.sh` tiered-MI automation.
+- [`scripts/provision-openrouter-key.sh`](../scripts/provision-openrouter-key.sh) — per-customer LLM key minting.
+- [`docs/INFRA_BOOTSTRAP_PATTERN.md`](INFRA_BOOTSTRAP_PATTERN.md) — Path C + Layer 2 bootstrap architecture.