Singleton-Solution · tayebmokni · May 26, 2026 · May 26, 2026
@@ -101,6 +101,73 @@ Good AI use looks like: you read the design doc, drafted with AI help, reviewed
 
 If you're not sure whether your contribution will be received well, open a `design-discussion` issue first and ask.
 
+## Security review checklist
+
+Every PR that touches an authenticated endpoint, an admin surface, a
+plugin host boundary, or a SQL query must walk through this checklist
+before requesting review. Tick each item in the PR description or
+explain why it doesn't apply.
+
+The reviewer is responsible for spot-checking the answers — items
+left blank or hand-waved ("N/A" with no reason) block merge.
+
+- [ ] **Input validation.** Every request payload is validated against
+  a typed shape (Go struct + `validate` tags / TS Zod) before it
+  reaches the business logic. Unexpected fields are rejected by
+  `additionalProperties: false` at the schema layer, not silently
+  ignored. Required fields are checked explicitly. Length, range,
+  and format limits are enforced server-side regardless of any
+  client-side validation.
+- [ ] **Sanitization.** Output that lands in HTML is escaped (React
+  does this by default; raw insertions via `dangerouslySetInnerHTML`
+  use `DOMPurify` from `apps/admin/src/components/SafeHTML.tsx`).
+  SQL is parametrized — no `fmt.Sprintf` into a query. Shell
+  invocations are avoided; when unavoidable, they use `exec.Command`
+  with an explicit argv (never `bash -c`).
+- [ ] **Auth check.** The endpoint requires authentication (or has a
+  documented public-by-design reason). Mounted behind
+  `auth.RequireSession` or equivalent. Anonymous endpoints have a
+  one-line comment explaining why.
+- [ ] **Capability gate.** Beyond authentication, every privileged
+  action goes through `policy.Can(...)` with a specific capability
+  constant. No role-string comparisons in the handler body. If a
+  new capability is introduced, it's added to
+  `packages/go/policy/capabilities.go` and to the default role
+  mapping in `defaults.go`.
+- [ ] **Rate limit.** Login, password reset, email verification,
+  and any high-value mutation surface go through the limiter from
+  `packages/go/ratelimit`. If a new bucket is added, the default
+  policy is documented and conservative (fail-closed when Redis is
+  unavailable, except where the doc explicitly approves fail-open).
+- [ ] **Audit emit.** Every privileged action emits an
+  `audit.Event`. Event type uses the dotted convention
+  (`auth.login.success`, `plugin.activated`). Severity matches the
+  matrix in `docs/06-auth-permissions.md` §13. Metadata is bounded
+  (no caller-controlled blobs).
+- [ ] **CSRF.** State-changing endpoints (POST/PUT/PATCH/DELETE) are
+  same-origin via session cookies + SameSite=Lax, OR they require
+  a CSRF token from `auth/csrf`. The reviewer confirms the
+  middleware is in the chain.
+- [ ] **Secret handling.** No credentials, tokens, or PII in log
+  lines. `slog` attrs use the structured form so the redactor in
+  `packages/go/log` can mask sensitive keys. Secrets read from
+  config use the masked accessor (`cfg.Auth.Pepper` is masked when
+  the config is dumped).
+- [ ] **Dependency vulns.** New direct deps are checked against
+  `osv-scanner` / `pnpm audit`. Transitive vulns are noted in the
+  PR description with a remediation plan (or a documented accept).
+  CI runs the scanners on every PR, but humans should still look at
+  the manifest diff.
+- [ ] **Error message info leak.** Error responses don't echo
+  internal paths, SQL fragments, or stack traces. The error code
+  is a short slug (`not_found`, `internal_error`); the human
+  message is generic. Detailed diagnostics live in the structured
+  log, never in the response body.
+
+If your change is a pure documentation update, a comment-only refactor,
+or a test-only patch, this section can be skipped — say so in the PR
+description.
+
 ## Reporting bugs
 
 Open an issue with the `bug` template. Include:

@@ -0,0 +1,155 @@
+# Multi-region deployment (v2)
+
+> Status: **design only**. This doc captures the config flags and
+> fan-out URLs the v2 multi-region surface will read from. The v1
+> binary ships a single-region wiring; the flags here are reserved so
+> the v2 cut can land without breaking existing operators' configs.
+>
+> Tracked in issue #138. Implementation issues fan out from this
+> document.
+
+## Scope
+
+A v2 multi-region GoNext deployment runs:
+
+- one **primary region** owning the writable Postgres + Redis
+  primary,
+- one or more **read replica regions** with Postgres physical replicas
+  + Redis cluster nodes,
+- a global CDN fronting the public theme,
+- a global anycast LB fronting the admin and API surfaces.
+
+This is not a multi-master setup. Writes always go to the primary
+region; the read replicas serve reads with bounded staleness.
+
+## Why this matters
+
+For tenants serving a global audience, the round-trip from a faraway
+client to the primary region is the dominant page-load cost. With
+read replicas in-region, the slow path is reduced to writes (which
+are rarer) and the public theme is served entirely from edge.
+
+For larger operators, multi-region is also the disaster-recovery
+story: a complete primary-region failure is recoverable by promoting
+a replica region.
+
+## Config flags
+
+All flags below live in the standard `packages/go/config` surface
+and read from environment variables.
+
+### Region identity
+
+| Flag                   | Default       | Notes                                                                  |
+| ---------------------- | ------------- | ---------------------------------------------------------------------- |
+| `GONEXT_REGION`        | `""`          | Required in v2. Short slug, e.g. `us-east`, `eu-west`. Used in logs.   |
+| `GONEXT_REGION_ROLE`   | `"primary"`   | One of `primary`, `replica`. Replicas refuse writes and surface 503.   |
+| `GONEXT_REGION_PEERS`  | `""`          | Comma-separated peer regions. Used for cache fan-out (see below).      |
+
+### Database read replicas
+
+| Flag                       | Default | Notes                                                                                              |
+| -------------------------- | ------- | -------------------------------------------------------------------------------------------------- |
+| `DATABASE_URL`             | -       | Primary, read-write. Required everywhere.                                                          |
+| `DATABASE_REPLICA_URL`     | `""`    | Optional. When set, read-only queries are routed here. Falls back to `DATABASE_URL` on error.      |
+| `DATABASE_REPLICA_MAX_LAG` | `5s`    | Reads served from a replica that's lagged > this duration fall back to the primary.                |
+
+In `GONEXT_REGION_ROLE=replica`, the API binary requires
+`DATABASE_REPLICA_URL`; otherwise reads of regional data would cross
+the WAN.
+
+### Redis cluster mode
+
+v1 uses a single Redis instance. v2 supports Redis Cluster for both
+sessions (sharded by token) and the Asynq queue (Asynq itself does
+not support cluster mode, so the queue Redis stays single-instance
+in the primary region).
+
+| Flag                  | Default            | Notes                                                                                       |
+| --------------------- | ------------------ | ------------------------------------------------------------------------------------------- |
+| `REDIS_URL`           | -                  | Cache + sessions. Optionally points at a cluster (`redis://host:6379?cluster=true`).        |
+| `REDIS_JOB_URL`       | inherits           | Asynq queue. Always single-instance, always in the primary region.                          |
+| `REDIS_CLUSTER_MODE`  | inferred from URL  | Force-override if the URL scheme is ambiguous.                                              |
+| `REDIS_REPLICA_READS` | `true`             | Whether to send reads to cluster replicas. Off for strong consistency.                      |
+
+### CDN headers
+
+The public theme is served behind a CDN. Multi-region deployments
+also serve the admin behind a CDN (cached `Vary: Cookie, Accept-Encoding`)
+so the initial document load is fast.
+
+| Header                | Set by                | Notes                                                                                  |
+| --------------------- | --------------------- | -------------------------------------------------------------------------------------- |
+| `Cache-Control`       | `apps/api`            | `public, max-age=60, s-maxage=300, stale-while-revalidate=86400` for theme reads.      |
+| `Surrogate-Key`       | `apps/api`            | `post:<id> author:<id> tag:<slug>`. Drives the targeted-invalidate fan-out below.      |
+| `Vary`                | `apps/api`            | `Accept, Accept-Encoding, Cookie` on theme; only `Accept-Encoding` on API JSON.        |
+| `X-GoNext-Region`     | `apps/api`            | Region slug, for client-side debug.                                                    |
+
+## Fan-out URLs
+
+When an editor publishes a post or otherwise mutates cache-relevant
+data, the API issues a **fan-out invalidation** to every peer
+region's CDN.
+
+### Targeted invalidate
+
+```
+POST https://cdn-<peer-region>.example.com/_invalidate
+Content-Type: application/json
+
+{
+  "surrogate_keys": ["post:42", "author:7"],
+  "issued_at": "2026-05-26T13:00:00Z",
+  "from_region": "us-east"
+}
+```
+
+- Each peer region acks within 2 s.
+- Failed fan-outs are retried via the Asynq `webhook` queue (already
+  shipped in v1).
+
+### Bulk invalidate
+
+For schema changes (theme reload, plugin install) that touch every
+cached surface, a bulk wildcard:
+
+```
+POST https://cdn-<peer-region>.example.com/_invalidate
+{ "surrogate_keys": ["*"], "from_region": "us-east" }
+```
+
+Used sparingly — clears every cache, expensive.
+
+## Failover
+
+Primary-region failure:
+
+1. **Promote a replica.** Postgres replica is promoted to primary;
+   `GONEXT_REGION_ROLE` is flipped to `primary` on that region's
+   replicas.
+2. **DNS cutover.** The write-path DNS record (`api-write.example.com`)
+   is repointed at the new primary.
+3. **Drain the queue Redis.** Asynq tasks queued in the old primary
+   region are lost (the queue Redis is single-instance per design).
+   Tenant-visible side effect: outbound webhooks pending at the
+   moment of failover may be skipped. Document this in the
+   compliance binder.
+4. **Reverse the replication.** The old primary, when it returns,
+   becomes a replica of the new primary.
+
+## Open questions
+
+- Cross-region session migration: today a session minted in
+  `us-east` is invisible to `eu-west` because each region has its
+  own Redis. The v2 cut needs either (a) global Redis-cluster
+  with cross-region replication, accepting the latency penalty, or
+  (b) signed session tokens that any region can validate without a
+  store lookup. Decision deferred to ADR (TBD).
+- Audit log replication: writes happen in the primary region, so
+  the canonical `audit_log` table lives there. Replica regions read
+  via the standard Postgres physical replication; no extra flags
+  needed. Confirm SOC 2 auditor is comfortable with this.
+- Media: today uploads land in object storage (S3-compatible) with
+  cross-region replication enabled at the provider level. The API
+  binary doesn't need to know — but the URL signing key must be
+  the same across regions.
@@ -0,0 +1,69 @@
+# Runbook: Audit log corruption
+
+> Status: The `audit_log` table is unreadable, has gaps, or its hash
+> chain has broken. This is a **compliance-relevant** incident — the
+> audit log is the source of truth for "who did what, when", and a
+> gap is a finding on every SOC 2 audit.
+
+## Symptom
+
+- `gonext_audit_emit_failures_total` is non-zero and rising.
+- `gonext audit tail` returns rows with `prev_hash != sha256(prev_row)`,
+  i.e. the tamper-evidence chain is broken.
+- Admins report "I revoked X yesterday but the audit log doesn't show
+  it."
+- Postgres errors: `ERROR: invalid page in block N of relation
+  audit_log`.
+- A migration that backfilled `audit_log` failed mid-way.
+
+## First 5 minutes
+
+1. **Stop the bleeding.** If audit writes are failing, every
+   privileged action is happening UNAUDITED. Either:
+   - flip the API to read-only via `OPS_READ_ONLY=true`, or
+   - point the audit emitter at an emergency in-memory buffer
+     (`AUDIT_EMERGENCY_BUFFER=true`) so writes are queued for replay
+     when Postgres recovers.
+2. **Page the security on-call.** This is their incident, not the
+   platform team's — the audit log is a security artifact.
+3. **Snapshot the current `audit_log`.** Before anyone runs a repair,
+   `pg_dump --table=audit_log` to a write-locked bucket. This is the
+   evidence chain.
+4. **Disable the sweeper.** The retention sweeper is going to make
+   the gap unrecoverable if it runs during the incident. Set
+   `AUDIT_SWEEPER_DISABLED=true` and roll API replicas.
+5. **Announce in #incidents AND #compliance.**
+
+## Mitigation
+
+- **If the corruption is at the storage level** (Postgres page
+  error): see also [`database-down.md`](./database-down.md). Restore
+  the table from the most recent backup that predates the corruption.
+- **If the chain broke because a row was manually edited:** the
+  table is now unauditable for everything after that row. Document
+  the gap (with row IDs) in the postmortem and treat it as a finding.
+  Do not attempt to "re-link" the chain by recomputing hashes — that
+  destroys the tamper evidence.
+- **If a backfill migration is the cause:** roll the migration back,
+  restore the pre-migration `audit_log` snapshot, replay the
+  emergency buffer if one was captured.
+- **Re-enable normal writes** only after security on-call signs off
+  that the chain is consistent again.
+
+## Escalation
+
+- **Security on-call:** primary owner. They drive comms with
+  compliance.
+- **DBA on-call:** for the Postgres-level repair.
+- **Customer success:** if any customer asks about an audit gap —
+  they need the canned response.
+
+## After-incident
+
+- Postmortem within 24 h (faster than normal because it's
+  compliance-relevant).
+- Append a "gap" record to the audit chain documenting the affected
+  time range and root cause.
+- Update the compliance binder with the incident reference.
+- If the cause was a code path that wrote to `audit_log` outside the
+  emitter, fix it and add a CI gate.
@@ -0,0 +1,68 @@
+# Runbook: Database down
+
+> Status: Postgres is unreachable. Most read and all write endpoints are
+> 5xx-ing. `/healthz` is still green on stateless replicas but `/readyz`
+> is red.
+
+## Symptom
+
+- `gonext_db_up == 0` for >= 60 s on every API replica.
+- A burst of `5xx` on `/api/v1/posts`, `/api/v1/users`, and every other
+  DB-backed route.
+- The admin UI shows the "API not available" banner on most surfaces.
+- Sessions still work (Redis), so signed-in admins reach the dashboard
+  but see "database unavailable" on every list view.
+- `pgx: failed to connect` lines in the API server logs.
+
+## First 5 minutes
+
+1. **Confirm scope.** `kubectl get pods -n postgres` (or the managed
+   provider's status page). Is the DB pod CrashLoopBackOff, or is the
+   network path between API and DB the problem?
+2. **Page the on-call DBA** if the DB itself is down. The API team
+   cannot fix a corrupted WAL — escalate immediately.
+3. **Confirm reads are degraded, not data loss.** Postgres replicas
+   should still answer reads via the connection-string fallback if
+   configured. Check `gonext_db_replica_up`.
+4. **Flip the read-only flag** in the API config if writes need to be
+   refused gracefully: `OPS_READ_ONLY=true` (graceful 503 with a clear
+   error body) instead of letting the DB driver time out on every
+   write.
+5. **Announce in #incidents** with the gonext_db_up dashboard panel
+   link and the current ETA.
+
+## Mitigation
+
+- **If primary is down + replicas are up:** promote a replica via the
+  managed provider's failover button. Update `DATABASE_URL` secret to
+  point at the new primary, roll API replicas.
+- **If the DB is up but unreachable from API:** check the network
+  policy (Kubernetes NetworkPolicy / security group). A recent
+  deployment may have tightened the egress rules — roll back the
+  network policy first, debug after.
+- **If WAL is corrupted or the disk filled:** see also
+  [`disk-full.md`](./disk-full.md). Bring the DB up read-only, take a
+  snapshot, then attempt repair.
+- **Worker drain:** the worker binary will keep retrying every job
+  that touches the DB. Either pause Asynq (`gonext jobs drain`
+  followed by re-enqueue when DB is back) or accept the retry
+  pressure on Redis.
+
+## Escalation
+
+- **DBA on-call:** for DB-level issues (WAL corruption, replication
+  lag, primary failover).
+- **Platform on-call:** for network reachability or managed-service
+  outages.
+- **CTO + customer success:** if downtime is > 15 min during business
+  hours.
+
+## After-incident
+
+- File a postmortem within 48 h. Required sections: timeline, root
+  cause, what worked, what didn't, action items with owners + due
+  dates.
+- If the cause was an upgrade or migration, update
+  `docs/operations/multi-region.md` with the lesson.
+- Add a Prometheus alert for the failure mode if one was missing —
+  every postmortem ships at least one new alert.