From abcb99173ef01074493a3e50597d70e61356eaf9 Mon Sep 17 00:00:00 2001 From: Mohamed Tayeb Mokni Date: Tue, 26 May 2026 13:18:18 +0200 Subject: [PATCH] docs(operations): add 7 incident runbooks + multi-region design + security checklist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #217, #220, #138. - docs/operations/runbooks/{database-down,redis-down,audit-corruption, plugin-trap-storm,mass-401,disk-full,oom}.md — symptom / first 5 min / mitigation / escalation / after-incident skeletons. - docs/operations/multi-region.md — v2 config flags (region role, replica URL, Redis cluster, CDN headers) + fan-out invalidation contract. - CONTRIBUTING.md — security review checklist (input validation, sanitization, auth, capability, rate limit, audit, CSRF, secrets, deps, error info leak). Co-Authored-By: Claude Opus 4.7 Signed-off-by: Mohamed Tayeb Mokni --- CONTRIBUTING.md | 67 ++++++++ docs/operations/multi-region.md | 155 ++++++++++++++++++ docs/operations/runbooks/audit-corruption.md | 69 ++++++++ docs/operations/runbooks/database-down.md | 68 ++++++++ docs/operations/runbooks/disk-full.md | 67 ++++++++ docs/operations/runbooks/mass-401.md | 66 ++++++++ docs/operations/runbooks/oom.md | 67 ++++++++ docs/operations/runbooks/plugin-trap-storm.md | 74 +++++++++ docs/operations/runbooks/redis-down.md | 68 ++++++++ 9 files changed, 701 insertions(+) create mode 100644 docs/operations/multi-region.md create mode 100644 docs/operations/runbooks/audit-corruption.md create mode 100644 docs/operations/runbooks/database-down.md create mode 100644 docs/operations/runbooks/disk-full.md create mode 100644 docs/operations/runbooks/mass-401.md create mode 100644 docs/operations/runbooks/oom.md create mode 100644 docs/operations/runbooks/plugin-trap-storm.md create mode 100644 docs/operations/runbooks/redis-down.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dfb74a43..a350c0ad 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -101,6 +101,73 @@ Good AI use looks like: you read the design doc, drafted with AI help, reviewed If you're not sure whether your contribution will be received well, open a `design-discussion` issue first and ask. +## Security review checklist + +Every PR that touches an authenticated endpoint, an admin surface, a +plugin host boundary, or a SQL query must walk through this checklist +before requesting review. Tick each item in the PR description or +explain why it doesn't apply. + +The reviewer is responsible for spot-checking the answers — items +left blank or hand-waved ("N/A" with no reason) block merge. + +- [ ] **Input validation.** Every request payload is validated against + a typed shape (Go struct + `validate` tags / TS Zod) before it + reaches the business logic. Unexpected fields are rejected by + `additionalProperties: false` at the schema layer, not silently + ignored. Required fields are checked explicitly. Length, range, + and format limits are enforced server-side regardless of any + client-side validation. +- [ ] **Sanitization.** Output that lands in HTML is escaped (React + does this by default; raw insertions via `dangerouslySetInnerHTML` + use `DOMPurify` from `apps/admin/src/components/SafeHTML.tsx`). + SQL is parametrized — no `fmt.Sprintf` into a query. Shell + invocations are avoided; when unavoidable, they use `exec.Command` + with an explicit argv (never `bash -c`). +- [ ] **Auth check.** The endpoint requires authentication (or has a + documented public-by-design reason). Mounted behind + `auth.RequireSession` or equivalent. Anonymous endpoints have a + one-line comment explaining why. +- [ ] **Capability gate.** Beyond authentication, every privileged + action goes through `policy.Can(...)` with a specific capability + constant. No role-string comparisons in the handler body. If a + new capability is introduced, it's added to + `packages/go/policy/capabilities.go` and to the default role + mapping in `defaults.go`. +- [ ] **Rate limit.** Login, password reset, email verification, + and any high-value mutation surface go through the limiter from + `packages/go/ratelimit`. If a new bucket is added, the default + policy is documented and conservative (fail-closed when Redis is + unavailable, except where the doc explicitly approves fail-open). +- [ ] **Audit emit.** Every privileged action emits an + `audit.Event`. Event type uses the dotted convention + (`auth.login.success`, `plugin.activated`). Severity matches the + matrix in `docs/06-auth-permissions.md` §13. Metadata is bounded + (no caller-controlled blobs). +- [ ] **CSRF.** State-changing endpoints (POST/PUT/PATCH/DELETE) are + same-origin via session cookies + SameSite=Lax, OR they require + a CSRF token from `auth/csrf`. The reviewer confirms the + middleware is in the chain. +- [ ] **Secret handling.** No credentials, tokens, or PII in log + lines. `slog` attrs use the structured form so the redactor in + `packages/go/log` can mask sensitive keys. Secrets read from + config use the masked accessor (`cfg.Auth.Pepper` is masked when + the config is dumped). +- [ ] **Dependency vulns.** New direct deps are checked against + `osv-scanner` / `pnpm audit`. Transitive vulns are noted in the + PR description with a remediation plan (or a documented accept). + CI runs the scanners on every PR, but humans should still look at + the manifest diff. +- [ ] **Error message info leak.** Error responses don't echo + internal paths, SQL fragments, or stack traces. The error code + is a short slug (`not_found`, `internal_error`); the human + message is generic. Detailed diagnostics live in the structured + log, never in the response body. + +If your change is a pure documentation update, a comment-only refactor, +or a test-only patch, this section can be skipped — say so in the PR +description. + ## Reporting bugs Open an issue with the `bug` template. Include: diff --git a/docs/operations/multi-region.md b/docs/operations/multi-region.md new file mode 100644 index 00000000..5a1edd3d --- /dev/null +++ b/docs/operations/multi-region.md @@ -0,0 +1,155 @@ +# Multi-region deployment (v2) + +> Status: **design only**. This doc captures the config flags and +> fan-out URLs the v2 multi-region surface will read from. The v1 +> binary ships a single-region wiring; the flags here are reserved so +> the v2 cut can land without breaking existing operators' configs. +> +> Tracked in issue #138. Implementation issues fan out from this +> document. + +## Scope + +A v2 multi-region GoNext deployment runs: + +- one **primary region** owning the writable Postgres + Redis + primary, +- one or more **read replica regions** with Postgres physical replicas + + Redis cluster nodes, +- a global CDN fronting the public theme, +- a global anycast LB fronting the admin and API surfaces. + +This is not a multi-master setup. Writes always go to the primary +region; the read replicas serve reads with bounded staleness. + +## Why this matters + +For tenants serving a global audience, the round-trip from a faraway +client to the primary region is the dominant page-load cost. With +read replicas in-region, the slow path is reduced to writes (which +are rarer) and the public theme is served entirely from edge. + +For larger operators, multi-region is also the disaster-recovery +story: a complete primary-region failure is recoverable by promoting +a replica region. + +## Config flags + +All flags below live in the standard `packages/go/config` surface +and read from environment variables. + +### Region identity + +| Flag | Default | Notes | +| ---------------------- | ------------- | ---------------------------------------------------------------------- | +| `GONEXT_REGION` | `""` | Required in v2. Short slug, e.g. `us-east`, `eu-west`. Used in logs. | +| `GONEXT_REGION_ROLE` | `"primary"` | One of `primary`, `replica`. Replicas refuse writes and surface 503. | +| `GONEXT_REGION_PEERS` | `""` | Comma-separated peer regions. Used for cache fan-out (see below). | + +### Database read replicas + +| Flag | Default | Notes | +| -------------------------- | ------- | -------------------------------------------------------------------------------------------------- | +| `DATABASE_URL` | - | Primary, read-write. Required everywhere. | +| `DATABASE_REPLICA_URL` | `""` | Optional. When set, read-only queries are routed here. Falls back to `DATABASE_URL` on error. | +| `DATABASE_REPLICA_MAX_LAG` | `5s` | Reads served from a replica that's lagged > this duration fall back to the primary. | + +In `GONEXT_REGION_ROLE=replica`, the API binary requires +`DATABASE_REPLICA_URL`; otherwise reads of regional data would cross +the WAN. + +### Redis cluster mode + +v1 uses a single Redis instance. v2 supports Redis Cluster for both +sessions (sharded by token) and the Asynq queue (Asynq itself does +not support cluster mode, so the queue Redis stays single-instance +in the primary region). + +| Flag | Default | Notes | +| --------------------- | ------------------ | ------------------------------------------------------------------------------------------- | +| `REDIS_URL` | - | Cache + sessions. Optionally points at a cluster (`redis://host:6379?cluster=true`). | +| `REDIS_JOB_URL` | inherits | Asynq queue. Always single-instance, always in the primary region. | +| `REDIS_CLUSTER_MODE` | inferred from URL | Force-override if the URL scheme is ambiguous. | +| `REDIS_REPLICA_READS` | `true` | Whether to send reads to cluster replicas. Off for strong consistency. | + +### CDN headers + +The public theme is served behind a CDN. Multi-region deployments +also serve the admin behind a CDN (cached `Vary: Cookie, Accept-Encoding`) +so the initial document load is fast. + +| Header | Set by | Notes | +| --------------------- | --------------------- | -------------------------------------------------------------------------------------- | +| `Cache-Control` | `apps/api` | `public, max-age=60, s-maxage=300, stale-while-revalidate=86400` for theme reads. | +| `Surrogate-Key` | `apps/api` | `post: author: tag:`. Drives the targeted-invalidate fan-out below. | +| `Vary` | `apps/api` | `Accept, Accept-Encoding, Cookie` on theme; only `Accept-Encoding` on API JSON. | +| `X-GoNext-Region` | `apps/api` | Region slug, for client-side debug. | + +## Fan-out URLs + +When an editor publishes a post or otherwise mutates cache-relevant +data, the API issues a **fan-out invalidation** to every peer +region's CDN. + +### Targeted invalidate + +``` +POST https://cdn-.example.com/_invalidate +Content-Type: application/json + +{ + "surrogate_keys": ["post:42", "author:7"], + "issued_at": "2026-05-26T13:00:00Z", + "from_region": "us-east" +} +``` + +- Each peer region acks within 2 s. +- Failed fan-outs are retried via the Asynq `webhook` queue (already + shipped in v1). + +### Bulk invalidate + +For schema changes (theme reload, plugin install) that touch every +cached surface, a bulk wildcard: + +``` +POST https://cdn-.example.com/_invalidate +{ "surrogate_keys": ["*"], "from_region": "us-east" } +``` + +Used sparingly — clears every cache, expensive. + +## Failover + +Primary-region failure: + +1. **Promote a replica.** Postgres replica is promoted to primary; + `GONEXT_REGION_ROLE` is flipped to `primary` on that region's + replicas. +2. **DNS cutover.** The write-path DNS record (`api-write.example.com`) + is repointed at the new primary. +3. **Drain the queue Redis.** Asynq tasks queued in the old primary + region are lost (the queue Redis is single-instance per design). + Tenant-visible side effect: outbound webhooks pending at the + moment of failover may be skipped. Document this in the + compliance binder. +4. **Reverse the replication.** The old primary, when it returns, + becomes a replica of the new primary. + +## Open questions + +- Cross-region session migration: today a session minted in + `us-east` is invisible to `eu-west` because each region has its + own Redis. The v2 cut needs either (a) global Redis-cluster + with cross-region replication, accepting the latency penalty, or + (b) signed session tokens that any region can validate without a + store lookup. Decision deferred to ADR (TBD). +- Audit log replication: writes happen in the primary region, so + the canonical `audit_log` table lives there. Replica regions read + via the standard Postgres physical replication; no extra flags + needed. Confirm SOC 2 auditor is comfortable with this. +- Media: today uploads land in object storage (S3-compatible) with + cross-region replication enabled at the provider level. The API + binary doesn't need to know — but the URL signing key must be + the same across regions. diff --git a/docs/operations/runbooks/audit-corruption.md b/docs/operations/runbooks/audit-corruption.md new file mode 100644 index 00000000..cfe44158 --- /dev/null +++ b/docs/operations/runbooks/audit-corruption.md @@ -0,0 +1,69 @@ +# Runbook: Audit log corruption + +> Status: The `audit_log` table is unreadable, has gaps, or its hash +> chain has broken. This is a **compliance-relevant** incident — the +> audit log is the source of truth for "who did what, when", and a +> gap is a finding on every SOC 2 audit. + +## Symptom + +- `gonext_audit_emit_failures_total` is non-zero and rising. +- `gonext audit tail` returns rows with `prev_hash != sha256(prev_row)`, + i.e. the tamper-evidence chain is broken. +- Admins report "I revoked X yesterday but the audit log doesn't show + it." +- Postgres errors: `ERROR: invalid page in block N of relation + audit_log`. +- A migration that backfilled `audit_log` failed mid-way. + +## First 5 minutes + +1. **Stop the bleeding.** If audit writes are failing, every + privileged action is happening UNAUDITED. Either: + - flip the API to read-only via `OPS_READ_ONLY=true`, or + - point the audit emitter at an emergency in-memory buffer + (`AUDIT_EMERGENCY_BUFFER=true`) so writes are queued for replay + when Postgres recovers. +2. **Page the security on-call.** This is their incident, not the + platform team's — the audit log is a security artifact. +3. **Snapshot the current `audit_log`.** Before anyone runs a repair, + `pg_dump --table=audit_log` to a write-locked bucket. This is the + evidence chain. +4. **Disable the sweeper.** The retention sweeper is going to make + the gap unrecoverable if it runs during the incident. Set + `AUDIT_SWEEPER_DISABLED=true` and roll API replicas. +5. **Announce in #incidents AND #compliance.** + +## Mitigation + +- **If the corruption is at the storage level** (Postgres page + error): see also [`database-down.md`](./database-down.md). Restore + the table from the most recent backup that predates the corruption. +- **If the chain broke because a row was manually edited:** the + table is now unauditable for everything after that row. Document + the gap (with row IDs) in the postmortem and treat it as a finding. + Do not attempt to "re-link" the chain by recomputing hashes — that + destroys the tamper evidence. +- **If a backfill migration is the cause:** roll the migration back, + restore the pre-migration `audit_log` snapshot, replay the + emergency buffer if one was captured. +- **Re-enable normal writes** only after security on-call signs off + that the chain is consistent again. + +## Escalation + +- **Security on-call:** primary owner. They drive comms with + compliance. +- **DBA on-call:** for the Postgres-level repair. +- **Customer success:** if any customer asks about an audit gap — + they need the canned response. + +## After-incident + +- Postmortem within 24 h (faster than normal because it's + compliance-relevant). +- Append a "gap" record to the audit chain documenting the affected + time range and root cause. +- Update the compliance binder with the incident reference. +- If the cause was a code path that wrote to `audit_log` outside the + emitter, fix it and add a CI gate. diff --git a/docs/operations/runbooks/database-down.md b/docs/operations/runbooks/database-down.md new file mode 100644 index 00000000..d39713f8 --- /dev/null +++ b/docs/operations/runbooks/database-down.md @@ -0,0 +1,68 @@ +# Runbook: Database down + +> Status: Postgres is unreachable. Most read and all write endpoints are +> 5xx-ing. `/healthz` is still green on stateless replicas but `/readyz` +> is red. + +## Symptom + +- `gonext_db_up == 0` for >= 60 s on every API replica. +- A burst of `5xx` on `/api/v1/posts`, `/api/v1/users`, and every other + DB-backed route. +- The admin UI shows the "API not available" banner on most surfaces. +- Sessions still work (Redis), so signed-in admins reach the dashboard + but see "database unavailable" on every list view. +- `pgx: failed to connect` lines in the API server logs. + +## First 5 minutes + +1. **Confirm scope.** `kubectl get pods -n postgres` (or the managed + provider's status page). Is the DB pod CrashLoopBackOff, or is the + network path between API and DB the problem? +2. **Page the on-call DBA** if the DB itself is down. The API team + cannot fix a corrupted WAL — escalate immediately. +3. **Confirm reads are degraded, not data loss.** Postgres replicas + should still answer reads via the connection-string fallback if + configured. Check `gonext_db_replica_up`. +4. **Flip the read-only flag** in the API config if writes need to be + refused gracefully: `OPS_READ_ONLY=true` (graceful 503 with a clear + error body) instead of letting the DB driver time out on every + write. +5. **Announce in #incidents** with the gonext_db_up dashboard panel + link and the current ETA. + +## Mitigation + +- **If primary is down + replicas are up:** promote a replica via the + managed provider's failover button. Update `DATABASE_URL` secret to + point at the new primary, roll API replicas. +- **If the DB is up but unreachable from API:** check the network + policy (Kubernetes NetworkPolicy / security group). A recent + deployment may have tightened the egress rules — roll back the + network policy first, debug after. +- **If WAL is corrupted or the disk filled:** see also + [`disk-full.md`](./disk-full.md). Bring the DB up read-only, take a + snapshot, then attempt repair. +- **Worker drain:** the worker binary will keep retrying every job + that touches the DB. Either pause Asynq (`gonext jobs drain` + followed by re-enqueue when DB is back) or accept the retry + pressure on Redis. + +## Escalation + +- **DBA on-call:** for DB-level issues (WAL corruption, replication + lag, primary failover). +- **Platform on-call:** for network reachability or managed-service + outages. +- **CTO + customer success:** if downtime is > 15 min during business + hours. + +## After-incident + +- File a postmortem within 48 h. Required sections: timeline, root + cause, what worked, what didn't, action items with owners + due + dates. +- If the cause was an upgrade or migration, update + `docs/operations/multi-region.md` with the lesson. +- Add a Prometheus alert for the failure mode if one was missing — + every postmortem ships at least one new alert. diff --git a/docs/operations/runbooks/disk-full.md b/docs/operations/runbooks/disk-full.md new file mode 100644 index 00000000..6137f221 --- /dev/null +++ b/docs/operations/runbooks/disk-full.md @@ -0,0 +1,67 @@ +# Runbook: Disk full + +> Status: A node (DB, Redis, media volume, API replica) has hit > 95% +> disk usage. Postgres refuses writes when its volume is full; media +> uploads fail; logs stop rotating. + +## Symptom + +- `node_filesystem_avail_bytes{mountpoint=...} / node_filesystem_size_bytes` < 0.05. +- Postgres logs: `ERROR: could not extend file "base/...": No space + left on device`. +- Media uploads return `507 Insufficient Storage`. +- The API server log file stops growing (logs are being dropped at + the journald layer). +- Recently: a runaway audit-log table, an Asynq dead-letter queue + that nobody drained, or media uploads with no retention. + +## First 5 minutes + +1. **Identify the affected volume.** `df -h` (or the cloud provider's + disk dashboard). Which mountpoint is full? +2. **Identify the largest consumer.** `du -sh /* | sort -h` on the + affected node. The usual suspects: + - `/var/lib/postgresql/data/base/...` — Postgres data. + - `/var/lib/redis` — Redis AOF/RDB. + - `/srv/media` — uploaded files. + - `/var/log` — log files that didn't rotate. +3. **Free space immediately.** Different tactics per cause: + - Postgres: `VACUUM FULL` is expensive but reclaims space; + truncate the `audit_log` to the last 30 days if it's the + bloat source (see retention policy in + [`audit-corruption.md`](./audit-corruption.md) before doing + this in haste). + - Redis: `BGREWRITEAOF` to compact the AOF; check + `redis-cli MEMORY DOCTOR` for outliers. + - Media: drop unreferenced files (use `gonext media gc`). + - Logs: `logrotate -f` or manually truncate old archives. +4. **Page the platform on-call.** Disk full is a platform + responsibility. +5. **Announce in #incidents** with the mountpoint, % used, and the + ETA for free space. + +## Mitigation + +- **Expand the volume.** Most managed providers support online + expansion; this is the fastest path. +- **Migrate to a larger node.** Slower (requires data migration), + but a permanent fix if the volume is at its provider-side ceiling. +- **Tighten retention.** If the bloat is `audit_log` or job history, + the retention policy was probably too generous. Tune the + retention windows in `/settings/privacy` (#225). +- **Prevent recurrence:** alert at 80% disk usage, not 95%. Add the + alert to `docs/operations/alerts.md` if missing. + +## Escalation + +- **Platform on-call:** primary owner. +- **DBA on-call:** if Postgres is the affected service. +- **CTO:** if the only mitigation is "wait for a larger node, which + takes hours" and impact is customer-visible. + +## After-incident + +- Postmortem within 48 h. Mandatory section: "Why didn't the 80% + alert fire?" (Was it missing? Misrouted? Acknowledged-and-forgotten?) +- Add or fix the alert. +- Set a calendar reminder to review disk growth quarterly. diff --git a/docs/operations/runbooks/mass-401.md b/docs/operations/runbooks/mass-401.md new file mode 100644 index 00000000..1588666e --- /dev/null +++ b/docs/operations/runbooks/mass-401.md @@ -0,0 +1,66 @@ +# Runbook: Mass 401s + +> Status: Every authenticated request is getting `401 Unauthorized`. +> Either the session store is broken, the cookie domain is misconfigured, +> or a signing-key rotation went wrong. + +## Symptom + +- `gonext_http_requests_total{status="401"}` spikes from baseline + (< 1%) to > 30% over a 1-2 minute window. +- Admin users report "I keep getting logged out." +- The admin UI shows the login screen even right after a successful + login (the cookie comes back but the next request rejects it). +- `auth.session.not_found` audit events spike. + +## First 5 minutes + +1. **Confirm scope.** Is it ALL users (likely a session store or + cookie-domain problem) or just some (likely a per-user state + issue)? +2. **Check Redis health.** See [`redis-down.md`](./redis-down.md) — + a Redis outage manifests as mass 401 because every session lookup + fails. +3. **Check the cookie config.** A recent deploy that changed + `Cookie-Domain`, `SameSite`, or `Secure` flag will silently + invalidate every existing session. Compare the cookie attributes + on the current response to a known-good capture from earlier in + the day. +4. **Check for signing-key rotation.** If the auth layer uses HMAC + over the session token (it does, per docs/06-auth-permissions.md + §5), rotating the signing key without a grace window invalidates + every session. Look for a recent secret rotation in the audit log. +5. **Page the auth on-call.** + +## Mitigation + +- **If Redis is the cause:** follow [`redis-down.md`](./redis-down.md). +- **If a cookie attribute change is the cause:** revert the deploy. + Existing sessions become valid again immediately. +- **If a key rotation is the cause:** restore the previous signing + key alongside the new one (the auth layer supports a key-set so + you can have both active during rotation). Sessions minted under + the old key validate again. Roll the new key out the next deploy + with the key-set logic in place. +- **Worst case — accept the session loss:** announce "you may need + to log in again" via the in-app banner, let users re-authenticate. + Only acceptable if the root cause is fixed and recurrence is + unlikely. + +## Escalation + +- **Auth on-call:** primary owner. +- **Platform on-call:** if the cause is upstream (LB, WAF) stripping + cookies. +- **Customer success:** if the impact lasts > 10 min during business + hours. + +## After-incident + +- Postmortem within 48 h. +- If the cause was a config change, add a CI check that flags + changes to the cookie envelope as a "session-invalidating change" + requiring two reviewers. +- If the cause was a key rotation without grace window, document + the rotation procedure in `docs/operations/multi-region.md` (and + in the runbook for the rotation itself). diff --git a/docs/operations/runbooks/oom.md b/docs/operations/runbooks/oom.md new file mode 100644 index 00000000..ed8ef768 --- /dev/null +++ b/docs/operations/runbooks/oom.md @@ -0,0 +1,67 @@ +# Runbook: OOM + +> Status: One or more API or worker replicas is being killed by the +> kernel OOM-killer. The pod restarts, holds for a few minutes, then +> dies again. + +## Symptom + +- `kube_pod_container_status_restarts_total{pod="api-..."}` is + rising. +- `kubectl describe pod api-...` shows `OOMKilled` as the last + termination reason. +- API p99 latency spikes during each restart cycle. +- `dmesg` on the node shows the OOM-killer killing the gonext-api + process. +- The Go runtime's heap (`go_memstats_heap_inuse_bytes`) is at or + near the container's memory limit. + +## First 5 minutes + +1. **Confirm scope.** Are all replicas OOMing (a code / load issue), + or a single replica (a bad node)? +2. **If single replica:** drain it. The replacement pod gets fresh + memory; the bad node's memory issues are someone else's problem + (or a flaky DIMM). +3. **If all replicas:** the binary is using more memory than the + container allows. Two possible causes: + - A recent deploy increased the working set (new feature, larger + cache, leak). + - Load has grown organically and the limit hasn't. +4. **Bump the container memory limit by 50%** as a temporary measure. + This buys time to diagnose without an outage. +5. **Page the platform on-call.** + +## Mitigation + +- **If a recent deploy is the cause:** roll back. The memory + regression is the bug; fix it on a follow-up, not at 3 AM. +- **If organic load is the cause:** horizontally scale (add + replicas) before vertically scaling (more memory per pod). The + Go runtime is happiest with smaller, replicated instances. +- **Profile the leak.** `go tool pprof + $API/debug/pprof/heap` (exposed only on internal `:9090`, never + the public port). Look for an unbounded slice in a hot path. +- **Tune GOMEMLIMIT.** The Go runtime is more aggressive about + collection when GOMEMLIMIT is set near the container limit; this + reduces the chance of an OOM even at the same working set. +- **Worker binary specifically:** check `gonext jobs queue` for a + task type with abnormal payload size. One bad job that pulls a + multi-megabyte blob into memory will OOM a worker. + +## Escalation + +- **Platform on-call:** primary owner. +- **Owner of the suspected code change:** to triage the regression. +- **CTO:** if the OOM cycle continues for > 30 min and customer + impact is sustained. + +## After-incident + +- Postmortem within 48 h. +- Capture the heap profile from the affected replica and attach it + to the postmortem. +- If the cause was a leak, file a bug with the profile attached + and assign to the relevant team. +- Update the deploy checklist: "did this change increase the + working set? confirm with a benchmark before rollout." diff --git a/docs/operations/runbooks/plugin-trap-storm.md b/docs/operations/runbooks/plugin-trap-storm.md new file mode 100644 index 00000000..8c57b1d3 --- /dev/null +++ b/docs/operations/runbooks/plugin-trap-storm.md @@ -0,0 +1,74 @@ +# Runbook: Plugin trap storm + +> Status: A plugin is repeatedly trapping (out-of-fuel, memory limit, +> ABI violation), generating thousands of trap events per minute and +> pushing the host into a degraded state. + +## Symptom + +- `gonext_plugin_traps_total{slug=...}` is rising fast (> 50/s). +- One plugin slug dominates the rate. +- API p99 latency on hook-running endpoints is up 2-5x. +- The audit log is filling with `plugin.trapped` events from a single + plugin. +- The Asynq `plugin` queue is backing up — every job from the + offending slug fails immediately. + +## First 5 minutes + +1. **Identify the offender.** `gonext_plugin_traps_total` has a + `slug` label. The dominant value is the offender. +2. **Deactivate the plugin via API** rather than CLI — the API path + audits the action, which the CLI does not: + ```bash + curl -X POST $API/api/v1/plugins//deactivate \ + -H "Cookie: sid=$SID" + ``` + If the API itself is degraded enough that the call won't go + through, fall back to `gonext plugin deactivate ` and emit + the audit event manually. +3. **Confirm the rate drops.** `gonext_plugin_traps_total{slug=...}` + should stop rising within 30 s of deactivation. +4. **Drain the Asynq plugin queue's failed bucket** so we don't + replay the offending jobs on the next restart: + `gonext jobs failed --queue=plugin --slug= --drain`. +5. **Announce in #incidents** with the slug, version, and approximate + number of traps observed. + +## Mitigation + +- **Plugin author needs notification.** Open a sev-2 issue against + the plugin's repo with the trap rate, version, and a sample stack + trace. +- **Quarantine the plugin** in the marketplace so other operators + don't install the broken version. Marketplace API: + `PATCH /api/v1/marketplace/plugins/` with `quarantined: true`. +- **Investigate the trap kind.** OOM and out-of-fuel are usually + bad plugin code; ABI violations are usually a host/plugin version + skew. Pull the actual trap message from a recent audit row. +- **Rollback to the previous plugin version** if the operator has + the bundle in their plugin storage and the previous version was + stable. The lifecycle manager supports + `gonext plugin rollback ` (issue #271, ship date dependent). + +## Escalation + +- **Plugin SRE:** primary owner. They drive the plugin author + conversation and the marketplace quarantine. +- **Platform on-call:** if the trap storm is degrading the host + binary (memory pressure, CPU saturation), they may need to + oversize the API replicas temporarily. +- **Customer success:** if the affected plugin is in wide use, they + need the canned response. + +## After-incident + +- Postmortem within 48 h. +- Add a Prometheus alert on + `rate(gonext_plugin_traps_total{slug=...}[1m]) > 10` if one wasn't + already firing. +- Update the plugin acceptance criteria (docs/02-plugin-system.md + §11) if the failure mode wasn't covered by the existing trap + budget. +- Consider raising the per-plugin fuel cap if the trap was legitimate + work — and lowering it if it was abuse. diff --git a/docs/operations/runbooks/redis-down.md b/docs/operations/runbooks/redis-down.md new file mode 100644 index 00000000..275075fb --- /dev/null +++ b/docs/operations/runbooks/redis-down.md @@ -0,0 +1,68 @@ +# Runbook: Redis down + +> Status: Redis is unreachable. Sessions, rate limits, and the job +> queue are all impacted. Postgres is fine, so cached reads degrade +> but the canonical data is intact. + +## Symptom + +- `gonext_redis_up == 0` for >= 60 s. +- Logged-in admins are bounced to `/login` because session lookups + fail (cookies present, but the manager returns `ErrNotFound`). +- Login attempts fail closed — the rate-limiter cannot count buckets + so the login handler returns 503 rather than risk an unrate-limited + surface. +- Asynq workers fall over with `dial tcp: connection refused`. +- `gonext_jobs_inflight` drops to 0 and stays there. + +## First 5 minutes + +1. **Confirm scope.** Managed Redis status page, or + `kubectl get pods -n redis`. Is it the pod, the network, or the + client config? +2. **Check session blast radius.** Every signed-in admin is logged + out the moment their request lands on a node that can't reach + Redis. Public traffic (theme reads, RSS) is unaffected. +3. **Page the on-call platform engineer.** Redis is shared between + sessions, rate limits, and the queue — a single-node outage + affects three independent subsystems. +4. **Pause the worker binary.** `kubectl scale deploy/worker + --replicas=0` so we don't generate Asynq reconnection storm noise + while Redis recovers. +5. **Announce in #incidents** with the impact summary (sessions out, + queue paused, logins refused). + +## Mitigation + +- **Failover to a Redis replica** if the cluster is configured for + HA. Update `REDIS_URL` secret to the new primary, roll API + worker + replicas. +- **If sessions need to come back ASAP and Redis is unrecoverable:** + point `REDIS_URL` at a fresh empty Redis. Existing sessions are + lost (everyone re-authenticates), but the admin is usable. Coordinate + with comms before doing this — it's user-visible. +- **If only the queue is needed:** stand up a temporary Redis for + Asynq while the original recovers, dual-write nothing (Asynq is + stateful — running two clusters simultaneously corrupts task + ordering). +- **Rate-limit fallback:** the limiter falls open during a Redis + outage, which means an attacker hitting `/api/v1/auth/login` is + unbounded. Mitigate via the WAF or cloud-side IP rate limit + until Redis returns. + +## Escalation + +- **Platform on-call:** primary owner. Redis infra is theirs. +- **Security on-call:** if the outage exceeds 10 min during business + hours — they need to know the rate-limit gate is open. +- **CTO:** if customer-visible session loss exceeds 30 min. + +## After-incident + +- Postmortem within 48 h. +- Capture how many users were logged out (sum of `Set-Cookie: + sid=; Max-Age=0` from the lb logs). +- Add a Prometheus alert for `gonext_redis_up == 0` for > 30 s if + one wasn't already firing. +- If the cause was an upgrade, document the lesson in the deploy + checklist.