diff --git a/docs/README.md b/docs/README.md index 78f24d2..fc51ccd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,7 @@ Every document listed here is maintained and describes the system as it currentl | [CONTRIBUTING](../CONTRIBUTING.md) | Contributors | The contribution rules: fork-first workflow, claiming an issue, branch naming, rebasing, pull requests, Definition of Ready and Done. Read before opening a PR. | | [SECURITY](../SECURITY.md) | Anyone reporting a vulnerability | How to disclose privately. Never open a public issue for a vulnerability. | | [AI provider egress policy](security/AI_PROVIDER_EGRESS.md) | Operators and backend contributors | Deployment-owned provider destination policy, DNS pinning, redirect handling, safe defaults, Compose boundary, and required production network controls. | +| [Self-hosted operations runbook](operations/SELF_HOSTED.md) | Operators running the Docker Compose stack | Required environment variables, security boundaries, unsupported deployment modes, and backup/restore expectations for the self-hosted stack. | | [CODE_OF_CONDUCT](../CODE_OF_CONDUCT.md) | Everyone | Expected conduct and how to report a violation. | | [System Overview](architecture/SYSTEM_OVERVIEW.md) | Contributors and maintainers | Current components, ingestion flow, persistence, consumers, trust boundaries, and architectural limitations. | | [Repository Intelligence](architecture/REPOSITORY_INTELLIGENCE.md) | Anyone changing analysis behaviour | What is extracted, what is deterministic versus heuristic, how facts are persisted, who consumes them, what consumers must not do, and where evidence and provenance stop. **Read this before touching analysis.** | diff --git a/docs/operations/SELF_HOSTED.md b/docs/operations/SELF_HOSTED.md new file mode 100644 index 0000000..371bbb8 --- /dev/null +++ b/docs/operations/SELF_HOSTED.md @@ -0,0 +1,122 @@ +# Self-hosted operations runbook + +This document describes running PARTHA's documented Docker Compose stack +(API + PostgreSQL + Redis) as it exists today. It is operational guidance, not +a claim of production hardening — read [Limitations and security](../../README.md#limitations-and-security) +first. + +## One command sequence + +From a clean checkout, with Docker and Docker Compose installed: + +```bash +npm run docker:config # validate the Compose file resolves +npm run docker:up # build and start api + postgres + redis +npm run docker:validate # full clean-clone acceptance gate (see below); tears the stack down when done +``` + +`npm run docker:validate` (`scripts/validate-compose.mjs`) is the repeatable +acceptance gate for this stack. From a clean checkout it: brings the stack up, +waits for `/ready`, re-runs and verifies Alembic migrations (including a +downgrade/upgrade round trip and a percent-encoded `DATABASE_URL` password — +the [#110](https://github.com/Second-Origin/PARTHA/issues/110) regression), +registers two disposable users and proves owner isolation (a non-owner gets +`404`, never `403`, on another user's repository), runs a full analysis to a +sealed `ri.v1` snapshot, restarts the `api` container and confirms the +repository, analysis result, and sealed snapshot all survive with no orphaned +job rows, proves a `pg_dump`/`pg_restore` round trip preserves a sealed +snapshot, and checks `docker compose logs` for the secrets it created. It +always tears the stack down (`docker compose down -v`), and exits non-zero on +any assertion failure. + +To bring the stack down without running the gate: `docker compose down -v`. + +## Required environment variables + +Outside `development`/`test`, the backend refuses to start unless these are +set (see `apps/backend/.env.example` for the full list with generation +commands): + +| Variable | Requirement | +| --- | --- | +| `AUTH_SECRET_KEY` | Required, at least 32 characters. Signs access tokens. | +| `AI_ENCRYPTION_KEY` | Required, a valid Fernet key. Encrypts each user's AI provider API key at rest. | +| `DATABASE_URL` | `sqlite:///…`, `postgresql://…`, or `postgresql+psycopg://…`. Compose sets this from `POSTGRES_USER`/`POSTGRES_PASSWORD`/`POSTGRES_DB`. A percent-encoded password (e.g. a literal `@` written as `%40`) is supported end-to-end, including by Alembic. | +| `REDIS_URL` | `redis://` or `rediss://`. Backs shared rate-limit budgets across API workers; Compose runs Redis for this. | +| `AI_EGRESS_MODE` | `hosted` (default, fail-safe) or `self_hosted`. See below. | +| `CORS_ORIGINS` | At least one valid `http(s)://` origin. No default that "just works" in production. | +| `AUTO_CREATE_TABLES` | Set `false` outside development/test so Alembic migrations are the sole source of schema truth (Compose already sets this). | + +Optional, with working defaults: `AI_EGRESS_ALLOWED_BASE_URLS`, +`AI_EGRESS_ALLOWED_CIDRS` (only meaningful when `AI_EGRESS_MODE=self_hosted`), +`RATE_LIMIT_*`, `ACCESS_TOKEN_TTL_SECONDS`, `REFRESH_TOKEN_TTL_SECONDS`, +`LOG_LEVEL`, `LOG_FORMAT`, `STORAGE_PATH`. + +Compose-only, not read directly by the backend: `POSTGRES_USER`, +`POSTGRES_PASSWORD`, `POSTGRES_DB` (feed `DATABASE_URL`). + +## Security boundaries + +- **Authentication and owner isolation.** Every non-auth route requires a + bearer access token; every repository, analysis, and intelligence resource + is owner-scoped. A request for another owner's resource returns `404`, not + `403` — it must not disclose that the resource exists. +- **Provider credentials.** Each user's AI provider API key is encrypted at + rest with `AI_ENCRYPTION_KEY` (Fernet). It is never logged or returned in a + response body. +- **AI egress allowlist and DNS pinning.** Outbound AI provider requests are + restricted to a deployment-owned allowlist with DNS pinning against + redirect and rebinding attacks. Read the + [AI provider egress policy](../security/AI_PROVIDER_EGRESS.md) in full + before configuring a custom or local (e.g. Ollama) provider endpoint — + a local/internal endpoint requires `AI_EGRESS_MODE=self_hosted` plus an + explicit administrator-owned base URL and CIDR allowlist; tenants cannot + widen this themselves. +- **Do not expose the development configuration to the public internet.** + The default Compose stack is local-development shaped (published + `5432`/`6379` ports, no TLS termination, no reverse proxy, no rate-limit + tuning for hostile traffic). Treat it as a trusted-network deployment + behind your own network boundary, not an internet-facing service. +- Report vulnerabilities privately per [SECURITY.md](../../SECURITY.md) — + never in a public issue. + +## Unsupported deployment modes + +PARTHA today is **pre-alpha, trusted-environment software**. The following +are explicitly not implemented or not hardened, and must not be assumed: + +- Public, multi-tenant exposure of a single deployment to untrusted users. +- Private-repository secure connect with per-connection permissions and + revocation — only public GitHub import and archive upload exist today. +- A separate analysis worker service or external job queue — analysis runs + on an in-process daemon thread inside the API, one job at a time. +- Organization/workspace membership, roles, or audit logs. +- Self-service account recovery. +- Automated backup scheduling, retention policy, or a backup/restore + operational runbook beyond the manual procedure below. +- Quotas, abuse controls, or production-grade monitoring beyond the + structured JSON logs and `/ready`/`/health` endpoints. + +## Backup and restore expectations + +Sealed `ri.v1` repository-intelligence snapshots (`ri_snapshots`, `ri_nodes`, +`ri_edges`, `ri_assertions`, `ri_observations`, `ri_evidence`, +`ri_derivations`, `ri_diagnostics`) are stored entirely in PostgreSQL, the +same as every other durable record (`repositories`, `analysis_jobs`, `users`). +There is no separate blob store or external snapshot artifact to back up. + +A standard `pg_dump`/`pg_restore` round trip of the PostgreSQL database +preserves sealed snapshots byte-for-byte — `npm run docker:validate` performs +exactly this round trip (dump the running database, restore into a scratch +database, and confirm a sealed snapshot's canonical graph hash is unchanged) +as part of its acceptance sequence. Example manual procedure against a +running Compose stack: + +```bash +docker compose exec postgres pg_dump -U partha partha > backup.sql +# ... later, against a fresh database ... +docker compose exec -T postgres psql -U partha -d partha < backup.sql +``` + +There is no automated backup schedule, retention policy, or point-in-time +recovery tooling — operators must run and store their own dumps. diff --git a/scripts/validate-compose.mjs b/scripts/validate-compose.mjs index 1a1f780..1bfe432 100644 --- a/scripts/validate-compose.mjs +++ b/scripts/validate-compose.mjs @@ -1,6 +1,16 @@ +import { randomBytes } from "node:crypto"; import { spawnSync } from "node:child_process"; import http from "node:http"; +const API_BASE_URL = "http://127.0.0.1:8000"; +const POSTGRES_USER = process.env.POSTGRES_USER || "partha"; +const POSTGRES_DB = process.env.POSTGRES_DB || "partha"; + +// A tiny, stable public repository (no supported source files) — already used +// as this API's own OpenAPI example, so it is known to import cleanly. +const FIXTURE_REPOSITORY_URL = "https://github.com/octocat/Hello-World"; +const FIXTURE_REPOSITORY_BRANCH = "master"; + function run(command, args) { const result = spawnSync(command, args, { stdio: "inherit", shell: false }); if (result.error) throw result.error; @@ -9,13 +19,43 @@ function run(command, args) { } } +function runCapture(command, args) { + const result = spawnSync(command, args, { stdio: ["ignore", "pipe", "inherit"], shell: false, encoding: "utf8" }); + if (result.error) throw result.error; + if (result.status !== 0) { + throw new Error(`${command} ${args.join(" ")} failed with exit code ${result.status}`); + } + return result.stdout; +} + +function dockerComposeExec(service, args) { + run("docker", ["compose", "exec", "-T", service, ...args]); +} + +function dockerComposeExecCapture(service, args) { + return runCapture("docker", ["compose", "exec", "-T", service, ...args]); +} + +function dockerComposeExecWithEnv(service, env, args) { + const envFlags = Object.entries(env).flatMap(([key, value]) => ["-e", `${key}=${value}`]); + run("docker", ["compose", "exec", "-T", ...envFlags, service, ...args]); +} + function sleep(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); } +function assert(condition, message) { + if (!condition) throw new Error(`Assertion failed: ${message}`); +} + +function randomSuffix() { + return randomBytes(4).toString("hex"); +} + function checkReady() { return new Promise((resolve) => { - const request = http.get("http://127.0.0.1:8000/ready", { timeout: 5000 }, (response) => { + const request = http.get(`${API_BASE_URL}/ready`, { timeout: 5000 }, (response) => { response.resume(); resolve(response.statusCode === 200); }); @@ -27,19 +67,348 @@ function checkReady() { }); } +async function waitUntilReady({ attempts = 30, intervalMs = 2000, label = "startup" } = {}) { + for (let attempt = 0; attempt < attempts; attempt += 1) { + if (await checkReady()) return; + await sleep(intervalMs); + } + run("docker", ["compose", "logs", "api", "postgres", "redis"]); + throw new Error(`Compose API did not become ready during ${label} (waited ${(attempts * intervalMs) / 1000}s).`); +} + +async function apiFetch(path, { method = "GET", token, body, expectedStatuses = [200] } = {}) { + const headers = {}; + if (token) headers.Authorization = `Bearer ${token}`; + let requestBody; + if (body !== undefined) { + headers["Content-Type"] = "application/json"; + requestBody = JSON.stringify(body); + } + const response = await fetch(`${API_BASE_URL}${path}`, { method, headers, body: requestBody }); + const text = await response.text(); + const payload = text ? JSON.parse(text) : null; + if (!expectedStatuses.includes(response.status)) { + throw new Error(`${method} ${path} expected status in [${expectedStatuses.join(", ")}], got ${response.status}: ${text}`); + } + return { status: response.status, payload }; +} + +// --- 1. Migrations --------------------------------------------------------- + +async function verifyMigrations(knownSecrets) { + console.log("[migrations] re-running alembic upgrade head against the live app database"); + dockerComposeExec("api", ["alembic", "upgrade", "head"]); + + console.log("[migrations] asserting core tables exist"); + const tableList = dockerComposeExecCapture("postgres", [ + "psql", + "-U", + POSTGRES_USER, + "-d", + POSTGRES_DB, + "-tAc", + "select tablename from pg_tables where schemaname='public'", + ]); + for (const table of ["repositories", "analysis_jobs", "ri_snapshots"]) { + assert(tableList.includes(table), `expected core table "${table}" to exist after migration`); + } + + console.log("[migrations] proving a downgrade/upgrade round trip is clean"); + dockerComposeExec("api", ["alembic", "downgrade", "-1"]); + dockerComposeExec("api", ["alembic", "upgrade", "head"]); + + await verifyPercentEncodedDatabaseUrl(knownSecrets); +} + +// Regression coverage for #110: Alembic's ConfigParser previously choked on a +// raw "%" in DATABASE_URL (e.g. a percent-encoded "@" in a password), because +// set_main_option() treats "%" as its own interpolation syntax unless escaped. +// This proves a real, live connection with such a password migrates cleanly, +// isolated in a scratch role/database so it never touches the app's own data. +async function verifyPercentEncodedDatabaseUrl(knownSecrets) { + console.log("[migrations] verifying a percent-encoded DATABASE_URL password (#110 regression)"); + const scratchUser = "migration_check_user"; + const scratchDb = "migration_check_db"; + const scratchPassword = `p@rtha-${randomSuffix()}`; + knownSecrets.push(scratchPassword); + const encodedPassword = encodeURIComponent(scratchPassword); + const scratchUrl = `postgresql+psycopg://${scratchUser}:${encodedPassword}@postgres:5432/${scratchDb}`; + + const createScratch = () => + run("docker", [ + "compose", + "exec", + "-T", + "postgres", + "psql", + "-U", + POSTGRES_USER, + "-d", + POSTGRES_DB, + "-v", + "ON_ERROR_STOP=1", + "-c", + `DROP DATABASE IF EXISTS ${scratchDb};`, + "-c", + `DROP ROLE IF EXISTS ${scratchUser};`, + "-c", + `CREATE ROLE ${scratchUser} LOGIN PASSWORD '${scratchPassword}';`, + "-c", + `CREATE DATABASE ${scratchDb} OWNER ${scratchUser};`, + ]); + const dropScratch = () => + run("docker", [ + "compose", + "exec", + "-T", + "postgres", + "psql", + "-U", + POSTGRES_USER, + "-d", + POSTGRES_DB, + "-v", + "ON_ERROR_STOP=1", + "-c", + `DROP DATABASE IF EXISTS ${scratchDb};`, + "-c", + `DROP ROLE IF EXISTS ${scratchUser};`, + ]); + + createScratch(); + try { + dockerComposeExecWithEnv("api", { DATABASE_URL: scratchUrl }, ["alembic", "upgrade", "head"]); + dockerComposeExecWithEnv("api", { DATABASE_URL: scratchUrl }, ["alembic", "downgrade", "-1"]); + dockerComposeExecWithEnv("api", { DATABASE_URL: scratchUrl }, ["alembic", "upgrade", "head"]); + } finally { + dropScratch(); + } +} + +// --- 2. Auth + owner isolation --------------------------------------------- + +async function registerUser(email, password) { + const { payload } = await apiFetch("/auth/register", { + method: "POST", + body: { email, password }, + expectedStatuses: [201], + }); + return payload; +} + +async function createFixtureRepository(token) { + const { payload } = await apiFetch("/repositories/github", { + method: "POST", + token, + body: { url: FIXTURE_REPOSITORY_URL, branch: FIXTURE_REPOSITORY_BRANCH }, + expectedStatuses: [201], + }); + return payload; +} + +async function verifyAuthAndOwnerIsolation(knownSecrets) { + console.log("[auth] registering two disposable users"); + const suffix = randomSuffix(); + const user1Email = `acceptance-user1-${suffix}@example.com`; + const user2Email = `acceptance-user2-${suffix}@example.com`; + const user1Password = `Accept-${suffix}-user1!`; + const user2Password = `Accept-${suffix}-user2!`; + knownSecrets.push(user1Password, user2Password); + + const user1 = await registerUser(user1Email, user1Password); + const user2 = await registerUser(user2Email, user2Password); + assert(user1.accessToken, "user1 registration did not return an access token"); + assert(user2.accessToken, "user2 registration did not return an access token"); + + console.log("[auth] user1 creating a fixture repository"); + const repository = await createFixtureRepository(user1.accessToken); + assert(repository.id, "repository creation did not return an id"); + + console.log("[auth] asserting owner isolation: user2 must get 404 on user1's repository"); + const isolationCheck = await apiFetch(`/repositories/${repository.id}`, { + token: user2.accessToken, + expectedStatuses: [404], + }); + assert(isolationCheck.status === 404, "owner isolation violated: user2 could read user1's repository"); + + return { user1, user2, repository }; +} + +// --- 3. Analysis lifecycle -------------------------------------------------- + +// Mirrors the frontend's resilient polling (fix/164, 1ca0daf): tolerate +// transient connection failures with capped exponential backoff, stop the +// instant a terminal status is observed, and give up cleanly at a deadline. +async function pollAnalysisUntilComplete(token, repositoryId, { timeoutMs = 120_000, intervalMs = 2000, maxNetworkRetries = 5 } = {}) { + const deadline = Date.now() + timeoutMs; + let networkRetries = 0; + while (Date.now() < deadline) { + let response; + try { + response = await apiFetch(`/analysis/${repositoryId}/status`, { token, expectedStatuses: [200] }); + } catch (error) { + networkRetries += 1; + if (networkRetries > maxNetworkRetries) { + throw new Error(`Analysis status polling lost connectivity after ${maxNetworkRetries} retries: ${error.message}`); + } + await sleep(Math.min(15_000, 1000 * 2 ** (networkRetries - 1))); + continue; + } + networkRetries = 0; + const { status } = response.payload; + if (status === "completed") return response.payload; + if (status === "failed" || status === "cancelled") { + throw new Error(`Analysis ended in terminal state "${status}": ${response.payload.error ?? "no error detail"}`); + } + await sleep(intervalMs); + } + throw new Error(`Analysis did not reach a completed status within ${timeoutMs}ms`); +} + +async function verifyAnalysisLifecycle(user1, repository) { + console.log("[analysis] starting analysis and polling for completion"); + await apiFetch(`/analysis/${repository.id}/start`, { method: "POST", token: user1.accessToken, expectedStatuses: [200] }); + const finalStatus = await pollAnalysisUntilComplete(user1.accessToken, repository.id); + assert(finalStatus.status === "completed", "analysis did not reach completed status"); + + console.log("[analysis] asserting a sealed ri.v1 snapshot exists for this revision"); + const manifest = await apiFetch(`/analysis/${repository.id}/revision-manifest`, { + token: user1.accessToken, + expectedStatuses: [200], + }); + const { manifest: body } = manifest.payload; + assert( + body.snapshotSchemaVersion === "ri.v1", + `expected a sealed ri.v1 snapshot, got schema version "${body.snapshotSchemaVersion}"`, + ); + assert(body.canonicalGraphHash, "sealed snapshot is missing its canonical graph hash"); + assert(body.sealedAt, "sealed snapshot is missing its sealedAt timestamp"); + assert( + manifest.payload.verificationState === "verified", + `expected verification state "verified", got "${manifest.payload.verificationState}"`, + ); + + return { snapshotId: body.snapshotId, canonicalGraphHash: body.canonicalGraphHash }; +} + +// --- 4. Restart / recovery --------------------------------------------------- + +async function verifyRestartRecovery(user1, repository, snapshot) { + console.log("[restart] restarting the api container"); + run("docker", ["compose", "restart", "api"]); + await waitUntilReady({ label: "post-restart" }); + + console.log("[restart] asserting the repository and completed analysis survived the restart"); + const repoCheck = await apiFetch(`/repositories/${repository.id}`, { token: user1.accessToken, expectedStatuses: [200] }); + assert(repoCheck.payload.id === repository.id, "repository missing or changed identity after restart"); + + const statusCheck = await apiFetch(`/analysis/${repository.id}/status`, { token: user1.accessToken, expectedStatuses: [200] }); + assert(statusCheck.payload.status === "completed", "analysis job lost its completed status after restart"); + + const manifestCheck = await apiFetch(`/analysis/${repository.id}/revision-manifest`, { + token: user1.accessToken, + expectedStatuses: [200], + }); + assert( + manifestCheck.payload.manifest.canonicalGraphHash === snapshot.canonicalGraphHash, + "sealed snapshot's canonical graph hash changed across restart (possible data corruption)", + ); + + console.log("[restart] asserting no orphaned or in-flight job rows at the database level"); + const jobStatuses = dockerComposeExecCapture("postgres", [ + "psql", + "-U", + POSTGRES_USER, + "-d", + POSTGRES_DB, + "-tAc", + `select status from analysis_jobs where repository_id='${repository.id}'`, + ]) + .split("\n") + .map((line) => line.trim()) + .filter(Boolean); + assert(jobStatuses.length === 1, `expected exactly one analysis job row for the repository, found ${jobStatuses.length}`); + assert(jobStatuses[0] === "completed", `orphaned or in-flight job detected after restart: status="${jobStatuses[0]}"`); +} + +// --- 5. Backup / restore ------------------------------------------------------ + +async function verifyBackupRestore(snapshot) { + console.log("[backup] proving a pg_dump/pg_restore round trip preserves the sealed snapshot"); + const restoreDb = "partha_restore_check"; + run("docker", [ + "compose", + "exec", + "-T", + "postgres", + "sh", + "-c", + `dropdb -U ${POSTGRES_USER} --if-exists ${restoreDb} && ` + + `createdb -U ${POSTGRES_USER} -O ${POSTGRES_USER} ${restoreDb} && ` + + `pg_dump -U ${POSTGRES_USER} ${POSTGRES_DB} | psql -U ${POSTGRES_USER} -d ${restoreDb}`, + ]); + try { + const restoredHash = dockerComposeExecCapture("postgres", [ + "psql", + "-U", + POSTGRES_USER, + "-d", + restoreDb, + "-tAc", + `select canonical_graph_hash from ri_snapshots where snapshot_id='${snapshot.snapshotId}'`, + ]).trim(); + assert( + restoredHash === snapshot.canonicalGraphHash, + `backup/restore round trip produced a different canonical_graph_hash (expected ${snapshot.canonicalGraphHash}, got ${restoredHash})`, + ); + } finally { + run("docker", ["compose", "exec", "-T", "postgres", "dropdb", "-U", POSTGRES_USER, "--if-exists", restoreDb]); + } +} + +// --- 6. Secrets in logs ------------------------------------------------------- + +function assertNoSecretsInLogs(knownSecrets) { + console.log("[secrets] checking docker compose logs for verbatim secret values"); + const logs = runCapture("docker", ["compose", "logs", "--no-color"]); + for (const secret of knownSecrets) { + if (secret && logs.includes(secret)) { + throw new Error("A known secret value from this run was found verbatim in `docker compose logs` output."); + } + } +} + +// --- main -------------------------------------------------------------------- + async function main() { run("docker", ["compose", "config"]); + const knownSecrets = []; + let failure = null; try { run("docker", ["compose", "up", "--build", "-d"]); - for (let attempt = 0; attempt < 30; attempt += 1) { - if (await checkReady()) return; - await sleep(2000); - } - run("docker", ["compose", "logs", "api", "postgres", "redis"]); - throw new Error("Compose API did not become ready."); + await waitUntilReady({ label: "initial startup" }); + + await verifyMigrations(knownSecrets); + const { user1, repository } = await verifyAuthAndOwnerIsolation(knownSecrets); + const snapshot = await verifyAnalysisLifecycle(user1, repository); + await verifyRestartRecovery(user1, repository, snapshot); + await verifyBackupRestore(snapshot); + + console.log( + "Acceptance sequence passed: migrations, auth/owner isolation, analysis lifecycle, restart/recovery, and backup/restore all verified.", + ); + } catch (error) { + failure = error; } finally { + try { + assertNoSecretsInLogs(knownSecrets); + } catch (secretsError) { + failure = failure ?? secretsError; + console.error(secretsError.message); + } run("docker", ["compose", "down", "-v"]); } + if (failure) throw failure; } main().catch((error) => {