feat(serving): PostgresControlPlaneStore — the scale profile ships (ADR 0010 slice 5)

.github/workflows/ci.yml

@@ -177,14 +177,30 @@ jobs:
           CLICKHOUSE_USER: agentflow
           CLICKHOUSE_PASSWORD: agentflow
           CLICKHOUSE_DB: agentflow
+      postgres:
+        # Live coverage for PostgresControlPlaneStore (ADR 0010 slice 5);
+        # test_control_plane_postgres_live.py skips itself when
+        # AGENTFLOW_TEST_PG_DSN is absent.
+        image: postgres:17
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_USER: agentflow
+          POSTGRES_PASSWORD: agentflow
+          POSTGRES_DB: agentflow
+        options: >-
+          --health-cmd "pg_isready -U agentflow"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
       - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
         with:
           python-version: "3.11"
       - name: Install dependencies
         run: |
-          pip install -e ".[dev,cloud]"
+          pip install -e ".[dev,cloud,postgres]"
           pip install -e "./sdk"
       - name: Prepare pytest temp directory
         run: mkdir -p .tmp
@@ -201,6 +217,7 @@ jobs:
           CLICKHOUSE_LIVE_USER: agentflow
           CLICKHOUSE_LIVE_PASSWORD: agentflow
           CLICKHOUSE_LIVE_DATABASE: agentflow
+          AGENTFLOW_TEST_PG_DSN: postgresql://agentflow:agentflow@localhost:5432/agentflow
         run: pytest tests/integration/ -v --tb=short
 
   helm-schema-live:

CHANGELOG.md

@@ -4,6 +4,63 @@ All notable changes to AgentFlow are documented in this file.
 
 ## [Unreleased]
 
+### Added — PostgresControlPlaneStore: the scale profile ships (ADR 0010 slice 5, 2026-07-03)
+
+- **New `src/serving/control_plane/postgres.py`** — all six control-plane
+  state classes as PostgreSQL tables behind the existing port, with the claim
+  semantics the embedded adapter only satisfies degenerately made real:
+  enqueue-win by `INSERT .. ON CONFLICT DO NOTHING` rowcount, queue/outbox
+  claims by `FOR UPDATE SKIP LOCKED` + a self-expiring `lease_expires_at`
+  (work-stealing across replicas, no leader election; a crashed owner's rows
+  become due again on lease expiry), invariant 8 as an ordinary transaction
+  (every store method is one transaction: commit on success, rollback on any
+  exception). Payloads stay TEXT/JSON-string so callers see the embedded
+  adapter's shapes. One connection per call — pooling stays out of ADR scope.
+  Selection: `AGENTFLOW_CONTROLPLANE_STORE=postgres` +
+  `AGENTFLOW_CONTROLPLANE_PG_DSN` (+ optional
+  `AGENTFLOW_CONTROLPLANE_LEASE_SECONDS`); the slice-1 `NotImplementedError`
+  ratchet is gone, and a missing DSN or missing `psycopg` fails the boot
+  loudly — never a silent fallback to embedded. `psycopg` is a new optional
+  extra (`pip install agentflow-runtime[postgres]`), the `redis` import
+  pattern.
+- **Webhook registrations (state class 5) move behind the port** — the
+  sharpest split-brain of the ADR's inventory was still a per-pod YAML read
+  outside the port after slices 1–4. New port methods
+  `load_webhook_registrations`/`save_webhook_registrations`;
+  `load_webhooks`/`save_webhooks`/`create_webhook`/`list_webhooks`/
+  `get_webhook`/`deactivate_webhook` now take `app` and resolve the store
+  inside (the same move the alert-rule helpers made in slice 2). The embedded
+  adapter keeps the byte-compatible `config/webhooks.yaml`.
+- **Alert-tick single-flight (ADR 0010 §2) wired into the dispatcher** — new
+  port methods `claim_alert_tick`/`complete_alert_tick`;
+  `AlertDispatcher.dispatch_alerts` claims each rule before evaluating (a
+  lost claim = another replica owns that rule's tick) and persists advanced
+  rule state **per rule** in the same transaction as the claim release — the
+  old full-set save would let two replicas advancing different rules clobber
+  each other's runtime state. Embedded grants every claim (one process), so
+  the single-replica profile behaves as before; a CRUD full-set save on
+  PostgreSQL upserts by id and does not release an in-flight claim.
+- **The postgres profile shares one store across every consumer**: `main.py`
+  injects the app-wide store into `AuthManager` and `OutboxProcessor` when
+  the profile is external (embedded keeps its historical private stores);
+  the analytics entry points (`analytics.py`, `routers/admin.py`,
+  `admin_ui.py`'s QPS tile) accept the store handle and route through
+  `AuthManager.store`, so usage/sessions land in PostgreSQL instead of a
+  per-pod DuckDB file.
+- **Verified live (standalone PostgreSQL 17.5, no Docker): 31/31 probes** —
+  the ADR's named suite (parallel claim exclusivity, lease-expiry re-drive,
+  restart re-drive, enqueue-win uniqueness, outbox↔dead-letter atomicity
+  incl. rollback halves, alert-tick single-flight) plus a full contract
+  parity sweep and an end-to-end app test (two boots on the postgres profile
+  see each other's webhook registration; usage accounting lands in PG):
+  `docs/perf/control-plane-pg-verify-2026-07-03.md`. The same suite runs in
+  CI against a new `postgres:17` service in the integration job and
+  self-skips where `AGENTFLOW_TEST_PG_DSN` is absent.
+- Helm is untouched by design: the values schema still pins
+  `controlPlane.store=embedded` and the chart still refuses multi-replica
+  renders — the enum extension and env/secret wiring are rollout slice 6,
+  which this slice unblocks.
+
 ### Added — API-usage accounting and session analytics behind the ControlPlaneStore port (ADR 0010 slice 4, 2026-07-02)
 
 - **`api_usage`** (per-tenant/per-key request counters) and **`api_sessions`**

docs/decisions/0010-control-plane-externalization-postgres.md

@@ -186,7 +186,17 @@ ClickHouse is consumed), and `psycopg` joins the optional dependencies.
 5. `PostgresControlPlaneStore` + live verification (standalone-PG probe
    suite: parallel claim exclusivity, lease expiry re-drive, restart
    re-drive, enqueue-win uniqueness, outbox↔dead-letter atomicity) +
-   CI integration coverage on the existing PG service.
+   CI integration coverage (a `postgres:17` service added to the CI
+   integration job). Executed 2026-07-03 with two scope additions the
+   extraction slices had left open, both required for §1's "all six state
+   classes" to hold: **webhook registrations** (class 5 — the sharpest
+   split-brain — was still a per-pod YAML read outside the port; the
+   registration CRUD now resolves the store from ``app``, embedded keeps the
+   byte-compatible YAML) and **§2's ``claim_alert_tick`` /
+   ``complete_alert_tick``** wired into the dispatcher (per-rule state
+   persistence — a full-set save would let two replicas clobber each other's
+   rule runtime state). Verified live: 31/31 probes,
+   `docs/perf/control-plane-pg-verify-2026-07-03.md`.
 6. Helm wiring (`controlPlane.store=postgres` profile: env + secret, schema
    enum extension — the render gate then admits multi-replica) and cutover
    plan Phase 3 execution: kind staging at `replicaCount=2`, verifying
@@ -213,8 +223,9 @@ ClickHouse is consumed), and `psycopg` joins the optional dependencies.
 - Per-pod event scanning is N× read amplification on the serving backend at
   scale (accepted; the scan is bounded and cheap, and consolidating scanners
   is a later topology refinement, cf. option 3).
-- Until slice 5 lands, multi-replica is simply impossible to render — a
-  deliberate fail-closed period.
+- Until slice 6 extends the chart, multi-replica is simply impossible to
+  render — a deliberate fail-closed period (the slice-5 adapter is app-side;
+  the schema enum still pins `embedded` until the helm profile ships).
 
 ## Follow-up
 

docs/perf/control-plane-pg-verify-2026-07-03.md

@@ -0,0 +1,65 @@
+# Control plane on PostgreSQL — live verification (ADR 0010 rollout slice 5)
+
+**Date:** 2026-07-03
+**Environment:** standalone PostgreSQL 17.5 (EDB windows-x64 binaries, no
+Docker, no service install: `initdb` + `pg_ctl`, port 55433, trust auth,
+user/db `agentflow`) — the same no-Docker standalone-PG recipe that verified
+the vault governance layer
+(`vault-pii-governance-pg-verify-2026-07-02.md`). Adapter under test:
+`src/serving/control_plane/postgres.py` (`PostgresControlPlaneStore`),
+driven by the probe suite the ADR names for this slice:
+`tests/integration/test_control_plane_postgres_live.py` with
+`AGENTFLOW_TEST_PG_DSN=postgresql://agentflow@127.0.0.1:55433/agentflow`.
+
+**Result: 31/31 probes passed** (`pytest`, 19.45s; psycopg 3.3.4). The same
+suite runs in CI against the `postgres:17` service container added to the
+`test-integration` job (it self-skips when the DSN env var is absent, the
+`test_clickhouse_backend_live.py` pattern).
+
+## The ADR's named probes
+
+| Probe (ADR 0010 § Rollout 5) | Test | Result |
+| --- | --- | --- |
+| Enqueue-win uniqueness | 8 threads race `enqueue_webhook_delivery` on one (webhook, event) → exactly **1** `True`, 1 row | passed |
+| Parallel claim exclusivity | 4 threads claim 10 due rows concurrently → no row handed out twice, none lost (`FOR UPDATE SKIP LOCKED`) | passed |
+| Lease-expiry re-drive | row claimed with a 0.4s lease, owner "crashes" → invisible while leased, claimable again after expiry; outcome writes clear the lease so backoff alone governs | passed |
+| Restart re-drive | pending row enqueued by one store instance is claimed by a **fresh** instance, canonical body verbatim | passed |
+| Outbox↔dead-letter atomicity (invariant 8) | `mark_outbox_sent` flips both rows in one transaction; with `dead_letter_events` dropped mid-scenario the outbox flip **rolls back** (row stays `pending`); `enqueue_outbox_replay` rolls back symmetrically | passed |
+| Alert-tick single-flight (§2) | second claimant loses; `complete_alert_tick` persists the advanced record and releases in one transaction; a stale claim self-expires; a concurrent CRUD full-set save does **not** release an in-flight claim | passed |
+
+## Contract parity sweep
+
+Every port method exercised against live PostgreSQL with the same assertions
+the embedded adapter's unit pins make: webhook outcome state machine
+(backoff → `dead` at max, success → `delivered`, park), oldest-first claim
+ordering under `limit`, attempt-log and alert-history roundtrips (newest
+first, JSON payload decoded), webhook-registration and alert-rule
+repositories (order preserved via a `position` column, full-set save =
+YAML-replace semantics, ids required), dead-letter reads (tenant scoping,
+reason filter, pagination, stats + trend), usage accounting (per tenant /
+per key / old-key-slot hour window), session analytics (idempotent
+insert-or-replace on `request_id`, usage/top-queries/top-entities/latency
+percentiles/anomalies/QPS shapes, malformed window → `ValueError`, QPS
+degrades to 0.0 on an unreachable server).
+
+## End to end: the app itself on the postgres profile
+
+`test_app_on_postgres_profile_shares_state_across_boots` boots the real
+FastAPI app twice with `AGENTFLOW_CONTROLPLANE_STORE=postgres`:
+
+- boot #1 resolves `PostgresControlPlaneStore` in the lifespan,
+  `AuthManager.store` **is** the shared app-wide store (slice 5 injection in
+  `main.py`), and `POST /v1/webhooks` registers a webhook;
+- boot #2 (a second pod, in production terms) sees that registration through
+  `GET /v1/webhooks` — the class-5 per-pod YAML split-brain is gone;
+- `api_usage` rows for the authenticated tenant landed in PostgreSQL, not in
+  a local DuckDB file.
+
+## Notes
+
+- Claim leases default to 300s (`AGENTFLOW_CONTROLPLANE_LEASE_SECONDS`
+  overrides); alert-tick leases are 120s (`AlertDispatcher.tick_lease_seconds`).
+- Payload columns are TEXT holding JSON strings — callers see the same
+  "string, you decode it" shape as on the embedded adapter.
+- One connection per method call, no pooling (out of ADR 0010 scope,
+  recorded follow-up).

docs/security-audit.md

@@ -80,7 +80,7 @@ Audit finding A-4 flagged the dynamic-SQL surface as "one careless edit away fro
 - Interpolated **identifiers** come from a fixed in-code allowlist, the semantic catalog, trusted backend config, live schema introspection (`PRAGMA table_info`), `_IDENTIFIER_RE`, or `sqlglot` validation.
 - Interpolated **values** are either bound as `?` parameters, fixed literals, integers, or regex-extracted tokens that exclude SQL metacharacters and are additionally quoted via `_quote_literal` / `_sql_str_literal`.
 
-No site interpolates unbound, unquoted request data. There are **no Class-A (migratable value-interpolation) sites remaining**: the hot entity/metric paths already bind values (`use_query_params` on DuckDB, `_quote_literal` elsewhere) and the operational routers already pass values through `?`. The remaining suppressions are Class-B (identifiers / structural fragments that cannot be parameterized).
+No site interpolates unbound, unquoted request data. There are **no Class-A (migratable value-interpolation) sites remaining**: the hot entity/metric paths already bind values (`use_query_params` on DuckDB, `_quote_literal` elsewhere) and the operational routers already pass values through `?`. The remaining suppressions are Class-B (identifiers / structural fragments that cannot be parameterized). The `PostgresControlPlaneStore` sites added by ADR 0010 slice 5 (`control_plane/postgres.py`, reviewed 2026-07-03) interpolate only a table name that is a module literal at exactly two call sites; every value binds via `%s`.
 
 The number of suppressions per file is pinned by `test_interpolated_sql_nosec_surface_is_pinned` — a new site (even inside an already-listed file) or a new file fails CI and forces a review. Each suppression's per-line rationale comment is enforced by `test_nosec_comments_carry_reason`.
 

helm/agentflow/templates/deployment.yaml

@@ -2,7 +2,7 @@
 {{- fail "DuckDB persistence requires a single writer replica: set replicaCount=1 and autoscaling.maxReplicas=1, or disable persistence and use ephemeral storage." }}
 {{- end }}
 {{- if and (or (gt (int .Values.replicaCount) 1) (and .Values.autoscaling.enabled (gt (int .Values.autoscaling.maxReplicas) 1))) (or (ne .Values.controlPlane.store "postgres") (ne .Values.serving.backend "clickhouse")) }}
-{{- fail "Multi-replica requires BOTH an external serving engine (serving.backend=clickhouse, ADR 0006/0007) AND an external control-plane store (controlPlane.store=postgres, ADR 0009/0010): the embedded per-pod store forks webhook/alert/outbox/usage state across replicas (duplicate deliveries, split alert history). Until the ADR 0010 rollout ships the postgres store, keep replicaCount=1 and autoscaling.maxReplicas=1." }}
+{{- fail "Multi-replica requires BOTH an external serving engine (serving.backend=clickhouse, ADR 0006/0007) AND an external control-plane store (controlPlane.store=postgres, ADR 0009/0010): the embedded per-pod store forks webhook/alert/outbox/usage state across replicas (duplicate deliveries, split alert history). The app-side postgres adapter shipped with ADR 0010 slice 5; until the chart profile for it lands (rollout slice 6), keep replicaCount=1 and autoscaling.maxReplicas=1." }}
 {{- end }}
 {{- $secretName := .Values.secrets.existingSecret | default (include "agentflow.fullname" .) }}
 apiVersion: apps/v1

helm/agentflow/values.yaml

@@ -134,13 +134,16 @@ serving:
     existingSecret: ""
     passwordKey: clickhouse-password
 
-# Control-plane state (ADR 0009 / ADR 0010): webhook queue+log, alert rules and
-# history, outbox, dead-letter and usage accounting. 'embedded' = per-pod
-# DuckDB + config files — correct for the single-replica profile, a split-brain
-# at replicaCount > 1 (duplicate deliveries, forked alert state). ADR 0010
-# externalizes this state to PostgreSQL behind a ControlPlaneStore port; until
-# that adapter ships, the values schema pins store=embedded (fail-closed
-# ratchet) and the chart refuses any multi-replica render.
+# Control-plane state (ADR 0009 / ADR 0010): webhook queue+log, registrations,
+# alert rules and history, outbox, dead-letter and usage accounting.
+# 'embedded' = per-pod DuckDB + config files — correct for the single-replica
+# profile, a split-brain at replicaCount > 1 (duplicate deliveries, forked
+# alert state). The PostgresControlPlaneStore adapter shipped with ADR 0010
+# rollout slice 5 (app-side: AGENTFLOW_CONTROLPLANE_STORE=postgres +
+# AGENTFLOW_CONTROLPLANE_PG_DSN); the chart profile for it (env + secret
+# wiring, schema enum extension) lands with rollout slice 6 — until then the
+# values schema pins store=embedded (fail-closed ratchet) and the chart
+# refuses any multi-replica render.
 controlPlane:
   store: embedded
 

pyproject.toml

@@ -37,6 +37,12 @@ cloud = [
     "boto3>=1.35,<2",
     "pyiceberg[pyiceberg-core]>=0.7,<1",
 ]
+postgres = [
+    # ADR 0010 slice 5: PostgresControlPlaneStore (scale profile). Optional
+    # exactly like redis — the embedded default profile needs none of it,
+    # and control_plane/postgres.py degrades to a clear boot error without it.
+    "psycopg[binary]>=3.2,<4",
+]
 load = [
     "locust>=2.29,<3",
     # benchmark_freshness.py: production QueryCache semantics (TTL + key
@@ -158,6 +164,11 @@ ignore = ["S101", "S311"]
 "src/serving/api/routers/lineage.py" = [
     "S608",
 ]
+# Interpolations are internal literals only (a lease-expiry SQL fragment,
+# optional filter clauses, table-name constants); all values bind via %s.
+"src/serving/control_plane/postgres.py" = [
+    "S608",
+]
 "src/serving/api/routers/slo.py" = [
     "S608",
 ]
@@ -284,7 +295,10 @@ editable-installs = [".[dev]"]
 editable-installs = [".[dev,cloud]"]
 
 [tool.agentflow.dependency-profiles.profiles.test-sdk]
-editable-installs = [".[dev,cloud]", "./sdk"]
+# postgres: the CI integration job drives the PostgresControlPlaneStore live
+# suite against its postgres:17 service (ADR 0010 slice 5), so it needs the
+# optional psycopg extra.
+editable-installs = [".[dev,cloud,postgres]", "./sdk"]
 
 [tool.agentflow.dependency-profiles.profiles.test-integrations]
 editable-installs = [".[dev,cloud]", "./sdk", "./integrations[mcp]"]