diff --git a/CHANGELOG.md b/CHANGELOG.md index 97803ef..82907b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,30 @@ All notable changes to AgentFlow are documented in this file. ## [Unreleased] +### Added — Operational serving split decided; ops-surfaces spec (ADR 0011, 2026-07-03) + +- **New [ADR 0011](docs/decisions/0011-ops-serving-split.md)** — the design + decision for the operational layer (`docs/domain.md` §4): every ops surface + (Order 360 timeline, stuck-orders worklist, exception inbox) composes + exactly the two existing ports — `QueryEngine`/`ServingBackend` for + analytical reads, `ControlPlaneStore` for transactional triage state — with + no third data path (no `query_engine._conn`, no vault DSN). Options + considered and rejected with reasons: everything-on-ClickHouse, + everything-on-PostgreSQL, direct vault reads for the customer block, + precomputed ops marts. The exception-triage overlay is recorded as the + seventh control-plane state class, extending ADR 0010's inventory. +- **New `docs/ops-surfaces-spec.md`** — the implementation contract for + slices D2–D4: SLA stage model with budgets as catalog data (a `stages:` + block in `contracts/entities/order.yaml`), stage-entry journal rows + (`orders.status` topic) as the stage clock with an honest `created_at` + fallback, the journal's `entity_id` axis made real on live writes, endpoint + contracts and response shapes for `/v1/entity/order/{id}/timeline`, + `/v1/ops/stuck-orders`, and `/v1/ops/exceptions` (+stats), reconciliation + checks R1/R2, the manual-resolutions counter, a pinned demo story + (ORD-20260404-1004 as the sole SLA breach), and twelve machine-checkable + invariants as the test ТЗ. +- Docs-only: no runtime behavior changes in this entry. + ### Added — PostgresControlPlaneStore: the scale profile ships (ADR 0010 slice 5, 2026-07-03) - **New `src/serving/control_plane/postgres.py`** — all six control-plane diff --git a/docs/architecture.md b/docs/architecture.md index af08d55..0218a40 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -188,6 +188,17 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis. > (`docs/perf/vault-pii-governance-verify-2026-07-02.md`). `sql_guard` remains, scoped > to what it can actually enforce: SELECT-only, no DML, the tenant table allow-list, and > the recursive-CTE shadow reject. Tracked in `road-to-9.8.md`. +> +> **Operational surfaces have a recorded serving split ([ADR 0011](decisions/0011-ops-serving-split.md), 2026-07-03).** +> The planned ops layer (Order 360 timeline, stuck-orders worklist, exception +> inbox — the workflows of [`domain.md`](domain.md) §4) composes exactly the two +> existing ports: analytical reads (entity point-reads, the `pipeline_events` +> journal, open-order scans) on the serving backend, transactional triage state +> (dead-letter lifecycle, the exception-triage overlay, the manual-work counter) +> on the `ControlPlaneStore` — no third data path. Endpoint contracts, the SLA +> stage model, and exception sources are pinned in +> [`docs/ops-surfaces-spec.md`](ops-surfaces-spec.md); implementation is staged +> (D2–D4). ## v1-v6 Capability Map diff --git a/docs/decisions/0011-ops-serving-split.md b/docs/decisions/0011-ops-serving-split.md new file mode 100644 index 0000000..699c86e --- /dev/null +++ b/docs/decisions/0011-ops-serving-split.md @@ -0,0 +1,212 @@ +# ADR 0011: Operational serving split — transactional reads on the control-plane store, analytical reads on the serving backend + +## Status + +Accepted - 2026-07-03 + +Design decision for the operational layer (the three ops surfaces of +[`domain.md`](../domain.md) §4: **Order 360 timeline**, **stuck-orders +worklist**, **exception inbox**). The companion specification that pins +endpoint contracts, the SLA stage model, and per-slice scope is +[`docs/ops-surfaces-spec.md`](../ops-surfaces-spec.md); implementation is +staged as slices D2–D4 (see § Rollout). + +## Context + +The operations team of the modeled business juggles five tools to answer one +question about one order (`domain.md` §4). The serving layer exists to replace +that with one API surface, and the three planned ops surfaces are its most +read-shape-diverse consumers yet: + +- **Order 360** is a *point-read composition*: one order row, its event + history, its customer, its failures — "everything about ORD-20260404-1001, + now, in one place". +- **Stuck-orders** is an *analytical scan*: every open order, joined to its + stage-entry time, filtered against per-stage SLA budgets, ranked by + overshoot. +- **Exception inbox** is a *transactional worklist*: failed events and + consistency findings with a triage lifecycle (acknowledge / resolve / + replay / dismiss) and a counter of absorbed manual work. + +The platform has exactly two data planes, with different semantics, both +already behind ports: + +| Plane | Port | Engines | Semantics | +|---|---|---|---| +| Serving / analytics | `ServingBackend` via `QueryEngine` (ADR 0006) | DuckDB (demo) / ClickHouse (scale) | Analytical reads: entity point-reads, metric aggregates, the `pipeline_events` journal scan (`QueryEngine.fetch_pipeline_events` — the freshness axis) | +| Control plane / transactional | `ControlPlaneStore` (ADR 0010) | embedded DuckDB+YAML (demo) / PostgreSQL (scale) | Mutable row state with claim semantics: webhook queue, alert state, outbox, **dead-letter lifecycle** (`failed → replay_pending → replayed` / `dismissed`), usage | + +### Verified inventory — where each surface's data lives today (`main=c2749dd`) + +| Data | Home | Access path | Notes | +|---|---|---|---| +| Order row, open-order set | `orders_v2` on the serving backend | `QueryEngine.get_entity` / catalog SQL | CH table is `ReplacingMergeTree ORDER BY order_id` — key point-reads are cheap on both engines | +| Per-order event history | `pipeline_events` journal | `QueryEngine.fetch_pipeline_events(entity_id=…)` | The journal has an `entity_id` axis, **but only the demo seed populates it** — the live pipeline writes journal rows without `entity_id` (`local_pipeline.py` insert sites, `clickhouse_sink.record_pipeline_event` has no such parameter). On CH the journal is `ORDER BY (tenant_id, topic, processed_at, event_id)`, so an `entity_id` lookup is scan-shaped — acceptable for one bounded order trail, wrong as a triage backbone | +| Customer golden record | `bv_customer_mdm` in the DV2 vault | engine-side governed views (ADR 0006 Phase 2) | PII-full, behind per-jurisdiction roles designed for warehouse principals. The serving store carries the PII-free projection `users_enriched` (the serving tier holds no PII by construction — 2026-07-01 decision) | +| Failed events + replay lifecycle | `dead_letter_events` + `outbox` | `ControlPlaneStore` | Tenant-scoped reads, `replay`/`dismiss` actions already live on `/v1/deadletter`; invariant 8 (outbox↔dead-letter transactional flip) preserved by the port | +| Webhook deliveries parked `dead` | webhook queue | `ControlPlaneStore` | Visible today only via per-webhook logs; no unified triage view | +| SLA budgets | nowhere yet | — | `contracts/entities/order.yaml` documents the status vocabulary but carries no stage budgets | +| OLTP hot tier (warehouse contour) | `ops_` PostgreSQL schemas (`pg_ops__*`) | DV2 promotion + LISTEN/NOTIFY | The warehouse-side archetype of the same split: "Postgres is a buffer, not a system of record; cross-branch joins live in CH, never in Postgres". Not wired to the serving API | + +### Constraints + +1. **The demo profile stays zero-dependency.** Both ports have embedded + adapters; the ops surfaces must run on them unchanged. +2. **The serving tier stays PII-free by construction.** Whatever the timeline + shows about a customer must come from the PII-free projection, not the + vault. +3. **Freshness is the point.** These surfaces are the business reading of the + event→metric axis (`domain.md` §4.1, oversell): a TTL-cached or + batch-materialized ops read would re-create the very lag the five programs + are being killed for. +4. **No third data path.** The 2026-07-02 architecture audit (§3.1, §3.6) + spent an entire phase (E) evicting direct `query_engine._conn` reads from + the control plane; the ops layer must not grow the seam back. + +## Options considered + +### 1. Serve everything from the analytics backend (ClickHouse at scale) + +Rejected. Triage is mutable row state with lifecycle transitions and actions — +this re-litigates ADR 0010 option 1 (asynchronous mutations, no row locks, +no transactional flip). Worse, the dead-letter lifecycle *already* lives +behind the control-plane port; reading a CH copy of it would fork the state +the replay/dismiss actions mutate. + +### 2. Serve everything from the transactional store (PostgreSQL at scale) + +Rejected. The worklist and aging aggregates are analytical scans over serving +data; serving them from PG means copying `orders_v2` and the journal into the +control-plane store — a second serving engine, against ADR 0006. The +warehouse's own hot tier is explicitly a rolling 30-day buffer, not history. + +### 3. Read the DV2 vault directly for the customer block + +Rejected for v1. Vault PII sits behind per-jurisdiction engine-side roles +(officers, analysts) that were deliberately designed for warehouse +principals, not for an always-on API service account; and the timeline's +customer block needs no PII — `users_enriched` (orders, spend, recency, +preferred category) is exactly the PII-free projection of the golden record. +A dedicated non-PII vault role for a richer MDM block is recorded as roadmap, +not v1. + +### 4. Precompute ops marts (materialized stuck-list / inbox tables) + +Rejected for v1. The open-order set is small and bounded; computing on read +keeps the surfaces at serving-store freshness (constraint 3) and adds no new +moving parts. Materialization remains a later optimization that must not +change the API contract. + +### 5. Two-port composition rule — chosen + +Formalize the split the platform already converged on (ADR 0010: "ClickHouse += analytics/serving, PostgreSQL = transactional state, Redis = cache/limits") +as a *rule for the ops layer*, executable today on the embedded profiles. + +## Decision + +1. **Every ops surface composes exactly the two existing ports.** Analytical + reads — entity point-reads, journal scans and history, open-order + aggregates — go through `QueryEngine` / `ServingBackend`. Transactional + state — triage lifecycles, actions, the manual-work counter — goes through + `ControlPlaneStore`. No ops code path touches `query_engine._conn`, a raw + engine connection, or a vault DSN (spec invariant I1, pinned by a + structural test like the slice-1 ratchet). + +2. **Per-surface mapping:** + + | Surface | Serving backend (analytics) | Control-plane store (transactional) | + |---|---|---| + | Order 360 timeline | order row · stage history + pipeline trail from the journal · `users_enriched` customer block | dead-letter detail for the order's failed events (journal rows are the index, the store is the truth) | + | Stuck-orders worklist | the whole computation (open orders × latest stage-entry row, budgets from the contract) | — | + | Exception inbox | journal only as the entity index for findings | item truth: dead-letter lifecycle, dead webhook deliveries, reconciliation findings + triage overlay, stats/counter | + +3. **The journal's entity axis becomes real.** The serving-store projection + writes stage-entry rows (topic `orders.status`, event_type + `order.status.`, `entity_id` = order id) on every order status + transition, and the existing validated/dead-letter journal writes carry + `entity_id` where it is derivable from the event payload + (`clickhouse_sink.record_pipeline_event` gains the parameter). Without + this, the timeline and the stage clock exist only for seeded demo rows — + the current live-write sites leave `entity_id` NULL. + +4. **SLA budgets are catalog data, not code.** A `stages:` block in + `contracts/entities/order.yaml` (ordered ladder, per-stage budget, + terminal markers) is parsed into the catalog's `EntityDefinition` and is + the only source of budgets for the worklist and the timeline's breach + flag. Agents and SDKs see the SLA model through the same catalog surface + as everything else. + +5. **The exception-triage overlay is the seventh control-plane state class.** + ADR 0010's inventory grows by one table (triage status for items that have + no native lifecycle: dead webhook deliveries, reconciliation findings), + implemented in both adapters. Items with a native lifecycle — dead-letter + events — are *mapped* into the inbox read model and are never duplicated + into the overlay; their actions stay on `/v1/deadletter`. + +6. **Consistency semantics, stated:** ops surfaces read at serving-store + freshness and are not metric-cached; the stage clock falls back to + `created_at` (and says so) for orders with no stage rows; reconciliation + checks run on inbox read with idempotent, dedupe-keyed finding upserts — + safe at `replicaCount > 1` under ADR 0010's semantics (reads are + stateless, writes are single-row transactions). + +7. **Scale-profile evolution stays a swap, not a rewrite.** Rebinding the + order point-read to a true OLTP hot tier (the `pg_ops__` pattern the + warehouse contour already demonstrates) is a recorded F-phase option + behind the same port boundary; the surface contracts in the spec must not + change if that rebinding happens. + +## Consequences + +### Positive + +- All three surfaces land on the zero-dependency demo profile — both ports + already have embedded adapters, so D2–D4 add no services. +- The audit's "no third path" concern becomes an enforced rule (structural + ratchet), not a review habit. +- Triage state inherits the PG adapter's transactional/claim semantics at + scale for free; the pinned control-plane regression suites transfer. +- The SLA model is data (contract), so budget changes are a config edit with + contract-versioning discipline, and the catalog/search surfaces expose it. + +### Negative + +- The stage clock depends on journal hygiene: orders written before the + stage-row writer shipped (or by writers that bypass it) degrade to the + `created_at` fallback — visible in the response (`clock: "fallback"`) + rather than hidden. +- Seeding stage rows moves the demo `error_rate` denominator (it counts all + journal rows in the window); the affected pinned tests are re-pinned by + arithmetic, per the B3 house rule — recorded in the spec, not discovered in + CI. +- The worklist is computed on read: O(open orders × journal window) per call. + Bounded and cheap at demo scale; the materialization option is recorded for + when it is not. +- Inbox reads trigger reconciliation checks against both stores — bounded + read amplification, accepted for v1 (no scheduler dependency). + +## Rollout + +- **D2** — Order 360: timeline endpoint, journal entity axis (§ Decision 3), + stage-entry seed trails, `error_rate` re-pins. +- **D3** — stuck-orders: `stages:` contract block + catalog parsing, + `/v1/ops/stuck-orders`, aging/summary. +- **D4** — exception inbox: triage overlay (state class 7), sources + (dead-letter mapping, dead webhook deliveries, reconciliation checks R1/R2), + `/v1/ops/exceptions` + stats + manual-resolutions counter. + +Each slice is one PR with the full suite green; endpoint contracts, schemas, +demo-story pins, and per-slice test obligations are in +[`docs/ops-surfaces-spec.md`](../ops-surfaces-spec.md). + +## Follow-up + +- F-phase option: order point-read rebinding to an OLTP hot tier behind the + same port (Decision 7). +- Per-channel SLA budgets need a channel column on serving orders — recorded + as out of v1 scope (`ops-surfaces-spec.md` § Non-goals). +- A non-PII vault role for a richer MDM customer block — roadmap (option 3). +- Container-ETA and marking-code exception sources (`domain.md` §4.2, §3) — + named roadmap sources for the inbox, blocked on `excel__`/WMS feeds landing + in the serving contour. diff --git a/docs/domain.md b/docs/domain.md index 1b52235..cafd6f8 100644 --- a/docs/domain.md +++ b/docs/domain.md @@ -14,7 +14,10 @@ Downstream consumers: - **Docs sweep** — README, `docs/architecture.md`, and the DV2 docs inherit the storyline and the vocabulary from §5. - **Operational layer design** — the three ops surfaces (order timeline, - stuck-orders worklist, exception inbox) serve the workflows in §4. + stuck-orders worklist, exception inbox) serve the workflows in §4; the + serving split is decided in + [ADR 0011](decisions/0011-ops-serving-split.md) and the surface contracts + are pinned in [`ops-surfaces-spec.md`](ops-surfaces-spec.md). --- diff --git a/docs/ops-surfaces-spec.md b/docs/ops-surfaces-spec.md new file mode 100644 index 0000000..9c9176c --- /dev/null +++ b/docs/ops-surfaces-spec.md @@ -0,0 +1,434 @@ +# Ops Surfaces Spec — Order 360, Stuck-Orders, Exception Inbox + +The implementation contract for the operational layer: the three surfaces +that replace the five-tool triage routine described in +[`domain.md`](domain.md) §4. The architectural decision they execute — which +data plane serves which read shape — is +[ADR 0011](decisions/0011-ops-serving-split.md); this document pins the +endpoint contracts, the SLA stage model, the exception sources, the demo +story, and the per-slice test obligations. + +Consumers: + +- **D2** — Order 360 timeline (§2, plus the shared foundations §1.2–§1.4 and + §1.6 it delivers). +- **D3** — stuck-orders worklist (§3, plus the contract `stages:` block + §1.5). +- **D4** — exception inbox (§4). +- §5 is the invariant list — the test ТЗ for all three slices, in the same + spirit as `generator-spec.md` §12. + +Ground rules inherited from ADR 0011: every surface composes exactly the +`QueryEngine`/`ServingBackend` port (analytical reads) and the +`ControlPlaneStore` port (transactional state); no `_conn`, no vault DSN; the +demo profile runs everything on the embedded adapters with zero new +dependencies. + +## 1. Shared foundations + +### 1.1 Stage model + +Stages are the business fulfilment statuses already pinned by +`contracts/entities/order.yaml`: + +``` +pending → confirmed → shipped → delivered + ↘ cancelled (from any non-terminal stage) +``` + +- **Ladder (non-terminal, in flow order):** `pending`, `confirmed`, + `shipped`. +- **Terminal:** `delivered`, `cancelled` — never stuck, never carry a budget. +- Stages are distinct from the *pipeline trail* (`order.created`, + `order.validated`, `order.served`, dead-letter/replay events): the trail + describes the platform moving data, stages describe the warehouse moving + goods. The timeline shows both, separately (§2.2). + +### 1.2 Stage-entry journal rows + +The `pipeline_events` journal is the stage clock. A stage entry is one +journal row: + +| Column | Value | +|---|---| +| `topic` | `orders.status` | +| `event_type` | `order.status.` (e.g. `order.status.confirmed`) | +| `entity_id` | the order id (`ORD-…`) | +| `tenant_id` | tenant of the write, as elsewhere in the journal | +| `processed_at` | the transition time | +| `latency_ms` | `NULL` (not a pipeline hop) | + +Writer: the serving-store projection site — wherever `orders_v2` is created +or its `status` transitions (local pipeline write path and its ClickHouse +mirror), one `orders.status` row is emitted alongside. Order creation emits +`order.status.pending`. The event-type namespace `order.status.*` is +deliberately disjoint from the ingestion vocabulary (`order.created`, +`order.updated`, …) so nothing that filters on ingestion types picks up +stage rows. + +### 1.3 Journal entity axis on live writes + +Today only the demo seed populates `pipeline_events.entity_id`; the live +write sites leave it NULL (`local_pipeline.py` validated/dead-letter inserts +name six columns without `entity_id`; `clickhouse_sink.record_pipeline_event` +has no such parameter). D2 fixes both: + +- validated and dead-letter journal writes carry `entity_id` extracted from + the event payload when derivable (`order_id` for `order.*` events, + `user_id` for `user.*`, `product_id` for `product.*`, `session_id` for + `session.*`); NULL when not derivable — never a synthesized id; +- `clickhouse_sink.record_pipeline_event` gains an `entity_id: str | None` + keyword (default `None` — existing callers stay valid). + +### 1.4 Stage clock resolution + +For an order whose current status is `S` (from `orders_v2`): + +1. `entered_at` = max `processed_at` over journal rows with + `topic='orders.status'`, `entity_id=`, `event_type='order.status.S'`; + `clock: "journal"`. +2. No such row → `entered_at = created_at` (the order row), `clock: + "fallback"`. Honest degradation for pre-existing or bypass-written orders; + the response says so rather than hiding it. +3. `in_stage_seconds = now − entered_at`; `breached = in_stage_seconds > + sla_minutes × 60` for ladder stages, `null` for terminal or unknown + stages. +4. A status outside the contract vocabulary resolves to `stage: unknown`, + `breached: null` — surfaced, never a crash (I4). + +### 1.5 SLA budgets — the `stages:` contract block + +Budgets are catalog data. `contracts/entities/order.yaml` gains: + +```yaml +stages: + - name: pending + sla_minutes: 30 + description: Confirmation SLA — marketplace orders auto-confirm within + minutes; a pending order older than this needs a payment/CRM decision + - name: confirmed + sla_minutes: 1440 + description: Ship-by SLA — FBS marketplace shipping deadlines drive the + 24h budget for warehouse handover + - name: shipped + sla_minutes: 7200 + description: Delivery SLA — 5 days courier/pickup before a customer + contact is due + - name: delivered + terminal: true + - name: cancelled + terminal: true +``` + +- List order = ladder order. `terminal: true` entries carry no + `sla_minutes`. +- Parsed into an optional `stages` field on the catalog's + `EntityDefinition` (default `None` — entities without the block behave as + today; catalog parsing must not require it). +- The block is the **only** source of budgets (I2): no stage literals or + budget constants in router/engine code. +- One ladder for all channels in v1. Per-channel budgets (B2B wholesale + vs FBS have genuinely different clocks) require a channel column on + serving orders — recorded in § Non-goals. +- Budget values above are grounded in the legend (`domain.md` §2): they are + the demo defaults, and changing them is a contract edit, not a code edit. + +### 1.6 Demo story (seed additions and their pinned consequences) + +D2 seeds stage trails for the 8 demo orders (both backends, mirrored — the +B3 discipline): each order gets `order.status.pending` at its `created_at`, +plus stage entries consistent with its current status, back-dated +plausibly between `created_at` and now. Pinned outcomes: + +- **ORD-20260404-1004** (pending, created 45 min ago) is the sole SLA breach + in the demo — 45 min in `pending` against a 30-min budget, overshoot 1.5×. + The stuck-orders default view returns exactly this order (I7). The + legend reading: a marketplace order stuck before confirmation — the "где + заказ?" question the ops manager asks first. +- **ORD-20260404-1001** (delivered) shows the full Order 360 story: complete + stage history, the existing pipeline trail (`order.created` → + `order.validated` → `order.served`), and the customer block for + `USR-10001` (I7). +- Every other open order sits inside its budget; terminal orders + (`delivered`/`cancelled`) never appear in the worklist. + +Seeding stage rows grows the journal, and the demo `error_rate` divides +dead-letter rows by **all** journal rows in the window — the pinned demo +value moves. Re-pin expected values by arithmetic from the new seed +(house rule since B3), never by copying observed output (I9). + +### 1.7 Tenancy and auth + +- All three surfaces resolve the tenant from request state exactly like the + existing entity/deadletter routes, and scope both ports' reads with it + (I8). +- Reads require a valid API key (any scope). Mutations (inbox + acknowledge/resolve) follow the dead-letter write rule: full-access keys + only; scoped keys get 403. + +### 1.8 Caching + +Ops responses are not metric-cached and set no cache headers: they are the +"now" surface (ADR 0011 constraint 3). The underlying entity/journal reads +are already cheap on both engines at the volumes involved. + +## 2. Surface 1 — Order 360 timeline (D2) + +**`GET /v1/entity/order/{order_id}/timeline`** — same router family as the +existing entity read (`agent_query.py` owns `/v1/entity/*`), same 404 +semantics (unknown order → 404, same shape as the entity route), same tenant +scoping. + +### 2.1 Composition (per ADR 0011 mapping) + +| Block | Source port | Read | +|---|---|---| +| `order` | serving | `QueryEngine.get_entity("order", id)` | +| `stage` + `stage_history` | serving | journal rows `topic='orders.status'`, `entity_id=`, ascending; clock per §1.4 | +| `pipeline_trail` | serving | journal rows for `entity_id=` with `topic != 'orders.status'`, ascending | +| `customer` | serving | `QueryEngine.get_entity("user", order.user_id)` — the PII-free serving projection of the MDM golden record (`users_enriched`); `null` when absent | +| `exceptions` | control plane | for trail rows with `topic='events.deadletter'`: `get_dead_letter_event(event_id, tenant)` — the journal is the index, the store is the truth | + +### 2.2 Response shape + +```json +{ + "order": { "order_id": "…", "user_id": "…", "status": "…", + "total_amount": 0, "currency": "RUB", "created_at": "…" }, + "stage": { "current": "pending", "entered_at": "…", + "in_stage_seconds": 2700, "sla_minutes": 30, + "breached": true, "clock": "journal" }, + "stage_history": [ { "status": "pending", "at": "…" } ], + "pipeline_trail": [ { "event_id": "…", "topic": "…", "event_type": "…", + "latency_ms": 12, "processed_at": "…" } ], + "customer": { "user_id": "…", "total_orders": 34, "total_spent": 0, + "first_order_at": "…", "last_order_at": "…", + "preferred_category": "grills" }, + "exceptions": [ { "event_id": "…", "failure_reason": "…", "status": "failed", + "occurred_at": "…", + "actions": { "replay": "/v1/deadletter/{id}/replay", + "dismiss": "/v1/deadletter/{id}/dismiss" } } ] +} +``` + +The customer block is a fixed field allow-list (the `users_enriched` +columns) — no PII field can appear in a timeline response by construction +(I3). + +### 2.3 D2 scope + +Endpoint + response models; §1.2 stage-row writer (both write paths); §1.3 +entity axis on live writes; §1.6 seed trails; `error_rate`/journal-count +re-pins by arithmetic; tests per §5. + +## 3. Surface 2 — stuck-orders worklist (D3) + +**`GET /v1/ops/stuck-orders`** — new `routers/ops.py` (prefix `/v1/ops`). + +### 3.1 Parameters + +| Param | Default | Meaning | +|---|---|---| +| `stage` | — | filter to one ladder stage | +| `include_within_sla` | `false` | `true` returns the whole open worklist, not only breaches | +| `page` / `page_size` | `1` / `50` (max 100) | dead-letter pagination conventions | + +### 3.2 Computation + +One serving-backend query shape (no per-order round-trips): open orders +(`status` in the ladder) left-joined to their latest `orders.status` row, +`entered_at` fallback per §1.4, budgets from the contract `stages:` block +(I2), `overshoot_ratio = in_stage_seconds / (sla_minutes × 60)`. Default +view: breaches only, ordered by `overshoot_ratio` desc. Computed on read at +serving-store freshness — no materialized state (ADR 0011 option 4). + +### 3.3 Response shape + +```json +{ + "items": [ { "order_id": "…", "user_id": "…", "status": "pending", + "entered_at": "…", "in_stage_seconds": 2700, + "sla_minutes": 30, "overshoot_ratio": 1.5, + "clock": "journal", "total_amount": 1890.0, + "currency": "RUB" } ], + "summary": { "open_by_stage": { "pending": 2, "confirmed": 2, "shipped": 1 }, + "breached_by_stage": { "pending": 1 } }, + "pagination": { "page": 1, "page_size": 50, "total": 1, "pages": 1 } +} +``` + +### 3.4 D3 scope + +Contract `stages:` block + catalog parsing (§1.5, optional field, tolerant +of absence); endpoint + summary; tests per §5. D3 depends on D2's stage rows +being seeded/written but degrades honestly without them (`clock: +"fallback"` from `created_at` — the demo breach story survives either way, +since ORD-1004's pending entry equals its `created_at`). + +## 4. Surface 3 — exception inbox (D4) + +**`GET /v1/ops/exceptions`** · **`GET /v1/ops/exceptions/stats`** · +**`POST /v1/ops/exceptions/{item_id}/acknowledge`** · +**`POST /v1/ops/exceptions/{item_id}/resolve`** + +One triage feed for "what failed and needs a human" (`domain.md` §4.3), +aggregating sources that today live on three different screens. + +### 4.1 Sources (v1) + +| # | Source | Truth | Lifecycle | Actions | +|---|---|---|---|---| +| 1 | Dead-letter events | `ControlPlaneStore` dead-letter methods (tenant-scoped) | **native**, mapped read-only: `failed → open`, `replay_pending → in_progress`, `replayed`/`dismissed → resolved` | links to `/v1/deadletter/{id}/replay` and `/dismiss`; inbox mutation of a `dl:` item → 409 (I6) | +| 2 | Webhook deliveries parked `dead` | webhook queue via a new port read (`list_dead_webhook_deliveries`, both adapters) | overlay (§4.2) | `acknowledge` / `resolve` | +| 3 | Reconciliation findings (§4.3) | computed on read, upserted into the overlay by dedupe key | overlay, plus **auto-resolve** when a finding no longer reproduces | `acknowledge` / `resolve` | + +Roadmap sources, named but not v1: container-ETA staleness (`excel__` +manifests), marking-code gaps at receiving (`domain.md` §3), SLO freshness +breaches (already alertable via `/v1/slo` + alert rules — the inbox links +rather than duplicates alerting). + +### 4.2 Triage overlay — control-plane state class 7 + +One table (embedded DuckDB + PostgreSQL adapters, extending ADR 0010's +inventory): + +``` +ops_exception_triage( + item_id TEXT PRIMARY KEY, -- stable id, §4.4 + tenant_id TEXT, + source TEXT, -- 'webhook_delivery' | 'reconciliation' + status TEXT, -- 'open' | 'acknowledged' | 'resolved' + first_seen_at / last_seen_at / resolved_at TIMESTAMP, + note TEXT -- optional operator note on resolve +) +``` + +Port methods (names final at implementation, semantics pinned here): +ensure-schema; get/list triage states for a set of item ids; set state +(single-row transactional upsert); upsert-finding (insert `open` or refresh +`last_seen_at`; **never** reopens a row an operator resolved unless the +finding reproduces after `resolved_at`); count manual resolutions for the +stats window. Dead-letter items get **no** overlay rows — their native +machine is the single source of truth (I6). + +### 4.3 Reconciliation checks (v1) + +Concretizing the "reconciliation" half of the phase-D plan — cross-store +consistency probes that today are nobody's job: + +- **R1 `journal_vs_store`** (severity `high`): over the check window, for + every `entity_id` seen in `orders.status` journal rows, the serving store + must have the order row and its `status` must not be *behind* the latest + journal stage. A mismatch means an event landed but the serving projection + didn't (or forked) — the silent failure mode of the store-and-journal + double write. Dedupe key: `r1::`. +- **R2 `stuck_replay`** (severity `medium`): dead-letter rows sitting in + `replay_pending` longer than a threshold (default: the control-plane lease + interval; env-tunable) — a replay was requested but its outbox entry never + completed the invariant-8 flip. Dedupe key: `r2:`. + +Checks run on inbox read (no scheduler dependency), read both ports +read-only, and write only overlay upserts — idempotent and dedupe-keyed, so +concurrent reads and `replicaCount > 1` are safe (I10). A finding absent in +the current run auto-resolves its `open` overlay row (status `resolved`, +note `auto-resolved: no longer reproduces`). + +### 4.4 Item model + +```json +{ "item_id": "dl:evt-004", + "source": "deadletter" | "webhook_delivery" | "reconciliation", + "severity": "high" | "medium" | "low", + "occurred_at": "…", "last_seen_at": "…", + "entity_ref": { "kind": "event" | "order" | "webhook", "id": "…" }, + "title": "…", "detail": "…", + "status": "open" | "in_progress" | "acknowledged" | "resolved", + "actions": [ … ] } +``` + +Stable ids: `dl:`, `wh::`, +`rc:` — the same underlying fact always maps to the same +`item_id` (I5). List params: `source`, `status` (default: everything not +`resolved`), pagination as §3.1. Severity defaults: dead-letter `high`, +dead webhook delivery `medium`, per-check for reconciliation. + +### 4.5 Stats and the manual-work counter + +`GET /v1/ops/exceptions/stats` returns counts by `source × status`, a +`last_24h` new-item count, and **`manual_resolutions`** for the window: the +number of human triage decisions absorbed by the platform — dead-letter +replays + dismisses (from native state transitions) + overlay +acknowledge/resolve actions. This is the kill-five-programs KPI +(`domain.md` §4): triage decisions that used to be five-screen detective +work, countable because they now happen in one feed. + +### 4.6 D4 scope + +Overlay table + port methods in both adapters; `list_dead_webhook_deliveries` +port read; R1/R2 checks; the three endpoints; a non-empty demo inbox story — +the two seeded dead-letter journal rows (`evt-004`, `evt-009`) get store +counterparts so the demo feed shows real items (affected dead-letter +stats/count pins re-pinned by arithmetic, I9); tests per §5. + +## 5. Invariants (the test ТЗ) + +Machine-checkable; each Dx slice lands the tests for the invariants it +touches, and G2 audits against the full list. + +- **I1 — no third path (structural ratchet).** No module under + `routers/ops*` and no timeline code references `query_engine._conn`, a raw + backend connection, or a vault DSN; ops surfaces import only the two + ports. Same test pattern as the ADR 0010 slice-1 dispatcher ratchet. +- **I2 — budgets only from the contract.** No stage-name or budget literal + in ops router/engine code paths; pointing the catalog at a fixture + contract with different `sla_minutes` changes worklist/timeline behavior + accordingly. +- **I3 — PII-free by construction.** Timeline and inbox response models are + fixed allow-lists; `first_name`/`last_name`/`email`/`phone`/`birth_date` + never appear (structural test over the response models, not a runtime + scrub). +- **I4 — stage vocabulary tolerance.** Stage rows use `topic='orders.status'` + and `event_type='order.status.'`; a status outside the contract + ladder yields `stage: unknown` / `breached: null`, never a 500. +- **I5 — stable item ids.** Same fact → same `item_id` across calls; + listings never show duplicates for one underlying fact. +- **I6 — native lifecycles are not duplicated.** `dl:` items have no overlay + rows; inbox mutation endpoints reject them with 409; their status in the + feed always mirrors the dead-letter store. +- **I7 — demo story pins.** Default worklist = exactly `ORD-20260404-1004` + (overshoot 1.5×, clock `journal`); `ORD-20260404-1001` timeline carries a + ≥3-row pipeline trail, full stage history, and the `USR-10001` customer + block; the demo inbox is non-empty. +- **I8 — tenant scoping.** Every surface scopes both ports' reads by the + request tenant; a foreign tenant's key sees none of the demo tenant's + orders/items (cross-tenant test per surface). +- **I9 — re-pin by arithmetic.** Seed changes (stage rows, dead-letter store + counterparts) re-pin affected expected values (`error_rate`, dead-letter + stats) computed from the seed, never copied from observed output. +- **I10 — read surfaces don't write serving state.** Ops endpoints perform + no serving-store writes; reconciliation writes only overlay upserts; + running checks concurrently is idempotent (dedupe keys). +- **I11 — live journal writes carry the entity axis.** Validated and + dead-letter journal writes (DuckDB path and ClickHouse mirror) set + `entity_id` for payloads where it is derivable; unit tests cover both + sites. +- **I12 — fallback honesty.** An order with no stage rows reports + `clock: "fallback"` with `entered_at = created_at` — asserted, so the + degraded mode stays visible instead of silently pretending to be a + journal clock. + +## 6. Non-goals (v1) + +- **Per-channel SLA ladders** — needs a channel column on serving orders; + single ladder v1 (§1.5). +- **Container-ETA / marking-code exception sources** — blocked on + `excel__`/WMS feeds reaching the serving contour (`domain.md` §4.2); + named roadmap in §4.1. +- **Vault MDM block in the timeline** — needs a dedicated non-PII vault + role (ADR 0011 option 3); `users_enriched` projection v1. +- **Materialized ops marts** — computed-on-read v1 (ADR 0011 option 4); the + API contract must survive later materialization unchanged. +- **Push notifications for breaches** — the alert subsystem already owns + paging; the inbox links to it rather than growing a second notifier. +- **A generic workflow engine** — the overlay is triage state (three + statuses, one note), not case management.