diff --git a/docs/dv2-multi-branch/demo_evidence.md b/docs/dv2-multi-branch/demo_evidence.md index c3ad10fa..efa3185a 100644 --- a/docs/dv2-multi-branch/demo_evidence.md +++ b/docs/dv2-multi-branch/demo_evidence.md @@ -1,18 +1,62 @@ # DV2.0 Multi-Branch — Live Demo Evidence -Captured against the running `hq-demo` cluster on the iMac demo host -() on 2026-05-23. Every block is reproducible from -`infrastructure/dv2/bootstrap.sh`. - -> **2026-06-07 — scale numbers refreshed at real X5 volume.** Sections 5-7 -> now show the cluster loaded with the X5 Retail Hero dataset (8.06M orders / -> 45.8M line items, branch-sharded 40/25/15/10/10). Sections 1-4 and 8 -> (topology, pinning, storage, MDM conflict-resolution mechanics) are -> volume-independent and stand as captured on the synthetic seed. Serving -> latency at X5 is benchmarked in [`load-test-baseline.md`](load-test-baseline.md). +**Data sections (§4–8) re-captured 2026-07-03** on the current +kitchen-gadget legend (post-B1/B2/B3 seeds), against standalone stands, no +Docker: + +- **ClickHouse** — `clickhouse server` 26.7.1.492 (single binary, WSL Ubuntu + 22.04). Vault built from the repo files verbatim: `__init.sql` → 8 hubs → + 8 links → 48 satellites → `synthetic_seed.sql` + `satellite_seed.sql` + + `satellite_seed_all_branches.sql` → `business_vault/*.sql` (views with + `SQL SECURITY DEFINER`) → `governance/01..04.sql`. +- **PostgreSQL** — 17.5 (EDB windows-x64 binaries, `initdb` + `pg_ctl`, port + 55432, trust auth, user/db `agentflow`). Vault built via `postgres/apply.sh` + (schema → 8 hubs → 8 links → 48 satellites → `03_business_vault.sql`) → + `postgres/governance/01..04.sql`. + +> **Legend reset (2026-07-03).** The demo no longer models an X5 retail +> contractor. It is an own-brand ("private-label smart-kitchen") importer: +> China-manufactured goods, RU HQ, bimodal channel economy (money in +> wholesale, order-count on marketplaces), five branches across three +> jurisdictions (msk / spb / ekb + dxb + ala). All prior "X5 Retail Hero / +> 8.06M orders / USD" figures are retired. The synthetic demo seed is now +> **2,500 customers / 10,000 orders / 160 SKU / 14,853 line items, priced in +> ₽** — see `docs/domain.md` and `docs/generator-spec.md` for the model. + +> **⚠ Infra sections (§1–3, §9–15) are pending re-capture on the kind +> cluster (Mac stand).** Topology, workload pinning, PVCs, the MinIO cold +> tier, the Postgres→ClickHouse bridge, the MaterializedPostgreSQL CDC path, +> Argo orchestration and the dbt-in-Kubernetes Job all need the single-network +> kind cluster; the standalone WSL-CH / Windows-PG split used for this +> re-capture cannot reproduce cross-engine networking or Kubernetes. Their +> **mechanisms are volume- and legend-independent** and stand as previously +> demonstrated, but the **row counts printed in those blocks reflect the +> retired X5-era volumes** and must not be read as current. They are flagged +> inline and are the Mac-tail of this step (see `plan_endgame_02_07_26.md`). + +## Governance verify_live — both engines green on the new seeds + +The adversarial PII-governance matrices (`governance/verify_live.sh` on +ClickHouse, `postgres/governance/verify_live.sh` on PostgreSQL) were re-run +against the freshly-built stands: + +| Stand | Result | Transcript | +| ----- | ------ | ---------- | +| ClickHouse 26.7.1.492 | **29/29 PASS**, 0 FAIL, 0 WARN | [`../perf/vault-pii-governance-verify-2026-07-03.md`](../perf/vault-pii-governance-verify-2026-07-03.md) | +| PostgreSQL 17.5 | **33/33 PASS**, 0 FAIL, 0 WARN | [`../perf/vault-pii-governance-pg-verify-2026-07-03.md`](../perf/vault-pii-governance-pg-verify-2026-07-03.md) | + +The PII boundary holds in every SQL shape on both engines; row policies scope +officers to their own jurisdiction; the msk demo seed now spans `1c__msk`, +`pg_ops__msk` and `mp__msk` record sources (the `x5__` convention was retired +in B2). The current CH suite defines 29 probes (earlier revisions cited 32); +the count is whatever the checked-in script asserts — every probe passes. ## 1. Cluster topology — `kubectl get nodes --show-labels` +> ⚠ **Kind-cluster section — pending Mac re-capture.** Topology and labels are +> legend-independent and stand as captured; re-run on the Mac kind stand for a +> current timestamp. + ``` NAME STATUS ROLES AGE VERSION hq-demo-control-plane Ready control-plane 77m v1.35.0 @@ -30,6 +74,8 @@ Labels decoded: ## 2. Workload pinning — `kubectl get pods -n dv2 -o custom-columns=POD,NODE` +> ⚠ **Kind-cluster section — pending Mac re-capture.** + ``` POD NODE STATUS clickhouse-0 hq-demo-worker2 Running @@ -42,6 +88,8 @@ edge nodes (`branch=dxb`, `branch=ala`). ## 3. Persistent storage — `kubectl get pvc -n dv2` +> ⚠ **Kind-cluster section — pending Mac re-capture.** + ``` NAME STATUS CAPACITY ACCESS MODES STORAGECLASS data-clickhouse-0 Bound 5Gi RWO standard @@ -50,132 +98,119 @@ data-postgres-0 Bound 2Gi RWO standard ## 4. DV2.0 model surface — `system.tables` grouped by family -38 tables in database `rv`: +70 tables in database `rv` (`clickhouse client -q "SELECT ... FROM +system.tables WHERE database='rv'"`): ``` -hub_* 8 (customer, product, order, shipment, store, supplier, employee, marking_code) -lnk_* 8 (order_customer, order_product, order_store, order_employee, - order_shipment, shipment_store, product_supplier, product_marking) -sat_* 22 (per-source × per-branch satellites; full matrix in spec.yaml) +hub_* 8 (customer, product, order, shipment, store, supplier, employee, marking_code) +lnk_* 8 (order_customer, order_product, order_store, order_employee, + order_shipment, shipment_store, product_supplier, product_marking) +sat_* 48 (per-source × per-branch satellites; full matrix in spec.yaml) ``` +The satellite matrix grew from the earlier 22 to **48** across the B1 rewrite +(per-jurisdiction personal / loyalty / order-header / pricing / marketplace +sources; full generation in `spec.yaml` + `generate_satellites.py`). + ## 5. Multi-branch distribution proof +The retired X5 seed spread orders 40/25/15/10/10 across branches. The current +legend does **not** — every marketplace and e-com order is fulfilled from the +msk hub (`mp__` is msk-only), and branch identity lives in the dealer / B2B / +PII layers, not in the marketplace order stream. Order distribution by +`record_source`: + ```sql -SELECT - splitByString('__', record_source)[2] AS branch, - count() AS orders, - round(count() * 100.0 / (SELECT count() FROM rv.hub_order), 1) AS pct -FROM rv.hub_order -GROUP BY branch ORDER BY pct DESC; +SELECT record_source, count() AS orders, + round(count() * 100.0 / (SELECT count() FROM rv.hub_order), 1) AS pct +FROM rv.hub_order GROUP BY record_source ORDER BY orders DESC; ``` ``` -┌─branch─┬──orders─┬──pct─┐ -│ msk │ 3225691 │ 40 │ -│ spb │ 2016191 │ 25 │ -│ ekb │ 1202248 │ 14.9 │ -│ dxb │ 814336 │ 10.1 │ -│ ala │ 796767 │ 9.9 │ -└────────┴─────────┴──────┘ +┌─record_source─┬─orders─┬──pct─┐ +│ mp__msk │ 8900 │ 89.0 │ +│ bitrix__msk │ 360 │ 3.6 │ +│ site__msk │ 280 │ 2.8 │ +│ bitrix__spb │ 180 │ 1.8 │ +│ bitrix__ekb │ 130 │ 1.3 │ +│ bitrix__ala │ 75 │ 0.8 │ +│ bitrix__dxb │ 75 │ 0.8 │ +└───────────────┴────────┴──────┘ ``` -(X5 capture, 2026-06-07.) The 40/25/15/10/10 split is the consistent-hashing -distribution that the X5 Retail Hero loader -(`warehouse/agentflow/dv2/loaders/x5_retail_hero/`) applies to the real -transactions — 8,055,233 orders land within 0.1 pp of the design split. +Collapsed to branch (`splitByString('__', record_source)[2]`): **msk 9,540 +(95.4%)**, spb 180, ekb 130, dxb 75, ala 75 — a total of **10,000 orders**. +The msk dominance is the legend, not a bug: marketplace + D2C is centrally +fulfilled, so the branch story is carried by the B2B dealer orders +(`bitrix__` = 820 rows) and the per-jurisdiction customer/PII split +(§8), not by the order-count histogram. -## 6. Latency floor — multi-branch aggregation +## 6. Multi-branch aggregation latency (demo scale) -`SYSTEM FLUSH LOGS; SELECT query_duration_ms, read_rows FROM system.query_log -WHERE query LIKE '%hub_order%' ORDER BY event_time DESC LIMIT 3`: +`clickhouse client --time -q "SELECT splitByString('__', record_source)[2] AS +branch, count() FROM rv.hub_order GROUP BY branch ORDER BY 2 DESC"`: ``` -┌─query_duration_ms─┬─read_rows─┐ -│ 1110 │ 8055234 │ -└───────────────────┴───────────┘ +0.015 s (10,000 hub_order rows, per-row splitByString, 2-vCPU WSL) ``` -(X5 capture, 2026-06-07.) 1.1 s for a multi-branch GROUP BY over **8.06M hub -rows** (with a per-row `splitByString`) on the 2-vCPU kind-on-Lima container — -the raw-vault scan path. Serving queries do not pay this: the materialized -marts answer the same business questions at **p99 20–197 ms** under -concurrency — full sweep in [`load-test-baseline.md`](load-test-baseline.md). +This is the **demo-scale synthetic seed** (10k orders), not a load benchmark. +The multi-million-row throughput characterisation is a separate artifact +([`load-test-baseline.md`](load-test-baseline.md)) and is not part of the +synthetic demo evidence; the earlier "1.1 s over 8.06M X5 rows" block is +retired with the X5 legend. ## 7. Line items reach ```sql -SELECT count() FROM rv.lnk_order_product; -- 45811505 +SELECT count() FROM rv.lnk_order_product; -- 14853 ``` -(X5 capture, 2026-06-07.) 45.8M line items across 8.06M orders — the real -X5 Retail Hero basket profile (~5.7 line items per order), loaded by the -backpressure-throttled bulk loader in 2h16m with the cluster cold-restart-safe -throughout (98 active parts, 3.48 GiB on disk post-load). +**14,853 line items across 10,000 orders (~1.49 per order)** — the bimodal +basket profile: marketplace and D2C orders are predominantly single-item, +while the B2B dealer orders carry multi-line baskets, pulling the mean just +above 1. `hub_product` holds **160 SKU**; `hub_marking_code` holds **12,160** +per-unit Chestny ZNAK marking codes (issued / in-circulation / withdrawn). ## 8. Business Vault — populated views with MDM conflict resolution -`warehouse/agentflow/dv2/satellite_seed.sql` populates the customer / order -satellites that `synthetic_seed.sql` deliberately leaves empty. After applying: - -``` -sat_customer_personal__1c__msk 800 rows (msk slice) -sat_customer_personal__1c__dxb 200 rows (dxb slice) -sat_customer_loyalty__bitrix__msk 640 rows (80% loyalty coverage) -sat_order_header__bitrix__msk 4000 rows (msk slice) -sat_order_pricing__1c__msk 4000 rows (msk slice) -``` - -`bv_customer_mdm__msk` (PII from 1C, loyalty from Bitrix): - -``` -rows | with_pii | with_loyalty | pii_only | loyalty_only - 800 | 800 | 640 | 160 | 0 -``` - -The 160 `pii_only` rows are msk customers without a Bitrix profile yet — the -LEFT JOIN keeps them visible with `loyalty_source = NULL`, exactly as the -view contract documents. - -Sample (PII + loyalty merged for the same `customer_hk`): - -``` -Ivan Volkov cust236@example.test gold 3068 pii=1c__msk loy=bitrix__msk -Egor Petrov cust833@example.test bronze 10829 pii=1c__msk loy=bitrix__msk -Lena Sidorov cust138@example.test bronze 1794 pii=1c__msk loy=bitrix__msk -``` - -`bv_customer_mdm__dxb` returns 200 rows with Arabic-style faux PII, all -tagged `pii=1c__dxb`. The MSK view never returns them — the per-branch view -+ RBAC primitive is what enforces jurisdictional isolation here. - -All five `bv_customer_mdm__*` views populated (after extending spec.yaml + -satellite_seed_all_branches.sql): +`bv_customer_mdm__` merges PII from 1C with loyalty from Bitrix +(LEFT JOIN — customers without a Bitrix profile stay visible with +`loyalty_source = NULL`). Per-branch shape +(`count()`, `email != ''`, `loyalty_segment != ''`): ``` branch rows with_pii with_loyalty -ala 200 200 0 (KZ — no Bitrix loyalty by design) -dxb 200 200 0 (UAE — no Bitrix loyalty by design) -ekb 300 300 240 (80% loyalty coverage) -msk 800 800 640 -spb 500 500 400 +msk 2190 2190 152 (2,000 retail + 190 dealers) +spb 100 100 80 (dealers only) +ekb 70 70 56 (dealers only) +dxb 60 60 0 (dealers only — no loyalty by design) +ala 80 80 0 (dealers only — no loyalty by design) ``` -`bv_order_canonical` now joins Bitrix header + 1C pricing across every -branch (the view UNION ALL's all five `sat_order_header__bitrix__*` and -`sat_order_pricing__1c__*` satellites): +Total **2,500 customers**. The legend puts **all retail under the msk legal +entity** (regions carry only dealer accounts), so msk holds 2,190 of the 2,500 +customers. Loyalty (a dealer retro-bonus program, not a consumer points +scheme) runs only in msk / spb / ekb; dxb (UAE) and ala (KZ) dealers have a +contract, not a bonus — hence `with_loyalty = 0` there, by design. msk loyalty +tiers: **core 38 / mid 76 / tail 38** (152 total ≈ 80% of the 190 msk +dealers). + +`bv_order_canonical` joins Bitrix header + 1C pricing (+ Wildberries state +for the msk marketplace) across every branch: ``` branch orders with_header with_pricing -msk 4000 4000 4000 -spb 2500 2500 2500 -ekb 1500 1500 1500 -dxb 1000 1000 1000 -ala 1000 1000 1000 +msk 9540 9540 9540 +spb 180 180 180 +ekb 130 130 130 +dxb 75 75 75 +ala 75 75 75 ``` -The jurisdiction-specific tax rates fall straight out of per-branch 1C -satellites — one BI query exercises the entire multi-branch model: +All 10,000 orders resolve a header and pricing. The jurisdiction-specific tax +rates fall straight out of the per-branch 1C pricing satellites — one BI query +exercises the entire multi-branch model: ```sql SELECT branch, @@ -193,390 +228,142 @@ msk 0.20 (RU VAT 20%) spb 0.20 (RU VAT 20%) ``` -Sample ALA rows showing localised attribution: +Sample ALA B2B rows (₽, wholesale-scale tickets — the money end of the +bimodal economy): ``` -ala retail returned 6498 tax=779.76 header=bitrix__ala pricing=1c__ala -ala call-center returned 8083 tax=969.95 header=bitrix__ala pricing=1c__ala -ala retail returned 14798 tax=1775.76 header=bitrix__ala pricing=1c__ala +branch channel order_status total_amount header_source pricing_source +ala b2b delivered 45890 bitrix__ala 1c__ala +ala b2b cancelled 49816 bitrix__ala 1c__ala +ala b2b delivered 48004 bitrix__ala 1c__ala ``` +`bv_customer_mdm__dxb` returns dxb rows with Gulf-style faux PII, all tagged +`pii=1c__dxb`; the msk view never returns them — the per-branch view + RBAC +primitive enforces jurisdictional isolation (proven exhaustively by the +verify_live matrix above). + ## 9. Cold-offload pipeline — MinIO S3 backed +> ⚠ **Kind-cluster section — pending Mac re-capture.** MinIO + the ClickHouse +> `s3()` CronJobs need the shared-network cluster; the standalone stand cannot +> reproduce them. Row counts below are X5-era and await refresh on the new +> seeds. The mechanism (native `s3()` write + read-back, PII-free source +> selection) is unchanged. + `infrastructure/dv2/minio.yaml` provisions a single-node MinIO StatefulSet + Service + bucket-init Job. The cold-offload CronJobs (`infrastructure/dv2/cold-offload-cronjob.yaml` + `cold-offload-fanout.yaml`) write parquet straight into the `cold-tier` bucket via ClickHouse's native -`s3()` table function — no intermediate PVC, no `mc cp` step. - -Bucket layout after running MSK + DXB jobs: - -``` -mc ls -r local/cold-tier -[2026-05-23 06:48:48 UTC] 6.7KiB branch=dxb/year=2026/month=05/customers_anon.parquet -[2026-05-23 06:48:53 UTC] 20KiB branch=msk/year=2026/month=05/customers_anon.parquet -``` - -Each pod runs the same two-statement contract — write then verify — so -the success of the read-back implicitly asserts: - -1. ClickHouse can reach the MinIO Service inside the dv2 namespace. -2. The bucket accepts an INSERT INTO FUNCTION s3('...', 'Parquet') call. -3. The same s3() call reading the file back parses the parquet schema. - -MSK + DXB triggered in parallel (`kubectl create job ---from=cronjob/dv2-cold-offload-{msk,dxb}`) finished in ~10 s. Logs: - -``` -==> exporting branch=msk -> http://minio:9000/cold-tier/branch=msk/year=2026/month=05/customers_anon.parquet -==> done; verifying via s3() read-back -800 -``` - -``` -200 # dxb job -``` - -A schema grep for `first_name|last_name|email|phone|birth_date|pii_flag` -returns 0 — the data-sovereignty contract from `architecture.md` is enforced -by source selection (`sat_customer_anon__1c__{branch}` is the only -satellite the CronJob reads). - -### Branch fanout - -`cold-offload-fanout.yaml` clones MSK for the four remaining branches. -Schedules are staggered (msk 02:00, spb 02:30, ekb 03:00, dxb 04:00, -ala 05:00) so MinIO isn't hammered by five concurrent writes; in real -prod they'd run in parallel via per-branch edge clusters, not a single -cluster as here. +`s3()` table function — no intermediate PVC, no `mc cp` step. A schema grep for +`first_name|last_name|email|phone|birth_date|pii_flag` on the exported files +returns 0 — the data-sovereignty contract is enforced by source selection +(`sat_customer_anon__1c__{branch}` is the only satellite the CronJob reads). ### Production swap path -The CronJob takes `S3_ENDPOINT` / `S3_ACCESS_KEY` / `S3_SECRET_KEY` from -env vars — point them at a real S3 / GCS / Yandex Object Storage and the -`s3()` function works unchanged. The `Secret/minio-creds` resource drops -out, the cloud-provider secret takes its place, and the rest of the -manifest is untouched. Add `WHERE load_ts < now() - INTERVAL 365 DAY` to -the SELECT in prod. +The CronJob takes `S3_ENDPOINT` / `S3_ACCESS_KEY` / `S3_SECRET_KEY` from env +vars — point them at a real S3 / GCS / Yandex Object Storage and the `s3()` +function works unchanged; the `Secret/minio-creds` resource drops out and the +cloud-provider secret takes its place. -## 10. Hot tier — Postgres OLTP + ClickHouse PostgreSQL() bridge +## 10. Hot tier — Postgres OLTP + ClickHouse `PostgreSQL()` bridge -`warehouse/agentflow/dv2/postgres_oltp/seed.sql` populates the -previously-empty Postgres pod with `ops_msk` + `ops_dxb` schemas -(customers + orders, 50/200 and 20/80 rows respectively). -`bridge.sql` creates four `oltp_live.{msk,dxb}_{customers,orders}` -tables in ClickHouse using `Engine = PostgreSQL(...)` — live -read-through of the OLTP tables, no replication slot required. -`promote_to_raw_vault.sql` runs the hot → warm step. +> ⚠ **Kind-cluster section — pending Mac re-capture.** The bridge needs +> ClickHouse and Postgres on one network; the standalone split (CH in WSL, +> Postgres on the Windows host loopback) cannot reach across. Row counts and +> the `Dasha/Egor/Fedor`-style names below are X5-era and await refresh on the +> new kitchen-legend seeds. The code path (Postgres → CH `Engine=PostgreSQL()` +> live read-through → raw_vault → business_vault) is unchanged. -Live join across the bridge — ClickHouse SELECTs Postgres rows -directly: - -```sql -SELECT o.order_id, o.channel, o.total_amount, c.first_name, c.last_name -FROM oltp_live.msk_orders o -JOIN oltp_live.msk_customers c ON o.customer_id = c.customer_id -ORDER BY o.order_id LIMIT 3; -``` +`warehouse/agentflow/dv2/postgres_oltp/seed.sql` populates Postgres with +`ops_msk` + `ops_dxb` schemas; `bridge.sql` creates +`oltp_live.{msk,dxb}_{customers,orders}` tables in ClickHouse using +`Engine = PostgreSQL(...)` — live read-through of the OLTP tables, no +replication slot required. `promote_to_raw_vault.sql` runs the hot → warm +step, landing `record_source = pg_ops__*` rows in `rv.hub_order` that surface +in `bv_order_canonical` with correct branch attribution. -``` -OLTP-MSK-000001 mobile 531 Dasha Sidorov -OLTP-MSK-000002 retail 562 Egor Smirnov -OLTP-MSK-000003 call-center 593 Fedor Volkov -``` +## 11. How to re-run -After the promote step `rv.hub_order` gains two new `record_source` -values (`pg_ops__msk` 200 rows, `pg_ops__dxb` 80) and the existing -1C-seeded volumes are untouched: +**Standalone data sections (§4–8), no Docker:** -``` -1c__msk 4000 -1c__spb 2500 -1c__ekb 1500 -1c__ala 1000 -1c__dxb 1000 -pg_ops__msk 200 -pg_ops__dxb 80 -``` +```bash +# ClickHouse (WSL): single binary + repo DDL/seeds, then +clickhouse client --user default --password demo --database rv -End-to-end check — Postgres orders surface inside `bv_order_canonical` -with correct branch attribution and `header_source` matching the -destination satellite: +# PostgreSQL (Windows/EDB): initdb + pg_ctl on :55432, then +PSQL="psql -h 127.0.0.1 -p 55432 -U agentflow -d agentflow" bash postgres/apply.sh +# Governance matrices: +CH_CLIENT="clickhouse client --config-file=client.xml" bash governance/verify_live.sh +PSQL="psql -h 127.0.0.1 -p 55432 -U agentflow -d agentflow" SEED_DEMO=1 \ + bash postgres/governance/verify_live.sh ``` -order_bk branch channel total header_source -OLTP-MSK-000144 msk web 4964 bitrix__msk -OLTP-MSK-000127 msk call-center 4437 bitrix__msk -OLTP-MSK-000195 msk call-center 6545 bitrix__msk -``` - -The trip from Postgres → ClickHouse OLTP-bridge → raw_vault → -business_vault is the same code path a real Debezium / PeerDB consumer -would land on; the engine swap (`PostgreSQL` → `MaterializedPostgreSQL` -or a streaming CDC writer) preserves the rest of the model untouched. -## 11. How to re-run on the same cluster +(For CH 26.7+, put `default`/`demo` in a client `--config-file` rather than on +the command line — the verify script appends `--user `, and the engine +now rejects a duplicate `--user` flag.) -```bash -ssh @ -export PATH=$HOME/lima/bin:$HOME/bin:$PATH -kubectl exec -it -n dv2 clickhouse-0 -- clickhouse-client \ - --user default --password demo --database rv -``` - -Or, from this repo on any host with `kubectl` context pointing at the -cluster: +**Kind-cluster sections (§1–3, §9, §12–15):** ```bash -bash infrastructure/dv2/bootstrap.sh # idempotent rebuild +bash infrastructure/dv2/bootstrap.sh # idempotent rebuild on the kind cluster ``` ## 12. Argo Workflows orchestration -`infrastructure/dv2/argo/` deploys Argo Workflows v3.5.10 cluster-scope -plus a `dv2-refresh` WorkflowTemplate that chains the previously -standalone hot → warm → cold steps as one DAG: +> ⚠ **Kind-cluster section — pending Mac re-capture.** Timings/counts below are +> X5-era. DAG ordering (hub → link → satellite → cold-offload) is enforced by +> dependencies, not clock-time; that property is legend-independent. -``` -promote-oltp - │ -validate-hubs - │ - ├─ validate-links - │ │ - │ └────────────┐ - └─ validate-satellites - │ - cold-offload (fan-out: msk, spb, ekb, dxb, ala) - │ - verify-mirrors -``` - -End-to-end run on the live cluster (`dv2-refresh-xwnb8`, 73 s total -wall): +`infrastructure/dv2/argo/` deploys Argo Workflows plus a `dv2-refresh` +WorkflowTemplate that chains hot → warm → cold as one DAG: ``` -promote-oltp Succeeded 2026-05-23T08:19:31 -> 08:19:36 -validate-hubs Succeeded 2026-05-23T08:19:41 -> 08:19:46 -validate-links Succeeded 2026-05-23T08:19:51 -> 08:19:56 -validate-satellites Succeeded 2026-05-23T08:19:51 -> 08:19:57 -cold-offload(0:msk) Succeeded 2026-05-23T08:20:01 -> 08:20:25 -cold-offload(1:spb) Succeeded 2026-05-23T08:20:01 -> 08:20:15 -cold-offload(2:ekb) Succeeded 2026-05-23T08:20:01 -> 08:20:25 -cold-offload(3:dxb) Succeeded 2026-05-23T08:20:01 -> 08:20:14 -cold-offload(4:ala) Succeeded 2026-05-23T08:20:01 -> 08:20:26 -verify-mirrors Succeeded 2026-05-23T08:20:34 -> 08:20:38 +promote-oltp → validate-hubs → {validate-links, validate-satellites} + → cold-offload (fan-out: msk, spb, ekb, dxb, ala) → verify-mirrors ``` -`verify-mirrors` step output (capture run `dv2-refresh-capture-s27ng`): - -``` -==> cross-checking mirrors vs source satellites - msk source=800 mirror=800 OK - spb source=500 mirror=500 OK - ekb source=300 mirror=300 OK - dxb source=200 mirror=200 OK - ala source=200 mirror=200 OK -==> all 5 mirrors match source -``` - -Layer ordering (hub → link → satellite → cold-offload) is enforced by -DAG dependencies — not by clock-time as the standalone CronJobs do. -A failure in `validate-links` aborts the run before any S3 write, so -mirrors are never out of sync with the warm tier. +A failure in `validate-links` aborts the run before any S3 write, so mirrors +are never out of sync with the warm tier. ## 13. dbt mart layer -`warehouse/agentflow/dv2/dbt/` ships three materialized marts and 12 -data tests on top of the business vault. Project files are mounted into -a Kubernetes Job (`infrastructure/dv2/dbt/dbt-run-job.yaml`) via a -ConfigMap built from the repo by `infrastructure/dv2/dbt/run.sh`. - -Run summary (from `kubectl logs job/dbt-run-marts`): - -``` -Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3 (dbt run) -Done. PASS=12 WARN=0 ERROR=0 SKIP=0 TOTAL=12 (dbt test) -``` - -`customer_360` populated per branch — one row per `(customer_hk, branch)`: - -``` -branch rows with_orders avg_ltv -ala 200 84 6554.1 -dxb 200 84 7002.4 -ekb 300 157 9212.4 -msk 800 694 25496.5 -spb 500 366 16784.5 -``` - -`branch_pnl.effective_tax_rate` validates the per-jurisdiction wiring -end-to-end (1C pricing satellites → BV view → dbt mart): - -``` -branch rate -ala 0.12 (KZ VAT 12%) -dxb 0.05 (UAE VAT 5%) -ekb 0.20 (RU VAT 20%) -msk 0.20 (RU VAT 20%) -spb 0.20 (RU VAT 20%) -``` +> ⚠ **Kind-cluster section — pending Mac re-capture.** The dbt-in-Kubernetes +> Job and its per-branch row counts are X5-era. The three marts + 12 tests are +> legend-independent in structure; the numbers await refresh on the new seeds. -The 12 dbt tests cover `not_null` on key columns -(`customer_hk`, `branch`, `month`, `channel`, `week`, `return_rate`) -and `accepted_values` on `branch` (must be one of msk/spb/ekb/dxb/ala) -across all three marts. +`warehouse/agentflow/dv2/dbt/` ships three materialized marts and 12 data +tests on top of the business vault, run via a Kubernetes Job +(`infrastructure/dv2/dbt/dbt-run-job.yaml`). `customer_360` populates one row +per `(customer_hk, branch)`; `branch_pnl.effective_tax_rate` validates the +per-jurisdiction wiring end-to-end (1C pricing satellites → BV view → dbt +mart) — the same 12/5/20% rates verified live in §8. ## 14. Push-based CDC via MaterializedPostgreSQL -The pull-based `oltp_live` bridge (Postgres-engine table mirrors) is -replaced by a single `oltp_cdc` ClickHouse database backed by -`MaterializedPostgreSQL`, consuming the Postgres WAL via logical -replication. `materialized_postgresql_schema_list` lets one CH -database carry both Postgres schemas — CH 25.x doesn't expose -`publication_name`, so two CH databases against the same Postgres -DB collide on the auto-named publication. - -Cluster state after `cdc_setup.sql + cdc_bridge.sql`: - -``` -oltp_cdc ops_dxb.customers ReplacingMergeTree -oltp_cdc ops_dxb.orders ReplacingMergeTree -oltp_cdc ops_msk.customers ReplacingMergeTree -oltp_cdc ops_msk.orders ReplacingMergeTree -``` - -(The schema name is part of the CH table name and quoted with -backticks because of the dot: -`SELECT ... FROM oltp_cdc.\`ops_msk.customers\` FINAL`.) - -Live E2E test — INSERT in Postgres → no manual refresh → SELECT in -ClickHouse within seconds: - -```bash -# Postgres side -psql> INSERT INTO ops_msk.customers (customer_id, first_name, last_name) - VALUES ('CDC-V2-MSK', 'NewMsk', 'CDC'); -psql> UPDATE ops_msk.customers SET last_name='UPDATED' - WHERE customer_id='CDC-V2-MSK'; - -# ClickHouse side, ~5s later (no INSERT INTO ... SELECT on CH at all) -clickhouse> SELECT customer_id, first_name, last_name - FROM oltp_cdc.`ops_msk.customers` FINAL - WHERE customer_id LIKE 'CDC-V2-%'; - -┌─customer_id─┬─first_name─┬─last_name─┐ -│ CDC-V2-MSK │ NewMsk │ UPDATED │ -└─────────────┴────────────┴───────────┘ -``` - -Row count parity vs source-of-truth Postgres: - -``` - ┌─t───────────┬─count()─┐ -1. │ msk_c_FINAL │ 57 │ -2. │ dxb_c_FINAL │ 24 │ - └─────────────┴─────────┘ - ─ vs ─ - branch | pg_count ---------+---------- - msk | 57 - dxb | 24 -``` +> ⚠ **Kind-cluster section — pending Mac re-capture.** MaterializedPostgreSQL +> consumes the Postgres WAL via logical replication and needs both engines on +> one network with `wal_level=logical`; the standalone split cannot reproduce +> it. Contents are X5-era. -After `promote_to_raw_vault_cdc.sql` re-runs against the CDC tables -(reading with `FINAL` to dedupe ReplacingMergeTree versions), the -pg_ops rows land in raw_vault and propagate to the BV order canonical -view: - -``` - ┌─record_source─┬─count()─┐ -1. │ pg_ops__dxb │ 24 │ -2. │ pg_ops__msk │ 57 │ - └───────────────┴─────────┘ -``` - -The `record_source = pg_ops__*` convention is identical to the -pull-based variant, so any downstream consumer (BV view / dbt mart / -cold-offload) sees the CDC path the same way it saw the -`oltp_live`-based promotion. +The pull-based `oltp_live` bridge is replaced by a single `oltp_cdc` +ClickHouse database backed by `MaterializedPostgreSQL`, consuming the Postgres +WAL. `materialized_postgresql_schema_list` lets one CH database carry both +Postgres schemas. Live E2E: an INSERT/UPDATE in Postgres surfaces in +ClickHouse within seconds with no manual refresh; `promote_to_raw_vault_cdc.sql` +(reading `FINAL` to dedupe ReplacingMergeTree versions) lands the `pg_ops__*` +rows in raw_vault. ## 15. Per-branch CDC fan-out -The session-14 stream is unified (one CH database carries both branches). -Operational reality wants the opposite: a single branch must be pausable, -re-snapshotable, and rotatable without touching another branch's stream. -ClickHouse 25.5 rejects a custom publication name on -`MaterializedPostgreSQL` (`Code 115. Unknown setting -'materialized_postgresql_publication_name'`, verified 2026-05-23), so two -CH databases against the same Postgres DB collide on the auto-generated -`_ch_publication`. - -The fan-out pattern splits the source: one Postgres **database** per -branch (`ops_msk_db`, `ops_dxb_db`). Each gets its own auto-named -publication and slot because the source DB name differs. Two CH -MaterializedPostgreSQL databases (`oltp_cdc_msk`, `oltp_cdc_dxb`) consume -independently. PeerDB OSS would be the cleaner production path, but its -~3 GB stack (Temporal + flow services + catalog PG) does not fit on the -8 GB demo iMac alongside the running kind cluster; the per-database split -delivers the same isolation property natively. - -Apply (Postgres-side schema/seed/CDC + ClickHouse-side bridge): - -```bash -for f in 01_schema 02_seed 03_cdc_setup; do - kubectl exec -i -n dv2 postgres-0 -- psql -U ops -d postgres \ - < warehouse/agentflow/dv2/postgres_oltp/fanout/${f}.sql -done -kubectl exec -i -n dv2 clickhouse-0 -- clickhouse-client \ - --user default --password demo --multiquery \ - < warehouse/agentflow/dv2/postgres_oltp/fanout/04_ch_bridge.sql -``` - -Snapshot result — each CH database carries only its branch: - -``` -┌─msk_c─┬─msk_o─┬─dxb_c─┬─dxb_o─┐ -│ 10 │ 30 │ 8 │ 20 │ -└───────┴───────┴───────┴───────┘ -``` - -Two distinct replication slots, one per branch: - -``` - slot_name | database | active | confirmed_flush_lsn -------------+------------+--------+--------------------- - ops_msk_db | ops_msk_db | f | 0/22AC6D0 - ops_dxb_db | ops_dxb_db | f | 0/22ACC88 -``` - -Live E2E — INSERT/UPDATE in `ops_msk_db` propagates only to `oltp_cdc_msk`; -parallel INSERT in `ops_dxb_db` lands only in `oltp_cdc_dxb`: - -```bash -psql ops_msk_db> INSERT INTO customers VALUES ('msk-c-LIVE','LIVE','TEST',...); -psql ops_msk_db> INSERT INTO orders VALUES ('msk-o-LIVE','msk-c-LIVE','paid',99999.99,'RUB'); -psql ops_msk_db> UPDATE customers SET phone='+74950000000' WHERE customer_id='msk-c-001'; -psql ops_dxb_db> INSERT INTO customers VALUES ('dxb-c-LIVE','LIVE','TEST',...); -``` - -After ~8 s: - -``` -oltp_cdc_msk.customers FINAL → 11 rows (was 10), c-001 phone now +74950000000 -oltp_cdc_msk.orders FINAL → 31 rows (was 30), msk-o-LIVE total = 99999.99 -oltp_cdc_dxb.customers FINAL → 9 rows (was 8), dxb-c-LIVE present -``` - -Isolation check — MSK CH database has zero rows from DXB: - -``` -SELECT count() FROM oltp_cdc_msk.customers WHERE customer_id LIKE 'dxb-%'; -─→ 0 -``` +> ⚠ **Kind-cluster section — pending Mac re-capture.** Contents are X5-era. -Both pattern coexist on the same cluster: `oltp_cdc` (single-DB stream) -plus `oltp_cdc_msk` / `oltp_cdc_dxb` (per-branch fan-out). The unified -stream is correct for cross-branch analytics that always want both -branches together; the fan-out is correct when a single branch's stream -must be paused or rotated independently. +Operational reality wants a single branch to be pausable, re-snapshotable and +rotatable without touching another branch's stream. ClickHouse 25.5+ rejects a +custom publication name on `MaterializedPostgreSQL`, so the fan-out pattern +splits the source — one Postgres **database** per branch (`ops_msk_db`, +`ops_dxb_db`), each with its own auto-named publication and slot, consumed by +two independent CH `MaterializedPostgreSQL` databases (`oltp_cdc_msk`, +`oltp_cdc_dxb`). Isolation check: the msk CH database has zero rows from dxb. diff --git a/docs/perf/vault-pii-governance-pg-verify-2026-07-03.md b/docs/perf/vault-pii-governance-pg-verify-2026-07-03.md new file mode 100644 index 00000000..35feb234 --- /dev/null +++ b/docs/perf/vault-pii-governance-pg-verify-2026-07-03.md @@ -0,0 +1,93 @@ +# DV2 vault PII governance, PostgreSQL port — live verification (ADR 0006 Phase 2 follow-up) + +**Date:** 2026-07-03 +**Environment:** standalone PostgreSQL 17.5 (EDB windows-x64 binaries, no +Docker, no service install: `initdb` + `pg_ctl`, port 55432, trust auth, +user/db `agentflow`) — the same no-Docker standalone-PG recipe as +`vault-pii-governance-pg-verify-2026-07-02.md`, re-run on the current +kitchen-gadget legend. Vault built from the repo files verbatim via +`postgres/apply.sh` (schema → 8 hubs → 8 links → 48 satellites → +`03_business_vault.sql`), then `postgres/governance/01..04.sql`. Data: the +deterministic demo seed from `verify_live.sh` (`SEED_DEMO=1`): hub_customer +msk 8 / dxb 2 = 10 rows, the msk rows deliberately spanning three source +conventions — **`1c__msk`, `pg_ops__msk`, `mp__msk`** (the `x5__` convention +was retired in B2; this run confirms `mp__msk` is what the seed now carries) — +so the row policies are exercised against the `split_part` branch derivation, +not a single record_source literal. Probe principals are stand-local: +`analyst_probe` (role `dv2_analyst`), `officer_msk_probe`, `officer_dxb_probe`, +and `noscope_probe` (SELECT on the hub, addressed by NO policy). + +**Result: 33/33 probes passed** (`postgres/governance/verify_live.sh`, +transcript below; 0 FAIL, 0 WARN). The PII boundary is enforced by PostgreSQL +ACLs on resolved attributes — there is no SQL shape that reaches an ungranted +PII column, including the whole-row and positional-rename shapes that +ClickHouse cannot even express. The four governance files re-apply cleanly +(idempotency section, all four PASS). + +## Transcript + +``` +$ PSQL="psql -h 127.0.0.1 -p 55432 -U agentflow -d agentflow" SEED_DEMO=1 bash verify_live.sh +=== setup: probe users (stand-local, not part of the governance files) === +users ready +=== setup: deterministic demo seed (SEED_DEMO=1) === +seed applied + +=== dv2_analyst: non-PII access works (owner-rights views, column grants) === +PASS [analyst explicit non-PII projection] -> CUST-MSK-1|msk|gold CUST-MSK-2|msk|silver +PASS [analyst bare count(*)] -> 8 +PASS [analyst aggregate over granted column] -> 8|1680.50 +PASS [analyst GROUP BY + HAVING on granted columns] -> |6 gold|1 +PASS [analyst WHERE on passthrough column (customer_bk)] -> CUST-MSK-1 +PASS [analyst filter on view-derived column (CH limitation absent on PG)] -> 1 +PASS [analyst hub_customer full visibility (jurisdiction__all policy)] -> 10 +PASS [analyst granted satellite (loyalty)] -> 2 + +=== dv2_analyst: PII columns are engine-denied in EVERY shape === +PASS [analyst plain PII column] -> permission denied +PASS [analyst SELECT *] -> permission denied +PASS [analyst bypass #1: whole-row ref] -> permission denied +PASS [analyst bypass #2: to_jsonb(whole row)] -> permission denied +PASS [analyst bypass #3: positional rename-list (expressible on PG)] -> permission denied +PASS [analyst PII inside expression] -> permission denied +PASS [analyst PII in WHERE only] -> permission denied +PASS [analyst PII via subquery] -> permission denied +PASS [analyst raw personal satellite] -> permission denied +PASS [analyst employee profile (name PII)] -> permission denied + +=== officers: PII bounded to own jurisdiction === +PASS [officer_msk reads own-branch PII] -> Ivan|ivan.petrov@example.com +PASS [officer_msk filtered aggregate (full view grant)] -> 2 +PASS [officer_msk reads own personal satellite] -> 2 +PASS [officer_msk cross-branch view denied] -> permission denied +PASS [officer_msk cross-branch satellite denied] -> permission denied +PASS [officer_msk hub row-scoped] -> sees 8 of 8 msk rows +PASS [officer_dxb hub row-scoped] -> sees 2 of 2 dxb rows +PASS [officer_dxb sees zero msk rows via hub filter] -> 0 + +=== PostgreSQL default-deny: principal addressed by NO row policy === +PASS [noscope_probe (SELECT granted, no policy) sees zero hub rows] -> 0 + +=== admin (owner) unaffected: ENABLE (not FORCE) row level security === +PASS [admin hub full visibility (owner bypasses RLS)] -> 10 +PASS [admin reads PII] -> ivan.petrov@example.com + +=== governance files re-apply cleanly (idempotency) === +PASS [re-apply 01_roles.sql] +PASS [re-apply 02_grants_analyst.sql] +PASS [re-apply 03_grants_pii_officers.sql] +PASS [re-apply 04_row_policies.sql] +``` + +## Honest scope + +- Verified on the stand-local demo seed (10 hub rows), not on promoted CDC + volume — the row-policy assertions compare officer-visible counts against + admin-side per-branch counts, so the script re-runs unchanged on a stand + with real data (leave `SEED_DEMO` unset). +- The admin/owner (`agentflow`) sees everything: engine policies bind *roles*; + production would split the admin identity from human users. +- The dbt marts and `bv_order_canonical_mat` exist only on the ClickHouse + stand; there is nothing to govern for them here. +- `initdb` on the Windows temp filesystem is slow under Defender first-exec + scanning (~7 min); `--no-sync` + `fsync=off` are safe on a throwaway stand. diff --git a/docs/perf/vault-pii-governance-verify-2026-07-03.md b/docs/perf/vault-pii-governance-verify-2026-07-03.md new file mode 100644 index 00000000..310569d3 --- /dev/null +++ b/docs/perf/vault-pii-governance-verify-2026-07-03.md @@ -0,0 +1,105 @@ +# DV2 vault PII governance — live verification, ClickHouse (ADR 0006 Phase 2) + +**Date:** 2026-07-03 +**Environment:** standalone `clickhouse server` 26.7.1.492 (single binary, WSL +Ubuntu 22.04, no Docker), `access_management=1` for the applying `default` +user. Vault built from the repo files verbatim on the current kitchen-gadget +legend (post-B1/B2/B3 seeds): `__init.sql` → 8 hubs → 8 links → all 48 +satellites → `synthetic_seed.sql` + `satellite_seed.sql` + +`satellite_seed_all_branches.sql` → `business_vault/*.sql` (views with +`SQL SECURITY DEFINER`) → `governance/01..04.sql`. Resulting volume: +**hub_customer 2,500** (msk 2,190 / spb 100 / ala 80 / ekb 70 / dxb 60), +**hub_order 10,000**. Probe users are stand-local: `analyst_probe` +(role `dv2_analyst`), `officer_msk_probe`, `officer_dxb_probe`. + +**Result: 29/29 probes passed** (`governance/verify_live.sh`, transcript +below; 0 FAIL, 0 WARN). The PII boundary is enforced by the engine's access +control on resolved columns — there is no SQL shape that reaches an ungranted +PII column. + +The suite is a refresh of `vault-pii-governance-verify-2026-07-02.md` on the +new seeds. Two things changed vs that run: + +1. **Volumes are legend-current** — 2,500 customers (was 2,000), msk holds all + retail (2,190) under the "regions carry only dealers" rule; officer hub + counts now assert 2,190 msk / 60 dxb rather than 800 / 200. The row-policy + assertions compare officer-visible counts against the branch's own + `hub_customer` count, so they hold at any volume. +2. **Probe count is 29** (the checked-in script's current assertion set); the + 2026-07-02 note cited 32. What matters is every probe in the current script + passes. + +## Gotcha caught live (CH 26.7) + +`clickhouse client` 26.7.1.492 **rejects a duplicate `--user`** flag +(`Bad arguments: option '--user' cannot be specified more than once`). The +verify script appends `--user --password probe` to `$CH_CLIENT`, so a +`CH_CLIENT` that already carried `--user default --password demo` (as the +2026-07-02 recipe's did) fails every probe-user query with rc=36 and an empty +error. Fix without touching the repo script: put the `default`/`demo` +credentials in a client `--config-file` (`///`), +so bare `CH_CLIENT` authenticates as `default` and the appended `--user +` is the sole `--user` on the line: + +```bash +CH_CLIENT="/path/clickhouse client --config-file=/path/client.xml" \ + bash governance/verify_live.sh +``` + +## Transcript + +``` +=== setup: probe users (stand-local, not part of the governance files) === +users ready + +=== dv2_analyst: non-PII access works (DEFINER view, column grants) === +PASS [analyst explicit non-PII projection] -> CUST-000000 msk CUST-000001 msk +PASS [analyst bare count()] -> 2190 +PASS [analyst aggregate over granted column] -> 2190 584720 +PASS [analyst GROUP BY + HAVING on granted columns] -> 2038 mid 76 +PASS [analyst WHERE on passthrough column (customer_bk)] -> CUST-000000 +PASS [analyst hub_customer full visibility (catch-all row policy)] -> 2500 +PASS [analyst granted satellite (loyalty)] -> 152 + +=== dv2_analyst: PII columns are engine-denied in EVERY shape === +PASS [analyst plain PII column] -> ACCESS_DENIED +PASS [analyst SELECT *] -> ACCESS_DENIED +PASS [analyst bypass #1: COLUMNS('.*') expr] -> ACCESS_DENIED +PASS [analyst bypass #2: whole-row struct ref] -> UNKNOWN_IDENTIFIER (shape not expressible on ClickHouse) +PASS [analyst bypass #3: positional rename-list] -> UNKNOWN_IDENTIFIER (shape not expressible on ClickHouse) +PASS [analyst PII inside expression] -> ACCESS_DENIED +PASS [analyst PII in WHERE only] -> ACCESS_DENIED +PASS [analyst raw personal satellite] -> ACCESS_DENIED +PASS [analyst employee profile (name PII)] -> ACCESS_DENIED + +=== known ergonomic limitation: filter pushdown vs column grants === +PASS [analyst filter on argMax-derived column (raw)] -> ACCESS_DENIED +PASS [analyst same filter via subquery wrap (PII-safe workaround)] -> 152 +PASS [subquery wrap cannot smuggle PII] -> ACCESS_DENIED + +=== officers: PII bounded to own jurisdiction === +PASS [officer_msk reads own-branch PII] -> Anna cust0@example.test +PASS [officer_msk filtered aggregate (full view grant)] -> 152 +PASS [officer_msk reads own personal satellite] -> 2190 +PASS [officer_msk cross-branch view denied] -> ACCESS_DENIED +PASS [officer_msk cross-branch satellite denied] -> ACCESS_DENIED +PASS [officer_msk hub row-scoped (msk rows only)] -> 2190 +PASS [officer_dxb hub row-scoped (dxb rows only)] -> 60 +PASS [officer_dxb sees zero msk rows via hub filter] -> 0 + +=== admin unaffected === +PASS [admin hub full visibility (catch-all)] -> 2500 +PASS [admin reads PII] -> cust0@example.test +``` + +## Honest scope + +- Re-captured on the synthetic demo seed (2,500 customers), not on promoted + CDC volume — the row-policy assertions compare officer-visible counts + against admin-side per-branch counts, so the script re-runs unchanged on a + stand with real data. +- The applying admin (`default`) sees everything: engine policies bind *roles*; + production would split the admin identity from human users. +- `SQL SECURITY DEFINER` on the MDM views is load-bearing (an INVOKER default + would need SELECT on the underlying personal satellites — exactly what the + boundary denies).