Skip to content

Commit fa4ba22

Browse files
brownjuly2003-codeJulia Edomskikhclaude
authored
G2 S6: Mac-tail cluster re-capture (live kind stand) (#164)
* docs(perf): G1 live smoke — bv_order_canonical on real PostgreSQL (17/17 PASS) * docs(dv2): re-capture demo_evidence §1-3 (topology/pinning/PVCs) on live kind stand * docs(dv2): re-capture demo_evidence §9 cold-offload (live MinIO s3, PII-free) * docs(dv2): re-capture demo_evidence §13 dbt marts (live SQL, 12/12 tests pass) * docs(dv2): re-capture demo_evidence §10 hot-tier bridge (live PostgreSQL() cross-pod) * docs(dv2): re-capture demo_evidence §12 DAG steps (live, Argo engine substitution documented) * docs(dv2): §14/15 CDC — honest attempted/blocked account (host contention), not fabricated * fix(dv2): cdc_setup grants CREATE on the source DB to rep_user MaterializedPostgreSQL self-bootstraps its publication with CREATE PUBLICATION executed as the replication user, and Postgres requires CREATE on the database for that. With CONNECT alone the replication handler retry-loops on "permission denied for database ops" — the CH database appears but carries no slot, no publication and no tables. Found live on the hq-demo kind stand 2026-07-06; the fan-out variant (fanout/03_cdc_setup.sql) always carried the equivalent grant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dv2): re-capture load-test-baseline on kitchen-legend seed (honest ambient-noise caveat) * docs(dv2): re-capture demo_evidence §14/§15 CDC live (single-DB + per-branch fan-out) Both sections now carry live output from the hq-demo kind stand (2026-07-06). §14: snapshot through the WAL pipeline (50/200/20/80), slot ops active with advancing flush LSN, publication with 4 tables, promotion collapsing to the same pg_ops__* hub counts as the §10 pull pass, live INSERT/UPDATE/DELETE round-trip, count()-vs-column-read tombstone semantics note, and the two-pass story (contention first, then the deterministic GRANT CREATE root cause fixed in cdc_setup.sql). §15: two per-branch Postgres DBs, two auto-named publications and slots, 10/30 + 8/20 snapshots, 12-second edit propagation with column-level detail, and a 0/0 cross-branch isolation check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dv2): B2-mp4 — re-record demo.cast/demo_voiced.mp4 live on current legend * docs(dv2): restore §14/§15 live CDC capture lost to a concurrent-session race Commit 1784d5e (B2-mp4 re-record, prepared in a parallel working session) was cut from a pre-2ed9ea3 copy of demo_evidence.md and unintentionally reverted §14/§15 to the earlier attempted-blocked account, and stamped the header with "§14/15 did not complete". Re-apply the full live capture from 2ed9ea3 verbatim and reword the header note to match reality (blocked first pass, then GRANT CREATE root cause found and the sections captured live). No other B2 content from 1784d5e is touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dv2): E4 — static render-gate re-verification; live 2-pod run honestly blocked by host contention --------- Co-authored-by: Julia Edomskikh <yulia.edomskikh@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 509902a commit fa4ba22

11 files changed

Lines changed: 629 additions & 193 deletions

‎docs/clickhouse-cutover-plan.md‎

Lines changed: 33 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -130,14 +130,39 @@ executed accidentally.
130130
unless `controlPlane.store=postgres` and `serving.backend=clickhouse` —
131131
see ADR 0010. Slice 6 (2026-07-04) released the enum ratchet to
132132
`[embedded, postgres]`.)*
133-
- [ ] **LIVE verify (kind, Docker — pending Mac/CI, E4 tail):**
134-
`scripts/k8s_staging_up.sh` on kind with the scale overlay,
135-
`k8s_smoke_test.sh` green, `replicaCount: 2` schedules without PVC
136-
contention; **plus the ADR 0010 replica-correctness checks** — exactly one
137-
delivery per (webhook, event) across two pods, one alert page per incident,
138-
a webhook registered via either pod visible to both. See the recipe below.
139-
*(Chart-side render verified locally 2026-07-04 via `helm template`/`lint`;
140-
the two-real-pods run needs Docker, unavailable on the authoring host.)*
133+
- [ ] **LIVE verify (kind, Docker — attempted on the Mac 2026-07-06, genuinely
134+
incomplete):** `scripts/k8s_staging_up.sh` on kind with the scale
135+
overlay, `k8s_smoke_test.sh` green, `replicaCount: 2` schedules without
136+
PVC contention; **plus the ADR 0010 replica-correctness checks** —
137+
exactly one delivery per (webhook, event) across two pods, one alert
138+
page per incident, a webhook registered via either pod visible to both.
139+
See the recipe below.
140+
*(Chart-side render verified locally 2026-07-04 via `helm template`/
141+
`lint`, and re-verified live-adjacent on the Mac 2026-07-06: `helm lint`
142+
clean; `helm template --set replicaCount=2 --set persistence.enabled=false`
143+
**without** `controlPlane.store=postgres`/`serving.backend=clickhouse`
144+
correctly fails at render time with the exact ADR 0010 gate error
145+
(`"Multi-replica requires BOTH an external serving engine ... AND an
146+
external control-plane store ..."`); the same render **with both halves
147+
set** succeeds with `replicas: 2` and the correct
148+
`AGENTFLOW_CONTROLPLANE_STORE`/`SERVING_BACKEND` env vars. The
149+
**two-real-pods live run** (building `agentflow/api`, deploying 2 pods,
150+
exercising `scripts/k8s_replica_correctness_verify.sh` against them) was
151+
attempted on the Mac kind stand but could not complete: this same
152+
session hit severe, sustained shared-host resource contention (the
153+
Mac runs this kind cluster alongside another project's ~9 containers;
154+
`/proc/loadavg` read 40–78 for extended stretches, `docker stats` showed
155+
a co-tenant's Temporal engine alone at up to 80% CPU, and this cluster's
156+
own control plane was crash-looping — `kube-apiserver` had restarted 10+
157+
times, `kube-controller-manager` 46+ times, confirmed via `crictl ps -a`
158+
directly on the node). Two separate `docker build` attempts for the
159+
fresh `agentflow/api` image (needed since the cached image predates
160+
ADR 0010 slice 5) stalled at near-zero CPU progress even after killing
161+
a competing orphaned build process. This is an honest "attempted,
162+
blocked by host resource crisis" outcome — not fabricated, and not
163+
abandoned without cause; the render-gate logic itself is now more
164+
thoroughly verified than before. Re-attempt on a quieter window on this
165+
host, or a dedicated instance.)*
141166

142167
### Phase 3 replica-correctness verify recipe (ADR 0010 slice 6)
143168

‎docs/decisions/0010-control-plane-externalization-postgres.md‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -208,13 +208,23 @@ ClickHouse is consumed), and `psycopg` joins the optional dependencies.
208208
`AGENTFLOW_CONTROLPLANE_STORE` + `AGENTFLOW_CONTROLPLANE_PG_DSN`; the scale
209209
profile is an overlay (`k8s/staging/values-staging-scale.yaml.example`), not
210210
a change to the zero-dependency default. Render side verified locally
211-
(`helm template`/`lint` + `tests/unit/test_helm_values_contract.py`). The
212-
**two-real-pods live run** (kind at `replicaCount=2` + the replica-correctness
213-
checks above) needs Docker and is the deferred Mac/CI tail —
214-
`scripts/k8s_replica_correctness_verify.sh` automates the pod-count and
215-
cross-pod registration-visibility checks; the delivery/alert checks follow
216-
the recipe in the cutover plan (their store-level guarantee is already
217-
live-verified by slice 5's 31/31 PG probes).
211+
(`helm template`/`lint` + `tests/unit/test_helm_values_contract.py`), and
212+
re-verified 2026-07-06 on the Mac kind stand: the render gate correctly
213+
rejects `replicaCount=2` without both halves of the gate (the exact ADR
214+
error text) and correctly admits it once both are set (`replicas: 2`,
215+
correct env vars). The **two-real-pods live run** (kind at
216+
`replicaCount=2` + the replica-correctness checks above) needs Docker and
217+
was attempted 2026-07-06 on the Mac stand — genuinely incomplete: the
218+
shared host hit severe resource contention this session (co-tenant CPU
219+
spikes up to 80%, this cluster's own control plane crash-looping,
220+
`kube-apiserver` 10+ restarts) that stalled the `agentflow/api` image build
221+
itself at near-zero CPU progress across two attempts. See
222+
`docs/clickhouse-cutover-plan.md` Phase 3 for the full honest account.
223+
`scripts/k8s_replica_correctness_verify.sh` remains ready to automate the
224+
pod-count and cross-pod registration-visibility checks whenever the live
225+
run can complete; the delivery/alert checks follow the recipe in the
226+
cutover plan (their store-level guarantee is already live-verified by
227+
slice 5's 31/31 PG probes).
218228

219229
## Consequences
220230

‎docs/dv2-multi-branch/demo.cast‎

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,24 @@
1-
{"version": 2, "width": 130, "height": 35, "timestamp": 1779531590, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}, "title": "DV2.0 multi-branch live demo"}
2-
[0.01269, "o", "\r\n\u001b[1;33m# === Beat 1 — Cluster topology (00:25) ===\u001b[0m\r\n"]
3-
[1.520249, "o", "\u001b[1;36m$\u001b[0m kubectl get nodes --show-labels | head -5\r\n"]
4-
[2.411272, "o", "NAME STATUS ROLES AGE VERSION LABELS\r\nhq-demo-control-plane Ready control-plane 6h18m v1.35.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,nodepool=hq-control\r\nhq-demo-worker Ready <none> 6h18m v1.35.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker,kubernetes.io/os=linux,nodepool=hq-data-tier-a,workload=postgres\r\nhq-demo-worker2 Ready <none> 6h18m v1.35.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker2,kubernetes.io/os=linux,nodepool=hq-data-tier-b,workload=clickhouse\r\n"]
5-
[4.921556, "o", "\r\n\u001b[1;33m# === Beat 2 — DV2.0 model surface (00:45) ===\u001b[0m\r\n"]
6-
[6.434524, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('_', name)[1] AS kind, count() FROM system.tables WHERE database='rv' GROUP BY kind ORDER BY kind\"\r\n"]
7-
[7.428901, "o", "bv\t6\r\nhub\t8\r\nlnk\t8\r\nsat\t39\r\n"]
8-
[9.95663, "o", "\r\n\u001b[1;33m# === Beat 3 — Multi-branch distribution (01:05) ===\u001b[0m\r\n"]
9-
[11.465034, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('__', record_source)[2] AS branch, count() AS orders, round(count()*100.0/(SELECT count() FROM rv.hub_order),1) AS pct FROM rv.hub_order GROUP BY branch ORDER BY pct DESC FORMAT PrettyCompact\"\r\n"]
10-
[12.475313, "o", " ┌─branch─┬─orders─┬──pct─┐\r\n1. │ msk │ 4200 │ 40.5 │\r\n2. │ spb │ 2500 │ 24.1 │\r\n3. │ ekb │ 1500 │ 14.5 │\r\n4. │ dxb │ 1160 │ 11.2 │\r\n5. │ ala │ 1000 │ 9.7 │\r\n └────────┴────────┴──────┘\r\n"]
11-
[14.995744, "o", "\r\n\u001b[1;33m# === Beat 4 — Business Vault MDM (01:25) ===\u001b[0m\r\n"]
12-
[16.50705, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --multiline --query \"SELECT 'msk' AS branch, count() AS rows, countIf(first_name != '') AS with_pii, countIf(loyalty_segment != '') AS with_loyalty FROM rv.bv_customer_mdm__msk UNION ALL SELECT 'dxb', count(), countIf(first_name != ''), countIf(loyalty_segment IS NOT NULL AND loyalty_segment != '') FROM rv.bv_customer_mdm__dxb FORMAT PrettyCompact\"\r\n"]
13-
[17.550896, "o", " ┌─branch─┬─rows─┬─with_pii─┬─with_loyalty─┐\r\n1. │ dxb │ 200 │ 200 │ 0 │\r\n2. │ msk │ 800 │ 800 │ 640 │\r\n └────────┴──────┴──────────┴──────────────┘\r\n"]
14-
[20.072279, "o", "\r\n\u001b[1;33m# === Beat 5a — Cold offload (01:50) ===\u001b[0m\r\n"]
15-
[21.576438, "o", "\u001b[1;36m$\u001b[0m kubectl create job --from=cronjob/dv2-cold-offload-msk cold-demo-3455 -n dv2\r\n"]
16-
[22.483005, "o", "job.batch/cold-demo-3455 created\r\n"]
17-
[32.996093, "o", "\r\n\u001b[1;33m# === Beat 5b — MinIO parquet listing ===\u001b[0m\r\n"]
18-
[34.50554, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 minio-0 -- mc ls -r local/cold-tier | tail -5\r\n"]
19-
[35.495859, "o", "[2026-05-23 09:44:37 UTC] 6.7KiB STANDARD branch=ala/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 09:44:32 UTC] 6.7KiB STANDARD branch=dxb/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 09:44:37 UTC] 8.6KiB STANDARD branch=ekb/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 10:20:13 UTC] 20KiB STANDARD branch=msk/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 09:44:32 UTC] 13KiB STANDARD branch=spb/year=2026/month=05/customers_anon.parquet\r\n"]
20-
[38.005981, "o", "\r\n\u001b[1;33m# === Closing (02:00) — Single bootstrap reproduces everything ===\u001b[0m\r\n"]
21-
[39.516429, "o", "\u001b[1;36m$\u001b[0m bash infrastructure/dv2/bootstrap.sh\r\n"]
22-
[41.521865, "o", "\r\n\u001b[1;33mDemo complete. See docs/dv2-multi-branch/demo_evidence.md for the full breakdown.\u001b[0m\r\n"]
1+
{"version": 2, "width": 130, "height": 35, "timestamp": 1783334766, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}, "title": "DV2.0 multi-branch live demo"}
2+
[0.033084, "o", "\r\n\u001b[1;33m# === Beat 1 — Cluster topology (00:25) ===\u001b[0m\r\n"]
3+
[1.554022, "o", "\u001b[1;36m$\u001b[0m kubectl get nodes --show-labels | head -5\r\n"]
4+
[2.805528, "o", "NAME STATUS ROLES AGE VERSION LABELS\r\nhq-demo-control-plane Ready control-plane 5h30m v1.32.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,nodepool=hq-control\r\nhq-demo-worker Ready <none> 5h29m v1.32.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker,kubernetes.io/os=linux,nodepool=hq-data-tier-a,workload=postgres\r\nhq-demo-worker2 Ready <none> 5h29m v1.32.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker2,kubernetes.io/os=linux,nodepool=hq-data-tier-b,workload=clickhouse\r\n"]
5+
[5.31246, "o", "\r\n\u001b[1;33m# === Beat 2 — DV2.0 model surface (00:45) ===\u001b[0m\r\n"]
6+
[6.826426, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('_', name)[1] AS kind, count() FROM system.tables WHERE database='rv' GROUP BY kind ORDER BY kind\"\r\n"]
7+
[13.226603, "o", "bv\t6\r\nhub\t8\r\nlnk\t8\r\nsat\t48\r\n"]
8+
[17.616754, "o", "\r\n\u001b[1;33m# === Beat 3 — Multi-branch distribution (01:05) ===\u001b[0m\r\n"]
9+
[19.126624, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('__', record_source)[2] AS branch, count() AS orders, round(count()*100.0/(SELECT count() FROM rv.hub_order),1) AS pct FROM rv.hub_order GROUP BY branch ORDER BY pct DESC FORMAT PrettyCompact\"\r\n"]
10+
[21.499089, "o", " ┌─branch─┬─orders─┬──pct─┐\r\n1. │ msk │ 9740 │ 94.7 │\r\n2. │ spb │ 180 │ 1.8 │\r\n3. │ dxb │ 155 │ 1.5 │\r\n4. │ ekb │ 130 │ 1.3 │\r\n5. │ ala │ 75 │ 0.7 │\r\n └────────┴────────┴──────┘\r\n"]
11+
[24.128329, "o", "\r\n\u001b[1;33m# === Beat 4 — Business Vault MDM (01:25) ===\u001b[0m\r\n"]
12+
[25.635284, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --multiline --query \"SELECT 'msk' AS branch, count() AS rows, countIf(first_name != '') AS with_pii, countIf(loyalty_segment != '') AS with_loyalty FROM rv.bv_customer_mdm__msk UNION ALL SELECT 'dxb', count(), countIf(first_name != ''), countIf(loyalty_segment IS NOT NULL AND loyalty_segment != '') FROM rv.bv_customer_mdm__dxb FORMAT PrettyCompact\"\r\n"]
13+
[37.092843, "o", " ┌─branch─┬─rows─┬─with_pii─┬─with_loyalty─┐\r\n1. │ dxb │ 80 │ 80 │ 0 │\r\n2. │ msk │ 2240 │ 2240 │ 152 │\r\n └────────┴──────┴──────────┴──────────────┘\r\n"]
14+
[40.056267, "o", "\r\n"]
15+
[40.056349, "o", "\u001b[1;33m# === Beat 5a — Cold offload (01:50) ===\u001b[0m\r\n"]
16+
[41.566622, "o", "\u001b[1;36m$\u001b[0m kubectl create job --from=cronjob/dv2-cold-offload-msk cold-demo-5534 -n dv2\r\n"]
17+
[42.711019, "o", "job.batch/cold-demo-5534 created\r\n"]
18+
[53.229094, "o", "\r\n\u001b[1;33m# === Beat 5b — MinIO parquet listing ===\u001b[0m\r\n"]
19+
[54.745112, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 minio-0 -- mc ls -r local/cold-tier | tail -5\r\n"]
20+
[62.23601, "o", "[2026-07-06 06:00:06 UTC] 4.1KiB STANDARD branch=ala/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 05:59:53 UTC] 3.7KiB STANDARD branch=dxb/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 05:59:49 UTC] 3.9KiB STANDARD branch=ekb/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 10:44:10 UTC] 50KiB STANDARD branch=msk/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 05:59:48 UTC] 4.6KiB STANDARD branch=spb/year=2026/month=07/customers_anon.parquet\r\n"]
21+
[64.753412, "o", "\r\n\u001b[1;33m# === Closing (02:00) — Single bootstrap reproduces everything ===\u001b[0m\r\n"]
22+
[66.268043, "o", "\u001b[1;36m$\u001b[0m bash infrastructure/dv2/bootstrap.sh\r\n"]
23+
[68.281547, "o", "\r\n"]
24+
[68.281596, "o", "\u001b[1;33mDemo complete. See docs/dv2-multi-branch/demo_evidence.md for the full breakdown.\u001b[0m\r\n"]

0 commit comments

Comments
 (0)