Skip to content

Commit 98bf1e7

Browse files
Merge pull request #175 from brownjuly2003-code/s8-freshness-e2e
docs(s8/s9/s10): Tier 1 real-path freshness, throughput, 2-pod topology
2 parents ea66fbb + ad11355 commit 98bf1e7

10 files changed

Lines changed: 1363 additions & 22 deletions

README.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# AgentFlow
22

3-
> Event-native metrics layer: business metrics that move when events happen — measured **1.1 s p50** event-to-metric freshness on production defaults. Live entity lookups, typed contracts, dual-language SDKs, and release-gated delivery for people, dashboards, services, and AI agents alike.
3+
> Event-native metrics layer: business metrics that move when events happen — measured **3.0 s p50** event-to-metric on the real Kafka→Flink→bridge path, **1.1 s p50** on the in-process demo shortcut. Live entity lookups, typed contracts, dual-language SDKs, and release-gated delivery for people, dashboards, services, and AI agents alike.
44
55
[![Release gate](https://img.shields.io/badge/release_gate-v2.0_published-brightgreen)](docs/dv2-multi-branch/RELEASE_STATUS.md)
66
[![codecov](https://codecov.io/gh/brownjuly2003-code/agentflow/branch/main/graph/badge.svg)](https://codecov.io/gh/brownjuly2003-code/agentflow)
@@ -11,7 +11,7 @@
1111

1212
BI on a replica answers yesterday's questions. Support, ops, and merch workflows need *current* orders, metrics, and health signals at the moment of decision — not a stale warehouse snapshot, not a pile of one-off service adapters, and not a cache that quietly serves 30-second-old numbers.
1313

14-
AgentFlow's axis is **event → live metric**: every metric declares which events move it (a contract-tested lineage graph), and the serving layer keeps reads fresh by invalidating its cache when events arrive — a measured behavior, not a slogan ([docs/freshness-benchmark.md](docs/freshness-benchmark.md)). One serving boundary on top of that axis:
14+
AgentFlow's axis is **event → live metric**: every metric declares which events move it (a contract-tested lineage graph), and the serving layer keeps reads fresh by invalidating its cache when events arrive — a measured behavior, not a slogan ([docs/freshness-benchmark.md](docs/freshness-benchmark.md), [real-path S8](docs/perf/freshness-e2e-realpath.md)). One serving boundary on top of that axis:
1515

1616
- streaming ingestion for operational events (validated, enriched, journaled)
1717
- a semantic layer that exposes entities, metrics, lineage, and query endpoints
@@ -22,7 +22,10 @@ Consumers are whoever needs the number now: humans, dashboards, downstream servi
2222

2323
## Highlights
2424

25-
- **Measured event-to-metric freshness** — an event entering the pipeline is reflected in `GET /v1/metrics/*` in **1.06 s p50 / 1.99 s p95** on production defaults (event-driven cache invalidation, no webhook registration), tunable to **238 ms p50**; a plain TTL cache on the same pipeline sits at ~15 s. Reproducible via `python scripts/benchmark_freshness.py`[freshness benchmark](docs/freshness-benchmark.md)
25+
- **Measured event-to-metric freshness** — two measured arms, not one number:
26+
- **Real path** (Kafka → Flink 2.3.0 → serving bridge → ClickHouse → `GET /v1/metrics/*` with Redis push invalidation): **3.02 s p50 / 5.70 s p95** (n=20, Mac/Colima) — [S8 e2e](docs/perf/freshness-e2e-realpath.md), `python scripts/benchmark_freshness_e2e.py`
27+
- **In-process demo shortcut** (`local_pipeline` → DuckDB, no Kafka/Flink): **1.06 s p50 / 1.99 s p95**, tunable to **238 ms p50**; TTL-only ~15 s — [demo benchmark](docs/freshness-benchmark.md), `python scripts/benchmark_freshness.py`
28+
Do not present the 1.06 s figure as the production streaming path.
2629
- **Lineage as a contract** — all six metrics declare their source events, serving table, and a 2.5 s p95 staleness budget in versioned contracts, exposed through `/v1/catalog` and `/v1/contracts` and pinned by tests against the actual write path
2730
- **Published release line through `v2.0.0`** on PyPI (`agentflow-runtime`, `agentflow-client`) and npm (`@yuliaedomskikh/agentflow-client`) via OIDC Trusted Publishers with SLSA provenance on every artifact
2831
- **Tested and gated** — 1,500+ unit tests plus a broad Windows no-Docker suite; CI enforces 13 required status checks (lint, schema, unit, integration, helm, perf, terraform, bandit, safety, npm-audit, trivy, contract, build-smoke) through branch protection

docs/freshness-benchmark.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,16 @@
55
66
Generated: `2026-06-06T10:10:41+03:00`
77

8-
> **S7 note (2026-07-09, not yet re-measured):** cache invalidation is no
9-
> longer hostage to the webhook dispatcher poll. Production wiring is push
10-
> from the serving bridge (Redis channel `agentflow:cache:metrics_invalidate`)
11-
> plus an independent journal scan in `MetricCacheController` — see
8+
> **S7/S8 note (2026-07-09):** cache invalidation is no longer hostage to the
9+
> webhook dispatcher poll. Production wiring is push from the serving bridge
10+
> (Redis channel `agentflow:cache:metrics_invalidate`) plus an independent
11+
> journal scan in `MetricCacheController` — see
1212
> [`serving-bridge.md`](serving-bridge.md#cache-invalidation-s7). The numbers
13-
> below still describe the pre-S7 poll path. **S8** will re-run this benchmark
14-
> end-to-end on the real Kafka→Flink→bridge path (and must use the N2-aware
15-
> ClickHouse timestamp convention).
13+
> **below** still describe the **in-process DuckDB shortcut** (pre-S7 poll-era
14+
> measurement of that arm). The full real path was re-measured in **S8**:
15+
> **3.02 s p50 / 5.70 s p95** event → `GET /v1/metrics/revenue` via
16+
> Kafka→Flink→bridge→ClickHouse — see
17+
> [`perf/freshness-e2e-realpath.md`](perf/freshness-e2e-realpath.md).
1618
1719
## What is measured
1820

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# E4 / S9 — live 2-pod topology proof (kind `hq-demo`)
2+
3+
Measured: `2026-07-09` on `deproject-mac` (Colima, kind cluster `hq-demo`).
4+
5+
## Goal
6+
7+
Prove the scale profile (`replicaCount=2`, `controlPlane.store=postgres`,
8+
`serving.backend=clickhouse`) does **not** split-brain webhook registrations
9+
across real pods — the sharpest class-5 failure mode from ADR 0010.
10+
11+
Automated checks: `scripts/k8s_replica_correctness_verify.sh` (Checks 1–2).
12+
Delivery/alert recipes remain separate (store guarantees already in
13+
`docs/perf/control-plane-pg-verify-2026-07-03.md`, 31/31).
14+
15+
## Topology
16+
17+
| Component | Placement |
18+
|-----------|-----------|
19+
| kind nodes | `hq-demo-control-plane`, `hq-demo-worker`, `hq-demo-worker2` (v1.32.2) |
20+
| AgentFlow pods | **2** Ready, anti-affinity preferred → one on `worker`, one on `worker2` |
21+
| Redis | 1 pod in `agentflow` |
22+
| PostgreSQL control plane | `postgres-0` in `dv2` → DB `agentflow_cp` |
23+
| ClickHouse serving | `clickhouse-0` in `dv2` |
24+
| Image | `agentflow/api:staging` loaded into kind (`kind load docker-image`) |
25+
26+
Helm:
27+
28+
```bash
29+
helm upgrade --install agentflow helm/agentflow \
30+
-f k8s/staging/values-staging.yaml \
31+
-f ~/values-hqdemo-scale.yaml \
32+
--namespace agentflow --wait --timeout 8m
33+
```
34+
35+
## Results
36+
37+
### Check 1 — two ready pods on postgres store
38+
39+
```
40+
PASS: 2 ready pods, all AGENTFLOW_CONTROLPLANE_STORE=postgres
41+
```
42+
43+
| Pod | Node | Ready |
44+
|-----|------|-------|
45+
| `agentflow-…-8q9ll` | `hq-demo-worker` | 1/1 |
46+
| `agentflow-…-kscc2` | `hq-demo-worker2` | 1/1 |
47+
48+
### Check 2 — Service-visible registration (script)
49+
50+
```
51+
PASS: webhook visible on all 8 round-robin reads across pods
52+
registered webhook_id=a24ccdd0-0b18-4eef-9286-bb13e5ddc3a4
53+
```
54+
55+
Access path used: `kubectl port-forward svc/agentflow 18080:8000`
56+
(NodePort socat to a fixed 30080 is brittle — this stand assigned **32456**).
57+
58+
### Cross-pod A→B (stronger probe)
59+
60+
Register on pod A, list on pod B (separate port-forwards):
61+
62+
```
63+
reg on A → id=323a26ea-f1f2-4778-97b5-d5062005135d
64+
list_A_has=yes
65+
list_B_has=yes
66+
CROSS_POD_OK: registration on A visible on B
67+
```
68+
69+
This is the split-brain test the embedded YAML store would fail.
70+
71+
## Stand notes / fixes applied during the run
72+
73+
1. **Free RAM** — stop compose Flink/Kafka stack before starting kind (Colima 6 GiB).
74+
2. **Image not on nodes**`kind load docker-image agentflow/api:staging --name hq-demo`.
75+
3. **Missing `psycopg`** — staging image lacked the optional `postgres` extra; install
76+
`psycopg[binary]` and re-commit the tag (scale profile requires it). Prefer baking
77+
`.[postgres]` into the staging Dockerfile for next builds.
78+
4. **`mapfile`** — macOS bash 3.2 has no `mapfile`; script updated to a portable loop
79+
in `scripts/k8s_replica_correctness_verify.sh`.
80+
5. **Kafka noise** — pods log rdkafka connect failures (no Kafka in this profile);
81+
readiness still green via `/v1/health`.
82+
83+
## Reproduce
84+
85+
```bash
86+
export PATH=$HOME/bin:/usr/local/bin:$PATH
87+
# kind nodes up; dv2 postgres+clickhouse Ready
88+
bash ~/s6_e4_stage2_deploy.sh # may need image load + psycopg first
89+
kubectl -n agentflow port-forward svc/agentflow 18080:8000 &
90+
BASE_URL=http://127.0.0.1:18080 bash scripts/k8s_replica_correctness_verify.sh
91+
# optional: register on pod A, list on pod B via two port-forwards
92+
```
93+
94+
## Status
95+
96+
**S9 / E4 automated topology proof: PASS** (Checks 1–2 + explicit A→B cross-pod).
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Event → Metric Freshness on the Real Path (S8)
2+
3+
> Generated by `scripts/benchmark_freshness_e2e.py`. Measured: `2026-07-09T15:33:39.729745+00:00`.
4+
> Machine-readable: `.artifacts/freshness/e2e-realpath-current.json`.
5+
6+
## What is measured
7+
8+
One sample is the wall-clock delay from producing an `order.created`
9+
event to Kafka `orders.raw` until `GET /v1/metrics/revenue` reflects
10+
that order's amount on the **full** real path:
11+
12+
```
13+
produce(orders.raw) → Flink stream_processor → events.validated → serving bridge → ClickHouse → (Redis push invalidation) → GET /v1/metrics/revenue
14+
```
15+
16+
This is the product axis claim (*event → live metric*) on the real
17+
streaming + bridge + serving path — not the in-process DuckDB shortcut
18+
and not the streaming-hop-only figure.
19+
20+
## System under test
21+
22+
- Host: `deproject-mac`, macOS 13.7.8 (Intel), Colima vz 6 GiB / 4 CPU
23+
- Python: `3.13.7`
24+
- Flink **2.3.0** (`agentflow-stream-processor` RUNNING, 1 TaskManager, 4 slots)
25+
- Kafka `confluentinc/cp-kafka:7.7.0` KRaft, bootstrap `127.0.0.1:19092`
26+
- ClickHouse 24.8 HTTP `127.0.0.1:8123`, Redis 7 for metric-cache push invalidation
27+
- Standalone bridge (`python -m src.processing.bridge_consumer`) + API
28+
(`SERVING_BACKEND=clickhouse`, demo seed on boot)
29+
- API: `http://127.0.0.1:8000` (metric `revenue`, window `24h`)
30+
- Source topic: `orders.raw`
31+
- Poll interval: 50 ms · timeout: 90.0 s
32+
- Warmup discarded: 2 · measured iterations: 20
33+
- Auth: dedicated high-rpm key (demo-key is 60 rpm and cannot sustain 50 ms polls)
34+
35+
## Results
36+
37+
| Metric | Real path (this run) | In-process DuckDB shortcut* | Streaming hop only** | Entity real-path (S6 live) |
38+
|--------|---------------------:|----------------------------:|---------------------:|---------------------------:|
39+
| p50 | **3.02 s** | 1.06 s | 2.50 s | 3.26 s (entity, n≈1) |
40+
| p95 | 5.70 s | 1.99 s | 10.11 s ||
41+
| p99 | 15.45 s || 15.42 s ||
42+
| min | 416 ms || 0.31 s ||
43+
| max | 17.89 s | 2.02 s | 16.09 s ||
44+
| mean | 3.76 s || 3.33 s ||
45+
| n / misses | 20 / 1 | 30 / 0 | 30 / 0 ||
46+
47+
\* `docs/freshness-benchmark.md``local_pipeline` → DuckDB → metric (pre-S7 poll-era).
48+
49+
\*\* `docs/perf/freshness-realpath-2026-06-30.md` — produce → `events.validated` only.
50+
51+
## Reading the numbers
52+
53+
- The real-path p50 includes Kafka produce, Flink Beam UDF hop, bridge
54+
apply into ClickHouse, Redis push (or journal-scan fallback) cache
55+
invalidation, and a metric SQL read — everything the product promise
56+
actually costs on a single-node Colima stand.
57+
- The in-process 1.06 s figure remains valid for the **demo/shortcut**
58+
path; do not present it as the production Kafka→Flink→bridge number.
59+
- S6's 3.26 s entity sample was a single live probe of the same path
60+
stopping at `GET /v1/entity`; this run measures the metric arm and
61+
reports full distribution.
62+
- N2 (ClickHouse naive DateTime = UTC) is required for sensible
63+
windowed metric SQL on non-UTC hosts; it is already in main.
64+
65+
## Reproduce
66+
67+
```bash
68+
# stack + bridge + API as in docs/serving-bridge.md / _NEXT_SESSION.md
69+
# Use a high-rpm API key: demo-key (60 rpm) cannot sustain 50 ms metric polls.
70+
python scripts/benchmark_freshness_e2e.py \
71+
--bootstrap 127.0.0.1:19092 --api-base http://127.0.0.1:8000 \
72+
--api-key <high-rpm-key> --iterations 20 --warmup 2
73+
```
74+
75+
## Samples (ms)
76+
77+
```
78+
[
79+
2231.5,
80+
2987.1,
81+
3300.7,
82+
2925.7,
83+
2713.8,
84+
3845.8,
85+
2489.1,
86+
2415.6,
87+
5061.7,
88+
777.0,
89+
3327.9,
90+
17885.7,
91+
2806.3,
92+
5029.1,
93+
415.6,
94+
4452.3,
95+
3191.1,
96+
3043.9,
97+
2918.4,
98+
3443.0
99+
]
100+
```

docs/perf/freshness-realpath-2026-06-30.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,11 @@ Measured: `2026-06-30`
88

99
> **Update 2026-07-09.** The missing link this document reports — "nothing
1010
> bridges `events.validated` into the serving store" — has since been built:
11-
> see [`../serving-bridge.md`](../serving-bridge.md). The numbers below are the
12-
> streaming-hop measurement as taken on 2026-06-30 and are left untouched; an
13-
> end-to-end *event → serving metric* measurement on the real path is a separate
14-
> run and is not reported here.
11+
> see [`../serving-bridge.md`](../serving-bridge.md). The numbers below remain
12+
> the streaming-hop measurement (produce → `events.validated`) as taken on
13+
> 2026-06-30. Full *event → metric* on the real path is **S8**:
14+
> [`freshness-e2e-realpath.md`](freshness-e2e-realpath.md)**3.02 s p50 /
15+
> 5.70 s p95** to `GET /v1/metrics/revenue`.
1516
1617
## Why this exists
1718

docs/perf/throughput-realpath.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Real-path throughput (S10): Kafka → Flink → bridge → ClickHouse
2+
3+
> Generated by `scripts/benchmark_throughput_realpath.py`. Measured: `2026-07-09T15:41:31+00:00`.
4+
> Machine-readable: `.artifacts/throughput/realpath-current.json`.
5+
6+
## What is measured
7+
8+
Burst produce of schema-valid `order.created` events onto `orders.raw`, through
9+
the live Flink `stream_processor` job, the serving bridge, and into ClickHouse.
10+
Counters come from the bridge Prometheus endpoint (`agentflow_bridge_events_*`).
11+
Latency samples poll `GET /v1/entity/order/{id}` for a subset of events with
12+
**unique** order ids (re-runs must not collide with prior stand data).
13+
14+
```
15+
produce(orders.raw) → Flink stream_processor → events.validated → serving bridge → ClickHouse
16+
```
17+
18+
This is **D2 / S10**: throughput on the real path, reported next to the S8
19+
latency number (3.02 s p50 event→metric unloaded). It is not the in-process
20+
DuckDB shortcut and not HTTP load-test RPS of the API.
21+
22+
## System under test
23+
24+
- Host: `deproject-mac`, macOS 13.7.8 (Intel), Colima vz 6 GiB / 4 CPU
25+
- Python: `3.13.7`
26+
- Flink **2.3.0** (`agentflow-stream-processor` RUNNING, 1 TaskManager, 4 slots)
27+
- Kafka KRaft HOST `127.0.0.1:19092`
28+
- ClickHouse 24.8 HTTP `127.0.0.1:8123`
29+
- Standalone bridge metrics `http://127.0.0.1:9108/metrics`
30+
- API for entity latency samples only
31+
- Count / pace: **400** events, unpaced burst
32+
- Latency samples: 10
33+
34+
## Results (canonical run)
35+
36+
| Arm | Value |
37+
|-----|------:|
38+
| Events produced | 400 |
39+
| Produce wall time | 0.57 s |
40+
| **Produce rate** | **699 events/s** |
41+
| Validated seen (Flink hop) | 400 |
42+
| Flink hop rate (produce start → catch-up) | 7.97 events/s |
43+
| Bridge applied delta | 400 |
44+
| Bridge duplicates delta | 0 |
45+
| Bridge apply failures delta | 0 |
46+
| **Bridge apply rate** | **7.97 events/s** |
47+
| Catch-up wall time | 50.2 s |
48+
| Lag start → end / peak sampled | 0 → 0 / 329 |
49+
50+
### Entity latency under burst load
51+
52+
| Metric | Value |
53+
|--------|------:|
54+
| n | 10 |
55+
| p50 | **19.5 s** |
56+
| p95 | 38.8 s |
57+
| min / max | 4.1 s / 43.6 s |
58+
| mean | 20.3 s |
59+
60+
### Consistency check (earlier 500-event burst)
61+
62+
| Arm | Value |
63+
|-----|------:|
64+
| Produce rate | 531 events/s |
65+
| Bridge apply rate | **8.24 events/s** |
66+
| Catch-up wall | 60.7 s |
67+
| Peak lag | 254 |
68+
69+
**8 events/s** bridge apply is stable across both runs on this stand.
70+
71+
## Reading the numbers
72+
73+
- **Produce rate (~700 events/s)** is what the driver put on Kafka. It is not
74+
the product ceiling.
75+
- **Bridge apply rate (~8 events/s)** is the product-relevant sustained number
76+
for event→serving on this single-node Colima stand: every applied event is
77+
in ClickHouse (idempotent journal). The S6 design
78+
([`serving-bridge.md`](../serving-bridge.md)) notes a serialized-writer
79+
ceiling via `_process_event` + scratch DuckDB — on this hardware the
80+
observed ceiling is **~8 events/s**, not hundreds.
81+
- Flink hop rate matches bridge apply rate in these runs: the bridge (and/or
82+
Flink Beam UDF hop under load) is the bottleneck; Kafka is not.
83+
- Peak lag ~300 means a burst of hundreds of events queues on
84+
`events.validated` until the bridge drains it.
85+
- Entity latency under burst (**p50 19.5 s**) is **queueing delay**, not the
86+
unloaded path cost. Unloaded S8 metric freshness is **3.02 s p50**;
87+
unloaded entity probe (S6) was **3.26 s**. Under a 400-event burst the tail
88+
waits behind the ~8 events/s drain.
89+
- Do not quote under-load entity p50 as the headline freshness number.
90+
91+
## Reproduce
92+
93+
```bash
94+
# stack + bridge as in docs/serving-bridge.md / _NEXT_SESSION.md
95+
python scripts/benchmark_throughput_realpath.py \
96+
--bootstrap 127.0.0.1:19092 --count 400 \
97+
--bridge-metrics http://127.0.0.1:9108/metrics \
98+
--api-base http://127.0.0.1:8000 --api-key <high-rpm-key> \
99+
--latency-samples 10 --latency-timeout-seconds 180
100+
```

0 commit comments

Comments
 (0)