|
| 1 | +# Real-path throughput (S10): Kafka → Flink → bridge → ClickHouse |
| 2 | + |
| 3 | +> Generated by `scripts/benchmark_throughput_realpath.py`. Measured: `2026-07-09T15:41:31+00:00`. |
| 4 | +> Machine-readable: `.artifacts/throughput/realpath-current.json`. |
| 5 | +
|
| 6 | +## What is measured |
| 7 | + |
| 8 | +Burst produce of schema-valid `order.created` events onto `orders.raw`, through |
| 9 | +the live Flink `stream_processor` job, the serving bridge, and into ClickHouse. |
| 10 | +Counters come from the bridge Prometheus endpoint (`agentflow_bridge_events_*`). |
| 11 | +Latency samples poll `GET /v1/entity/order/{id}` for a subset of events with |
| 12 | +**unique** order ids (re-runs must not collide with prior stand data). |
| 13 | + |
| 14 | +``` |
| 15 | +produce(orders.raw) → Flink stream_processor → events.validated → serving bridge → ClickHouse |
| 16 | +``` |
| 17 | + |
| 18 | +This is **D2 / S10**: throughput on the real path, reported next to the S8 |
| 19 | +latency number (3.02 s p50 event→metric unloaded). It is not the in-process |
| 20 | +DuckDB shortcut and not HTTP load-test RPS of the API. |
| 21 | + |
| 22 | +## System under test |
| 23 | + |
| 24 | +- Host: `deproject-mac`, macOS 13.7.8 (Intel), Colima vz 6 GiB / 4 CPU |
| 25 | +- Python: `3.13.7` |
| 26 | +- Flink **2.3.0** (`agentflow-stream-processor` RUNNING, 1 TaskManager, 4 slots) |
| 27 | +- Kafka KRaft HOST `127.0.0.1:19092` |
| 28 | +- ClickHouse 24.8 HTTP `127.0.0.1:8123` |
| 29 | +- Standalone bridge metrics `http://127.0.0.1:9108/metrics` |
| 30 | +- API for entity latency samples only |
| 31 | +- Count / pace: **400** events, unpaced burst |
| 32 | +- Latency samples: 10 |
| 33 | + |
| 34 | +## Results (canonical run) |
| 35 | + |
| 36 | +| Arm | Value | |
| 37 | +|-----|------:| |
| 38 | +| Events produced | 400 | |
| 39 | +| Produce wall time | 0.57 s | |
| 40 | +| **Produce rate** | **699 events/s** | |
| 41 | +| Validated seen (Flink hop) | 400 | |
| 42 | +| Flink hop rate (produce start → catch-up) | 7.97 events/s | |
| 43 | +| Bridge applied delta | 400 | |
| 44 | +| Bridge duplicates delta | 0 | |
| 45 | +| Bridge apply failures delta | 0 | |
| 46 | +| **Bridge apply rate** | **7.97 events/s** | |
| 47 | +| Catch-up wall time | 50.2 s | |
| 48 | +| Lag start → end / peak sampled | 0 → 0 / 329 | |
| 49 | + |
| 50 | +### Entity latency under burst load |
| 51 | + |
| 52 | +| Metric | Value | |
| 53 | +|--------|------:| |
| 54 | +| n | 10 | |
| 55 | +| p50 | **19.5 s** | |
| 56 | +| p95 | 38.8 s | |
| 57 | +| min / max | 4.1 s / 43.6 s | |
| 58 | +| mean | 20.3 s | |
| 59 | + |
| 60 | +### Consistency check (earlier 500-event burst) |
| 61 | + |
| 62 | +| Arm | Value | |
| 63 | +|-----|------:| |
| 64 | +| Produce rate | 531 events/s | |
| 65 | +| Bridge apply rate | **8.24 events/s** | |
| 66 | +| Catch-up wall | 60.7 s | |
| 67 | +| Peak lag | 254 | |
| 68 | + |
| 69 | +≈ **8 events/s** bridge apply is stable across both runs on this stand. |
| 70 | + |
| 71 | +## Reading the numbers |
| 72 | + |
| 73 | +- **Produce rate (~700 events/s)** is what the driver put on Kafka. It is not |
| 74 | + the product ceiling. |
| 75 | +- **Bridge apply rate (~8 events/s)** is the product-relevant sustained number |
| 76 | + for event→serving on this single-node Colima stand: every applied event is |
| 77 | + in ClickHouse (idempotent journal). The S6 design |
| 78 | + ([`serving-bridge.md`](../serving-bridge.md)) notes a serialized-writer |
| 79 | + ceiling via `_process_event` + scratch DuckDB — on this hardware the |
| 80 | + observed ceiling is **~8 events/s**, not hundreds. |
| 81 | +- Flink hop rate matches bridge apply rate in these runs: the bridge (and/or |
| 82 | + Flink Beam UDF hop under load) is the bottleneck; Kafka is not. |
| 83 | +- Peak lag ~300 means a burst of hundreds of events queues on |
| 84 | + `events.validated` until the bridge drains it. |
| 85 | +- Entity latency under burst (**p50 19.5 s**) is **queueing delay**, not the |
| 86 | + unloaded path cost. Unloaded S8 metric freshness is **3.02 s p50**; |
| 87 | + unloaded entity probe (S6) was **3.26 s**. Under a 400-event burst the tail |
| 88 | + waits behind the ~8 events/s drain. |
| 89 | +- Do not quote under-load entity p50 as the headline freshness number. |
| 90 | + |
| 91 | +## Reproduce |
| 92 | + |
| 93 | +```bash |
| 94 | +# stack + bridge as in docs/serving-bridge.md / _NEXT_SESSION.md |
| 95 | +python scripts/benchmark_throughput_realpath.py \ |
| 96 | + --bootstrap 127.0.0.1:19092 --count 400 \ |
| 97 | + --bridge-metrics http://127.0.0.1:9108/metrics \ |
| 98 | + --api-base http://127.0.0.1:8000 --api-key <high-rpm-key> \ |
| 99 | + --latency-samples 10 --latency-timeout-seconds 180 |
| 100 | +``` |
0 commit comments