Skip to content

STATUS.md

Latest commit

 

History

History
160 lines (140 loc) · 12.5 KB

STATUS.md

File metadata and controls

160 lines (140 loc) · 12.5 KB
Raw

Engineering Status

Updated: 2026-07-30 (clean-checkout PyFlink OCI build + submission smoke PASS; Flink Kubernetes Operator + Helm golden-topology deploy PASS; core-only API and hardened runtime-image remediation locally PASS; golden topology remains a production candidate, not production accepted) · release line v2.0.0. Numbers below come only from measured, in-repo evidence — see the linked reports for methodology and reproduction commands.

AgentFlow's product axis — event → live metric on the real streaming path (Kafka → PyFlink → events.validated → serving bridge → ClickHouse → API with Redis push invalidation) — is implemented, measured, and documented. The Iceberg materializer, operator-compatible image, and Helm workload are implemented and locally contract-tested. On 2026-07-30 a clean-checkout OCI build and real Flink job submission smoke passed on deproject-mac, and a later clean kind + Operator + Helm acceptance of exact HEAD 36ed1ec also passed (stable hold, growing checkpoints, zero leadership flaps; Kafka on that stand used live runtime fixes, later captured in the tracked kind acceptance scaffold k8s/acceptance/kafka-kraft.yaml — not production Kafka). Live Iceberg materialization, lake-to-serving E2E, recovery, soak, rollback, and external security acceptance remain open. The earlier 4 h @ 100 eps result remains valid evidence for its measured pre-materializer path, not for the still-unaccepted full golden topology.

Closing CI hardening on 2026-07-30 also verified the Python 3.11/3.12/3.13 compatibility lanes, made pyiceberg optional for the core-only API path, aligned SDK examples with Ruff 0.16 Markdown formatting, and removed pip/setuptools/wheel from the final API image after pip check. A later dependency-resolution gate now keeps the Iceberg write path complete with Python 3.13-compatible pyiceberg-core==0.7.0 and holds the optional MCP integration on its supported 1.x API. A clean core-only HTTP smoke returned 200 for health, entity, and NL query. An independent Mac rebuild scanned with Trivy 0.70.0 at zero HIGH/CRITICAL findings; see security-runtime-image-trivy-2026-07-30.md. The dependency failure, resolution, and clean Mac verification are recorded in dependency-compatibility-2026-07-30.md. All repository-owned coverage floors remain blocking; Codecov upload is non-blocking reporting until the repository is enabled in that external service, as recorded in operations/codecov-setup.md. The exact post-remediation GitHub SHA must still complete all required checks before the external closure gate is claimed.

Project lifecycle: closure candidate. Engineering scope is frozen; the production-candidate boundary, future acceptance program, and release gates are recorded in PROJECT_CLOSURE.md.

Proven

Claim Result Evidence
Real-path freshness e2e 3.02 s p50 / 5.70 s p95 (n=20) perf/freshness-e2e-realpath.md
In-process demo freshness 1.06 s p50 / 1.99 s p95 freshness-benchmark.md
Real-path throughput measured produce ~700 eps; bridge apply is the ceiling (see below) perf/throughput-realpath.md
2-pod control plane on kind webhook registered on pod A visible on pod B; verify script PASS perf/e4-2pod-topology-2026-07-09.md
E4 Checks 1–4 (2 pods, delivery + alert single-page) PASS on kind perf/e4-check4-alert-single-page-2026-07-17.md
4 h endurance soak (real path + API reads) bounded lag (peak 2 915 → 0), bridge RSS/FD flat, one faulted batch replayed exactly-once by the journal guard, zero cache drift perf/soak-s11-2026-07-10.md
At-scale on own data (S13) 51.2 M rows / 2.87 M orders / 4 years of legend history, analyst queries 20–730 ms, all 17 at-scale correctness checks pass (10 row reconciliations + 5 §12 invariants + 2 distributions) incl. full-scan GTIN validation; §12's 12 invariants are pinned in full by 15 unit tests perf/scale-own-data-2026-07-11.md
Security pass (offline/unit remainder) closed; third-party pen-test not claimed security-s12-2026-07-09.md, security-audit.md
Multi-tenant ClickHouse write key adversarial two-tenant suite green on live CH 25.3 (CI test-integration + audit stand) security-audit.md, tests/integration/test_clickhouse_tenant_isolation_live.py
Clean-checkout PyFlink OCI build + submission smoke PASS clean-checkout OCI build + submission smoke on 2026-07-30 — image built, JobID RUNNING, not Operator/E2E perf/golden-flink-submission-2026-07-30.md
Flink Kubernetes Operator + Helm golden deploy PASS clean kind Operator/Helm deploy of verified image on exact HEAD 36ed1ec — CR/job stable, checkpoints 2→23, leader flaps 0, not lake E2E perf/golden-operator-acceptance-2026-07-30.md
Hardened API runtime image (local acceptance) core-only API import/HTTP smoke PASS; Trivy 0.70.0 reports 0 HIGH/CRITICAL after runtime installer removal security-runtime-image-trivy-2026-07-30.md
Python cloud/MCP dependency compatibility (local + isolated Mac) 2170 unit/property tests PASS; MCP 1.29 + PyIceberg 0.11.1/core 0.7.0 clean environment and 39 focused tests PASS dependency-compatibility-2026-07-30.md

2026-07-23 audit closure

Area Locally verified state
Claims and documentation machine-readable topology, runtime, SDK, Python, and quality claims agree; 7 validator tests pass and strict MkDocs builds
Runtime and deployment pinned PyFlink 2.3 OCI definition and Helm FlinkDeployment agree; Helm lint passes and 32 Helm contract tests pass
CDC and materializers fail-closed CDC attribution plus separate Iceberg and ClickHouse consumers; 43 CDC and 56 lake/serving component tests pass
SDKs checked Python/TypeScript capability matrix; TypeScript typecheck and all 50 Vitest tests pass
Lifecycle and query paths role-aware lifecycle, deterministic partial search status, canonical tenant-scoped session job, and bounded HTTP readiness deadline are covered by targeted tests
Acceptance boundary clean build/submission smoke and Operator/Helm deploy now measured (2026-07-30); live Iceberg, lake-to-serving E2E, restore/replay, fresh soak/rollback, external pen-test, and npm environment approval remain unverified

Bridge write-path throughput — drain ceiling measured

The bridge apply rate is the honest product ceiling; it has been raised in measured steps on the same Mac compose stand (Kafka → Flink → bridge → CH):

Step Bridge apply State
Baseline (per-event apply) ~8 eps measured
Q1.2 — ClickHouse-only sink, no scratch lake 11.4 eps measured
Q1.3 — multi-row batch apply 22.9 eps measured — perf/throughput-realpath-q13-2026-07-09.md
Q1.4 — batched session/user read-modify-writes (constant round-trips per batch) 87.4 eps measured — perf/throughput-realpath-q14-2026-07-10.md
Stretch try — 2000-event drain on same Mac class 107.3 eps measured — perf/throughput-realpath-100eps-try-2026-07-17.md
Paced 10 min @ 100 eps produce 96.5 apply / 97.1 flink / 100 produce measured — perf/throughput-realpath-paced100-2026-07-17.md
Paced 1 h @ 100 eps produce 99.5 apply / 99.5 flink / 100 produce measured — perf/throughput-realpath-paced100-1h-2026-07-17.md
Paced 4 h @ 100 eps produce (r4) 99.9 apply / 99.9 flink / 100.0 produce — 1 440 000 events, dup = 0, failures = 0, lag 0 → 0, 0 Flink restarts measured — perf/throughput-realpath-paced100-4h-r4-2026-07-19.md

The series target of ≥ 80 eps is met on the 400-burst profile. A 2000-event drain cleared at 107.3 eps. Paced ingress at 100 eps held for 10 min, 1 h, and — closing the multi-hour gate — 4 h (r4: produced = validated = applied = 1 440 000 exactly, 0 duplicates, 0 failures, lag ends at 0, zero Flink restarts). The two failed 4 h attempts before r4 are written up honestly in the r4 report (stand disk exhaustion; bench producer losing a broker self-fence) — neither was an apply-path defect. Semantics of the batched path are in serving-bridge.md.

Known issues

  • Multi-tenant ClickHouse — proven live (audit P0-1). The boundary is the tenant_id column, leading each serving table's write key on both stores (ADR-004). DuckDB remains covered by example and property suites; ClickHouse is covered by tests/integration/test_clickhouse_tenant_isolation_live.py on live server 25.3 (CI test-integration service + audit Mac stand). Cross-tenant lookups 404, aggregates stay tenant-scoped, and assert_tenant_key() refuses an old single-column sorting key. Broader isolation across every external dependency is still out of scope — see security-audit.md.

  • API RSS growth under steady load — fixed and verified live (was 175 MB → 1.67 GB over the 4 h soak; the bridge stayed flat). The webhook dispatcher re-materialized the whole pipeline_events journal every 2 s and the scan/push dedup sets grew one entry per event forever; journal scans are now cursor-bounded and the seen-sets capped (issue #183, details in serving-bridge.md). Unit scale: per-scan allocation flat ≤ 0.8 MB against a journal growing 50 k → 400 k rows (was 35.5 → 283.6 MB). Live re-verification 2026-07-11: 97 min at the soak read/apply profile against a 1.37 M-row journal — RSS slope +7.5 MB/h, plateaued (was ~+370 MB/h monotonic); perf/rss-reverify-183-2026-07-11.md.

Post-closure operations and future work

The items below are not active engineering backlog for the closing release. They require a separately authorized acceptance, deployment, or breaking-release program.

  1. Golden-topology acceptance — clean-checkout OCI build + submission smoke is PASS (perf/golden-flink-submission-2026-07-30.md) and clean kind + Kubernetes Operator + Helm deployment of the verified image on exact HEAD 36ed1ec is also PASS (perf/golden-operator-acceptance-2026-07-30.md). Remaining acceptance work: two-tenant Kafka → PyFlink → Iceberg → ClickHouse → API smoke; checkpoint restore/replay without duplicate (tenant_id, event_id) rows; fresh 4 h soak plus backup/rollback rehearsal. Acceptance-scaffold Kafka reproducibility debt (kind runtime fixes for enableServiceLinks / controller quorum voters) is recorded in the Operator evidence and is not a production-acceptance claim.
  2. External gates — third-party penetration test with remediation/retest, production CDC onboarding and credentials, public production-grade benchmark, and GitHub Environment npm with approval protection.
  3. P2-6 packaging (breaking) — Phase 0 inventory + defaults done; Phase 1 (tree/agentflow_runtime + shim) waits for a release branch: plans/p2-6-runtime-namespace-migration.md.
  4. Flink-runtime dependency bump — the pinned apache-flink==2.3.0 job environment holds a safety ignore for a non-fixable transitive pyarrow advisory (isolated to the Flink image, core pins pyarrow>=17); retire the ignore when the upstream flink/beam chain allows it.

Keep this file to one page. Add a number only after the measurement doc it links to exists; retired claims move to the changelog.