diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a303c61..0294022c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,32 @@ All notable changes to AgentFlow are documented in this file. ## [Unreleased] +### Changed — one Flink version across pip extra and container runtime (audit 07.07 F2) + +- **The Docker runtime moves 2.2.1 → 2.3.0**, matching the `[flink]` extra + (`apache-flink==2.3.0`, bumped in #87 and smoke-validated since): the + `flink_jobs` image ARG (`FLINK_VERSION`, which also drives the pip install + and the `flink-s3-fs-hadoop` plugin symlink) and both official cluster + images in `docker-compose.yml` (`flink:2.3.0-java17`). Developing locally + against a 2.3.0 PyFlink while the cluster ran 2.2.1 could surface behaviour + that did not exist on the deployed minor; the two lines are now one. +- **The Kafka connector stays `flink-sql-connector-kafka-5.0.0-2.2`** — the + externalised connector is versioned `-` and no + `-2.3` build is published. Flink keeps `@Public` API stable within a major, + so the 2.2 artifact runs on a 2.3 cluster; the Dockerfile says so, and + `flink-smoke` submits `stream_processor.py` to a real Kafka+MinIO+Flink + cluster on every PR, so the combination is asserted rather than assumed. +- `docs/architecture.md` Technology Choices now reads Flink 2.3. Historical + records (CHANGELOG entries, `docs/perf/freshness-realpath-2026-06-30.md`) + keep the version they were measured on. + +### Changed — the published SDK is under the same lint gate as `src/` (audit 07.07 F1) + +- `ci.yml` runs `ruff check` and `ruff format --check` over `sdk/` as well; + the one-off reformat touched `async_client.py`, `cli.py`, `client.py` + (line-collapse only). `make lint` / `make format` now cover + `src/ tests/ scripts/ sdk/`, so local and CI agree. + ## [2.0.0] - 2026-07-06 ### Fixed — single-container demo deploys pin the DuckDB serving backend (G2 S7, 2026-07-06) diff --git a/docker-compose.yml b/docker-compose.yml index dbf46718..fb68b36a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -47,7 +47,7 @@ services: # ── Flink ───────────────────────────────────────────────────── flink-jobmanager: - image: flink:2.2.1-java17 + image: flink:2.3.0-java17 ports: - "8081:8081" command: jobmanager @@ -73,7 +73,7 @@ services: retries: 5 flink-taskmanager: - image: flink:2.2.1-java17 + image: flink:2.3.0-java17 depends_on: flink-jobmanager: condition: service_healthy diff --git a/docs/architecture.md b/docs/architecture.md index 47b5f8fd..958befbf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -95,7 +95,7 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis. |-----------|--------|-----------|-------------------| | Streaming | Kafka 3.7 (KRaft) | Pulsar | Ecosystem maturity, MSK managed service | | CDC capture | Debezium + Kafka Connect | Python-native connectors | Mature Postgres/MySQL CDC, built-in offsets/schema history, one ops model | -| Processing | Flink 2.2 | Spark Structured Streaming | True event-time, lower latency, native watermarks | +| Processing | Flink 2.3 | Spark Structured Streaming | True event-time, lower latency, native watermarks | | Storage | Iceberg 1.5 | Delta Lake | Vendor-neutral, hidden partitioning, time-travel | | Local query | DuckDB | SQLite | Columnar, fast analytics, Iceberg support | | Orchestration | Dagster | Airflow | Software-defined assets, better testing, type safety | diff --git a/src/processing/flink_jobs/Dockerfile b/src/processing/flink_jobs/Dockerfile index 7c125b3c..dc569d9f 100644 --- a/src/processing/flink_jobs/Dockerfile +++ b/src/processing/flink_jobs/Dockerfile @@ -1,7 +1,11 @@ FROM python:3.11-slim-bookworm@sha256:721dc13fd1be0a771e54b72097634291d628d0007dee9da777e2ce676a9c998f -ARG FLINK_VERSION=2.2.1 +ARG FLINK_VERSION=2.3.0 ARG SCALA_VERSION=2.12 +# The externalised Kafka connector is versioned - and +# its newest published build is 5.0.0-2.2 — there is no -2.3 artifact. Flink +# keeps @Public API stable inside a major, so the 2.2 build runs on a 2.3 +# cluster; flink-smoke submits the real job against it to keep that honest. ARG PYFLINK_KAFKA_JAR_VERSION=5.0.0-2.2 RUN apt-get update \