Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,32 @@ All notable changes to AgentFlow are documented in this file.

## [Unreleased]

### Changed — one Flink version across pip extra and container runtime (audit 07.07 F2)

- **The Docker runtime moves 2.2.1 → 2.3.0**, matching the `[flink]` extra
(`apache-flink==2.3.0`, bumped in #87 and smoke-validated since): the
`flink_jobs` image ARG (`FLINK_VERSION`, which also drives the pip install
and the `flink-s3-fs-hadoop` plugin symlink) and both official cluster
images in `docker-compose.yml` (`flink:2.3.0-java17`). Developing locally
against a 2.3.0 PyFlink while the cluster ran 2.2.1 could surface behaviour
that did not exist on the deployed minor; the two lines are now one.
- **The Kafka connector stays `flink-sql-connector-kafka-5.0.0-2.2`** — the
externalised connector is versioned `<connector>-<flink minor>` and no
`-2.3` build is published. Flink keeps `@Public` API stable within a major,
so the 2.2 artifact runs on a 2.3 cluster; the Dockerfile says so, and
`flink-smoke` submits `stream_processor.py` to a real Kafka+MinIO+Flink
cluster on every PR, so the combination is asserted rather than assumed.
- `docs/architecture.md` Technology Choices now reads Flink 2.3. Historical
records (CHANGELOG entries, `docs/perf/freshness-realpath-2026-06-30.md`)
keep the version they were measured on.

### Changed — the published SDK is under the same lint gate as `src/` (audit 07.07 F1)

- `ci.yml` runs `ruff check` and `ruff format --check` over `sdk/` as well;
the one-off reformat touched `async_client.py`, `cli.py`, `client.py`
(line-collapse only). `make lint` / `make format` now cover
`src/ tests/ scripts/ sdk/`, so local and CI agree.

## [2.0.0] - 2026-07-06

### Fixed — single-container demo deploys pin the DuckDB serving backend (G2 S7, 2026-07-06)
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ services:

# ── Flink ─────────────────────────────────────────────────────
flink-jobmanager:
image: flink:2.2.1-java17
image: flink:2.3.0-java17
ports:
- "8081:8081"
command: jobmanager
Expand All @@ -73,7 +73,7 @@ services:
retries: 5

flink-taskmanager:
image: flink:2.2.1-java17
image: flink:2.3.0-java17
depends_on:
flink-jobmanager:
condition: service_healthy
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis.
|-----------|--------|-----------|-------------------|
| Streaming | Kafka 3.7 (KRaft) | Pulsar | Ecosystem maturity, MSK managed service |
| CDC capture | Debezium + Kafka Connect | Python-native connectors | Mature Postgres/MySQL CDC, built-in offsets/schema history, one ops model |
| Processing | Flink 2.2 | Spark Structured Streaming | True event-time, lower latency, native watermarks |
| Processing | Flink 2.3 | Spark Structured Streaming | True event-time, lower latency, native watermarks |
| Storage | Iceberg 1.5 | Delta Lake | Vendor-neutral, hidden partitioning, time-travel |
| Local query | DuckDB | SQLite | Columnar, fast analytics, Iceberg support |
| Orchestration | Dagster | Airflow | Software-defined assets, better testing, type safety |
Expand Down
6 changes: 5 additions & 1 deletion src/processing/flink_jobs/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
FROM python:3.11-slim-bookworm@sha256:721dc13fd1be0a771e54b72097634291d628d0007dee9da777e2ce676a9c998f

ARG FLINK_VERSION=2.2.1
ARG FLINK_VERSION=2.3.0
ARG SCALA_VERSION=2.12
# The externalised Kafka connector is versioned <connector>-<flink minor> and
# its newest published build is 5.0.0-2.2 — there is no -2.3 artifact. Flink
# keeps @Public API stable inside a major, so the 2.2 build runs on a 2.3
# cluster; flink-smoke submits the real job against it to keep that honest.
ARG PYFLINK_KAFKA_JAR_VERSION=5.0.0-2.2

RUN apt-get update \
Expand Down