fix(stand): bounded logs + restart policies + Flink failure-rate restart budget (P2.1-P2.3) - #214
Conversation
… budget P2.1: x-logging anchor (json-file, 50m x 3) applied to every service in all seven compose files — the 2026-07-18 soak filled the stand disk with unrotated json-file logs (TM alone 3.8G) and took the pipeline down; this makes the ephemeral daemon.json host fix permanent in the repo. P2.2: restart: unless-stopped on flink-jobmanager/flink-taskmanager in docker-compose.flink.yml (one-shot provision jobs keep default "no") — the same soak lost its TM to a container death and hung in NoResourceAvailable for the rest of the run. P2.3: bounded failure-rate restart strategy in both Flink jobs' build_pipeline (3 failures / 5 min, 10s delay, env-overridable) instead of the default infinite fixed-delay — a persistently broken job now parks in FAILED instead of re-emitting from checkpoint forever (soak r1: 116 restarts, 292k duplicates downstream). Unit suite: 2026 passed locally, incl. new restart-strategy tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DORA Metrics
|
|
Do not merge yet — P2.3 will fail on the real runtime. The runtime is apache-flink 2.3.0 (pinned in Fix recipe (small): replace from pyflink.common import Configuration
config = Configuration()
config.set_string("restart-strategy.type", "failure-rate")
config.set_string("restart-strategy.failure-rate.max-failures-per-interval", os.getenv("FLINK_RESTART_MAX_FAILURES_PER_INTERVAL", "3"))
config.set_string("restart-strategy.failure-rate.failure-rate-interval", "5 min") # env-overridable likewise
config.set_string("restart-strategy.failure-rate.delay", "10 s")
env.configure(config)and update the two test fixtures to fake |
…e for Flink 2.x StreamExecutionEnvironment.set_restart_strategy was removed in Flink 2.x (FLIP-381); on the pinned apache-flink 2.3.0 runtime build_pipeline() raised AttributeError at submission. Both jobs now set the failure-rate budget via Configuration.set_string + env.configure (verified against release-2.3.0 sources: Configuration re-exported in pyflink.common, env.configure at stream_execution_environment.py:379). Env overrides keep the same variables, now rendered as "<n> ms" duration strings. Test fakes drop set_restart_strategy and the RestartStrategies module entirely so a regression back to the removed API fails in unit tests, not only on the live cluster; fixtures fake Configuration and capture env.configure instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Makes the 2026-07-18 soak remediation permanent in the repo instead of living only in the Mac host's daemon.json, and bounds Flink's restart behavior.
x-logginganchor (json-file,max-size: 50m,max-file: 3) on every service in all sevendocker-compose*.yml. Root cause of the failed 4h soak was the stand disk filling with unrotated container logs (TM alone 3.8G).restart: unless-stoppedonflink-jobmanager/flink-taskmanagerindocker-compose.flink.yml. One-shot provision jobs (kafka-init,minio-init,flink-job-runner) deliberately keep the defaultno— restarting a completed job-runner would resubmit the Flink job.RestartStrategies.failure_rate_restart(3, 5min, 10s)(env-overridable) in both jobs'build_pipeline. With the default infinite fixed-delay strategy, a job whose environment is persistently broken restarts forever and re-emits from the last checkpoint each attempt (soak r1: 116 restarts → 292k duplicate events downstream). Now it parks in FAILED where alerting takes over.Verification: all compose files re-parsed with anchors resolved (every service carries the exact logging config);
tests/unit2026 passed locally including new restart-strategy assertions for both jobs; ruff check + format clean.Live
docker killTM → auto-restart check needs the Mac stand (currently powered down) — the flink-smoke CI lane exercises the merged compose config.🤖 Generated with Claude Code