Skip to content

fix(stand): bounded logs + restart policies + Flink failure-rate restart budget (P2.1-P2.3) - #214

Merged
brownjuly2003-code merged 3 commits into
mainfrom
fix/p2-stand-hardening
Jul 20, 2026
Merged

fix(stand): bounded logs + restart policies + Flink failure-rate restart budget (P2.1-P2.3)#214
brownjuly2003-code merged 3 commits into
mainfrom
fix/p2-stand-hardening

Conversation

@brownjuly2003-code

Copy link
Copy Markdown
Owner

Makes the 2026-07-18 soak remediation permanent in the repo instead of living only in the Mac host's daemon.json, and bounds Flink's restart behavior.

  • P2.1 — log rotation: x-logging anchor (json-file, max-size: 50m, max-file: 3) on every service in all seven docker-compose*.yml. Root cause of the failed 4h soak was the stand disk filling with unrotated container logs (TM alone 3.8G).
  • P2.2 — restart policy: restart: unless-stopped on flink-jobmanager/flink-taskmanager in docker-compose.flink.yml. One-shot provision jobs (kafka-init, minio-init, flink-job-runner) deliberately keep the default no — restarting a completed job-runner would resubmit the Flink job.
  • P2.3 — bounded restart strategy: RestartStrategies.failure_rate_restart(3, 5min, 10s) (env-overridable) in both jobs' build_pipeline. With the default infinite fixed-delay strategy, a job whose environment is persistently broken restarts forever and re-emits from the last checkpoint each attempt (soak r1: 116 restarts → 292k duplicate events downstream). Now it parks in FAILED where alerting takes over.

Verification: all compose files re-parsed with anchors resolved (every service carries the exact logging config); tests/unit 2026 passed locally including new restart-strategy assertions for both jobs; ruff check + format clean.

Live docker kill TM → auto-restart check needs the Mac stand (currently powered down) — the flink-smoke CI lane exercises the merged compose config.

🤖 Generated with Claude Code

… budget

P2.1: x-logging anchor (json-file, 50m x 3) applied to every service in all
seven compose files — the 2026-07-18 soak filled the stand disk with
unrotated json-file logs (TM alone 3.8G) and took the pipeline down; this
makes the ephemeral daemon.json host fix permanent in the repo.

P2.2: restart: unless-stopped on flink-jobmanager/flink-taskmanager in
docker-compose.flink.yml (one-shot provision jobs keep default "no") — the
same soak lost its TM to a container death and hung in NoResourceAvailable
for the rest of the run.

P2.3: bounded failure-rate restart strategy in both Flink jobs'
build_pipeline (3 failures / 5 min, 10s delay, env-overridable) instead of
the default infinite fixed-delay — a persistently broken job now parks in
FAILED instead of re-emitting from checkpoint forever (soak r1: 116
restarts, 292k duplicates downstream).

Unit suite: 2026 passed locally, incl. new restart-strategy tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown

DORA Metrics

  • Window: last 30 days
  • Branch: main
  • Deployment frequency: 112 total / 26.13 per week
  • Lead time for changes: avg 0.6h / median 0.0h
  • Change failure rate: 75.89% (85/112)
  • MTTR: 20.13h across 5 incident(s)

@brownjuly2003-code

Copy link
Copy Markdown
Owner Author

Do not merge yet — P2.3 will fail on the real runtime.

The runtime is apache-flink 2.3.0 (pinned in src/processing/flink_jobs/requirements.txt, asserted in the Dockerfile). In Flink 2.x StreamExecutionEnvironment.set_restart_strategy was removed (FLIP-381) — verified against release-2.3.0 sources: pyflink/common/restart_strategy.py still exists, but the env method is gone, so build_pipeline() raises AttributeError at submission. Unit tests pass because the fixture fakes the env; expect flink-smoke on this PR to fail — that failure is this, not the compose changes.

Fix recipe (small): replace env.set_restart_strategy(RestartStrategies.failure_rate_restart(...)) in both jobs with the Flink 2.x config route, which release-2.3.0 supports (def configure(self, configuration: Configuration) at stream_execution_environment.py:379):

from pyflink.common import Configuration

config = Configuration()
config.set_string("restart-strategy.type", "failure-rate")
config.set_string("restart-strategy.failure-rate.max-failures-per-interval", os.getenv("FLINK_RESTART_MAX_FAILURES_PER_INTERVAL", "3"))
config.set_string("restart-strategy.failure-rate.failure-rate-interval", "5 min")   # env-overridable likewise
config.set_string("restart-strategy.failure-rate.delay", "10 s")
env.configure(config)

and update the two test fixtures to fake pyflink.common.Configuration + capture env.configure instead of set_restart_strategy. P2.1/P2.2 (compose logging/restart) are unaffected.

JuliaEdom and others added 2 commits July 20, 2026 16:44
…e for Flink 2.x

StreamExecutionEnvironment.set_restart_strategy was removed in Flink 2.x
(FLIP-381); on the pinned apache-flink 2.3.0 runtime build_pipeline()
raised AttributeError at submission. Both jobs now set the failure-rate
budget via Configuration.set_string + env.configure (verified against
release-2.3.0 sources: Configuration re-exported in pyflink.common,
env.configure at stream_execution_environment.py:379). Env overrides
keep the same variables, now rendered as "<n> ms" duration strings.

Test fakes drop set_restart_strategy and the RestartStrategies module
entirely so a regression back to the removed API fails in unit tests,
not only on the live cluster; fixtures fake Configuration and capture
env.configure instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@brownjuly2003-code
brownjuly2003-code merged commit d0b7461 into main Jul 20, 2026
27 checks passed
@brownjuly2003-code
brownjuly2003-code deleted the fix/p2-stand-hardening branch July 20, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants