fix(telegram): honor getUpdates 429 retry_after and back off poll restarts - #22
Open
muqiao215 wants to merge 1 commit into
Open
fix(telegram): honor getUpdates 429 retry_after and back off poll restarts#22muqiao215 wants to merge 1 commit into
muqiao215 wants to merge 1 commit into
Conversation
…tarts Root cause of the fleet-wide "active but unresponsive" outage: a burst of transient getUpdates errors (ServerDisconnectedError / Connection reset / Request timeout / 502) made controlmesh mark the transport dirty, stop_polling, rebuild the session, and immediately re-issue getUpdates -- with no backoff and without honoring TelegramRetryAfter.retry_after. That hammering tripped Telegram's getUpdates flood control (429, observed on moonrise: 'Flood control exceeded on GetUpdates. Retry in 5 seconds'), which then locked inbound delivery out for an escalating window. Every chat stopped receiving replies while the process and cron kept running (hourly model-cache refreshes prove the loop was alive), so systemd showed active(running)/Errors:0 and a restart 'fixed' it. Fixes: - Capture retry_after from TelegramRetryAfter in _note_poll_failed and store it on _TelegramPollDiagnostics (reset on success). - Before each poll-restart rebuild, sleep max(retry_after, exponential backoff) via _poll_restart_backoff_seconds() (0.5*2^n capped at 60s). 429 wins when larger. - Cap _keep_inbound_claim_alive renewals at _MAX_CLAIM_LIFETIME_SECONDS (1800s): a hung frontstage run can no longer renew the lease forever, so recover_stale_claims() can finally reclaim a permanently-blocked lane. - Emit a periodic (<=60s) INFO summary of pending/blocked backlog from the poll watchdog so the next occurrence is visible at runtime, not only at restart. Tests: unit tests for _poll_restart_backoff_seconds (exponential growth, 60s cap, retry_after precedence, clamping) and _TelegramPollDiagnostics retry_after lifecycle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (the real one)
Fleet-wide "active but unresponsive" outage. On every host
controlmesh.serviceshowedactive (running)/Errors: 0and the asyncio loop was alive the whole time (hourly model-cache refreshes continued 06:22→13:22), yet no bot replied to any user and a restart "fixed" it.A burst of transient
getUpdateserrors (ServerDisconnectedError,Connection reset by peer,Request timeout,502 Bad Gateway) made controlmesh mark the transport dirty →stop_polling()→ rebuild the aiohttp session → immediately re-issuegetUpdates— with no backoff and without honoringTelegramRetryAfter.retry_after(_recoverable_poll_reasonreturned"recoverable_http_429",_rebuild_poll_transportjust recreated the session,run()re-polled at once).That reconnect hammering tripped Telegram's
getUpdatesflood control — observed on moonrise:Once the 429 spiral starts,
getUpdateskeeps getting rejected for an escalating window, so no inbound updates are fetched → every chat appears dead while cron/model-cache keep the process "alive". This matches every observed fact (loop alive, service active, all users silent, restart resolves).The earlier PR #21 (cron subprocess group-kill) is a separate robustness fix and does not address this incident.
Fixes
In
controlmesh/messenger/telegram/app.py:TelegramRetryAfter.retry_after— captured in_note_poll_failed, stored on_TelegramPollDiagnostics.last_retry_after_seconds, reset on success._poll_restart_backoff_seconds(consecutive_failures, retry_after_seconds)=max(retry_after, min(60, 0.5*2^n));run()now sleeps this long before each_rebuild_poll_transport().retry_afterwins when Telegram explicitly asked us to wait._keep_inbound_claim_alivestops renewing after_MAX_CLAIM_LIFETIME_SECONDS(1800s). A hung frontstage run can no longer renew the 30s lease forever, sorecover_stale_claims()can finally reclaim a permanently-blocked lane (theblocked_lanes=1 / recovered_stale_claims=0shadow seen at restart).pending/blocked_lanes/oldest_age/unhealthy, so the next occurrence is visible at runtime instead of only at restart.Tests (
tests/messenger/telegram/test_poll_backoff.py)_poll_restart_backoff_seconds: no-failure→0, exponential growth (1→2→4→16), 60s cap from n=7,retry_afterprecedence, exponential precedence, negative-input clamping, zero-retry-after with failures still backs off._TelegramPollDiagnostics:note_poll_failedrecordsretry_after,note_poll_succeededresets it, default is 0, non-positive values clamped.Verified offline via
py_compile+ a standalone run of the backoff formula (n: 0→0s, 1→1s, 3→4s, 5→16s, 7→60s, 20→60s; retry_after=12 → 12s).