fix(telegram): stop shared topic authority outages from crashing the daemon - #3974
Merged
Yeachan-Heo merged 1 commit intoAug 7, 2026
Merged
Conversation
Heyaaz
force-pushed
the
fix/telegram-topic-authority-crash
branch
from
August 7, 2026 00:51
b6e07e5 to
bf240f5
Compare
…daemon
The liveness heartbeat renewed a session's topic lease through a
single-argument .then() whose rejection escaped to the process-level
fatal handler whenever the shared topic authority was momentarily
unavailable. Production logs show the resulting daemon deaths as
repeated 'Uncaught exception' / 'Unhandled rejection: shared topic
authority unavailable' entries (one session crashed the daemon, then
eight hours of dead-owner state with no notifications delivered).
- Heartbeat lease renewal now reports the failure via a rejection
handler and retries on the next heartbeat instead of escaping.
- Startup topic-registry load failure is reported and the daemon
continues with an empty registry instead of dying before serving.
- The four catch{throw} authority-failure sites preserve the
underlying error as Error cause for diagnostics.
- The compensation fence stops retrying a failed persist every 250ms
forever: it gives up after 40 attempts (~10s) so a shutdown can
quiesce and the registry stops churning; the next scan/session
pass retries.
- DAEMON_GENERATION bumped to 55 (run() lifecycle change).
Regression test drives the liveness heartbeat against a dead
authority and asserts no unhandled rejection escapes.
Heyaaz
force-pushed
the
fix/telegram-topic-authority-crash
branch
from
August 7, 2026 01:30
bf240f5 to
df4f1f8
Compare
Yeachan-Heo
approved these changes
Aug 7, 2026
Yeachan-Heo
left a comment
Owner
There was a problem hiding this comment.
Terminal red-team (exact head df4f1f8)
Verdict: MERGE_READY / APPROVE
- Contract: shared topic authority outages must not crash the Telegram daemon; generation bump to 55 with tests.
- Scope: telegram-daemon + contract + tests + generation manifest.
- Exact-head CI green; mergeable=clean. Generation pin advances 54→55 for this daemon contract change (supersedes #3972 pin intentionally).
- Independent of ACP #3950 and perf #3764.
Merge authorized.
Yeachan-Heo
pushed a commit
that referenced
this pull request
Aug 7, 2026
Yeachan-Heo
pushed a commit
that referenced
this pull request
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stop the Telegram notifications daemon from dying when the shared topic authority is momentarily unavailable.
Root cause: the liveness heartbeat (5s interval, one per session) renewed a session's topic lease via
void this.#renewTopicLease(logicalSessionId).then(renewed => {...})— a single-argument.then()with no rejection handler.#renewTopicLeaseis an async function that adopts#persistTopicMutation's promise, so any authority failure (compareAndSet/read throwingshared topic authority unavailable) rejected that promise with nothing attached and escaped to the process-level fatal handler.Production evidence (gjc 0.12.15, 2026-08-06/07): repeated
Uncaught exception/Unhandled rejection: Error: shared topic authority unavailableentries with a bare throw-site stack +processTicksAndRejections, plussession scan failed,Telegram replay admission failed,shutdown persistence failed,shutdown was not durably quiescedwhile the authority was down.Fixes (all in
packages/coding-agent/src/sdk/bus/telegram-daemon.ts):run()'sawait this.loadTopics()is wrapped; a failed load warns and the daemon continues with an empty registry (mirrors the runScan retry philosophy from fix(notifications): stop a failed reconciliation pass from exiting the Telegram daemon #3907). Connected sessions re-create topics viaensureTopic; persist passes read the winner back from the authority.catch { throw new Error("shared topic authority unavailable") }sites now throw with{ cause: error }so the underlying failure isn't lost.#superviseCompensationFence'sfor(;;)250ms retry now gives up after 40 attempts (~10s) with a warn; previously it spun forever while the authority was down, churning the registry and blocking shutdown quiescence.DAEMON_GENERATION54→55 (protectedrun()lifecycle change); generation-guard manifest regenerated.Why
A single topic-authority outage killed the daemon (5s heartbeat × sessions), leaving every session topic as an unarchived shell answering nothing — observed as eight hours of dead-owner state with no notifications delivered.
Testing
a topic lease renewal failure on the liveness heartbeat is reported instead of escaping as an unhandled rejection) drivesconnectSession+hello(client_ping_pong)+event_replay_resultto register the heartbeat interval, arms a deadcompareAndSetauthority, fires the tick. Against the unpatched code it fails with exactly the production signature (shared topic authority unavailable+processTicksAndRejectionsunhandled rejection); with the patch it passes (no escape, warn logged).bun test packages/coding-agent/test/notifications-*.test.ts— 1430/1430 pass (54 files, includes the 556-test telegram-daemon suite)tsc -p tsconfig.json --noEmit(packages/coding-agent) — cleanbiome checkon touched files — cleanbun scripts/telegram-daemon-generation-guard.ts --validate-current-tree— clean (manifest regenerated for therundeclaration digest)GJC verdict
devbun checkpasses (biome + tsc on touched package)