Skip to content

fix(telegram): stop shared topic authority outages from crashing the daemon - #3974

Merged
Yeachan-Heo merged 1 commit into
Yeachan-Heo:devfrom
Heyaaz:fix/telegram-topic-authority-crash
Aug 7, 2026
Merged

fix(telegram): stop shared topic authority outages from crashing the daemon#3974
Yeachan-Heo merged 1 commit into
Yeachan-Heo:devfrom
Heyaaz:fix/telegram-topic-authority-crash

Conversation

@Heyaaz

@Heyaaz Heyaaz commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

What

Stop the Telegram notifications daemon from dying when the shared topic authority is momentarily unavailable.

Root cause: the liveness heartbeat (5s interval, one per session) renewed a session's topic lease via void this.#renewTopicLease(logicalSessionId).then(renewed => {...}) — a single-argument .then() with no rejection handler. #renewTopicLease is an async function that adopts #persistTopicMutation's promise, so any authority failure (compareAndSet/read throwing shared topic authority unavailable) rejected that promise with nothing attached and escaped to the process-level fatal handler.

Production evidence (gjc 0.12.15, 2026-08-06/07): repeated Uncaught exception / Unhandled rejection: Error: shared topic authority unavailable entries with a bare throw-site stack + processTicksAndRejections, plus session scan failed, Telegram replay admission failed, shutdown persistence failed, shutdown was not durably quiesced while the authority was down.

Fixes (all in packages/coding-agent/src/sdk/bus/telegram-daemon.ts):

  1. Heartbeat lease renewal now passes a rejection handler — the failure is warn-logged and retried on the next heartbeat instead of killing the daemon.
  2. Startup registry loadrun()'s await this.loadTopics() is wrapped; a failed load warns and the daemon continues with an empty registry (mirrors the runScan retry philosophy from fix(notifications): stop a failed reconciliation pass from exiting the Telegram daemon #3907). Connected sessions re-create topics via ensureTopic; persist passes read the winner back from the authority.
  3. Cause preservation — the four catch { throw new Error("shared topic authority unavailable") } sites now throw with { cause: error } so the underlying failure isn't lost.
  4. Bounded compensation fence#superviseCompensationFence's for(;;) 250ms retry now gives up after 40 attempts (~10s) with a warn; previously it spun forever while the authority was down, churning the registry and blocking shutdown quiescence.
  5. DAEMON_GENERATION 54→55 (protected run() lifecycle change); generation-guard manifest regenerated.

Why

A single topic-authority outage killed the daemon (5s heartbeat × sessions), leaving every session topic as an unarchived shell answering nothing — observed as eight hours of dead-owner state with no notifications delivered.

Testing

  • Reproduction: new regression test (a topic lease renewal failure on the liveness heartbeat is reported instead of escaping as an unhandled rejection) drives connectSession + hello(client_ping_pong) + event_replay_result to register the heartbeat interval, arms a dead compareAndSet authority, fires the tick. Against the unpatched code it fails with exactly the production signature (shared topic authority unavailable + processTicksAndRejections unhandled rejection); with the patch it passes (no escape, warn logged).
  • bun test packages/coding-agent/test/notifications-*.test.ts — 1430/1430 pass (54 files, includes the 556-test telegram-daemon suite)
  • tsc -p tsconfig.json --noEmit (packages/coding-agent) — clean
  • biome check on touched files — clean
  • bun scripts/telegram-daemon-generation-guard.ts --validate-current-tree — clean (manifest regenerated for the run declaration digest)

GJC verdict

gajae.pr-review-verdict.v1 merge-approved sha256:3562971edf08fd92bddda2efc469fbed670b2c74b0c59f6931916c775afbd87e reviewer:critic evidence:bun test packages/coding-agent/test/notifications-*.test.ts (1430/1430) + tsc --noEmit + generation-guard --validate-current-tree + four independent critic passes on exact head df4f1f8cb205aea5da9d356bd885aec9e4fbcc05 (rebase onto upstream/dev 9079f1b9a; CHANGELOG conflict resolved keeping both entries; fix code tree-identical to critic-approved bf240f526)

  • Target branch is dev
  • bun check passes (biome + tsc on touched package)
  • Tested locally
  • CHANGELOG updated (if user-facing)
  • Verdict above matches the exact PR head, not an earlier commit

@Heyaaz
Heyaaz force-pushed the fix/telegram-topic-authority-crash branch from b6e07e5 to bf240f5 Compare August 7, 2026 00:51
…daemon

The liveness heartbeat renewed a session's topic lease through a
single-argument .then() whose rejection escaped to the process-level
fatal handler whenever the shared topic authority was momentarily
unavailable. Production logs show the resulting daemon deaths as
repeated 'Uncaught exception' / 'Unhandled rejection: shared topic
authority unavailable' entries (one session crashed the daemon, then
eight hours of dead-owner state with no notifications delivered).

- Heartbeat lease renewal now reports the failure via a rejection
  handler and retries on the next heartbeat instead of escaping.
- Startup topic-registry load failure is reported and the daemon
  continues with an empty registry instead of dying before serving.
- The four catch{throw} authority-failure sites preserve the
  underlying error as Error cause for diagnostics.
- The compensation fence stops retrying a failed persist every 250ms
  forever: it gives up after 40 attempts (~10s) so a shutdown can
  quiesce and the registry stops churning; the next scan/session
  pass retries.
- DAEMON_GENERATION bumped to 55 (run() lifecycle change).

Regression test drives the liveness heartbeat against a dead
authority and asserts no unhandled rejection escapes.
@Heyaaz
Heyaaz force-pushed the fix/telegram-topic-authority-crash branch from bf240f5 to df4f1f8 Compare August 7, 2026 01:30

@Yeachan-Heo Yeachan-Heo left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Terminal red-team (exact head df4f1f8)

Verdict: MERGE_READY / APPROVE

  • Contract: shared topic authority outages must not crash the Telegram daemon; generation bump to 55 with tests.
  • Scope: telegram-daemon + contract + tests + generation manifest.
  • Exact-head CI green; mergeable=clean. Generation pin advances 54→55 for this daemon contract change (supersedes #3972 pin intentionally).
  • Independent of ACP #3950 and perf #3764.

Merge authorized.

@Yeachan-Heo
Yeachan-Heo merged commit 40b02f6 into Yeachan-Heo:dev Aug 7, 2026
30 checks passed
Yeachan-Heo pushed a commit that referenced this pull request Aug 7, 2026
Keep #3974 shared-topic-authority gen 55 notes and claim gen 56 for the
#3846 lazy native authority cut. Update generation contract tests.

Tested: telegram-daemon-generation-guard v43 bump verified
Yeachan-Heo pushed a commit that referenced this pull request Aug 7, 2026
Keep #3974 shared-topic-authority gen 55 notes and claim gen 56 for the
#3846 lazy native authority cut. Update generation contract tests.

Tested: telegram-daemon-generation-guard v43 bump verified
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants