Skip to content

fix(telegram): yield orphan pollers on sustained getUpdates 409 - #3891

Closed
innocarpe wants to merge 1 commit into
Yeachan-Heo:devfrom
innocarpe:fix/issue-3587-telegram-orphan-poller
Closed

fix(telegram): yield orphan pollers on sustained getUpdates 409#3891
innocarpe wants to merge 1 commit into
Yeachan-Heo:devfrom
innocarpe:fix/issue-3587-telegram-orphan-poller

Conversation

@innocarpe

Copy link
Copy Markdown
Contributor

Summary

Bounded secondary fence for #3587 (not full closure): after a streak of consecutive Telegram getUpdates 409 conflicts, the poller emits conflict_yield.

  • A process that no longer holds durable ownership exits through the normal shutdown / ownership-release path (no peer signal/kill).
  • A still-owning daemon keeps serving (resets the streak) so thrash from an orphan cannot push the legitimate owner out.
  • Ownership reclaim / PID-reuse provenance (ownershipLockIsReclaimable) is intentionally unchanged — the acquisition-time ambiguity that can spawn a second daemon beside a live original remains open under the owner’s “not implementation-ready” disposition.

Refs #3587 (use Fixes only when the full provenance/spawn-fence contract lands).

Why this slice

Owner admission accepts dual-poller/409/stale-button evidence and allows a bounded sustained-409 self-defense as a secondary fence, not as a substitute for ownership correctness. Full spawn-fence/provenance work is out of scope here.

Changes

  • TelegramUpdatePoller: consecutive 409 counter → conflict_yield after POLL_CONFLICT_YIELD_AFTER (8); success / non-409 resets streak; resetConflictStreak() for owner keep-serving.
  • Daemon run() loop: on conflict_yield, renew ownership sidecar — keep serving if still owner, else break into normal shutdown.
  • DAEMON_GENERATION 51 → 52 + generation manifest attestation.
  • Unit tests (mocked Bot API; no live Telegram credentials).

Test plan

  • bun test packages/coding-agent/test/notifications-telegram-daemon.test.ts -t "409|yield|sustained|resetConflict|generation 52"
  • Broader telegram suites: notifications-telegram-daemon*.test.ts, topic-registry, lifecycle ownership, baseline, CAS — 645 pass
  • bun --cwd=packages/coding-agent run check (biome + tsc)
  • bun scripts/telegram-daemon-generation-guard.ts --validate-current-tree
  • bun test scripts/telegram-daemon-generation-guard.test.ts48 pass
  • Live multi-session dual-poller on a real bot token (not required for this mockable slice)

Risks

  • Both thrashing processes that still believe they own (pathological dual-owner state) could keep serving; that is already an ownership bug outside this fence.
  • Owner that is thrashing against a long-lived orphan will log yield + continue; orphan should exit once it fails ownership renew or hits yield without ownership. Residual dual-poller time is bounded by heartbeat interval + yield threshold, not “all day” backoff-only thrash.
  • Does not prevent the second spawn on live-PID + incarnation-mismatch reclaim; that remains the open provenance/spawn-fence work for a later owner PR.

Credits

Field evidence and contract analysis: pinion05 (#3587, predecessor #3584).

@innocarpe

Copy link
Copy Markdown
Contributor Author

Label request (fork PR lacks label write permission): please apply bug, area:ux, effort:S.

@innocarpe
innocarpe force-pushed the fix/issue-3587-telegram-orphan-poller branch from 928d315 to 58bfccd Compare August 6, 2026 02:38
@innocarpe

Copy link
Copy Markdown
Contributor Author

Rebased onto latest dev. Real conflicts in daemon generation pins: claimed DAEMON_GENERATION 54 (dev already had 52/53). Manifest digests regenerated.

@innocarpe
innocarpe force-pushed the fix/issue-3587-telegram-orphan-poller branch from 58bfccd to b245ac5 Compare August 6, 2026 03:39
@innocarpe

Copy link
Copy Markdown
Contributor Author

Rebased onto latest dev (clean replay).

@yazzang-homelab yazzang-homelab left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent architect review.

The design question that decides this PR is whether conflict_yield can evict a legitimate owner. It cannot, and the reason is load-bearing enough to write down:

if (pollResult.kind === "conflict_yield") {
  if (await this.renewOwnershipHeartbeat()) { this.poller.resetConflictStreak(); this.loopBackoff.reset(); continue; }
  break;
}

renewOwnerHeartbeatSidecar returns false only for definite non-ownership — missing state file, invalid pid, incarnation mismatch, ownerId/acquisitionId mismatch, stoppedAt set, or a lock that does not match state. Every indeterminate case throws instead: readJson (telegram-daemon.ts:641) returns undefined only on ENOENT and rethrows every other error, so an EIO/EACCES/EBUSY hiccup or malformed JSON propagates out of renewOwnershipHeartbeat, lands in the loop's existing catch, and the daemon backs off and keeps serving.

That distinction is exactly what #3844 and #3910 were about — a momentarily unavailable authority must not be read as "I am not the owner". Here it isn't. Had readJson swallowed errors into undefined, this same code would have been a self-eviction bug under disk pressure. Worth a sentence in the code comment, since the safety of break depends entirely on a property of a function two files away.

The rest holds up:

  • Self-defense only: no kill, no signal, no mutation of the peer's state. A thrashing orphan cannot displace the owner; it can only remove itself.
  • The still-owner branch resets both the conflict streak and the loop backoff, so the steady state is "re-verify ownership every N conflicts" rather than a one-shot decision. Correct for a race that may resolve on its own.
  • Exit goes through break, so the outer finally runs the normal shutdown — heartbeat timer stop, joinExclusive, ownership release. It does not process.exit out from under the release path.
  • No hot loop: the 409 backoff lives inside the poller (POLL_BACKOFF_MS), so skipping the loop-tail sleep(10) on this path costs nothing.
  • DAEMON_GENERATION 53 → 54 with the manifest regenerated, so the protected-declaration guard is satisfied.
  • CHANGELOG entry is under ## [Unreleased] (line 13, section opens line 3) — correct, and worth noting since several open PRs currently have entries misfiled into released sections.

Two things to address, neither blocking the logic:

  1. continue skips the control-stop check. The loop tail is if (await this.controlStopRequested()) break; await this.runtime.sleep(10);continue jumps over both. A /stop issued during a sustained-409 streak is deferred by one full poll cycle. Bounded and self-correcting, but since this path can now repeat indefinitely while ownership holds, the deferral can repeat too. Checking controlStopRequested() before continue, or restructuring so the yield path falls through to the tail, removes it.

  2. DAEMON_GENERATION = 54 collides with #3844. Both open PRs bump 53 → 54. Whichever merges first takes 54 and the other fails telegram-daemon-generation-guard with protected Telegram lifecycle change requires a strictly higher DAEMON_GENERATION — the same collision @Yeachan-Heo caught on #3844 against #3834 at 50. Nothing to change here unilaterally; it needs merge sequencing, and the second one through rebases and bumps to 55. #3844 is mine, so if this lands first I will take 55.

gajae.pr-review-verdict.v1 merge-approved sha256:b245ac5981611c02018966d72ae5ae3508c03a70 reviewer:architect evidence:read of telegram-daemon.ts:641-648,1327-1376,9655-9682,12343-12375 at this head

Dual live pollers thrash forever on Telegram's single getUpdates slot,
flooding sessions with stale-button notices. After a bounded consecutive
409 streak the poller emits conflict_yield; a non-owner exits through
normal shutdown/ownership release, while a still-owning daemon keeps
serving. Never signals or kills a peer PID. Ownership reclaim /
PID-reuse provenance is intentionally unchanged.

Rebased onto latest dev: DAEMON_GENERATION is now 54 (dev already claimed
52 for Yeachan-Heo#3761 diagnostics and 53 for ask multi-select).

Refs Yeachan-Heo#3587

Lore-id: 3587b409
Constraint: must not signal/kill based on PID+incarnation ambiguity alone
Constraint: still-owning shared daemon must not be retired by thrash
Rejected: reclaim-path spawn fence | provenance contract not admitted yet
Rejected: unconditional yield of durable owner | orphans could push owner out
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: unit mocks for yield/reset/non-409; generation pin 54; coding-agent guard validate
Not-tested: live multi-session Telegram dual-poller on real bot token
@innocarpe
innocarpe force-pushed the fix/issue-3587-telegram-orphan-poller branch from b245ac5 to 2522ced Compare August 6, 2026 11:27
@innocarpe

Copy link
Copy Markdown
Contributor Author

Rebased onto latest dev. CHANGELOG: kept both #3859 and #3587 unreleased bullets.

@Yeachan-Heo

Copy link
Copy Markdown
Owner

Closing during the emergency maintenance freeze. This PR is not in the retained critical or maintainer-owned set. Do not open a replacement PR unless a maintainer explicitly directs it.


[repo owner's gaebal-gajae (clawdbot) 🦞]

@Yeachan-Heo Yeachan-Heo closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants