Skip to content

Fix confirmed-delivery wake loss across session restart - #950

Merged
olegbrok merged 1 commit into
mainfrom
agent/recover-undelivered-wakes
Jul 31, 2026
Merged

Fix confirmed-delivery wake loss across session restart#950
olegbrok merged 1 commit into
mainfrom
agent/recover-undelivered-wakes

Conversation

@bradbrok

@bradbrok bradbrok commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Closes #949.

What changed

Why

A long healthy turn could outlive the 600s scheduler receipt window. The resulting negative receipt was requeued into the same transport deque, which voluntary context_restart then canceled and discarded. Journald recorded the loss but no owner-facing channel surfaced it.

Behavior and boundaries

  • Replay is lifecycle-gated: another cron fire in the old session cannot drain the persisted backlog.
  • Catch-up is sequential behind orientation; older persisted wakes precede a simultaneously ready live cohort.
  • Transient failure of the oldest row halts replay and leaves the whole FIFO for the next boundary; terminal zombie rows retire loudly and allow later live rows to advance.
  • Auto-disabled one-shots remain replayable and retire atomically after one exact positive receipt.
  • The callback-start synchronization is bounded and receipt-independent; an older per-agent delivery lock cannot make the scheduler tick wait indefinitely.
  • Broader Scheduler fires are lost when a daemon restart lands near the fire: no atomic last_run persist, no startup catch-up (3 casualties on 06-11) #747 cron-window catch-up is intentionally out of scope.
  • No deployment is performed from this lane. No live daemon SQLite was opened; all database checks use fixture files.

Verification

Comment thread src/pinky_daemon/scheduler.py Fixed
@olegbrok

Copy link
Copy Markdown
Collaborator

Verify-lane criterion for the replacement packet (recording here since it's the surface both lanes re-read; also sent to Kuzya directly):

"Stop replay at first oldest failure" must distinguish two cases or the outbox gains a new single point of loss:

  • TRANSIENT failure of the oldest row (delivery attempt failed, session busy/dying): halt FIFO, retry at the next lifecycle boundary. No overtake — per Murzik's strict-FIFO finding.
  • TERMINAL verdict on the oldest row (schedule deleted/operator-disabled → PERSISTED_WAKE_ZOMBIE_DROPPED): retire the row loudly and let FIFO advance. A poisoned head row must not wedge every wake queued behind it.

Required regression if not already present: zombie at head of queue + live pending rows behind it → zombie dropped with the literal, live rows deliver in order.

Also concur with treating the post-freeze CodeQL inline finding as unwaived — adjudicate it on the replacement SHA where it re-runs.

🤖 Opened by Barsik

@olegbrok
olegbrok force-pushed the agent/recover-undelivered-wakes branch from c1fbbc7 to d473dbf Compare July 31, 2026 14:45
@olegbrok
olegbrok force-pushed the agent/recover-undelivered-wakes branch from d473dbf to f8a67d5 Compare July 31, 2026 15:06
@olegbrok
olegbrok marked this pull request as ready for review July 31, 2026 15:54
@olegbrok
olegbrok merged commit 16b7789 into main Jul 31, 2026
15 of 16 checks passed
@olegbrok
olegbrok deleted the agent/recover-undelivered-wakes branch July 31, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Confirmed-delivery wakes die behind long busy turns; UNDELIVERED requeues don't survive session restart (detected, not recovered, not alerted)

3 participants