Skip to content

fix(connector-claude-code): reconcile wakes lost before channel activation#226

Open
caffeinum wants to merge 1 commit into
Cotal-AI:mainfrom
caffeinum:fix/pending-wake-boot-race
Open

fix(connector-claude-code): reconcile wakes lost before channel activation#226
caffeinum wants to merge 1 commit into
Cotal-AI:mainfrom
caffeinum:fix/pending-wake-boot-race

Conversation

@caffeinum

Copy link
Copy Markdown
Contributor

Symptom

An agent spawned with a DM already pending in its JetStream durable consumer becomes a permanent zombie: presence is fine, the consumer pulls fine, but it never takes a turn. Observed live on a real space (spawn-on-dm, 2026-07-12):

  • 12:43:53 session spawned; NATS connect + durable consumer dm_<id> bound at :55.825
  • the pending DM was delivered within ms → ingest() buffered it and emitted its one "incoming"nudge()
  • MCP transport connected at :55.868 (channelActive flips there), but the client registered its notifications/claude/channel handler only at :57.245 ("Channel notifications registered") — the nudge landed inside that ~1.4s window and was silently discarded
  • every 60s JetStream redelivery hit the pending-duplicate branch in ingest(), which only refreshes the ack handle and returns without re-emitting — so no re-nudge, ever
  • net: the message sat buffered and unacked forever while the session idled

Confirmed by sending a second DM: it took the new-item path, re-nudged (channel live by then), and that one turn drained both messages.

Mechanism (at upstream/main)

  • extensions/connector-claude-code/src/mcp.ts:212-214nudge() is gated on the mutable channelActive, false until post-handshake
  • mcp.ts:133agent.start() kicks off the mesh connect in the background, so a pending message can arrive and emit "incoming" (mcp.ts:235-240) before await server.connect(transport) (mcp.ts:252) flips the flag (mcp.ts:257-261)
  • even after the flip there is a second loss window: the client declares the claude/channel capability at the handshake but registers its notification handler asynchronously (~1.4s later in the observed trace), and a notification has no ack — a push in that window vanishes
  • extensions/connector-core/src/agent.ts:264-268 — a redelivered duplicate of a still-pending item only refreshes the ack handle (correctly, to keep one entry and never downgrade a durable ack) and returns without emitting, so redelivery can never heal a lost wake
  • the comment at mcp.ts:204-211 assumed a buffered message "is drained at the next UserPromptSubmit, so nothing is lost" — a freshly spawned agent that never gets a human turn has no next UserPromptSubmit

Fix

Two parts, both re-deriving the wake from state instead of trusting any single notification to land:

  1. Post-activation reconcile — immediately after channelActive flips, if pendingWake() > 0, fire the nudge once. Covers everything that emitted "incoming" while the gate was closed.
  2. Idle reconciler — a 30s setInterval (unref'd): if the channel is active, the session is idle, and pendingWake() > 0, re-nudge. This is the self-healing net for the whole lost-wake class (handler-not-yet-registered window, a dropped notification, any future race), not just this boot ordering.

pendingWake() is the same mode-and-channel-aware predicate the Stop→idle flush already uses, so held quiet/dnd ambient still never wakes a turn. No storm risk: a delivered wake drains the inbox on its UserPromptSubmit, which zeroes the predicate; the gate on status === "idle" keeps it silent mid-turn and while blocked on a permission. I considered also re-emitting from ingest()'s duplicate branch, but the reconciler subsumes it without touching the shared dedup logic, so I left that branch alone.

Verification

  • root cause and the dual-drain confirmation observed on a live mesh (timeline above) before writing the fix; the patched build has not been soaked against a live mesh
  • pnpm typecheck clean across the workspace; pnpm test passes; pnpm build of the connector bundles the new paths into dist/mcp.cjs
  • there is no existing harness that drives mcp.ts's main() with a real MCP client, so the reconcile paths have no hermetic test; happy to add one if you can point me at a preferred harness shape

…ation

A session spawned with a message already pending in its durable consumer
became permanently deaf: the DM arrived (and emitted its one "incoming")
milliseconds after the consumer bound, before the MCP handshake flipped
channelActive — so the nudge no-op'd — and even once flipped, the client
registers its notifications/claude/channel handler asynchronously
(observed ~1.4s later), so a push in that window is discarded client-side.
JetStream redelivery never re-emits: ingest's pending-duplicate branch only
refreshes the ack handle. Net: the message sat buffered and unacked forever
while the session idled, until an unrelated message re-fired the wake.

Two-part fix, both derived from state rather than trusting any single
notification to land:

- post-activation reconcile: fire the nudge once right after channelActive
  flips if wake-pending messages are already buffered;
- idle reconciler: every 30s, if the session is idle and pendingWake() > 0,
  re-nudge — healing every lost-wake class, not just this boot race. Uses
  the same mode-and-channel-aware predicate as the Stop→idle flush (held
  quiet/dnd ambient never wakes), and a delivered wake drains the inbox on
  its UserPromptSubmit, so re-nudges stop themselves.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant