fix(daemon): reconstruct sessions across restarts and raise idle timeout to 120m - #92
Merged
Merged
Conversation
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Jun 12, 2026
rgao-coreweave
force-pushed
the
fix/daemon-session-reconstruction
branch
from
July 1, 2026 16:55
50d5279 to
fb95bfc
Compare
rgao-coreweave
added a commit
that referenced
this pull request
Jul 1, 2026
…107) ## Summary A turn's root `invoke_agent claude-code` span is created at UserPromptSubmit and ended only at Stop or SessionEnd. Every other daemon exit path — inactivity timeout, SIGTERM/SIGINT/SIGHUP, restart/config-change control message — ran `shutdown()`, which flushed the exporter but never ended open turn spans. Their already-ended children (completed tool spans, finalized chat spans, closed subagent spans) had exported; the still-open root had not. Result: rootless traces (tool spans with no user turn, and no agent/conversation identity, both of which live on the root). Local daemon log evidence: 49 shutdowns fired with an open turn span (~88 leaked roots over the log's life; ~44 in the last 7 days), almost all `inactivity`, plus `SIGTERM` and `control message` (restart). Extract the SessionEnd finalization into `finalizeSession()` and call it for every live session inside the shutdown `drain`, before `provider.shutdown()` flushes, so an interrupted turn keeps its exported root. `shutdown()` now delegates to a testable `drain()` (everything except `process.exit`). Complementary to (does not overlap) #93 (holds the daemon open while in-flight, but still drops the root when it does exit), #92 (reconstructs sessions after a restart), #87 (removes the terminal-SIGHUP kill). ## Test plan - `node --import tsx --test tests/daemon-shutdown-finalizes-turn.test.ts` — mid-turn `drain` exports the root turn span (+ open subagent span) under one trace; SessionEnd still exports its root. - `npm run check` (build + full suite): green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
rgao-coreweave
force-pushed
the
fix/daemon-session-reconstruction
branch
2 times, most recently
from
July 1, 2026 17:59
c2096d7 to
1f5b34b
Compare
rgao-coreweave
changed the title
rgao-coreweave
marked this pull request as ready for review
July 1, 2026 18:21
drtangible
approved these changes
Jul 2, 2026
The daemon idles out after ~10 min and holds all session state in memory, seeded only at SessionStart. A Claude Code session that outlives a daemon restart sends its next UserPromptSubmit to a fresh daemon that never saw its SessionStart, which logged "Unknown session" and silently dropped tracing for the rest of that session (159 such errors over 14 days in one local log; 52 distinct sessions observed straddling a restart). Reconstruct the session from the transcript_path every hook event carries, seeding the turn counter from the turns already on disk so numbering continues. Makes the daemon tolerant of its own restarts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The idle timeout only fires when nothing is in flight (INFLIGHT_HOLD_MAX_MS already keeps an active turn/tool/team alive), so it purely governs how long an idle daemon stays warm. At 10m a normal think-time gap reaped the daemon and stranded the resumed session on a fresh one, the dominant "Unknown session" trigger this PR also reconstructs from. 120m keeps the daemon warm across the gaps in a working session (long build, meeting, lunch); longer idle still reaps and is recovered by reconstruction. Env-overridable via WEAVE_INACTIVITY_MS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rgao-coreweave
force-pushed
the
fix/daemon-session-reconstruction
branch
from
July 2, 2026 16:59
1f5b34b to
1651521
Compare
…amed options Address review on #92: pull newSessionState out of the class (it never touched `this`), take a single options object instead of 7 positional args (cwd/source/initialRequestModel were easy to swap), and narrow the CLI version via an extracted const so the `as string` cast drops out. No behavior change; build + full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes that harden the daemon against its own idle-reap, which was stranding sessions and dropping tracing (464 `Unknown session` errors locally; 43 in the last 3 days).
Reconstruct session state on unknown sessions. The daemon keeps all session state in memory, seeded only at SessionStart, and self-reaps when idle. Claude Code only emits SessionStart on startup/resume/clear/compact, so a session that outlives a reap sends its next UserPromptSubmit to a fresh daemon that never saw its SessionStart, logs `Unknown session`, and drops the rest of the session silently. Reconstruct from the `transcript_path` on the event (resolve conversation id via the forkedFrom chain, seed the turn counter from turns on disk), so the daemon tolerates its own restarts.
Raise the idle-reap timeout 10m to 120m. The timeout only fires when nothing is in flight (INFLIGHT_HOLD_MAX_MS already keeps an active turn/tool/team alive, fix(daemon): hold daemon open while a turn or tool is in flight #93), so it purely governs how long an idle daemon stays warm. 10m reaped across normal working-session gaps and caused most of the `Unknown session` drops; 120m keeps the daemon warm across a long build / meeting / lunch, and longer idle gaps are recovered by (1). Env-overridable via `WEAVE_INACTIVITY_MS`. Tradeoff: an idle daemon lingers up to 2h and a stale version runs longer after `plugin update` until restart (surfaced by the config-drift warning; `weave-claude-code restart` clears it).
Reconstruction is anchored at UserPromptSubmit (always the first event of a resumed session, and where the error was logged); a turn interrupted mid-flight by the reap is not recovered by design (its open spans died with the old process). Turn-number seeding rides the transcript parser that #56 tightens.
Test plan
🤖 Generated with Claude Code