Skip to content

fix(daemon): hold daemon open while a turn or tool is in flight - #93

Merged
rgao-coreweave merged 3 commits into
mainfrom
fix/daemon-idle-during-inflight
Jul 1, 2026
Merged

fix(daemon): hold daemon open while a turn or tool is in flight#93
rgao-coreweave merged 3 commits into
mainfrom
fix/daemon-idle-during-inflight

Conversation

@rgao-coreweave

@rgao-coreweave rgao-coreweave commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

checkInactivity only held the daemon open for in-flight cross-session team work. A plain long-running tool or turn (longer than the idle timeout, with no other session active) tripped the timeout mid-flight: the daemon exited, dropped the still-open spans, and the resumed work landed on a fresh daemon.

Extend the hold to any session with an open turn span, a pending tool call, or a tracked subagent, under the same ceiling.

Test plan

  • node --import tsx --test tests/daemon-idle-inflight.test.ts

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@drtangible drtangible left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 👍 👍

rgao-coreweave added a commit that referenced this pull request Jul 1, 2026
…107)

## Summary
A turn's root `invoke_agent claude-code` span is created at UserPromptSubmit and ended only at Stop or SessionEnd. Every other daemon exit path — inactivity timeout, SIGTERM/SIGINT/SIGHUP, restart/config-change control message — ran `shutdown()`, which flushed the exporter but never ended open turn spans. Their already-ended children (completed tool spans, finalized chat spans, closed subagent spans) had exported; the still-open root had not. Result: rootless traces (tool spans with no user turn, and no agent/conversation identity, both of which live on the root).

Local daemon log evidence: 49 shutdowns fired with an open turn span (~88 leaked roots over the log's life; ~44 in the last 7 days), almost all `inactivity`, plus `SIGTERM` and `control message` (restart).

Extract the SessionEnd finalization into `finalizeSession()` and call it for every live session inside the shutdown `drain`, before `provider.shutdown()` flushes, so an interrupted turn keeps its exported root. `shutdown()` now delegates to a testable `drain()` (everything except `process.exit`).

Complementary to (does not overlap) #93 (holds the daemon open while in-flight, but still drops the root when it does exit), #92 (reconstructs sessions after a restart), #87 (removes the terminal-SIGHUP kill).

## Test plan
- `node --import tsx --test tests/daemon-shutdown-finalizes-turn.test.ts` — mid-turn `drain` exports the root turn span (+ open subagent span) under one trace; SessionEnd still exports its root.
- `npm run check` (build + full suite): green.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
rgao-coreweave and others added 3 commits July 1, 2026 10:18
checkInactivity only kept the daemon alive for in-flight cross-session team
work. A plain long-running tool or turn (longer than the inactivity timeout,
with no other session active) tripped the timeout mid-flight: the daemon
exited, dropped the still-open turn/tool spans, and the resumed work landed on
a fresh daemon. Observed locally as reboots within 3s of an inactivity
shutdown.

Extend the hold to any session with an open turn span, a pending tool call, or
a tracked subagent, under the same TEAM_INFLIGHT_MAX_MS ceiling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This PR broadens the ceiling to bound ordinary in-flight work (open turn /
pending tool / tracked subagent) via hasInFlightWork(), not just cross-session
team correlation. The TEAM_ prefix no longer describes its scope, so rename it
and generalize the defining comment to cover both holds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The daemon integration harness (startTestDaemon / waitUntil / TestDaemon) that
this PR's idle-inflight test depends on was introduced by the session-recon
branch. Now that this PR targets main directly rather than stacking on it, carry
the harness here so the test stands alone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rgao-coreweave
rgao-coreweave force-pushed the fix/daemon-idle-during-inflight branch from f576eda to be99a66 Compare July 1, 2026 17:20
@rgao-coreweave
rgao-coreweave changed the base branch from fix/daemon-session-reconstruction to main July 1, 2026 17:20
@rgao-coreweave
rgao-coreweave merged commit 0ce7b3a into main Jul 1, 2026
4 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants