fix(turn): keep eBPF-sourced turns whose opening call was missed#168
Conversation
A Claude Code agent turn is N calls in a tool loop (tool_use × (N-1) → end_turn). Claude Code is a stateless client: every request carries the full conversation history, INCLUDING the current roundtrip's tool_result. Consequence for the turn's FIRST call (whose messages[0] is the user's opening text): the messages array ends on a user:tool_result, so the opening user text is buried at messages[0]. is_user_turn_start today inspects 'the last non-system message' → a tool_result → Some(false). So NO call in the turn reports user_turn_start=true; the tracker's has_user_start guard fails (emit_or_discard) and the ENTIRE turn is dropped as TurnDiscardedNoUserStart — every call stays at the LlmCalls level with turn_id=None. Confirmed on production: one Claude Code process (PID-attributed via eBPF) issued 8 same-session calls (tool_use×7 → end_turn), all turn_id=None. Three regression tests capture the broken behavior so a fix has a red signal. The CC tool-loop first-call case asserts Some(false) today (FIXME marker; flip to Some(true) when fixed). Two control cases (plain-text opening = true; pure tool_result continuation = false) lock in the behaviors that must not regress.
When an eBPF-captured Claude Code agent turn's OPENING call is missed by the source (connection-setup / uprobe-attach timing — observed in production: one PID issued 8 same-session calls tool_use×7 → end_turn, all turn_id=None), every captured call is a continuation. Claude Code is stateless, so each request carries full messages history plus the current roundtrip's tool_result; is_user_turn_start correctly reports Some(false) for all of them (the opening call — the only one whose messages[-1] is user text — isn't among them). The partition then has no main-agent user-turn-start and emit_or_discard dropped the ENTIRE turn as TurnDiscardedNoUserStart. The calls do carry a shared eBPF ProcessInfo (pid/comm/exe), which is proof they are one process's contiguous agent traffic. Add a partition_has_pid_attribution fallback to has_user_start: when no main-agent user-start survived but the partition's main-agent calls share process attribution, keep the partition (TurnKeptByPidAttribution) instead of discarding. The opening call is gone, but the rest of the turn is real and would otherwise be lost entirely. Safety: - Turn boundaries unchanged — still driven by is_turn_terminal (end_turn etc.); PID never participates in boundary partitioning. - Sub-agent guard preserved — fallback only considers subagent_name.is_none() calls, so sub-agent-only partitions are still dropped even if they carry a PID. - Inert for passive taps — non-eBPF calls carry process=None, so the fallback never fires; existing passive-tap behavior is unchanged. - is_user_turn_start semantics untouched — no profile / wire-api change. Applied symmetrically in refresh_active_snapshot so the in-progress registry view matches the persisted Completed (metric bumped only at finalize, not on every snapshot refresh, to avoid double-counting).
There was a problem hiding this comment.
I have enough to write a thorough review. Let me verify one key thing: whether the old extract_user_input reads messages.last() (which would mean a tool_result continuation yields user_input=None), confirming the phantom-row concern for kept-by-PID turns.
Looking at extract_user_input at line 624: it reads messages.last(). For a continuation call whose messages[-1] is user with tool_result blocks only, the filter for type == "text" yields an empty vec, joins to "", trims to empty, returns None. So user_input is None for every continuation call. That means a kept-by-PID partition will produce a turn with user_input_preview = None and user_call_id = None — exactly the "phantom Incomplete with user_input=None" shape that the sub-agent guard was added to prevent. The difference is this is now intentional for eBPF-missed turns. The UI already handles null gracefully ("—"). This is acceptable but worth flagging as a known consequence.
Now I have everything I need to write the review.
Summary
PR #168 fixes a real production drop: when the eBPF source misses an agent turn's opening call (the only call whose messages[-1] is user text), every captured call is a continuation with is_user_turn_start=false, and emit_or_discard was discarding the entire partition as TurnDiscardedNoUserStart. The fix adds a partition_has_pid_attribution fallback: if no main-agent user-start survived but the partition's calls carry eBPF process attribution, keep the turn (new TurnKeptByPidAttribution counter). The logic is sound, the sub-agent guard is correctly preserved, the fallback is inert for passive taps (no process), and it's applied symmetrically in refresh_active_snapshot. The test pair (kept-with-PID, discarded-without-PID) pins both branches. Recommendation: APPROVE with one non-blocking concern about user_input_preview shape and one verification question about PID sharedness.
Blocking
None.
Suggestions
- server/h-turn/src/tracker.rs:688 —
partition_has_pid_attributionchecks onlycall.process.is_some(), not that the PID is shared across the partition's calls. The commit message and theinternal_metrics.rs:217comment both describe the rationale as "a shared eBPF ProcessInfo (pid/comm/exe), which is proof they are one process's contiguous agent traffic," but the implementation accepts a partition where a single call has a PID and the rest have none. In practice all eBPF calls for a session usually carry the same PID, so this is unlikely to misfire — but the code is looser than the documented invariant. Either tighten to "≥2 calls share the same pid" or soften the comments to "any main-agent call carries process attribution." Low-impact; flagging because the doc and the code disagree. - server/h-turn/src/tracker.rs:853 — A kept-by-PID partition will produce a
Completedturn withuser_input_preview = Noneanduser_call_id = None.extract_user_input(anthropic.rs:624) readsmessages.last(), which for every continuation call is atool_result-only user message → filtered to empty →None. Theuser_pickfallback at line 855 then finds no main-agent call withuser_input.is_some(), so both fields stayNone. This is the same "phantom row with user_input=None" shape the sub-agent guard was added to suppress (see comment at line 724), now deliberately emitted for eBPF-missed turns. The UI handlesnullgracefully (agent-turns.tsx:111renders "—"), so this is acceptable — but worth a one-line comment inbuild_turnnoting that kept-by-PID turns intentionally haveuser_input_preview=None, so a future reader doesn't reintroduce a guard that re-breaks this path. - server/h-turn/src/tracker.rs:730 — Minor:
refsis now collected unconditionally before the discard check, whereas before it was only built after thehas_user_startshort-circuit passed. For the discard path this does one extra smallVecallocation per finalized partition. Negligible, but if the hot path matters thehas_user_startcheck could scancalls.iter()directly first and buildrefsonly when emitting. Not worth changing unless this is a measured hotspot.
Questions
- server/h-turn/src/tracker.rs:694 —
partition_has_pid_attributionaccepts anycall.process.is_some(), but the commit subject says "shared eBPF ProcessInfo." Is the intent to keep any PID-attributed partition, or only partitions where ≥2 calls share a PID? If the latter, the test at line 1387 (two PID'd calls) doesn't cover the "single PID'd call + others without" edge case, and the predicate should comparepidvalues.
Verified
- Schema mirror:
TurnKeptByPidAttributionis a backend-only internal metric (counter inh-common/src/internal_metrics.rs:218), not a persisted column or a serde field onAgentTurn— noconsole/src/types/api.tsmirror needed. Confirmed noServiceRow/AgentTurnfield changed. - Metric registration: added to the
register_workerlist in all four sites that enumerate Turn metrics —stage.rs:73,test_support.rs:30,tracker.rs:1010(test_metrics),tests/integration.rs:884,tests/reorder.rs:37. No site that bumps the counter is missing the registration. - Symmetry between
refresh_active_snapshotandemit_or_discard: both usemain_agent_user_turn_start+partition_has_pid_attributionwith the same!has_user_start &&guard (tracker.rs:308 and :743). Metric is correctly bumped only inemit_or_discard, not in the refresh path — avoids double-counting, as the comment at :305 notes. - Sub-agent guard preserved: both new helpers (
main_agent_user_turn_start,partition_has_pid_attribution) filter onsubagent_name.is_none(), so a sub-agent-only partition with a PID is still discarded. Existingsubagent_only_partition_with_user_turn_start_is_discardedtest at :1318 is unchanged. - Passive-tap inertness:
LlmCall.processisOption<ProcessInfo>(model.rs:83); passive-tap calls set it toNone, sopartition_has_pid_attributionreturnsfalsefor them — theno_pid_no_user_start_still_discardedtest at :1427 pins this. is_user_turn_startsemantics untouched: the anthropic.rs diff is comment + test-fixture only (renamedcc_turn_first_call_request→cc_continuation_call_request, replaced the Chinese-text fixture with an English one, and flipped the FIXME test to pinSome(false)). No change to the function body at :653. Confirmed the three tests now assert the documented behavior.- No sensitive-info leakage: the original commit's test fixture contained Chinese text (
"修改9000上的litellm,把对外GLM5.1转给GLM5.2") that looked like a real operator instruction; the second commit replaced it with generic English ("update the gateway config to route model A to model B"). No IPs, hostnames, credentials, or machine paths in the diff.
🤖 Reviewed by the review bot • workflow run
There was a problem hiding this comment.
No sensitive-info leakage. Note the first commit's test body originally contained "修改9000上的litellm,把对外GLM5.1转给GLM5.2" (referencing port 9000 and internal service names "litellm/GLM") but the second commit replaced it with a generic "update the gateway config to route model A to model B" — so the final state is clean. Good.
Now I have everything I need. Let me verify the one remaining concern: whether partition_has_pid_attribution being called with an empty iterator could ever matter (defensive). Looking at emit_or_discard, calls.is_empty() returns early at line 721, so the predicate only runs on non-empty partitions. In refresh_active_snapshot, partition.is_empty() returns early at line 295-297. Safe.
One last thing — let me confirm the refs Vec allocation in emit_or_discard isn't a regression: before, has_user_start iterated calls.iter() directly with a closure; now it materializes refs: Vec<&AgentCall> first (line 730) then reuses it for build_turn. This is fine — the refs Vec was already allocated just below before the change; the diff just moves it up. No perf concern.
Summary
PR #168 adds a partition_has_pid_attribution fallback to the turn-tracker's discard rule: when an eBPF-captured agent turn's opening call was missed (so no main-agent is_user_turn_start=true survives), but the partition's main-agent calls carry a process attribution, the turn is kept (TurnKeptByPidAttribution) instead of dropped. The fix is scoped correctly — turn boundaries are still driven by is_turn_terminal, the sub-agent guard is preserved, and the fallback is inert for passive taps (which carry process=None). Symmetric application in refresh_active_snapshot keeps the in-progress registry view consistent with the persisted turn. Tests are thorough and self-verifying. The approach is sound and the safety argument holds. Recommendation: APPROVE with one non-blocking observation on intent-vs-implementation wording.
Suggestions
- server/h-turn/src/tracker.rs:688–695 —
partition_has_pid_attributionis documented (and the commit message describes) as detecting a shared/common PID across the partition's calls ("a shared PID still proves these are one process's contiguous agent traffic", "carry a common eBPF ProcessInfo"). The implementation only checksany(|ic| ... process.is_some())— it never compares PIDs across calls. A single call with any PID passes; two calls with different PIDs also pass. In practice eBPF attribution is per-flow (stamped once from the first attributed packet intcp.rs:166), so a(source_id, session_id)partition's calls will almost always share one PID anyway — so this is unlikely to mis-fire. But the doc/comment claims a stronger invariant than the code enforces. Either tighten the code to actually verify PIDs match (e.g.calls.map(...).collect::<BTreeSet<_>>().len() == 1), or relax the doc to say "any main-agent call carries eBPF attribution (which is per-flow, hence shared)".
Questions
- server/h-turn/src/tracker.rs:744–750 — A partition kept purely by PID attribution will emit a turn with
user_input_preview = Noneanduser_call_id = None(no user-start call exists;tool_resultcontinuation bodies yield nouser_inputinbuild_turn'suser_pickfallback at line 855). This is the same "phantom row with no user_input" shape thatsubagent_only_partition_with_user_turn_start_is_discardedguards against for sub-agents — here it's deliberately allowed through for main-agent PID-attributed calls. Is the intent that the console should display these turns (likely with a blank user-input column), or should they carry a flag inmetadata(e.g.{"kept_by_pid_attribution": true}) so the UI / ops can distinguish "real turn with missing opening" from "turn whose user input we captured"? The metric exists for ops observability, but the persisted row has no marker.
Verified
- Metric registration consistency:
TurnKeptByPidAttributionis added todefine_metrics!(internal_metrics.rs:218), and to everyregister_workercall site that lists its siblingsTurnDiscardedNoUserStart— stage.rs:73 (production turn workers), test_support.rs:30, tracker.rs:1010 (test_metrics), integration.rs:884, reorder.rs:37. No site that registers the sibling misses the new one. - "Passive taps carry no process" safety claim: traced
processpopulation — only set via eBPF inh-capture/src/ebpf/mod.rs:248andh-protocol/src/tcp.rs:166(which stamps frompkt.process, itself eBPF-only). All passive-tap / test construction sites setprocess: None. The fallback genuinely cannot fire for passive captures. - Sub-agent guard preserved:
partition_has_pid_attributionfiltersic.agent.subagent_name.is_none(), so a sub-agent-only partition with PIDs still fails the predicate and is discarded.main_agent_user_turn_startapplies the same guard. - Symmetry between
emit_or_discardandrefresh_active_snapshot: both now computehas_user_start || kept_by_pidwith identical predicates; the metric is bumped only inemit_or_discard(correctly noted in the inline comment to avoid double-counting on the per-ingest snapshot refresh path). - No body-scan SQL smell: no DuckDB/ClickHouse query changes in this PR.
- No schema/TS drift: no Rust serde type fields changed; the new metric is internal-only.
- No sensitive-info leakage: scanned the full diff (
bd463e3...261d6f5) for RFC1918/CGNAT IPs, private-key blocks, passwords, and machine paths — none. The first commit's Chinese test string referencing internal service names was replaced by a generic English string in the second commit. is_user_turn_startsemantics untouched: read the function at anthropic.rs:653 — the PR adds only tests around it, no logic change to the verdict itself. The pinnedSome(false)for the CC continuation shape is correct given the function inspects the last non-system message.
🤖 Reviewed by the review bot • workflow run
Problem
An eBPF-captured Claude Code agent turn whose opening call is missed by the source (connection-setup / uprobe-attach timing) is dropped entirely — every call in the turn ends up at the LlmCalls level with
turn_id=None, never assembled into anAgentTurn.Root cause
Claude Code is a stateless client: every request carries the full
messageshistory plus the current roundtrip'stool_result. So within one turn:messages[-1]is user text) is the only call that reportsis_user_turn_start = Some(true).messages[-1] = user:tool_result, sois_user_turn_startcorrectly returnsSome(false).When the eBPF source misses that opening call, no captured call reports a user-turn-start. The tracker's
has_user_startguard inemit_or_discardthen fails, and the entire partition is dropped asTurnDiscardedNoUserStart— including the real continuation calls and theend_turnterminator.is_user_turn_startis not buggy here; the verdict is right, the opening call just isn't among the captured calls.Fix
The captured continuation calls do carry a shared eBPF
ProcessInfo(pid/comm/exe) — proof they are one process's contiguous agent traffic. Add apartition_has_pid_attributionfallback tohas_user_start:TurnKeptByPidAttribution) instead of discarding it.Applied symmetrically in
refresh_active_snapshotso the in-progress registry view matches the persistedCompleted(the metric is bumped only at finalize, not on every snapshot refresh, to avoid double-counting).Safety
is_turn_terminal(end_turnetc.); PID never participates in partitioning.subagent_name.is_none()calls, so sub-agent-only partitions are still dropped even if they happen to carry a PID.process = None, so the fallback never fires; existing passive-tap behavior is unchanged.is_user_turn_startsemantics untouched — no profile / wire-api change.Known limitation
A turn kept via this fallback has
user_input_preview = None(the opening call that would supply the user text was missed).final_answer_preview,call_ids, and token statistics are unaffected (they derive from the terminal call, not the user-start call). Extracting user text frommessages[0]is a profile-layer follow-up, out of scope here.Commits
6d42328— reproduction tests (continuation-shape verdict + control cases)c92be76— the fallback fix + h-turn unit tests (PID fallback keeps turn; no-PID control still discards)Verification
cargo test --workspacegreen (incl.--features fault-injectionconcurrent tests).pid_attribution_keeps_turn_when_opening_call_missed(fallback fires, turn kept,TurnKeptByPidAttributionbumped) andno_pid_no_user_start_still_discarded(control: passive-tap partition still discarded).