proxy-pair grouping misses turns split across profiles by a header-stripping proxy leg

[vaderyang]

Summary

A single logical agent turn observed across two network vantage points — where one leg is captured before a forwarding proxy and the other after it — is split into two independent AgentTurns that never get folded by the proxy_pair grouping mechanism, even though that mechanism exists precisely for this case.

Reproduction (production)

One logical opencode session (OpenAI Chat wire, GLM model) was captured as two turns with identical content but different agent_kind / session_id:

leg	profile	session_id source	session_id	agent_kind
proxy ingress (client → proxy)	opencode	x-session-affinity header	ses_…	opencode
proxy egress (proxy → upstream)	generic	tool-id synthesis (header stripped)	call_…	generic

Both turns carry ~the same calls (verified: 26 of 26 stable message-content fingerprints match across the two turns) and overlap in time, but the proxy egress leg lost the opencode session header — the forwarding proxy (LiteLLM, using the AsyncOpenAI Python SDK) rebuilds the request and drops x-session-affinity and replaces the User-Agent. With no opencode header/UA, the egress leg falls through to GenericProfile, which synthesizes a completely unrelated session_id from the tool-call anchor.

A secondary effect: each leg is itself captured 2× (ingress + loopback/docker bridge mirror), inflating call_count on both turns (and by different amounts — 51 vs 49 — since the mirror counts diverge).

Root cause

server/h-turn/src/proxy_pair.rs::content_fingerprint:

fn content_fingerprint(c: &PairCandidate) -> (&str, &str, u32, u64, u64) {
    (c.session_id, c.agent_kind, c.call_count, c.total_input_tokens, c.total_output_tokens)
}

group_all buckets candidates by this fingerprint before time-clustering. The doc comment (proxy_pair.rs:199–200) states the design assumption:

session_id is the strongest signal — agent profiles already content-hash on first user message, so cross-proxy legs land in the same session by construction.

That assumption holds when both legs hit the same profile (both keep the header, or both lose it and both synthesize identically). It breaks when one leg keeps the header (→ opencode session) and the other loses it (→ generic synthesized session): the two session_ids are unrelated strings, so the legs land in different fingerprint buckets and group_all never compares them.

Every other fingerprint field is also unstable in this case:

• agent_kind differs (opencode vs generic) — same root cause.
• call_count / tokens differ — mirror double-capture inflates each leg by a different amount.

Why the fix isn't trivial

The one signal that is stable across profiles and immune to double-capture is the request body content of the turn's opening call. Verified on the reproduction: both turns' first user-call body hashes to the identical d41dd507…. But:

• user_input_preview (the existing content column on agent_turns) is NULL on both turns here — the opening call's extract_user_input returns None for the tool-result-tail continuation shape (same class of issue as the eBPF opening-call-missed case in fix(turn): keep eBPF-sourced turns whose opening call was missed #168), so the existing column can't carry it.
• PairCandidate is a narrow DB projection with no body / body-hash field today.

So a fix needs a new stable content signal persisted on the turn (e.g. a first_call_content_hash column populated at build_turn time), projected into PairCandidate, and used in content_fingerprint in place of session_id/agent_kind. That touches schema (migration) + tracker + storage projection across all three backends — non-trivial, hence filing before implementing.

Open questions (before implementing)

1. Frequency. This is one observed case (a LiteLLM-fronted opencode deployment). Is the header-stripping proxy leg common enough across deployments to justify a schema migration, or should a lighter-weight fix be preferred (e.g. body re-read at pair-query time, no new column)?
2. Double-capture scope. The 2× mirror inflation per leg is a separate capture-layer concern. This issue is about the cross-turn split only; the mirror inflation should likely be its own issue if it warrants work.

Related

• fix(turn): keep eBPF-sourced turns whose opening call was missed #168 — eBPF opening-call-missed turn drop (different mechanism: eBPF source, PID fallback). Both are "turn correlation correctness", but fix(turn): keep eBPF-sourced turns whose opening call was missed #168 is fixed via tracker has_user_start fallback; this one is proxy_pair fingerprint scope.

[github-actions]

Thanks for this report — it's a genuinely good catch, and the investigation you did up front (down to pointing at the exact fingerprint field and the doc-comment assumption it breaks) made it easy to verify.

I read through server/h-turn/src/proxy_pair.rs and confirmed exactly what you describe: content_fingerprint (proxy_pair.rs:206) keys on (session_id, agent_kind, call_count, total_input_tokens, total_output_tokens), group_all buckets by that key before any time-clustering, and the doc comment at proxy_pair.rs:199–200 really does lean on the assumption that both legs land in the same session "by construction." Your counterexample — one leg keeps x-session-affinity (→ opencode profile, real session id) and the other loses it behind a LiteLLM rebuild (→ generic profile, synthesized call_… session) — cleanly breaks that assumption and the two turns never get compared. I also checked PairCandidate (proxy_pair.rs:108) and query_pair_candidates in h-storage-duckdb/src/turns.rs:639: there's no body / body-hash field on the projection today, so your "why the fix isn't trivial" reading is accurate too.

Before I queue this for work, I'd like to align with you on the two questions you flagged, because they change the scope a lot:

1. Which fix shape do you want? You laid out two: (a) persist a new first_call_content_hash column populated at build_turn time, projected into PairCandidate, and used in content_fingerprint in place of session_id/agent_kind — that's a schema migration across DuckDB and ClickHouse plus touches in h-turn, h-storage, h-storage-duckdb, h-storage-clickhouse, and the migration tests; or (b) a lighter-weight version that re-reads the opening call's body at pair-query time and hashes it inside the sweeper, with no new column. (b) is smaller but pushes a body read into the sweeper hot path. Do you have a preference, or should I start with (b) as a probe and only escalate to (a) if the sweeper cost is a problem?
2. How common is the header-stripping leg? You mentioned it's one observed case (a LiteLLM-fronted opencode deployment). Is this something you're seeing repeatedly in your capture, or a one-off? That calibrates whether the schema migration is worth it vs. the lighter fix.
3. Acceptance shape. Just to make sure I'd fix the right thing: the outcome you want is that the two legs in your repro collapse into a single ProxyGroup with one proxy_in and one proxy_out member (mirrors the haproxy_three_leg test), correct? And you're fine leaving the 2× mirror-inflation per leg as a separate issue, as you suggested?

As a near-term workaround while we decide: if you control the LiteLLM config, having it forward x-session-affinity (or any stable header the opencode profile already keys on) to the upstream would let both legs land in the same profile and the existing content_fingerprint would fold them — no code change needed. Not always possible, I know, but worth flagging.

Happy to scope this with you — once you answer 1–3 I can turn it into something concrete and queued.

Maintainer controls

This isn't auto-queued for the dev agent. A maintainer can add agent:try to have it attempted anyway, or agent:skip to mute re-triage. After editing the issue, re-add agent:assess to run triage again.

[vaderyang]

PR #170 is developed. It will be merge to trunk in a right release