feat(correlation): propagate request ID and trace context across all legs - #654
Draft
cdoern wants to merge 5 commits into
Draft
feat(correlation): propagate request ID and trace context across all legs#654cdoern wants to merge 5 commits into
cdoern wants to merge 5 commits into
Conversation
|
Unsigned commits: ca5fa2d. Please sign your commits. |
…legs A single AI request reaches the backend over two paths: the forwarded request the proxy passes upstream, and the delegated callouts the proxy originates itself while handling it (Files API metadata and content fetches). Neither carried trace context, so the two arrived as unrelated traffic and a slow or failing request could not be attributed to a layer. Add a shared TraceContext that resolves `x-request-id` and a W3C `traceparent` once per downstream request, and stamp both onto every leg: - New `trace_context` filter propagates them to the forwarded request. - Delegated callouts inject them unconditionally, outside the operator-configured `forward_headers` allowlist, since correlation that depends on per-filter configuration silently goes missing. Each leg draws its own span-id from the shared trace-id, keeping delegation latency separable from inference latency. Two details worth noting for review: Identifier resolution reads `extra_request_headers` as well as the downstream headers. The `request_id` builtin writes its generated ID there rather than back into `ctx.request.headers`, so a downstream-only lookup finds an ID only when the client happened to supply one. Agreement across hops is established through request extensions rather than header inspection. Delegated callouts can run in the StreamBuffer pre-read phase, ahead of header-phase filters, where injected values are not visible; whichever hop resolves first initializes the shared context and later hops reuse it regardless of filter order. Client-supplied `traceparent` values are validated before use and discarded when malformed, rather than forwarded into the telemetry backend unchecked. Building the callout header set once per request instead of once per callout also removes repeated allowlist copies for requests that resolve several file references. Trace context is propagated but not yet exported: no OpenTelemetry pipeline exists in this repo, so these headers make requests correlatable without producing spans. Delegated MCP calls and delegation-latency metrics remain to be covered. Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
force-pushed
the
correlation-propagation
branch
from
August 5, 2026 20:20
ca5fa2d to
881e6f6
Compare
praxis-bot
reviewed
Aug 6, 2026
praxis-bot
left a comment
Collaborator
There was a problem hiding this comment.
Review Summary
Clean, well-documented feature with thorough test coverage. The module-level doc in correlation/mod.rs is excellent -- the resolution precedence and the agreement-across-hops rationale are exactly what a future maintainer needs. Three medium-severity items below.
| Severity | Count |
|---|---|
| Critical | 0 |
| Large | 0 |
| Medium | 3 |
Contributor
|
Good direction, thanks @cdoern |
cdoern
added a commit
to cdoern/ai
that referenced
this pull request
Aug 12, 2026
Address praxis-bot review findings on praxis-proxy#654. Resolve the Files API callout headers once at the filter boundary and thread them through current-input resolution and both state-history paths, so every delegated callout of one request shares a span-id regardless of whether history references are cache hits. Guard the defensive ID sanitization against emitting the all-zero trace-id and span-id that W3C Trace Context section 2.2.2 forbids, and document the get_or_init precondition on TraceContext::from_filter_context with a debug log when a caller resolves without it.
Address praxis-bot review findings on praxis-proxy#654. Resolve the Files API callout headers once at the filter boundary and thread them through current-input resolution and both state-history paths, so every delegated callout of one request shares a span-id regardless of whether history references are cache hits. Guard the defensive ID sanitization against emitting the all-zero trace-id and span-id that W3C Trace Context section 2.2.2 forbids, and document the get_or_init precondition on TraceContext::from_filter_context with a debug log when a caller resolves without it. Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
force-pushed
the
correlation-propagation
branch
from
August 12, 2026 14:14
679b75d to
57c5d27
Compare
The file-search callout filter merged onto main independently of the correlation work, and the two combined into a semantic conflict that git resolved cleanly. Before correlation, `post_json_bytes` applied the operator's `forward_headers` allowlist itself via `build_header_map`, so passing the downstream header map straight in was correct. Correlation moved allowlist application out to the caller — the parameter became a pre-built header set that the client now clones verbatim — but the file-search path still passed `&ctx.request.headers`. The result was that vector-store searches ignored their own configured allowlist and shipped the client's entire downstream header map, credentials included, to the vector store, while carrying no correlation. Build the callout header set once per request at the filter boundary, as the Files API path already does, and thread it through the fan-out. The pending-call check moves ahead of planning so trace context is established only for requests that actually call out, and before the plan borrows the context. `SearchPlan::has_pending_calls` is replaced by a shared `has_pending_calls` predicate, so the boundary check and the plan cannot drift apart. The regression test asserts all three halves: allowlisted headers arrive, non-allowlisted ones do not, and correlation is present. Signed-off-by: Charlie Doern <cdoern@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A single AI request reaches the backend over two paths:
Neither carried trace context. The two legs arrived at the backend as unrelated traffic, so a slow or failing request could not be attributed to a layer.
Change
A shared
TraceContextresolvesx-request-idand a W3Ctraceparentonce per downstream request, and both are stamped onto every leg:trace_contextfilter propagates them to the forwarded request.forward_headersallowlist — correlation that depends on per-filter configuration silently goes missing.Each leg draws its own span-id from the shared trace-id, which is what keeps delegation latency separable from inference latency.
Notes for review
Identifier resolution reads
extra_request_headers, not just downstream headers. Therequest_idbuiltin writes its generated ID there rather than back intoctx.request.headers. A downstream-only lookup finds an ID only when the client happened to supply one — and silently no-ops otherwise, which is the common case. There is a test named for it.Agreement across hops goes through request extensions, not header inspection. Delegated callouts can run in the
StreamBufferpre-read phase, ahead of header-phase filters, where injected values are not visible. Whichever hop resolves first initializes the shared context; later hops reuse it regardless of filter order. An earlier header-based attempt produced two different trace-ids for one request, caught by the integration test in this PR.Client-supplied
traceparentvalues are validated and discarded when malformed rather than forwarded into the telemetry backend unchecked.One assertion was deliberately relaxed. The delegated callouts of one request share a span, because correlation is resolved once at the filter boundary — file resolution is one delegation hop regardless of how many files it fetches. The test asserts that instead of per-callout spans, and that the forwarded leg's span is disjoint from it. Happy to switch to per-callout spans if reviewers prefer.
Incidental improvement: the callout header set is now built once per request instead of once per callout, removing repeated allowlist copies for requests that resolve several file references.
Scope
Trace context is propagated but not yet exported — no OpenTelemetry pipeline exists in this repo, so these headers make requests correlatable without producing spans. Wiring an exporter, covering delegated MCP calls, and recording a delegation-latency metric remain to be done.
Testing
traceparentvalidation, including a table of malformed inputstrace_contextfilterexamples/configs/trace-context.yaml, generated filter docs, full workspace suite andclippy -D warningsclean