Skip to content

feat(correlation): propagate request ID and trace context across all legs - #654

Draft
cdoern wants to merge 5 commits into
praxis-proxy:mainfrom
cdoern:correlation-propagation
Draft

feat(correlation): propagate request ID and trace context across all legs#654
cdoern wants to merge 5 commits into
praxis-proxy:mainfrom
cdoern:correlation-propagation

Conversation

@cdoern

@cdoern cdoern commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Problem

A single AI request reaches the backend over two paths:

  • forwarded — the client's request, proxied upstream
  • delegated — callouts the proxy originates itself while handling it (Files API metadata and content fetches)

Neither carried trace context. The two legs arrived at the backend as unrelated traffic, so a slow or failing request could not be attributed to a layer.

Change

A shared TraceContext resolves x-request-id and a W3C traceparent once per downstream request, and both are stamped onto every leg:

  • New trace_context filter propagates them to the forwarded request.
  • Delegated callouts inject them unconditionally, outside the operator-configured forward_headers allowlist — correlation that depends on per-filter configuration silently goes missing.

Each leg draws its own span-id from the shared trace-id, which is what keeps delegation latency separable from inference latency.

Notes for review

Identifier resolution reads extra_request_headers, not just downstream headers. The request_id builtin writes its generated ID there rather than back into ctx.request.headers. A downstream-only lookup finds an ID only when the client happened to supply one — and silently no-ops otherwise, which is the common case. There is a test named for it.

Agreement across hops goes through request extensions, not header inspection. Delegated callouts can run in the StreamBuffer pre-read phase, ahead of header-phase filters, where injected values are not visible. Whichever hop resolves first initializes the shared context; later hops reuse it regardless of filter order. An earlier header-based attempt produced two different trace-ids for one request, caught by the integration test in this PR.

Client-supplied traceparent values are validated and discarded when malformed rather than forwarded into the telemetry backend unchecked.

One assertion was deliberately relaxed. The delegated callouts of one request share a span, because correlation is resolved once at the filter boundary — file resolution is one delegation hop regardless of how many files it fetches. The test asserts that instead of per-callout spans, and that the forwarded leg's span is disjoint from it. Happy to switch to per-callout spans if reviewers prefer.

Incidental improvement: the callout header set is now built once per request instead of once per callout, removing repeated allowlist copies for requests that resolve several file references.

Scope

Trace context is propagated but not yet exported — no OpenTelemetry pipeline exists in this repo, so these headers make requests correlatable without producing spans. Wiring an exporter, covering delegated MCP calls, and recording a delegation-latency metric remain to be done.

Testing

  • 9 unit tests for identifier resolution and traceparent validation, including a table of malformed inputs
  • 7 unit tests for the trace_context filter
  • 2 integration tests over the example config asserting both legs share a trace-id under distinct spans, for generated and client-supplied traces
  • Example config examples/configs/trace-context.yaml, generated filter docs, full workspace suite and clippy -D warnings clean

@praxis-bot-app

praxis-bot-app Bot commented Aug 5, 2026

Copy link
Copy Markdown

Unsigned commits: ca5fa2d. Please sign your commits.

…legs

A single AI request reaches the backend over two paths: the forwarded
request the proxy passes upstream, and the delegated callouts the proxy
originates itself while handling it (Files API metadata and content
fetches). Neither carried trace context, so the two arrived as unrelated
traffic and a slow or failing request could not be attributed to a layer.

Add a shared TraceContext that resolves `x-request-id` and a W3C
`traceparent` once per downstream request, and stamp both onto every leg:

- New `trace_context` filter propagates them to the forwarded request.
- Delegated callouts inject them unconditionally, outside the
  operator-configured `forward_headers` allowlist, since correlation that
  depends on per-filter configuration silently goes missing.

Each leg draws its own span-id from the shared trace-id, keeping
delegation latency separable from inference latency.

Two details worth noting for review:

Identifier resolution reads `extra_request_headers` as well as the
downstream headers. The `request_id` builtin writes its generated ID
there rather than back into `ctx.request.headers`, so a downstream-only
lookup finds an ID only when the client happened to supply one.

Agreement across hops is established through request extensions rather
than header inspection. Delegated callouts can run in the StreamBuffer
pre-read phase, ahead of header-phase filters, where injected values are
not visible; whichever hop resolves first initializes the shared context
and later hops reuse it regardless of filter order.

Client-supplied `traceparent` values are validated before use and
discarded when malformed, rather than forwarded into the telemetry
backend unchecked.

Building the callout header set once per request instead of once per
callout also removes repeated allowlist copies for requests that resolve
several file references.

Trace context is propagated but not yet exported: no OpenTelemetry
pipeline exists in this repo, so these headers make requests correlatable
without producing spans. Delegated MCP calls and delegation-latency
metrics remain to be covered.

Signed-off-by: Charlie Doern <cdoern@redhat.com>
@cdoern
cdoern force-pushed the correlation-propagation branch from ca5fa2d to 881e6f6 Compare August 5, 2026 20:20

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Clean, well-documented feature with thorough test coverage. The module-level doc in correlation/mod.rs is excellent -- the resolution precedence and the agreement-across-hops rationale are exactly what a future maintainer needs. Three medium-severity items below.

Severity Count
Critical 0
Large 0
Medium 3

Comment thread apis/src/openai/responses/file_resolve/mod.rs
Comment thread apis/src/correlation/mod.rs Outdated
Comment thread apis/src/correlation/mod.rs Outdated
@leseb

leseb commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Good direction, thanks @cdoern

cdoern added a commit to cdoern/ai that referenced this pull request Aug 12, 2026
Address praxis-bot review findings on praxis-proxy#654.

Resolve the Files API callout headers once at the filter boundary and
thread them through current-input resolution and both state-history
paths, so every delegated callout of one request shares a span-id
regardless of whether history references are cache hits.

Guard the defensive ID sanitization against emitting the all-zero
trace-id and span-id that W3C Trace Context section 2.2.2 forbids, and
document the get_or_init precondition on
TraceContext::from_filter_context with a debug log when a caller
resolves without it.
Address praxis-bot review findings on praxis-proxy#654.

Resolve the Files API callout headers once at the filter boundary and
thread them through current-input resolution and both state-history
paths, so every delegated callout of one request shares a span-id
regardless of whether history references are cache hits.

Guard the defensive ID sanitization against emitting the all-zero
trace-id and span-id that W3C Trace Context section 2.2.2 forbids, and
document the get_or_init precondition on
TraceContext::from_filter_context with a debug log when a caller
resolves without it.

Signed-off-by: Charlie Doern <cdoern@redhat.com>
@cdoern
cdoern force-pushed the correlation-propagation branch from 679b75d to 57c5d27 Compare August 12, 2026 14:14
@cdoern
cdoern requested a review from praxis-bot August 12, 2026 14:28
cdoern added 2 commits August 12, 2026 12:58
The file-search callout filter merged onto main independently of the
correlation work, and the two combined into a semantic conflict that
git resolved cleanly.

Before correlation, `post_json_bytes` applied the operator's
`forward_headers` allowlist itself via `build_header_map`, so passing
the downstream header map straight in was correct. Correlation moved
allowlist application out to the caller — the parameter became a
pre-built header set that the client now clones verbatim — but the
file-search path still passed `&ctx.request.headers`.

The result was that vector-store searches ignored their own configured
allowlist and shipped the client's entire downstream header map,
credentials included, to the vector store, while carrying no
correlation.

Build the callout header set once per request at the filter boundary,
as the Files API path already does, and thread it through the fan-out.
The pending-call check moves ahead of planning so trace context is
established only for requests that actually call out, and before the
plan borrows the context.

`SearchPlan::has_pending_calls` is replaced by a shared
`has_pending_calls` predicate, so the boundary check and the plan
cannot drift apart.

The regression test asserts all three halves: allowlisted headers
arrive, non-allowlisted ones do not, and correlation is present.

Signed-off-by: Charlie Doern <cdoern@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants