Skip to content

fix(llmobs): prevent evaluator spans from leaking into experiments index#17172

Draft
FouadWahabi wants to merge 1 commit intomainfrom
fouad.wahabi/fix-experiment-evaluator-scope-leak
Draft

fix(llmobs): prevent evaluator spans from leaking into experiments index#17172
FouadWahabi wants to merge 1 commit intomainfrom
fouad.wahabi/fix-experiment-evaluator-scope-leak

Conversation

@FouadWahabi
Copy link
Copy Markdown
Contributor

@FouadWahabi FouadWahabi commented Mar 27, 2026

Summary

  • Evaluator LLM spans were incorrectly inheriting the experiment span's baggage and getting routed to the experiments index instead of the standard LLMObs index
  • Added _clear_trace_context() call between task and evaluator execution in _process_and_evaluate to clear both the default tracer and LLMObs context providers
  • Added test verifying that task spans get scope=experiments while evaluator spans do not

Root Cause

In _run_tasks_with_evaluators, the _process_and_evaluate coroutine runs _process_record (task) and _evaluate_record (evaluators) sequentially within the same asyncio Task. The experiment span created during task execution sets EXPERIMENT_ID_KEY as baggage on its trace context. After the experiment span finishes, the trace context may still carry this baggage when evaluator code runs, causing any LLM spans created by evaluators to inherit the scope=experiments attribute and get routed to the wrong index.

Test plan

  • New test test_experiment_evaluator_spans_not_in_experiments_scope verifies task spans have scope=experiments while evaluator spans do not
  • Existing experiment tests continue to pass

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

Codeowners resolved as

ddtrace/llmobs/_experiment.py                                           @DataDog/ml-observability
tests/llmobs/test_experiments.py                                        @DataDog/ml-observability

@FouadWahabi FouadWahabi force-pushed the fouad.wahabi/fix-experiment-evaluator-scope-leak branch from 1506319 to a584393 Compare March 27, 2026 17:52
Clear trace context between task and evaluator execution so that LLM
spans created inside evaluators don't inherit the experiment span's
baggage and get incorrectly routed to the experiments index.
@FouadWahabi FouadWahabi force-pushed the fouad.wahabi/fix-experiment-evaluator-scope-leak branch from a584393 to 908beb2 Compare March 30, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant