feat(otel): Enable improved OTel trace compatibility via opt-in configuration by zacharycmontoya · Pull Request #18495 · DataDog/dd-trace-py

zacharycmontoya · 2026-06-06T00:22:32Z

Description

This PR adds a configuration to process spans as closely as possible to the OTel SDK. The result is that using OpenTelemetry Tracing instrumentations allows DD SDKs to produce OTLP spans with nearly identical span fields.

When enabled, this updates the dd-trace-py behavior in the following ways:

Configures the libdatadog native trace exporter with enable_otel_trace_semantics() (see the related libdatadog PR for specific attribute changes)
OtelSpan.set_attribute no longer remaps span tags such as http.response.status_code or service.name so they will maintain their intended key and value type
OtelSpan.record_exception no longer sets span attributes error.message, error.type, or error.stack. These exist on the exception span event and will be recognized by Error Tracking
OtelSpan.record_exception sets the attribute exception.stacktrace on the exception span event to align with OpenTelemetry
The OTel span will not have span attribute span.kind

This PR also enables native span events when OTLP traces export is enabled via OTEL_TRACES_EXPORTER=otlp because OTLP spans have a first-class events field.

Relies on DataDog/libdatadog#2091 to implement the OTLP trace exporter changes

// TODO: Fix the exception.type change so it only applies when the OTel compatibility mode is enabled

Testing

Unit tests are included for the updated OTel test span behavior and the native span events configuration

Risks

Besides the native span events change, the remaining changes are opt-in so the risk is low.

Additional Notes

Currently the libdatadog dependency is set to a development branch which contains the updated TraceExporterBuilder::enable_otel_trace_semantics() API

…LED. When enabled, the tracer will not attempt to remap known OpenTelemetry attributes when handling OTEL spans and instead preserve them. A unit test has been added to assert that `otelspan.set_attribute("http.response.status_code", 200)` maintains both the original attribute key AND value (an integer)

…. Requires an updated libdatadog dependency as this is configured via a new interop call in the TraceExporterBuilder

…ontoya/otlp-compatibility-mode to consume the new native changes

…ty is enabled

…export Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ER=otlp Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ve span events Adds the _trace_native_span_events private field to the type stub for AgentConfig and adds a test case documenting that multi-exporter OTEL_TRACES_EXPORTER strings (e.g. "otlp,zipkin") intentionally do NOT trigger auto-enable, only the exact value "otlp" does. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…TRACES_EXPORTER=otlp

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… behaviors: - Do not set error tracking tags on the main span (they exist on the span event) - Add the attribute 'exception.stacktrace' to the exception span event

…n tags

…here builtins is not included in the module name Note: This could be considered a breaking change as the signature of the error will be updated.

datadog-datadog-prod-us1-2 · 2026-06-06T00:22:52Z

Tests

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 11 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741491-d2b8243-musllinux_1_2_x86_64, 1]

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]

View all 11 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: c68a85b | Docs | Datadog PR Page | Give us feedback!}

…races export feature is enabled

cit-pr-commenter-54b7da · 2026-06-08T21:01:19Z

Codeowners resolved as

src/native/Cargo.lock                                                   @DataDog/apm-core-python

…EVENTS feature

pr-commenter · 2026-06-08T21:34:53Z

Benchmarks

Benchmark execution time: 2026-06-12 01:52:04

Comparing candidate commit c68a85b in PR branch zach.montoya/otel-trace-compatibility with baseline commit 58ac3af in branch main.

Found 0 performance improvements and 4 performance regressions! Performance is the same for 613 metrics, 10 unstable metrics.

scenario:httppropagationinject-ids_only

🟥 execution_time [+1.452µs; +1.632µs] or [+7.100%; +7.980%]

scenario:iastaspects-swapcase_aspect

🟥 execution_time [+40.273µs; +46.630µs] or [+11.797%; +13.659%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+102.159µs; +110.289µs] or [+23.795%; +25.689%]

scenario:span-start

🟥 execution_time [+1.404ms; +1.553ms] or [+9.060%; +10.025%]

marcotc · 2026-06-08T22:17:08Z

-        # We will not set the exception.stacktrace attribute, this will reduce the size of the span event
+        # https://github.com/open-telemetry/opentelemetry-python/blob/v1.25.0/opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py#L1018
+        module = type(exception).__module__
+        qualname = type(exception).__qualname__


Using __qualname__ is a semantic change from using __name__.

I think that __qualname__ is more accurate: it captures nested exception class names, but a change nevertheless.

I think we'd need someone with more Python experience to know the write answer here.
But leaving it as __name__ is always an option, to avoid changing behaviour.

@mabdinur do you have an opinion here?
The otel spec for exception.type says (link):

The type of the exception (its fully-qualified class name, if applicable).

__qualname__ will return a fully nested exception class, unlike __name__, but it's a change on the error type.

I'm happy to keep the name update behind a feature flag, but I think we should update our tracer to write the fully qualified type name

…fix and the updated compatibility mode

mabdinur · 2026-06-09T16:05:50Z

+    def enable_otel_trace_compatibility(self) -> TraceExporterBuilder:
+        """
+        Enable OTel trace compatibility mode: omit DD-specific per-span attributes
+        (service.name, operation.name, resource.name, span.type) from the OTLP payload.


Is this an exhaustive list? I expect we'll add more values as we align more with otel sematnics.

Suggested change

(service.name, operation.name, resource.name, span.type) from the OTLP payload.

(ex: service.name, operation.name, resource.name, span.type) from the OTLP payload.

This should almost be exhaustive. In my testing at https://github.com/DataDog/evalya/pull/767 this almost fully aligns FastAPI spans when using the OTel instrumentation, so if there are any further attributes that need to be modified then I expect they will be small in number

mabdinur · 2026-06-09T16:13:30Z

@@ -0,0 +1,71 @@
+# Design: Suppress DD error tags in `record_exception` when OTel compatibility mode is enabled


Are these doc changes intentional?

No, I'll remove the docs file changes

…ed on the OTel span kind

…w mode as well

…e new compat mode

…CS_ENABLED

zacharycmontoya and others added 14 commits June 4, 2026 13:34

Enable OTel Trace Compatibility mode in the libdatadog Trace Exporter…

0874c68

…. Requires an updated libdatadog dependency as this is configured via a new interop call in the TraceExporterBuilder

Update the libdatadog dependency to use the development branch zach.m…

464b866

…ontoya/otlp-compatibility-mode to consume the new native changes

Also make sure "span.kind" is not set when the OTel Trace Compatibili…

26dc16e

…ty is enabled

Update to latest libdatadog dev branch

1a82df7

docs: add design spec for auto-enabling native span events with OTLP …

efa2d8f

…export Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test: add failing tests for trace_native_span_events auto-enable

7dfb352

feat(tracing): auto-enable native span events when OTEL_TRACES_EXPORT…

3922538

…ER=otlp Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: add release note for auto-enable native span events when OTEL_…

7ae4645

…TRACES_EXPORTER=otlp

docs: add design spec for record_exception otel compatibility fix

fb14217

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Update OtelSpan.record_exception with the following OTel Trace Compat…

81c91c8

… behaviors: - Do not set error tracking tags on the main span (they exist on the span event) - Add the attribute 'exception.stacktrace' to the exception span event

Update libdatadog reference to include exclusion of error message spa…

0989ea0

…n tags

Update the exception type to match the OpenTelemetry implementation w…

abfa938

…here builtins is not included in the module name Note: This could be considered a breaking change as the signature of the error will be updated.

marcotc reviewed Jun 8, 2026

View reviewed changes

Comment thread ddtrace/internal/settings/_agent.py Outdated

marcotc reviewed Jun 8, 2026

View reviewed changes

Comment thread ddtrace/internal/opentelemetry/span.py Outdated

zacharycmontoya added 2 commits June 8, 2026 13:59

Address PR feedback: Always enable native span events when the OTLP t…

cd7db37

…races export feature is enabled

Address PR feedback: Fix the link to OTel SDK record_exception

fff0e0f

Update the release note after auto-enabling the DD_TRACE_NATIVE_SPAN_…

7d61d78

…EVENTS feature

marcotc reviewed Jun 8, 2026

View reviewed changes

zacharycmontoya added 3 commits June 8, 2026 16:28

Update the libdatadog commit that includes both the 128-bit trace ID …

751731e

…fix and the updated compatibility mode

Update cargo lock file

d87af59

Revert: RC updates in order to consume dev build of libdatadog

34175dd

mabdinur reviewed Jun 9, 2026

View reviewed changes

Update the OTel trace compat mode to update the Datadog span type bas…

a2d179a

…ed on the OTel span kind

zacharycmontoya force-pushed the zach.montoya/otel-trace-compatibility branch from a6560fb to a2d179a Compare June 9, 2026 17:32

Revert span kind change so we set the "span.kind" attribute in the ne…

baf6ae8

…w mode as well

zacharycmontoya added 3 commits June 9, 2026 11:40

Update to the latest libdatadog

afc49e3

Update the span.set_status to override the span's error message in th…

f313e2a

…e new compat mode

Refactor DD_TRACE_OTEL_COMPATIBILITY_ENABLED to DD_TRACE_OTEL_SEMANTI…

3869cc9

…CS_ENABLED

zacharycmontoya force-pushed the zach.montoya/otel-trace-compatibility branch from 10a9453 to 3869cc9 Compare June 12, 2026 01:12

Update libdatadog to the dev branch after the change to the native layer

c68a85b

zacharycmontoya mentioned this pull request Jun 12, 2026

feat(tracer): add DD_TRACE_OTEL_SEMANTICS_ENABLED for OTLP export DataDog/dd-trace-go#4889

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(otel): Enable improved OTel trace compatibility via opt-in configuration#18495

feat(otel): Enable improved OTel trace compatibility via opt-in configuration#18495
zacharycmontoya wants to merge 26 commits into
mainfrom
zach.montoya/otel-trace-compatibility

zacharycmontoya commented Jun 6, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 6, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 8, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jun 8, 2026 •

edited

Loading

Uh oh!

marcotc Jun 8, 2026

Uh oh!

marcotc Jun 9, 2026

Uh oh!

zacharycmontoya Jun 9, 2026

Uh oh!

mabdinur Jun 9, 2026

Uh oh!

zacharycmontoya Jun 9, 2026

Uh oh!

Uh oh!

mabdinur Jun 9, 2026

Uh oh!

zacharycmontoya Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	(service.name, operation.name, resource.name, span.type) from the OTLP payload.
	(ex: service.name, operation.name, resource.name, span.type) from the OTLP payload.

		@@ -0,0 +1,71 @@
		# Design: Suppress DD error tags in `record_exception` when OTel compatibility mode is enabled

Conversation

zacharycmontoya commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Risks

Additional Notes

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 6, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

pr-commenter Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:httppropagationinject-ids_only

scenario:iastaspects-swapcase_aspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:span-start

Uh oh!

marcotc Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

marcotc Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

mabdinur Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mabdinur Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zacharycmontoya commented Jun 6, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jun 6, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

cit-pr-commenter-54b7da Bot commented Jun 8, 2026 •

edited

Loading

pr-commenter Bot commented Jun 8, 2026 •

edited

Loading