Set `DD_TRACE_OTEL_ENABLED=true` by default for all integration tests by andrewlock · Pull Request #8370 · DataDog/dd-trace-dotnet

andrewlock · 2026-03-25T17:06:55Z

Summary of changes

Set DD_TRACE_OTEL_ENABLED=true by default in integration tests
Add additional excludes for activity handlers with equivalent custom instrumentation

Reason for change

We had an escalation recently, which highlighted that we were missing some entries in the IgnoreActivityHandler. To avoid hitting similar issues in the future, we can set DD_TRACE_OTEL_ENABLED=true by default, so we know as soon as a new ActivitySource lights up.

Implementation details

A lot of trial and error here, mostly setting DD_TRACE_OTEL_ENABLED=true in the TracingIntegrationTest base class and seeing what breaks 😅 That pointed to a variety of extra handlers and differences in behaviour:

Couchbase.DotnetSdk.OpenTelemetryRequestTracer, I don't think we actually need this one strictly, but it showed up while I was trying to fix persistent issues with Couchbase3, so I think it makes sense to exclude it
- The real issue I had is that some early versions of Couchbase (3.0.0 - 3.2.0) create activities using new Activity(), which means there's no ActivitySource associated, and so we have no way to filter them 💀
- Rather than fight with that, and because those versions are deprecated, just disabled OTel integration for these specific tested versions
connector-net - this is the ActivitySource for MySql.Data (we already excluded the one for MySqlConnector)
RabbitMQ.Client.* - these were the ones that caused the original issue, and were breaking DSM
Experimental.System.Net.Security - this one came in .NET 9, and was causing extra spans in gRPC and Yarp tests
Grpc.Net.Client - The gRPC client has had an ActivitySource for a long time 😅
Yarp.ReverseProxy - ...as has Yarp

In addition, there were some extra "fixes" to the tests required:

For the HttpMessageHandler tests, where the integration is disabled, we were previously verifying that W3C headers weren't injected, but they will be if OTel is enabled, so just relaxed the restrictions there.
For OpenTelemetrySdkTests.SubmitsOtlpLogs, the data changes depending on whether otel is enabled or not, so just reset it to the default for simplicity rather than wrestle with it (I spent some time ping-ponging snapshots before I gave up 😅)
Updated the gRPC snapshots to add grpc.method and grpc.status_code which are now added to the aspnetcore spans

Test coverage

Hopefully self explanatory 😅

Other details

Fixes https://datadoghq.atlassian.net/browse/DSMS-138

Technically, this could be a breaking change for some people, so maybe we should revisit making the ignore activity handler configurable?

pr-commenter · 2026-03-25T17:52:20Z

Benchmarks

Benchmark execution time: 2026-03-30 18:21:37

Comparing candidate commit 7390b47 in PR branch andrew/otel-everywhere with baseline commit 5ff69b3 in branch master.

Found 11 performance improvements and 6 performance regressions! Performance is the same for 261 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472

🟥 throughput [-6690.644op/s; -6278.648op/s] or [-7.593%; -7.126%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

🟩 execution_time [-20.127ms; -19.880ms] or [-16.240%; -16.040%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

🟩 execution_time [-102.609ms; -100.658ms] or [-51.975%; -50.987%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

🟥 execution_time [+22.971ms; +25.695ms] or [+22.891%; +25.606%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

🟥 throughput [-200.207op/s; -145.336op/s] or [-11.265%; -8.178%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472

🟥 execution_time [+220.174µs; +229.759µs] or [+19.249%; +20.087%]
🟥 throughput [-146.453op/s; -140.907op/s] or [-16.751%; -16.117%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0

🟥 execution_time [+20.871ms; +26.814ms] or [+10.533%; +13.533%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

🟩 execution_time [-94.250µs; -45.659µs] or [-16.579%; -8.032%]
🟩 throughput [+182.638op/s; +343.806op/s] or [+10.374%; +19.528%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472

🟩 allocated_mem [-8.196KB; -8.189KB] or [-14.293%; -14.281%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

🟩 execution_time [-6.480µs; -2.760µs] or [-13.000%; -5.536%]
🟩 throughput [+1304.159op/s; +2706.334op/s] or [+6.402%; +13.285%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

🟩 execution_time [-7.430µs; -3.102µs] or [-13.858%; -5.785%]
🟩 throughput [+1102.296op/s; +2382.978op/s] or [+5.795%; +12.527%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

🟩 execution_time [-15.959ms; -14.655ms] or [-14.414%; -13.236%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

🟩 throughput [+14951864.878op/s; +16051181.276op/s] or [+6.636%; +7.123%]

dd-trace-dotnet-ci-bot · 2026-03-26T12:11:20Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8370) and master.

⚠️ Potential regressions detected

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	193.20 ± (193.07 - 194.03) ms	211.20 ± (211.11 - 212.16) ms	+9.3%	❌⬆️
.NET Framework 4.8 - Bailout
duration	197.04 ± (197.25 - 198.32) ms	217.98 ± (217.27 - 218.18) ms	+10.6%	❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1165.14 ± (1172.42 - 1183.27) ms	1253.28 ± (1251.27 - 1258.67) ms	+7.6%	❌⬆️

Full Metrics Comparison

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	71.43 ± (71.50 - 71.78) ms	71.80 ± (71.85 - 72.19) ms	+0.5%	✅⬆️
.NET Framework 4.8 - Bailout
duration	76.02 ± (75.97 - 76.33) ms	76.39 ± (76.30 - 76.69) ms	+0.5%	✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1071.89 ± (1072.00 - 1078.73) ms	1071.70 ± (1072.70 - 1078.42) ms	-0.0%	✅
.NET Core 3.1 - Baseline
process.internal_duration_ms	22.35 ± (22.30 - 22.40) ms	22.25 ± (22.21 - 22.30) ms	-0.4%	✅
process.time_to_main_ms	83.67 ± (83.50 - 83.84) ms	83.03 ± (82.87 - 83.20) ms	-0.8%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.90 ± (10.90 - 10.91) MB	10.91 ± (10.91 - 10.92) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	12 ± (12 - 12)	12 ± (12 - 12)	+0.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	22.22 ± (22.18 - 22.25) ms	22.36 ± (22.32 - 22.40) ms	+0.6%	✅⬆️
process.time_to_main_ms	85.11 ± (84.95 - 85.28) ms	85.13 ± (84.91 - 85.35) ms	+0.0%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.95 ± (10.94 - 10.95) MB	10.95 ± (10.94 - 10.95) MB	-0.0%	✅
runtime.dotnet.threads.count	13 ± (13 - 13)	13 ± (13 - 13)	+0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	223.16 ± (221.99 - 224.32) ms	220.45 ± (219.41 - 221.50) ms	-1.2%	✅
process.time_to_main_ms	532.05 ± (530.99 - 533.11) ms	532.02 ± (530.70 - 533.34) ms	-0.0%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	48.24 ± (48.20 - 48.27) MB	48.28 ± (48.25 - 48.30) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.7%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	20.94 ± (20.91 - 20.97) ms	20.87 ± (20.84 - 20.91) ms	-0.3%	✅
process.time_to_main_ms	71.92 ± (71.77 - 72.08) ms	71.71 ± (71.55 - 71.86) ms	-0.3%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.62 ± (10.62 - 10.62) MB	10.62 ± (10.62 - 10.63) MB	+0.0%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 6 - Bailout
process.internal_duration_ms	20.92 ± (20.89 - 20.96) ms	20.88 ± (20.85 - 20.91) ms	-0.2%	✅
process.time_to_main_ms	73.13 ± (72.97 - 73.29) ms	72.88 ± (72.73 - 73.03) ms	-0.3%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.66 ± (10.66 - 10.67) MB	10.74 ± (10.73 - 10.74) MB	+0.7%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	387.20 ± (384.70 - 389.69) ms	387.13 ± (384.75 - 389.52) ms	-0.0%	✅
process.time_to_main_ms	530.58 ± (529.61 - 531.55) ms	532.00 ± (531.01 - 532.99) ms	+0.3%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	50.23 ± (50.21 - 50.26) MB	50.33 ± (50.31 - 50.36) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	-0.4%	✅
.NET 8 - Baseline
process.internal_duration_ms	19.16 ± (19.12 - 19.19) ms	19.23 ± (19.20 - 19.26) ms	+0.4%	✅⬆️
process.time_to_main_ms	71.28 ± (71.13 - 71.43) ms	72.02 ± (71.89 - 72.15) ms	+1.0%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.69 ± (7.68 - 7.69) MB	7.69 ± (7.68 - 7.70) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 8 - Bailout
process.internal_duration_ms	19.25 ± (19.21 - 19.28) ms	19.18 ± (19.15 - 19.22) ms	-0.3%	✅
process.time_to_main_ms	72.88 ± (72.74 - 73.02) ms	72.57 ± (72.42 - 72.72) ms	-0.4%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.75 ± (7.74 - 7.75) MB	7.73 ± (7.73 - 7.74) MB	-0.1%	✅
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	308.00 ± (305.65 - 310.34) ms	306.48 ± (304.40 - 308.55) ms	-0.5%	✅
process.time_to_main_ms	490.98 ± (490.24 - 491.72) ms	493.07 ± (492.21 - 493.93) ms	+0.4%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	37.26 ± (37.24 - 37.28) MB	37.28 ± (37.26 - 37.30) MB	+0.0%	✅⬆️
runtime.dotnet.threads.count	27 ± (27 - 27)	27 ± (27 - 27)	-0.0%	✅

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	193.20 ± (193.07 - 194.03) ms	211.20 ± (211.11 - 212.16) ms	+9.3%	❌⬆️
.NET Framework 4.8 - Bailout
duration	197.04 ± (197.25 - 198.32) ms	217.98 ± (217.27 - 218.18) ms	+10.6%	❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1165.14 ± (1172.42 - 1183.27) ms	1253.28 ± (1251.27 - 1258.67) ms	+7.6%	❌⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms	187.87 ± (187.50 - 188.24) ms	209.82 ± (209.26 - 210.38) ms	+11.7%	✅⬆️
process.time_to_main_ms	81.24 ± (80.95 - 81.54) ms	91.63 ± (91.32 - 91.93) ms	+12.8%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.11 ± (16.07 - 16.14) MB	15.90 ± (15.88 - 15.92) MB	-1.3%	✅
runtime.dotnet.threads.count	20 ± (19 - 20)	20 ± (20 - 20)	+1.7%	✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms	186.53 ± (186.24 - 186.82) ms	209.52 ± (208.94 - 210.10) ms	+12.3%	✅⬆️
process.time_to_main_ms	81.95 ± (81.79 - 82.12) ms	92.75 ± (92.46 - 93.03) ms	+13.2%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.20 ± (16.17 - 16.24) MB	15.99 ± (15.97 - 16.01) MB	-1.3%	✅
runtime.dotnet.threads.count	21 ± (21 - 21)	21 ± (21 - 21)	+1.1%	✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	395.41 ± (394.15 - 396.68) ms	422.54 ± (420.92 - 424.16) ms	+6.9%	✅⬆️
process.time_to_main_ms	525.23 ± (524.34 - 526.13) ms	569.66 ± (568.22 - 571.11) ms	+8.5%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	59.05 ± (58.94 - 59.17) MB	59.25 ± (59.20 - 59.30) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	-0.1%	✅
.NET 6 - Baseline
process.internal_duration_ms	192.21 ± (191.88 - 192.55) ms	212.84 ± (212.32 - 213.36) ms	+10.7%	✅⬆️
process.time_to_main_ms	70.09 ± (69.91 - 70.26) ms	79.06 ± (78.70 - 79.42) ms	+12.8%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.17 ± (16.03 - 16.31) MB	16.18 ± (16.16 - 16.20) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	20 ± (20 - 20)	+3.7%	✅⬆️
.NET 6 - Bailout
process.internal_duration_ms	192.26 ± (191.93 - 192.59) ms	213.75 ± (213.18 - 214.32) ms	+11.2%	✅⬆️
process.time_to_main_ms	71.05 ± (70.94 - 71.15) ms	80.57 ± (80.33 - 80.81) ms	+13.4%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.22 ± (16.09 - 16.35) MB	16.21 ± (16.18 - 16.23) MB	-0.1%	✅
runtime.dotnet.threads.count	19 ± (19 - 20)	21 ± (20 - 21)	+6.7%	✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	600.34 ± (597.60 - 603.08) ms	597.66 ± (595.44 - 599.89) ms	-0.4%	✅
process.time_to_main_ms	523.80 ± (522.95 - 524.65) ms	575.82 ± (574.39 - 577.26) ms	+9.9%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	62.08 ± (62.00 - 62.17) MB	61.59 ± (61.49 - 61.68) MB	-0.8%	✅
runtime.dotnet.threads.count	30 ± (30 - 30)	31 ± (31 - 31)	+1.5%	✅⬆️
.NET 8 - Baseline
process.internal_duration_ms	190.16 ± (189.86 - 190.46) ms	210.78 ± (210.20 - 211.37) ms	+10.8%	✅⬆️
process.time_to_main_ms	69.51 ± (69.33 - 69.68) ms	78.16 ± (77.90 - 78.42) ms	+12.4%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.84 ± (11.81 - 11.88) MB	11.57 ± (11.55 - 11.59) MB	-2.3%	✅
runtime.dotnet.threads.count	18 ± (18 - 18)	19 ± (19 - 19)	+3.7%	✅⬆️
.NET 8 - Bailout
process.internal_duration_ms	190.07 ± (189.67 - 190.47) ms	210.83 ± (210.30 - 211.36) ms	+10.9%	✅⬆️
process.time_to_main_ms	70.62 ± (70.48 - 70.76) ms	79.65 ± (79.40 - 79.89) ms	+12.8%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.88 ± (11.85 - 11.90) MB	11.64 ± (11.62 - 11.66) MB	-2.0%	✅
runtime.dotnet.threads.count	19 ± (19 - 19)	20 ± (20 - 20)	+4.8%	✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	519.62 ± (517.02 - 522.21) ms	571.87 ± (563.98 - 579.76) ms	+10.1%	✅⬆️
process.time_to_main_ms	480.70 ± (480.02 - 481.38) ms	529.05 ± (528.04 - 530.05) ms	+10.1%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	50.96 ± (50.93 - 50.99) MB	51.41 ± (51.31 - 51.51) MB	+0.9%	✅⬆️
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	+0.1%	✅⬆️

Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts

FakeDbCommand (.NET Framework 4.8)

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (72ms)  : 69, 75
    master - mean (72ms)  : 70, 74

    section Bailout
    This PR (8370) - mean (76ms)  : 75, 78
    master - mean (76ms)  : 74, 78

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (1,076ms)  : 1035, 1116
    master - mean (1,075ms)  : 1027, 1124

FakeDbCommand (.NET Core 3.1)

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (112ms)  : 109, 115
    master - mean (113ms)  : 110, 116

    section Bailout
    This PR (8370) - mean (114ms)  : 112, 117
    master - mean (114ms)  : 112, 116

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (791ms)  : 774, 807
    master - mean (792ms)  : 771, 814

FakeDbCommand (.NET 6)

gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (99ms)  : 95, 102
    master - mean (99ms)  : 95, 103

    section Bailout
    This PR (8370) - mean (100ms)  : 97, 103
    master - mean (100ms)  : 98, 103

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (947ms)  : 906, 988
    master - mean (946ms)  : 910, 982

FakeDbCommand (.NET 8)

gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (99ms)  : 96, 102
    master - mean (98ms)  : 95, 101

    section Bailout
    This PR (8370) - mean (99ms)  : 97, 102
    master - mean (100ms)  : 98, 102

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (830ms)  : 792, 869
    master - mean (829ms)  : 793, 865

HttpMessageHandler (.NET Framework 4.8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (212ms)  : 206, 217
    master - mean (194ms)  : 189, 198

    section Bailout
    This PR (8370) - mean (218ms)  : crit, 213, 222
    master - mean (198ms)  : 192, 203

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (1,255ms)  : crit, 1201, 1309
    master - mean (1,178ms)  : 1095, 1261

HttpMessageHandler (.NET Core 3.1)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (311ms)  : 303, 320
    master - mean (277ms)  : 272, 283

    section Bailout
    This PR (8370) - mean (312ms)  : crit, 303, 320
    master - mean (277ms)  : 273, 280

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (1,033ms)  : crit, 1008, 1057
    master - mean (953ms)  : 929, 977

HttpMessageHandler (.NET 6)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (301ms)  : 294, 308
    master - mean (271ms)  : 266, 275

    section Bailout
    This PR (8370) - mean (304ms)  : crit, 297, 310
    master - mean (271ms)  : 268, 275

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (1,206ms)  : 1168, 1244
    master - mean (1,155ms)  : 1111, 1199

HttpMessageHandler (.NET 8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8370) - mean (299ms)  : 292, 307
    master - mean (270ms)  : 265, 274

    section Bailout
    This PR (8370) - mean (301ms)  : crit, 294, 308
    master - mean (270ms)  : 265, 275

    section CallTarget+Inlining+NGEN
    This PR (8370) - mean (1,134ms)  : crit, 1012, 1257
    master - mean (1,034ms)  : 988, 1080

github-actions · 2026-03-26T15:34:12Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

4 occurrences of :

+      grpc.method: /greet.tester.Greeter/StreamingBothWays,
+      grpc.status_code: 0,

4 occurrences of :

+      grpc.method: /greet.tester.Greeter/StreamingFromClient,
+      grpc.status_code: 0,

8 occurrences of :

+      grpc.method: /greet.tester.Greeter/ErroringMethod,
+      grpc.status_code: 1,

8 occurrences of :

+      grpc.method: /greet.tester.Greeter/ErroringMethod,
+      grpc.status_code: 15,

8 occurrences of :

+      grpc.method: /greet.tester.Greeter/ErroringMethod,
+      grpc.status_code: 5,

8 occurrences of :

+      grpc.method: /greet.tester.Greeter/ErroringMethod,
+      grpc.status_code: 2,

4 occurrences of :

+      grpc.method: /greet.tester.Greeter/StreamingFromServer,
+      grpc.status_code: 0,

8 occurrences of :

+      grpc.method: /greet.tester.Greeter/Unary,
+      grpc.status_code: 0,

8 occurrences of :

+      grpc.method: /greet.tester.Greeter/VerySlow,
+      grpc.status_code: 4,

…ty sources)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7390b47b29

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-30T17:44:04Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/OpenTelemetrySdkTests.cs

            SetEnvironmentVariable("OTEL_RESOURCE_ATTRIBUTES", "service.name=OtlpLogsService,deployment.environment=testing");
+            // TODO: if we want to have this enabled (which we probably _should_)
+            // then it requires updating a bunch of snapshots, because it impacts the "flags" field
+            EnvironmentHelper.CustomEnvironmentVariables.Remove("DD_TRACE_OTEL_ENABLED");


Set DD_TRACE_OTEL_ENABLED explicitly for OTLP logs test

In SubmitsOtlpLogs, removing DD_TRACE_OTEL_ENABLED from CustomEnvironmentVariables makes the sample process fall back to the parent process environment instead of a deterministic test value. Because process startup inherits ambient variables and DD_TRACE_OTEL_ENABLED is not part of the profiler env vars that get cleared, this test can silently run with OTel enabled on machines/agents that export the variable, causing snapshot-dependent behavior and flaky outcomes. Set it explicitly (e.g., false) in this test instead of removing it.

Useful? React with 👍 / 👎.

zacharycmontoya · 2026-03-30T18:04:04Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/OpenTelemetrySdkTests.cs

            SetEnvironmentVariable("DD_SERVICE", "OtlpLogsService");
            SetEnvironmentVariable("OTEL_RESOURCE_ATTRIBUTES", "service.name=OtlpLogsService,deployment.environment=testing");
+            // TODO: if we want to have this enabled (which we probably _should_)
+            // then it requires updating a bunch of snapshots, because it impacts the "flags" field


We can update this in a separate PR if the number of snapshots is high 👍🏼

zacharycmontoya · 2026-03-30T18:06:22Z

tracer/src/Datadog.Trace/Activity/Handlers/IgnoreActivityHandler.cs

        private static readonly string[] SourcesNames =
        {
            "Couchbase.DotnetSdk.RequestTracer",
+            "Couchbase.DotnetSdk.OpenTelemetryRequestTracer",


Do you remember why we chose to "listen and ignore" sources rather than "do not subscribe" to sources? I suspect we should be doing the latter

zacharycmontoya · 2026-03-30T20:44:23Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/HttpMessageHandlerTests.cs

-                    headers.Should().NotContainKey(W3CTraceContextPropagator.TraceParentHeaderName);
-                    headers.Should().NotContainKey(W3CTraceContextPropagator.TraceStateHeaderName);
+                    // These are still injected when we have otel enabled, but they're not added by us
+                    // TODO: Verify whether the injected values are actually _correct_ - they _should_ be,


I did a local test and this behavior only happens starting in .NET 7. Unfortunately, the span-id on the headers matches the span-id of the ignored activity. More details:

The ActivitySource with name System.Net.Http will generate an activity named System.Net.Http.HttpRequestOut. This source is not covered by our IgnoreActivityHandler so this is handled by our DefaultActivityHandler and ultimately we "ignore" the activity and do not create a corresponding Datadog span because it passes our IgnoreActivityHandler.ShouldIgnoreByOperationName(activity.OperationName) check.

We then call IgnoreActivityHandler.IgnoreActivity and the the TraceId and ParentSpanId are rewritten (notably we do not touch the SpanId). After our ActivityHandlers respond to the new Activity creation, it is injected into the headers as traceparent={activity.Id} and the result is the span-id part of the traceparent is incorrect because we never recorded a span with that span-id name. When tracing is enabled we end up overriding the traceparent header with our own span information, but when tracing is disabled then this information goes through untouched.

In reality, this behavior will only affect cases where we want to disable tracing, which includes the export of traces or other telemetry data that the user would view as noise. We can do a follow-up PR to resolve this issue because it's not new, but we should fix this.

zacharycmontoya

LGTM, with a comment about some follow-up work we can do

Set DD_TRACE_OTEL_ENABLED=true by default for all integration tests

6329bfd

github-actions bot added the area:tests unit tests, integration tests label Mar 25, 2026

andrewlock changed the title ~~Set DD_TRACE_OTEL_ENABLED=true by default for all integration tests~~ Set DD_TRACE_OTEL_ENABLED=true by default for all integration tests Mar 25, 2026

andrewlock added 4 commits March 26, 2026 11:11

Disable OTel for "Otel disabled" tests

134b2f8

Ignore RabbitMQ activities

35db4c5

Exclude MySql.Data activity source

f1c3d36

Add missing Couchbase ActivitySource

67c86c3

andrewlock added 2 commits March 26, 2026 15:32

Update some snapshots (I don't know why)

f80ad4c

Try to see what's going on with couchbase

f96c00d

andrewlock added 8 commits March 27, 2026 17:55

Expect 5 spans, just to see what's going on

3e471f1

Fix disable HTTP tests

05774d8

Add flags back (could have ping-pong issue here)

42c7024

Don't run otel tests against couchbase < 3.2.0 (they don't use activi…

2e7cd1e

…ty sources)

Add Experimental.System.Net.Security to ignored handler

9fbe610

Revert uneeded change

1a641f5

Fix couchbase tests

e8c744e

Remove DD_TRACE_OTEL_ENABLED from OpenTelemetrySdkTests.SubmitsOtlpLogs

ed6038d

andrewlock added type:bug breaking-change area:opentelemetry OpenTelemetry support labels Mar 30, 2026

andrewlock added 2 commits March 30, 2026 18:23

Add Grpc.Net.Client to ignore handlers

32b74a1

Add Yarp.ReverseProxy to ignored handler

7390b47

andrewlock marked this pull request as ready for review March 30, 2026 17:39

andrewlock requested review from a team as code owners March 30, 2026 17:39

andrewlock requested a review from a team as a code owner March 30, 2026 17:39

andrewlock requested a review from anna-git March 30, 2026 17:39

chatgpt-codex-connector bot reviewed Mar 30, 2026

View reviewed changes

zacharycmontoya reviewed Mar 30, 2026

View reviewed changes

zacharycmontoya approved these changes Mar 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set `DD_TRACE_OTEL_ENABLED=true` by default for all integration tests#8370

Set `DD_TRACE_OTEL_ENABLED=true` by default for all integration tests#8370
andrewlock wants to merge 17 commits intomasterfrom
andrew/otel-everywhere

andrewlock commented Mar 25, 2026 •

edited

Loading

Uh oh!

pr-commenter bot commented Mar 25, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Uh oh!

dd-trace-dotnet-ci-bot bot commented Mar 26, 2026 •

edited

Loading

FakeDbCommand

HttpMessageHandler

Uh oh!

github-actions bot commented Mar 26, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 30, 2026

Uh oh!

zacharycmontoya Mar 30, 2026

Uh oh!

zacharycmontoya Mar 30, 2026

Uh oh!

zacharycmontoya Mar 30, 2026

Uh oh!

zacharycmontoya left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

andrewlock commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

pr-commenter bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

Uh oh!

dd-trace-dotnet-ci-bot bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

⚠️ Potential regressions detected

HttpMessageHandler

FakeDbCommand

HttpMessageHandler

Uh oh!

github-actions bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Snapshots difference summary

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andrewlock commented Mar 25, 2026 •

edited

Loading

pr-commenter bot commented Mar 25, 2026 •

edited

Loading

dd-trace-dotnet-ci-bot bot commented Mar 26, 2026 •

edited

Loading

github-actions bot commented Mar 26, 2026 •

edited

Loading