Skip to content

Activity R&D#8357

Draft
andrewlock wants to merge 6 commits intomasterfrom
andrew/r_and_d/activity
Draft

Activity R&D#8357
andrewlock wants to merge 6 commits intomasterfrom
andrew/r_and_d/activity

Conversation

@andrewlock
Copy link
Copy Markdown
Member

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Mar 24, 2026

Benchmarks

Benchmark execution time: 2026-03-24 11:40:01

Comparing candidate commit 8a21358 in PR branch andrew/r_and_d/activity with baseline commit 390e11e in branch master.

Found 7 performance improvements and 9 performance regressions! Performance is the same for 257 metrics, 15 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟩 execution_time [-84.475ms; -84.392ms] or [-41.034%; -40.994%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟩 execution_time [-88.694ms; -88.575ms] or [-44.613%; -44.553%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

  • 🟩 throughput [+108647.579op/s; +112366.725op/s] or [+12.201%; +12.618%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • 🟥 execution_time [+11.137ms; +16.914ms] or [+5.617%; +8.531%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • 🟥 execution_time [+20.320ms; +21.009ms] or [+11.068%; +11.443%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟩 execution_time [-20.706ms; -15.296ms] or [-9.510%; -7.025%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+54.043ms; +60.964ms] or [+32.078%; +36.186%]
  • 🟥 throughput [-124.741op/s; -80.287op/s] or [-8.208%; -5.283%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+60.017ms; +66.427ms] or [+39.651%; +43.886%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0

  • 🟩 execution_time [-268.094µs; -262.439µs] or [-16.413%; -16.067%]
  • 🟩 throughput [+117.394op/s; +120.013op/s] or [+19.175%; +19.603%]

scenario:Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery net6.0

  • 🟩 execution_time [-16.452ms; -14.294ms] or [-7.851%; -6.821%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

  • 🟥 execution_time [+11.465ms; +12.950ms] or [+5.795%; +6.546%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

  • 🟥 throughput [-61355.819op/s; -55448.623op/s] or [-11.232%; -10.150%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+42.423ms; +45.363ms] or [+26.825%; +28.683%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

  • 🟥 throughput [-61670.086op/s; -49790.158op/s] or [-8.375%; -6.761%]

andrewlock and others added 6 commits March 24, 2026 13:26
…replacement

Introduces a new opt-in approach (DD_TRACE_OTEL_ACTIVITY_INTERCEPTION_ENABLED=true)
that intercepts System.Diagnostics.Activity methods via CallTarget instead of using
the managed ActivityListener. Goals: reduce memory usage by eliminating Activity's
internal tag storage duplication and the ConcurrentDictionary span lookup.

New CallTarget integrations:
- ActivityStartIntegration: intercepts Activity.Start() to create a Span and link
  it to the Activity via GetCustomProperty/SetCustomProperty ('__dd_span__' key)
- ActivityStopIntegration: intercepts Activity.Stop() to finish the Span with
  correct timing, extracting links/events/status at stop time
- ActivityAddTagStringIntegration: intercepts AddTag(string, string?) to write
  directly to the Span and skip Activity's internal tag list
- ActivityAddTagObjectIntegration: intercepts AddTag(string, object?) similarly
- ActivitySetTagIntegration: intercepts SetTag(string, object?) similarly
- ActivitySetStatusIntegration: intercepts SetStatus() to map OTel status to
  Datadog error tags, bypassing Activity's internal status field
- ActivityDisplayNameIntegration: intercepts set_DisplayName to set Span.ResourceName

Supporting infrastructure:
- ActivityCustomPropertyAccessor<TTarget>: zero-allocation cached delegates for
  reading/writing the Scope via Activity's custom property API
- ActivitySourceFilter: shared filter for source names already handled by other
  Datadog integrations (mirrors IgnoreActivityHandler.SourcesNames)

Configuration changes:
- Added DD_TRACE_OTEL_ACTIVITY_INTERCEPTION_ENABLED feature flag to TracerSettings,
  supported-configurations.yaml, and generated ConfigurationKeys
- Instrumentation.cs: skips managed ActivityListener when interception is enabled
- MutableSettings.cs: keeps OpenTelemetry integration enabled under either mode

Other changes:
- IActivity5: added GetCustomProperty/SetCustomProperty to the duck type interface
- OtlpHelpers: added ExtractLinksAndEventsFromActivity helper for stop integration
- ResourceAttributeProcessorHelper: uses custom property lookup when interception
  is enabled, falling back to ConcurrentDictionary for the legacy listener path

All integrations registered in InstrumentationDefinitions.g.cs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 1 — Fix IsAllDataRequested:
ActivityStartIntegration.CreateAndLinkScope now sets activity5.IsAllDataRequested = true
after linking the Span. The managed ActivityListener did this implicitly via its Sample
callback returning AllData; since we skip the listener when interception is enabled, we
must set it explicitly so user code guarded by `if (activity.IsAllDataRequested)` runs.

Phase 2 — Getter interceptions (redirect reads to the Span):
The setter integrations use skipMethodBody to bypass Activity's internal storage, which
means reading from the Activity would return stale/empty values. Five new getter
integrations restore correct observable state by reading from the linked Span:

- ActivityDisplayNameGetterIntegration: get_DisplayName → span.ResourceName ?? span.OperationName
- ActivityStatusGetterIntegration: get_Status → ActivityStatusCode reconstructed from
  span's "otel.status_code" tag (Enum.ToObject handles the foreign enum conversion)
- ActivityStatusDescriptionGetterIntegration: get_StatusDescription → span's
  "otel.status_description" tag
- ActivityTagsGetterIntegration: get_Tags → all span string tags enumerated via
  ITags.EnumerateTags into List<KVP<string,string?>>
- ActivityTagObjectsGetterIntegration: get_TagObjects → same but boxed as object?,
  matching Activity.TagObjects' IEnumerable<KVP<string,object?>> contract

Note on TagObjects: numeric tags set via SetMetric are not reflected since the Span
stores them separately from string tags. This is an acceptable R&D limitation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add the 5 new Activity getter integration classes to the GetIntegrationId
switch in the generated InstrumentationDefinitions file so they are
correctly mapped to IntegrationId.OpenTelemetry at runtime.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add SubmitsTracesWithInterception to both NetActivitySdkTests and
OpenTelemetrySdkTests. Each variant enables DD_TRACE_OTEL_ACTIVITY_INTERCEPTION_ENABLED
instead of DD_TRACE_OTEL_ENABLED and shares the same snapshot file as
the existing SubmitsTraces test, asserting that the CallTarget-based
interception approach produces output identical to the managed
ActivityListener approach.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@andrewlock andrewlock force-pushed the andrew/r_and_d/activity branch from 8a21358 to b35deb9 Compare March 24, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant