[OpenTelemetry] Enable OpenTelemetry stitching with Logs for Dataflow worker by stankiewicz · Pull Request #39625 · apache/beam

stankiewicz · 2026-08-05T08:43:58Z

This change will add trace, spanId and sampled fields supported by Logging if Spans are available in current context and if feature is enabled (disabled by default).

Outcome is trace link visible in logging and logging visible in trace:

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2026-08-05T09:38:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.15%. Comparing base (f2c622b) to head (945ffdd).
⚠️ Report is 8 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #39625      +/-   ##
============================================
- Coverage     58.15%   58.15%   -0.01%     
+ Complexity    13085    13084       -1     
============================================
  Files          2521     2521              
  Lines        264850   264863      +13     
  Branches      10788    10788              
============================================
- Hits         154023   154020       -3     
- Misses       105051   105066      +15     
- Partials       5776     5777       +1

Flag	Coverage Δ
java	`64.25% <ø> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

scwhittle

Do you also want to add this to LogRecordHandler in BeamFnLoggingClient.java?
That is used with the dataflow portable runner harness.

scwhittle · 2026-08-05T12:28:23Z

            .setTimestamp(Instant.ofEpochMilli(record.getMillis()))
            .setSeverity(severityFor(record.getLevel()));
-
+    SpanContext spanContext = Span.current().getSpanContext();


maybe better to just get the current span context if logOpenTelemetryTraceSpanIdAndSampled is enabled in case there is some overhead like a thread-local

good point, will do

scwhittle · 2026-08-05T12:28:35Z

      writeIfNotEmpty(generator, "logger", record.getLoggerName());
      writeIfNotEmpty(generator, "exception", formatException(record.getThrown()));
+      SpanContext spanContext = Span.current().getSpanContext();
+      if (logOpenTelemetryTraceSpanIdAndSampled.get() && spanContext.isValid()) {


scwhittle · 2026-08-05T12:30:00Z

      writeIfNotEmpty(generator, "exception", formatException(record.getThrown()));
+      SpanContext spanContext = Span.current().getSpanContext();
+      if (logOpenTelemetryTraceSpanIdAndSampled.get() && spanContext.isValid()) {
+        generator.writeStringField("trace", spanContext.getTraceId());


does isValid imply that the trace id and span id are not empty? otherwise use writeIfNotEmpty

correct, it means that there is something non empty and not default (different to "00000000000000000000000000000000")

scwhittle · 2026-08-05T12:30:30Z

+      if (logOpenTelemetryTraceSpanIdAndSampled.get() && spanContext.isValid()) {
+        generator.writeStringField("trace", spanContext.getTraceId());
+        generator.writeStringField("spanId", spanContext.getSpanId());
+        generator.writeBooleanField("traceSampled", spanContext.isSampled());


would it be better to reduce overhead by just writing this if it is true and omitting it if false?

maybe this should be trace_sampled as well instead of camel-case to match custom_data?

hmm the documented agent format is also inconsistent https://docs.cloud.google.com/logging/docs/structured-logging#structured_logging_special_fields

uses trace_sampled and spanId

but json is format used between dataflow worker and agent that reads data and sends asynchronously to clod logging. Happy to change field names.

on skipping writing when sampled=false - sounds like good idea, will implement that as well.

scwhittle · 2026-08-05T12:31:12Z

      @Nullable DataflowExecutionState executionState,
      ImmutableMap<String, String> defaultResourceLabels) {
    Struct.Builder payloadBuilder = Struct.newBuilder();
+    //


github-actions · 2026-08-05T13:07:14Z

Assigning reviewers:

R: @kennknowles for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

… direct logging and file based

scwhittle · 2026-08-07T09:10:09Z

+        if (spanContext.isValid() && spanContext.isSampled()) {
+          generator.writeStringField("trace", spanContext.getTraceId());
+          generator.writeStringField("spanId", spanContext.getSpanId());
+          generator.writeBooleanField("trace_sampled", spanContext.isSampled());


I meant to just omit
"trace_sampled": false
since that is probably common and we could just infer it was false.

This is a little different since we're also not logging the other fields.
Is there still value in adding the trace and spanId to the log if it is not-sampled? If not, this seems better to avoid unnecessary writes/parsing. but perhaps there could be some use for these fields, if they could correlate to other logs even if it isn't sampled? That is indicated by https://docs.cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#FIELDS.trace_sampled

I think it would be better then to log trace/spanId if the context is valid. And just conditionally log trace_sampled if it is valid and true. Alternatively we could add another option controlling if we only log sampled trace/spanids.

thanks, will update that today.

stankiewicz requested a review from scwhittle August 5, 2026 08:44

github-actions Bot added java runners dataflow labels Aug 5, 2026

scwhittle requested changes Aug 5, 2026

View reviewed changes

github-actions Bot added the Next Action: Reviewers label Aug 5, 2026

stankiewicz force-pushed the tace_logs_stitch branch from 945ffdd to 20804a3 Compare August 5, 2026 13:19

Enable OpenTelemetry stiching with Logs for Dataflow worker, both for…

873e645

… direct logging and file based

stankiewicz force-pushed the tace_logs_stitch branch from 20804a3 to 873e645 Compare August 5, 2026 13:51

stankiewicz requested a review from scwhittle August 5, 2026 13:51

scwhittle requested changes Aug 7, 2026

View reviewed changes

Uh oh!

Conversation

stankiewicz commented Aug 5, 2026

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov Bot commented Aug 5, 2026

Codecov Report

Uh oh!

scwhittle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stankiewicz Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stankiewicz Aug 5, 2026 •

edited

Loading