Fix trace ID collision in DDEntryPoint.continue()

[brianmowen]

When using natchez-datadog with cats-effect in a high-concurrency scenario, multiple unrelated requests can end up with the same trace ID. This causes their spans to appear as siblings in Datadog, making distributed tracing unreliable.

Root Cause

In DDEntryPoint.scala, the root() method correctly uses ignoreActiveSpan():

override def root(name: String, options: Span.Options): Resource[F, Span[F]] = {
  def initSpan(): F[ot.Span] =
    Sync[F]
      .delay(tracer.buildSpan(name).ignoreActiveSpan())  // ✓ Correct
      // ...
}

However, continue() does not use ignoreActiveSpan():

override def continue(name: String, kernel: Kernel, options: Span.Options): Resource[F, Span[F]] = {
  def initSpan(): F[ot.Span] =
    Sync[F]
      .delay {
        val spanContext = tracer.extract(Format.Builtin.HTTP_HEADERS, new TextMapAdapter(kernel.toJava))
        tracer.buildSpan(name).asChildOf(spanContext)  // ✗ Missing ignoreActiveSpan()
      }
      // ...
}

The Datadog tracer uses thread-local storage for the "active span". When continue() is called:

1. If the kernel has no tracing headers (common for external requests), tracer.extract() returns null
2. asChildOf(null) is a no-op for setting the parent from headers
3. Without ignoreActiveSpan(), the tracer falls back to using whatever span is currently "active" on that thread
4. In fiber-based runtimes like cats-effect, fibers don't have a 1:1 mapping to threads
5. A fiber may run on a thread that has another fiber's span as the "active span"
6. Result: The new span inherits the wrong trace ID

Why continueOrElseRoot Doesn't Help

override def continueOrElseRoot(name: String, kernel: Kernel, options: Span.Options): Resource[F, Span[F]] =
  continue(name, kernel, options).flatMap {
    case null => root(name, options)  // Fallback never triggers!
    case span => span.pure[Resource[F, *]]
  }

The fallback to root() only happens if the span itself is null. But continue() always returns a valid span - it just has the wrong parent. The null check doesn't trigger for missing/invalid headers.

Reproduction

We created a minimal reproduction that demonstrates the bug:

test("DDTracer trace ID collision under concurrency") {
  DDTracer.entryPoint[IO] { builder =>
    IO.delay(builder.serviceName("test").build())
  }.use { ep =>
    for {
      traceIds <- Ref.of[IO, List[Option[String]]](List.empty)
      
      // Process 100 concurrent requests
      _ <- (1 to 100).toList.parTraverse { _ =>
        ep.continueOrElseRoot("request", Kernel(Map.empty)).use { span =>
          span.traceId.flatMap(id => traceIds.update(_ :+ id))
        }
      }
      
      ids <- traceIds.get
    } yield {
      val uniqueIds = ids.flatten.distinct
      // BUG: uniqueIds.size is ~20, not 100!
      assertEquals(uniqueIds.size, 100)
    }
  }
}

Results:

• Expected: 100 unique trace IDs
• Actual: ~20 unique trace IDs (varies based on thread scheduling)

Proposed Fix

Add ignoreActiveSpan() to continue() and only set the parent from the extracted spanContext if it's not null:

override def continue(name: String, kernel: Kernel, options: Span.Options): Resource[F, Span[F]] = {
  def initSpan(): F[ot.Span] =
    Sync[F]
      .delay {
        val spanContext = tracer.extract(
          Format.Builtin.HTTP_HEADERS,
          new TextMapAdapter(kernel.toJava)
        )
        val builder = tracer.buildSpan(name).ignoreActiveSpan()  // ← ADD THIS
        if (spanContext != null) builder.asChildOf(spanContext)
        else builder
      }
      .flatTap(addSpanKind(_, options.spanKind))
      .flatMap(options.links.foldM(_)(addLink[F](tracer)))
      .flatMap(builder => Sync[F].delay(builder.start()))

  // ... rest unchanged
}

Environment

• natchez-datadog version: 0.3.8
• Scala version: 2.13.18
• cats-effect version: 3.x
• Runtime: Fiber-based (cats-effect IO)

Impact

This bug affects any application using natchez-datadog with:

• Concurrent request handling
• Requests that don't carry incoming trace headers (external requests)
• Fiber-based runtimes (cats-effect, ZIO via interop, etc.)

[kubukoz]

That looks good, just one nitpick for the tests.

[kubukoz]

question: would a Ref suffice here? It doesn't looks like we're using evalModify or others, here nor in the other test.

[brianmowen]

Definitely, updated here d184575