Destination service span enrichment issue

[hegerchr]

In the following situation, an unexpected behaviour is observed. A web frontend is instrumented with OTel JS for browser. The requests from the browser first hits a Kubernetes ingress controller that routes the requests to the backend services based on the {language} in the URL path /api/{language}/rolldice. The destination service span enrichment sets the IP address as destination service. The result is that in this scenario, the service map cannot properly resolve the backend services.

The ingress controller does URL rewriting like this:

# Java service API
- path: /api/java(/|$)(.*)
  pathType: ImplementationSpecific
  backend:
    service:
      name: java-service
      port:
        number: 80

Is it possible to take the path into account for the destination service?

I have experimented with a custom span processor in the frontend to set the peer.service attribute. If this is set to the value of service.name attribute of the backend service, it works as expected. However, maintaining such a mapping in the frontend might be unfeasible.

[kDuin]

I am also having an issue with service_destination missing, as it breaks the Kibana APM Application dependency view:

Description

After upgrading from Elastic Stack 9.1.4 / EDOT Collector 9.1.0 to Elastic Stack 9.3.1 / EDOT Collector 9.3.1, APM no longer shows dependent services for our applications in Kibana.

The underlying problem appears to be that service_destination metrics stop being generated after span.name is no longer added to span attributes during Elastic enrichment.

Impact

In Kibana APM, dependent services are missing or incomplete because service_destination metrics are no longer produced.

This is a user-visible regression, even though raw exit spans are still present.

What we observed

• Raw exit spans still arrive correctly.
• Other derived metrics are still produced.
• service_destination metrics drop to zero after the upgrade.
• Kibana then cannot show depending/dependent services as expected.

Suspected regression

The strongest candidate is this change in elastic/opentelemetry-lib:

• Commit: 823b73711ec71d213b613fd2134c74cafe4e5269
• Message: Update transaction enrichment to clear the span ID and span Name. Remove span ID and Name enrichment (#254)

From our investigation, older processor/enricher code populated span.name into span attributes, which downstream metric generation relied on.

Once span.name stopped being enriched, service_destination was no longer emitted.

Why this seems to matter

service_destination generation appears to depend on span.name being present in span attributes.

Later changes in elastic/opentelemetry-collector-components seem to support this interpretation:

• be8f12df9c3bb7f001da90cfef7f8a182a6a4c06 - add transaction.root enrichment in connector
• 16187f626e6bc06fceef826d7d9f7fd6567be142 - add span.name enrichment in connector

The latter commit explicitly suggests that span.name had to be restored because it was no longer added by the processor.

Expected behavior

Either:

1. opentelemetry-lib should continue to preserve the enrichment contract required for downstream service_destination generation, or
2. the downstream components and Kibana expectations should be updated in a coordinated way so dependent services do not disappear.

Actual behavior

After the enrichment change, service_destination is no longer generated, and Kibana APM loses dependent service visibility.

Request

Can someone confirm whether removing span.name enrichment in opentelemetry-lib was expected to be a breaking change for APM dependency/service destination metrics?