otel/16: run conformance against Datadog

[zhongkechen]

Stacked on #41.

Summary

• implement Datadog v2 span-search queries with OAuth bearer authentication from DATADOG_ACCESS_TOKEN, JSON:API request envelopes, cursor pagination, correlation discovery, and full-trace retrieval
• normalize real Datadog span-search payloads, including nested custom attributes, 128-bit OpenTelemetry trace IDs, parent IDs, timestamps, span kind, and OpenTelemetry status
• run Datadog beside X-Ray and Dash0 in the Java, Python, and TypeScript OpenTelemetry suites using the hosted OTLP/HTTP protobuf intake
• add HTTP/protobuf exporter support to the Java example while retaining gRPC for Dash0 and the S3 collector
• document Datadog query and CI credential configuration and cover backend, normalization, exporter, and workflow behavior

Stateful long-running workflows remain unchanged because Dash0 does not run there either.

Secrets

• DATADOG_ACCESS_TOKEN authenticates Datadog API queries
• DATADOG_OTLP_HEADERS supplies the Datadog OTLP intake headers

Verification

• hatch run test:all (313 passed)
• hatch run types:check
• hatch fmt --check packages scripts
• hatch run yaml:lint packages/aws-durable-execution-conformance-tests-otel/test-requirements
• parsed all 15 workflow YAML files
• built the Java OpenTelemetry examples with Maven against SDK 2.1.1-SNAPSHOT

[github-actions]

The new query.trace_id discovery branch (@otel.trace_id:"…" search) is the only newly-added code path in _lookup with no test coverage. Every other new path added in this PR — arn-based discovery, cursor pagination, repeated-cursor rejection, _native_trace_id follow-up query, and factory access-token handling — has a dedicated test, but no test constructs a TelemetryQuery with trace_id set for the Datadog backend. As a result a regression in the @otel.trace_id query string, or in how matching_trace/_native_trace_id behave when query.trace_id is populated, would ship undetected. Add a test that drives find_trace with TelemetryQuery(..., trace_id=<128-bit id>) and asserts the first search body's filter.query is @otel.trace_id:"<normalized>" and that the correlated full trace is returned.

[github-actions]

Status precedence prevents UNSET from ever surfacing, and no UNSET_STATUS disparity is declared.

outer.get("status") or attributes.get("otel.status_code") gives Datadog's coarse top-level status priority over the precise OTel status code. Datadog derives the span-level status field from the error flag, so a span that OpenTelemetry marked UNSET arrives here as status: "ok" → normalize_status → "OK", and the more accurate otel.status_code ("Unset") is never consulted. The whole point of introducing the otel.status_code fallback in this PR is to recover UNSET/OK/ERROR fidelity, but this ordering defeats it.

Because DatadogBackend.feature_disparities is only {SPAN_LINKS} (unlike XRayBackend, which also declares UNSET_STATUS), _matches_span_status requires an exact status match. Requirements assert status: UNSET on real spans (e.g. test-requirements/otel-invocation/otel-invocation-18.yaml:136 and the long-running suites). With the Datadog job now running on every push/PR (previously it was workflow_dispatch-only), those UNSET selectors will fail against live Datadog data.

Fix: prefer the OTel code, e.g.

status=normalize_status(
    attributes.get("otel.status_code") or outer.get("status"),
),

or, if Datadog genuinely cannot represent UNSET, add BackendFeatureDisparity.UNSET_STATUS to feature_disparities (matching X-Ray). Either way, add a test covering a payload where both top-level status and otel.status_code are present — the current tests only ever set one, so this precedence path is untested.

[github-actions]

Datadog OTLP endpoint doubles the signal path for the Python/TypeScript community layers.

DATADOG_OTLP_ENDPOINT is passed via --otel-endpoint, which the community exporter profile maps to OTEL_EXPORTER_OTLP_ENDPOINT in the template (with OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf). For the Python (opentelemetry-exporter-otlp-proto-http) and TypeScript (@opentelemetry/exporter-trace-otlp-proto) auto-instrumentation layers, that variable is a base URL to which the SDK appends the signal path /v1/traces. With the value https://otlp.datadoghq.com/v1/traces, spans are exported to https://otlp.datadoghq.com/v1/traces/v1/traces, which Datadog rejects, so no traces are ingested and these jobs fail with telemetry timeouts.

Note the asymmetry: the Java example (examples/java/.../OtelConformanceHandler.java) reads the same env var and passes it to OtlpHttpSpanExporter.builder().setEndpoint(...), which expects the full signal URL — so Java requires the /v1/traces suffix while Python/TS require its absence. The established Dash0 jobs confirm the base-URL convention: DASH0_OTLP_ENDPOINT: https://ingress.us-west-2.aws.dash0.com (no /v1/traces).

Because a single endpoint value can't satisfy both interpretations, either supply the base host (https://otlp.datadoghq.com) to Python/TS and give Java the full path separately, or have the Java handler honor OTel base-URL semantics. This same value appears in .github/workflows/typescript-opentelemetry-suite.yml:245 and .github/workflows/java-opentelemetry-suite.yml:236.

[github-actions]

OTLP export endpoint includes /v1/traces, which the Python/TS community exporters double-append.

DATADOG_OTLP_ENDPOINT is passed to --otel-endpoint, which the community profile writes verbatim into OTEL_EXPORTER_OTLP_ENDPOINT (exporters.py:123; template.yaml:87). With OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf, the OTel SDK treats OTEL_EXPORTER_OTLP_ENDPOINT as a base URL and appends the signal path /v1/traces (unlike OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, which is used as-is). So Python and TypeScript will POST to https://otlp.datadoghq.com/v1/traces/v1/traces, which 404s — no traces are ingested and every Datadog case times out. The existing Dash0 job confirms this convention: it sets the base https://ingress.us-west-2.aws.dash0.com (no path) and the SDK appends /v1/traces.

The same value works for Java only because its handler calls OtlpHttpSpanExporter.setEndpoint(...), which expects the full path and does not append. A single shared value cannot satisfy both mechanisms.

Fix: for Python/TS use the base URL https://otlp.datadoghq.com (or set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to the full path), and keep the full /v1/traces path only for Java's setEndpoint.

[github-actions]

Same OTLP double-path issue as the Python suite.

DATADOG_OTLP_ENDPOINT=https://otlp.datadoghq.com/v1/traces flows through --otel-endpoint into OTEL_EXPORTER_OTLP_ENDPOINT (exporters.py:123; typescript/template.yaml:87-89, http/protobuf). The community JS exporter treats that variable as a base URL and appends /v1/traces, producing .../v1/traces/v1/traces and a 404, so no spans reach Datadog and the run times out. Use the base URL https://otlp.datadoghq.com here (Java keeps the full path because its handler passes it straight to OtlpHttpSpanExporter.setEndpoint).

[github-actions]

OTLP endpoint includes /v1/traces, which the Python http/protobuf exporter appends to again → doubled path, no spans ingested.

--otel-endpoint "$DATADOG_OTLP_ENDPOINT" is mapped to OTEL_EXPORTER_OTLP_ENDPOINT (the general OTLP endpoint) by CommunityExporterProfile.configure (exporters.py:123), and this template hard-codes OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf. For the general endpoint variable, the OpenTelemetry Python HTTP exporter unconditionally appends the signal path /v1/traces, so https://otlp.datadoghq.com/v1/traces becomes https://otlp.datadoghq.com/v1/traces/v1/traces. Exports then fail and the Datadog backend times out finding no correlated trace.

Note the existing Dash0 job supplies a path-less base (https://ingress.us-west-2.aws.dash0.com) precisely because the SDK appends /v1/traces. The Java example is unaffected because its handler calls OtlpHttpSpanExporter.setEndpoint() with the full URL (used verbatim, no path appended) — so the same value is correct for Java but wrong here.

Fix: use the base URL for the Python (and TypeScript) jobs:

Suggested change

	DATADOG_OTLP_ENDPOINT: https://otlp.datadoghq.com/v1/traces
	DATADOG_OTLP_ENDPOINT: https://otlp.datadoghq.com

(or set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead, which is used as-is).

[github-actions]

OTLP endpoint includes /v1/traces, which the Node http/protobuf exporter appends to again → doubled path, no spans ingested.

--otel-endpoint "$DATADOG_OTLP_ENDPOINT" is mapped to OTEL_EXPORTER_OTLP_ENDPOINT (exporters.py:123) and this template hard-codes OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf. The @opentelemetry/exporter-trace-otlp-proto exporter appends /v1/traces to the general endpoint, so https://otlp.datadoghq.com/v1/traces resolves to https://otlp.datadoghq.com/v1/traces/v1/traces, exports fail, and the backend times out. The Dash0 job uses a path-less base for this reason; the Java handler is unaffected because it calls setEndpoint() with the full URL verbatim.

Fix: use the base URL for the TypeScript job:

Suggested change

	DATADOG_OTLP_ENDPOINT: https://otlp.datadoghq.com/v1/traces
	DATADOG_OTLP_ENDPOINT: https://otlp.datadoghq.com

(or set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, which is used as-is).

[github-actions]

Codex AI review

• [P1] Preserve key-based redaction for truncated bodies — packages/aws-durable-execution-conformance-tests-otel/src/aws_durable_execution_conformance_tests_otel/backends/_common.py:44: Truncation makes otherwise valid JSON unparsable, so redact() receives a string and does not redact values under keys such as token or api_key. This can expose server-returned credentials in CI diagnostics. Omit unparsable bodies or sanitize them with key-aware redaction; add a truncated JSON secret test.

• [P1] Unwrap Datadog's nested custom-attribute envelope — packages/aws-durable-execution-conformance-tests-otel/src/aws_durable_execution_conformance_tests_otel/backends/datadog.py:57: Datadog v2 responses can place custom fields under data[].attributes.attributes.custom. Flattening attributes directly produces names like custom.durable.execution.arn and custom.otel.trace_id, so correlation fails and Datadog polling times out. Flatten the nested custom object as root attributes while retaining support for the direct shape, and test the API-schema shape.

Reviewed commit eb64e8549865cf2e93bd49e91ca52db28dbd8f05. Workflow run

[github-actions]

Claude AI review

I reviewed only the PR diff (backend/normalizer/polling/redaction changes plus the three suite workflows, Java exporter/template, docs, and tests) against the base revision. The Python-side implementation is careful and well unit-tested (pagination cursor loops, retryable 429 handling, redacted HTTP error bodies, nested-attribute flattening, 128-bit trace IDs, status normalization). Unit coverage is good. I found one integration-level correctness issue and two residual risks worth noting.

Finding: hardcoded OTLP endpoint includes /v1/traces, which double-appends for the Python and TypeScript community layers

• .github/workflows/python-opentelemetry-suite.yml:188 — DATADOG_OTLP_ENDPOINT: https://otlp.datadoghq.com/v1/traces
• .github/workflows/typescript-opentelemetry-suite.yml:245 — DATADOG_OTLP_ENDPOINT: https://otlp.datadoghq.com/v1/traces

Both values are passed via --otel-endpoint, which the community exporter maps to the OTEL_EXPORTER_OTLP_ENDPOINT environment variable in examples/python/template.yaml and examples/typescript/template.yaml (both hardcode OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf). Per the OTLP spec, OTEL_EXPORTER_OTLP_ENDPOINT is a base URL and the exporter appends the signal path /v1/traces; only OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is treated as the full path. So the auto-configured community layer will POST to https://otlp.datadoghq.com/v1/traces/v1/traces, no spans will be ingested, and every Python/TypeScript Datadog case will time out in find_trace and fail — defeating the job's purpose.

Two independent signals corroborate this:

• The existing Dash0 jobs pass a base URL with no /v1/traces suffix (https://ingress.us-west-2.aws.dash0.com) into the same variable, relying on the exporter to append the path.
• Java is unaffected precisely because its handler (OtelConformanceHandler.createExporter) calls OtlpHttpSpanExporter.builder().setEndpoint(otlpEndpoint), whose contract requires the full path including /v1/traces. The single shared endpoint value is correct for Java's explicit setEndpoint but wrong for the env-var-driven Python/TS layers.

Fix: for the Python and TypeScript suites, drop the path (DATADOG_OTLP_ENDPOINT: https://otlp.datadoghq.com) so the layer appends /v1/traces, or deliver it via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead. Java can keep the full-path value. Because the value now differs per language, this can't stay a single shared constant.

Residual test risk (could not be validated without a live Datadog run)

• No integration proof for the endpoint change. The PR's verification (hatch run test:all, type/format/lint, Maven build) exercises only unit-level parsing and workflow-text assertions; no test would catch the double-/v1/traces above, and test_datadog_jobs_run_beside_dash0_and_use_separate_credentials only asserts the literal endpoint string, not its transport semantics. The PR description confirms Datadog was not actually run end-to-end.
• Status precedence in normalize_datadog (backends/datadog.py). status=normalize_status(outer.get("status") or attributes.get("otel.status_code")) prefers Datadog's derived top-level status over the faithful OTel otel.status_code. If a real span-search payload sets a top-level status: "ok" for a span whose true OTel status is UNSET, it normalizes to OK; since DatadogBackend does not declare BackendFeatureDisparity.UNSET_STATUS, requirements expecting UNSET (e.g. otel-invocation-2/3/9/…) would fail. The added tests only cover payloads where the top-level status is absent, so this precedence is unverified against real data; preferring otel.status_code when present would be safer.

Reviewed commit eb64e8549865cf2e93bd49e91ca52db28dbd8f05. Workflow run