telemetry: output_attributes drops Usage fields beyond input/output tokens

[Kamilbenkirane]

Summary

telemetry.output_attributes(output) currently emits only gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. Every other field on the typed Usage object is dropped before reaching the span.

Current state

src/celeste/telemetry.py:125-136:

def output_attributes(output: Output[Any]) -> dict[str, Any]:
    attrs: dict[str, Any] = {}
    input_tokens = getattr(output.usage, "input_tokens", None)
    output_tokens = getattr(output.usage, "output_tokens", None)
    if isinstance(input_tokens, int):
        attrs["gen_ai.usage.input_tokens"] = input_tokens
    if isinstance(output_tokens, int):
        attrs["gen_ai.usage.output_tokens"] = output_tokens
    if output.finish_reason is not None and output.finish_reason.reason is not None:
        attrs["gen_ai.response.finish_reasons"] = (output.finish_reason.reason,)
    return attrs

Gap

Usage field (per modality subclass)	Emitted today	Relevant GenAI semconv attribute
input_tokens	yes	gen_ai.usage.input_tokens
output_tokens	yes	gen_ai.usage.output_tokens
total_tokens (TextUsage)	no	(no standard; derive from input + output)
reasoning_tokens (TextUsage)	no	gen_ai.usage.reasoning_tokens
Modality-specific fields (ImageUsage, VideoUsage, AudioUsage, EmbeddingsUsage)	no	varies by modality

Provider extras that pass through provider mixin parsing but never reach output_attributes:

• Anthropic / Gemini / OpenAI cached-prompt token counts (semconv: gen_ai.usage.cached_input_tokens)
• Anthropic cache-creation tokens
• Reasoning-token splits where providers report them separately

Why it matters

Spans on Cloud Trace / Honeycomb / similar carry a strict subset of what callers can read via output.usage. Cost analysis, cache-hit ratios, and reasoning-vs-output token ratios cannot be answered from telemetry alone — they require reading output.usage directly. The typed surface and the telemetry emission have drifted.

Out of scope here

Fixing this is not part of # (the streaming-vs-non-streaming parity for input_tokens / output_tokens). That change keeps the existing two-attribute set and only ensures the streaming path emits them too. Widening the attribute set is a separate change with its own decisions:

• which GenAI semconv attributes to adopt (some are recent / experimental)
• how to handle modality-specific Usage fields that semconv does not cover
• whether to emit total_tokens despite no semconv standard

[Kamilbenkirane]

This issue is being bundled into a broader V2 telemetry change covering:

1. Usage fields beyond input/output (this issue) — total_tokens, reasoning_tokens, cached_input_tokens widened on output_attributes, plus modality-specific fields under celeste.usage.*.
2. GenAI metric histograms — gen_ai.client.token.usage and gen_ai.client.operation.duration for sampling-resilient dashboards.
3. Opt-in content events — gen_ai.input.messages / gen_ai.output.messages gated behind OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. Multimodal artifacts captured by URL reference, never inline bytes.

The PR for this bundled change will close this issue on merge.