feat(telemetry): widen usage attrs, add metrics histograms, opt-in content events#273
Merged
Merged
Conversation
…ntent events V2 telemetry expansion bundling three improvements on top of #270/#272: 1. Span attributes: output_attributes() now emits gen_ai.usage.total_tokens, reasoning_tokens, and cached_input_tokens when the typed Usage carries them. Off-spec modality fields fall through to celeste.usage.<field>. Adds the missing cached_tokens field to TextUsage so the data already produced by anthropic / openai / cohere / deepseek / openresponses providers actually reaches the span (previously dropped silently). Extends Google Gemini and chatcompletions provider mixins to surface cached-prompt tokens too. 2. Metrics: registers two GenAI semconv histograms — gen_ai.client.token.usage (one record per token category via gen_ai.token.type dimension) and gen_ai.client.operation.duration (with error.type on failures). Sampling- resilient token-rate and latency dashboards no longer require span scanning. Wired into _predict and _TracedStream._finalize. 3. Content events: opt-in via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT env flag, default off for PII safety. When enabled, spans carry gen_ai.input.messages and gen_ai.output.messages events with the semconv {role, parts: [...]} shape. Text + reasoning + tool calls inline; multimodal artifacts (image/audio/video/document) by URL reference, never inline bytes. 19 new tests across unit_tests/test_telemetry_metrics.py and test_telemetry_content_events.py, plus extended attribute coverage in test_telemetry_streaming.py. 630 unit tests pass. Closes #271.
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
…ed finalize Three small cleanups on top of the V2 telemetry expansion, no behavior change: 1. `_GEN_AI_USAGE_FIELDS` and `_GEN_AI_TOKEN_TYPES` now key on `UsageField` enum members instead of bare strings. Catches typos at import; aligns with the vocabulary providers already populate. 2. New public `add_input_event(span, inputs)` and `add_output_event(span, output)` helpers replace the cross-module calls to private `_input_messages_event` / `_output_messages_event`. Underscore-prefixed names stay as the dict-builder primitives; the public helpers handle `if event is not None: span.add_event(...)`. 3. New `record_output(span, output, attributes, duration, error=None)` extracts the four-step finalize sequence (set output_attributes + emit content event + record token usage + record duration) shared by `_predict` and `_TracedStream._finalize`. Also: drop dead `hasattr(type(usage), "model_fields")` guards (Usage is always Pydantic BaseModel by io.py contract); skip `bool` in the numeric usage iter (bool is a subclass of int — paranoid guard against ever populating a Usage boolean by accident). 630 tests still pass.
…e inner Stream's The previous `_TracedStream.__iter__` delegated to `iter(self._inner)`, which makes Python iterate the inner Stream directly via its blocking-portal `__iter__`. That bypasses the wrapper's `__anext__` — so for sync streaming consumers, the span never finalized, TTFC was never recorded, and metrics were never emitted. Fix: `_TracedStream.__iter__` now spins its own portal and drives `self.__anext__` (the wrapper's), mirroring `Stream.__iter__`. On exhaustion, exception, or generator-close (consumer break / GC), the finally block calls `self.aclose()` which finalizes the span via the existing async path. Caught by smoke-testing `celeste.text.sync.stream.generate(...)` against a real Groq llama-3.1-8b-instant call: previously the span was missing from finished spans entirely; now it appears alongside the async-streaming span with the same attribute set.
…flake Three review agents flagged the +800 LOC PR for: triplicated test stream classes, redundant fixtures, parametrizable repetition, and one Windows-fragile timing assertion. Applied: - New `tests/unit_tests/_telemetry_helpers.py` holds the canonical `TelemetryUsage` / `TelemetryOutput` / `TelemetryStream` / `async_iter` used by all three telemetry test files. - New `tests/unit_tests/conftest.py` exposes the shared `exporter` fixture + `start_test_span` helper. - `test_telemetry_metrics.py` parametrizes `record_token_usage` (3 cases) and `record_operation_duration` (success / failure). Drops the redundant `test_input_and_output_recorded_separately` (subsumed by the parametrized `all_token_types` case). - `test_telemetry_metrics.py:191` now asserts `>= 0` on duration sum — Windows `time.monotonic()` resolution can return 0 for sub-millisecond in-memory streams; the previous `> 0` failed CI on Windows runners. - `telemetry.py`: drop `add_output_event` (only ever called by `record_output` — inlined). Trim verbose comments above `_GEN_AI_USAGE_FIELDS`, `_GEN_AI_TOKEN_TYPES`, `_CAPTURE_CONTENT`, and the two messages-event helpers down to one line each. Net: -210 modified LOC, +72 new helper LOC. PR diff drops from +861 to ~+725. Coverage unchanged (630 tests pass).
…xcept + import time from client User pushed back on `_predict` wrapping its entire body in try/except just to record `gen_ai.client.operation.duration` on the error path. The right home for that is a context manager inside `telemetry.py`. `gen_ai_span(model=, provider=, protocol=, modality=)` opens the span via `tracer.start_as_current_span`, captures the start time on enter, and in `finally` records the operation duration with `error.type` populated when the body raised. It yields `(span, request_attrs)` so `_predict` can call `add_input_event` and `record_output` against them. Result: - `_predict` is straight-line code: no try/except, no manual time math. - `client.py` no longer imports `time`. - `record_output` no longer takes `duration_seconds` / `error` params — duration is the span's lifecycle concern, not the output recorder's. - Streaming side (`_TracedStream._finalize`) updates symmetrically: records duration directly, then calls the simplified `record_output`. Real-call validated against gemini-3.1-flash-lite-preview text non-streaming + async streaming: same span attributes, same metric histograms, same content events as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
V2 GenAI telemetry expansion on top of #270 and #272. Closes #271.
Three improvements bundled — all in
telemetry.pyplus minimal provider mixin updates:output_attributes()now emits the full GenAI semconv set when populated:gen_ai.usage.total_tokens,gen_ai.usage.reasoning_tokens,gen_ai.usage.cached_input_tokens. Off-spec modality fields (e.g.images_generated,audio_seconds) fall through toceleste.usage.<field>so they're queryable without pollutinggen_ai.*.gen_ai.client.token.usage(withgen_ai.token.typedimension:input/output/reasoning/cached_input) andgen_ai.client.operation.duration(witherror.typeon failures). Sampling-resilient — dashboards aren't broken by span sampling.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=trueenablesgen_ai.input.messages/gen_ai.output.messagesevents on spans. Default off (PII safety). Text + reasoning + tool calls inline; image/audio/video/document artifacts as{type, uri, mime_type}URL references — never inline bytes.Provider mixin updates
TextUsage.cached_tokenswas missing — providers (anthropic, cohere, deepseek, openai responses) were already producing the data viaUsageField.CACHED_TOKENS, but it was silently dropped atTextUsage(**raw)since the field didn't exist. Adding the field flows the data through tooutput_attributes, which emits it as the semconvgen_ai.usage.cached_input_tokens. Also extends Googlegenerate_contentand OpenAIchatcompletionsmixins to surfacecachedContentTokenCount/prompt_tokens_details.cached_tokens.Behavior
[otel]extra not installed): all paths route through existing_NoOp*stand-ins (now_NoOpMeter/_NoOpHistogramadded). Zero overhead, zero behavior change.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true: content events emit too.Out of scope
incubatingstatus that may shift.Test plan
uv run ruff check src tests— cleanuv run ruff format --check src tests— cleanuv run mypy src/celesteanduv run mypy tests/— both cleanuv run pytest tests/unit_tests/ -m "not integration" -q— 630 passed (611 existing + 19 new)