feat(telemetry): GenAI OpenTelemetry spans on _predict and _stream#270
Merged
Conversation
celeste-python now emits GenAI v1.41 semantic-convention spans on every ModalityClient._predict and ._stream call when a TracerProvider is configured. No changes to public API; default installs gain zero new deps; spans no-op when no provider is set. - New src/celeste/telemetry.py: lazy OTel import + no-op fallback, provider/modality maps, request_attributes, span_name, output_attributes, trace_stream async generator - client.py _predict: wrap body in tracer.start_as_current_span; OTel's context manager handles exception/lifecycle - client.py _stream: detached span + use_span(end_on_exit=True) inside the iterator wrapper; records time_to_first_chunk on first chunk - pyproject: optional [otel] extra adding opentelemetry-api>=1.30; per-file ANN401 ignore for telemetry.py (mirrors OTel API surface) Closes #269.
Add a 'Tracing with OpenTelemetry' README section covering: - uv add 'celeste-ai[otel]' - minimal TracerProvider + OTLP/Console exporter snippet - expected span name and attribute schema - OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental opt-in - attribute drift caveat until GenAI semconv stabilizes
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
CI runs mypy without the [otel] extra installed, so 'from opentelemetry import trace' fails the type check even though the runtime import is guarded by try/except. Mirror the existing pattern used for httpx / httpx_sse / google.* / etc.
This reverts commit b87c05d.
…y ignore The previous fix (`ignore_missing_imports` for opentelemetry.*) was a workaround that disabled real type-checking on OTel symbols. The opentelemetry-api package ships py.typed and is fully type-checkable when installed. Add opentelemetry-api to the dev dependency group so CI's mypy sees the real types. Drop the ignore_missing_imports override.
Earlier commit moved opentelemetry-api into the dev group as a workaround. Cleaner: mirror the existing 'extras-per-job' pattern (the test job already does `uv sync --extra gcp`). The type-check job now `uv sync --extra otel` so mypy sees the real OTel types. Reverts the dev-group addition.
4 tasks
Kamilbenkirane
added a commit
that referenced
this pull request
May 5, 2026
…ntent events (#273) * feat(telemetry): widen usage attrs, add metrics histograms, opt-in content events V2 telemetry expansion bundling three improvements on top of #270/#272: 1. Span attributes: output_attributes() now emits gen_ai.usage.total_tokens, reasoning_tokens, and cached_input_tokens when the typed Usage carries them. Off-spec modality fields fall through to celeste.usage.<field>. Adds the missing cached_tokens field to TextUsage so the data already produced by anthropic / openai / cohere / deepseek / openresponses providers actually reaches the span (previously dropped silently). Extends Google Gemini and chatcompletions provider mixins to surface cached-prompt tokens too. 2. Metrics: registers two GenAI semconv histograms — gen_ai.client.token.usage (one record per token category via gen_ai.token.type dimension) and gen_ai.client.operation.duration (with error.type on failures). Sampling- resilient token-rate and latency dashboards no longer require span scanning. Wired into _predict and _TracedStream._finalize. 3. Content events: opt-in via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT env flag, default off for PII safety. When enabled, spans carry gen_ai.input.messages and gen_ai.output.messages events with the semconv {role, parts: [...]} shape. Text + reasoning + tool calls inline; multimodal artifacts (image/audio/video/document) by URL reference, never inline bytes. 19 new tests across unit_tests/test_telemetry_metrics.py and test_telemetry_content_events.py, plus extended attribute coverage in test_telemetry_streaming.py. 630 unit tests pass. Closes #271. * refactor(telemetry): UsageField enum keys, public event helpers, shared finalize Three small cleanups on top of the V2 telemetry expansion, no behavior change: 1. `_GEN_AI_USAGE_FIELDS` and `_GEN_AI_TOKEN_TYPES` now key on `UsageField` enum members instead of bare strings. Catches typos at import; aligns with the vocabulary providers already populate. 2. New public `add_input_event(span, inputs)` and `add_output_event(span, output)` helpers replace the cross-module calls to private `_input_messages_event` / `_output_messages_event`. Underscore-prefixed names stay as the dict-builder primitives; the public helpers handle `if event is not None: span.add_event(...)`. 3. New `record_output(span, output, attributes, duration, error=None)` extracts the four-step finalize sequence (set output_attributes + emit content event + record token usage + record duration) shared by `_predict` and `_TracedStream._finalize`. Also: drop dead `hasattr(type(usage), "model_fields")` guards (Usage is always Pydantic BaseModel by io.py contract); skip `bool` in the numeric usage iter (bool is a subclass of int — paranoid guard against ever populating a Usage boolean by accident). 630 tests still pass. * fix(telemetry): sync iteration drives the wrapper's __anext__, not the inner Stream's The previous `_TracedStream.__iter__` delegated to `iter(self._inner)`, which makes Python iterate the inner Stream directly via its blocking-portal `__iter__`. That bypasses the wrapper's `__anext__` — so for sync streaming consumers, the span never finalized, TTFC was never recorded, and metrics were never emitted. Fix: `_TracedStream.__iter__` now spins its own portal and drives `self.__anext__` (the wrapper's), mirroring `Stream.__iter__`. On exhaustion, exception, or generator-close (consumer break / GC), the finally block calls `self.aclose()` which finalizes the span via the existing async path. Caught by smoke-testing `celeste.text.sync.stream.generate(...)` against a real Groq llama-3.1-8b-instant call: previously the span was missing from finished spans entirely; now it appears alongside the async-streaming span with the same attribute set. * refactor: dedupe telemetry test fixtures, fix Windows time.monotonic flake Three review agents flagged the +800 LOC PR for: triplicated test stream classes, redundant fixtures, parametrizable repetition, and one Windows-fragile timing assertion. Applied: - New `tests/unit_tests/_telemetry_helpers.py` holds the canonical `TelemetryUsage` / `TelemetryOutput` / `TelemetryStream` / `async_iter` used by all three telemetry test files. - New `tests/unit_tests/conftest.py` exposes the shared `exporter` fixture + `start_test_span` helper. - `test_telemetry_metrics.py` parametrizes `record_token_usage` (3 cases) and `record_operation_duration` (success / failure). Drops the redundant `test_input_and_output_recorded_separately` (subsumed by the parametrized `all_token_types` case). - `test_telemetry_metrics.py:191` now asserts `>= 0` on duration sum — Windows `time.monotonic()` resolution can return 0 for sub-millisecond in-memory streams; the previous `> 0` failed CI on Windows runners. - `telemetry.py`: drop `add_output_event` (only ever called by `record_output` — inlined). Trim verbose comments above `_GEN_AI_USAGE_FIELDS`, `_GEN_AI_TOKEN_TYPES`, `_CAPTURE_CONTENT`, and the two messages-event helpers down to one line each. Net: -210 modified LOC, +72 new helper LOC. PR diff drops from +861 to ~+725. Coverage unchanged (630 tests pass). * refactor(telemetry): extract gen_ai_span context manager — drop try/except + import time from client User pushed back on `_predict` wrapping its entire body in try/except just to record `gen_ai.client.operation.duration` on the error path. The right home for that is a context manager inside `telemetry.py`. `gen_ai_span(model=, provider=, protocol=, modality=)` opens the span via `tracer.start_as_current_span`, captures the start time on enter, and in `finally` records the operation duration with `error.type` populated when the body raised. It yields `(span, request_attrs)` so `_predict` can call `add_input_event` and `record_output` against them. Result: - `_predict` is straight-line code: no try/except, no manual time math. - `client.py` no longer imports `time`. - `record_output` no longer takes `duration_seconds` / `error` params — duration is the span's lifecycle concern, not the output recorder's. - Streaming side (`_TracedStream._finalize`) updates symmetrically: records duration directly, then calls the simplified `record_output`. Real-call validated against gemini-3.1-flash-lite-preview text non-streaming + async streaming: same span attributes, same metric histograms, same content events as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Native OpenTelemetry GenAI v1.41.0 spans on every
ModalityClient._predictand_streamcall. Provider-SDK instrumentation packages (OpenInference, openllmetry, opentelemetry-instrumentation-openai-v2) cannot reach Celeste calls because Celeste bypasses provider SDKs; the right home for native LLM-semantic spans is celeste-python itself.Closes #206.
What changed
src/celeste/telemetry.py(~170 LOC) with lazy OTel import + private no-op fallback, GenAI provider/modality enum maps, and four pure helpers (request_attributes,span_name,output_attributes,trace_stream).src/celeste/client.py:ModalityClient._predict— body wrapped inwith telemetry.tracer.start_as_current_span(...) as span:. OTel's context manager auto-handles exception recording, status, and span end. Output attributes set on success before return.src/celeste/client.py:ModalityClient._stream— detached span viatracer.start_span(...), thentelemetry.trace_stream(sse_iterator, span)wraps the SSE iterator withuse_span(span, end_on_exit=True)for clean lifecycle (completion / exception /GeneratorExitabandonment, no try/except in our code).pyproject.toml— optional[otel]extra (opentelemetry-api>=1.30); per-fileANN401ignore for**/telemetry.pymirroring the existing**/client.pyignore (forwarding shims needAny)..github/workflows/ci.yml— type-check job installs[otel]extra so mypy sees the real OTel types (mirrors the existing[gcp]install in the test job).Verified — real Gemini call
{ "name": "chat gemini-3.1-flash-lite-preview", "attributes": { "gen_ai.request.model": "gemini-3.1-flash-lite-preview", "gen_ai.provider.name": "gcp.gemini", "gen_ai.operation.name": "chat", "gen_ai.usage.input_tokens": 13, "gen_ai.usage.output_tokens": 1, "gen_ai.response.finish_reasons": ["STOP"] } }Out of scope (future)
gen_ai.input.messages/output.messages, env-var gated).gen_ai.client.token.usage,gen_ai.client.operation.duration).operation: Operationkwarg on_predict/_streamfor off-spec span fidelity (image edit vs generate, audio speak, video edit).Behavior matrix
celeste-ai(no extra)celeste-ai[otel]ProxyTracerreturnsNonRecordingSpan. Zero overhead.celeste-ai[otel]gen_ai.*attributes per spec.Test plan
uv sync --all-extrasuv run mypy -p celesteclean (333 source files)uv run ruff check src/celestecleanuv run pytest tests/unit_tests/ -m "not integration"— 604 passedConsoleSpanExporter(output above)