diff --git a/README.md b/README.md index 5fadd79..b667a92 100644 --- a/README.md +++ b/README.md @@ -15,12 +15,12 @@ content by default. - OpenTelemetry GenAI spans: `gen_ai.chat`, `gen_ai.embeddings`, `gen_ai.execute_tool`, `gen_ai.agent`, and `gen_ai.workflow` - Provider adapters for OpenAI chat, OpenAI-compatible APIs, Anthropic, Cohere, AWS Bedrock, Google Vertex, and Ollama -- Promptfoo result conversion with run, case, dataset, score, assertion, provenance, and evidence metadata +- Framework adapters for Promptfoo, RAGAS, and DeepEval with run, case, dataset, score, provenance, and evidence metadata - RAG scoring for context precision, recall, faithfulness, MRR, NDCG, citation coverage, top-k relevance, and context-token use - Privacy controls for opt-in content capture, redaction-to-string, redaction-to-fingerprint, truncation flags, and event caps - A versioned `eval2otel.v1` contract backed by conformance fixtures - Operational telemetry about Eval2Otel itself: conversion count, warnings, dropped events, redactions, truncations, and duration -- A Python contract scaffold for teams that want the same Eval2Otel payload shape outside TypeScript +- A Python SDK preview with optional OTLP spans, content events, PII redaction, and provider instrumentation hooks ## Install @@ -164,12 +164,16 @@ Supported adapter modes: Every adapter result includes structured warnings and raw payload evidence hashes, so conversion failures can be reported without dumping raw payloads. -## Promptfoo Adapter +## Framework Adapters Promptfoo results can be converted directly into Eval2Otel results: ```ts -import { convertPromptfooToEvalResults } from 'eval2otel'; +import { + convertDeepEvalToEvalResults, + convertPromptfooToEvalResults, + convertRagasToEvalResults, +} from 'eval2otel'; const { evalResults, warnings } = convertPromptfooToEvalResults(promptfooJson, { runId: 'promptfoo-nightly', @@ -191,6 +195,31 @@ The adapter preserves Promptfoo success, score, assertion counts, failed assertion warnings, metric names, run identity, case identity, and payload hashes. +RAGAS and DeepEval exports use the same conversion shape: + +```ts +const ragas = convertRagasToEvalResults(ragasJson, { + runId: 'ragas-nightly', + datasetId: 'rag-evals', + defaultModel: 'gpt-4o-mini', +}); + +const deepeval = convertDeepEvalToEvalResults(deepevalJson, { + runId: 'deepeval-nightly', + includeExplanations: true, + defaultModel: 'gpt-4o-mini', +}); + +for (const result of [...ragas.evalResults, ...deepeval.evalResults]) { + eval2otel.processEvaluation(result); +} +``` + +RAGAS rows populate RAG metrics such as context precision, context recall, +answer relevance, and faithfulness. DeepEval rows preserve metric scores, +failed-metric warnings, expected-output fingerprints, and retrieval context as +RAG chunk evidence. + ## RAG Telemetry RAG payloads can include retrieval inputs, chunk metadata, explicit eval scores, @@ -338,23 +367,25 @@ Each provider-native line should look like: {"startTime":1725170000000,"endTime":1725170001200,"request":{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]},"response":{"id":"chatcmpl-1","object":"chat.completion","model":"gpt-4o-mini","choices":[{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}} ``` -## Python Scaffold +## Python SDK Preview -The TypeScript package is the production emitter today. The Python directory -contains a small contract-first scaffold with the same `eval2otel.v1` -provenance/evidence/report vocabulary: +The Python package mirrors the `eval2otel.v1` provenance, evidence, and +conversion-report vocabulary. It can run contract-only with no OpenTelemetry +dependency, or emit real spans when the optional OTel extras are installed: ```bash +pip install -e "python[otel]" PYTHONPATH=python python3 -m unittest discover -s python/tests ``` ```python -from eval2otel import Eval2Otel +from eval2otel import instrument_all -client = Eval2Otel(service_name="pytest", semconv_version="1.37.0") +client = instrument_all() report = client.process_evaluation({ "id": "py-case-1", "model": "gpt-4o-mini", + "system": "openai", "operation": "chat", "request": {"model": "gpt-4o-mini"}, "response": {}, @@ -364,6 +395,18 @@ report = client.process_evaluation({ assert report.contract_version == "eval2otel.v1" ``` +`instrument_all()` honors the common OTLP environment variables plus: + +- `EVAL2OTEL_SERVICE_NAME` +- `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` +- `EVAL2OTEL_SAMPLE_RATE` +- `EVAL2OTEL_REDACT_PII` +- `EVAL2OTEL_PROVIDERS` + +Provider hooks are optional. If provider packages and compatible +OpenLLMetry/OpenInference instrumentors are installed, Eval2Otel invokes them; +otherwise it returns structured handles explaining what was available. + See [python/README.md](./python/README.md). ## Configuration diff --git a/docs/contract/eval2otel-v1.md b/docs/contract/eval2otel-v1.md index 672eee8..c617544 100644 --- a/docs/contract/eval2otel-v1.md +++ b/docs/contract/eval2otel-v1.md @@ -43,6 +43,7 @@ attributes intentionally aligned with: exists yet; - Eval2Otel-owned `evalops.*` contract, privacy, provenance, and evidence attributes; +- framework-specific `eval.*` diagnostic attributes; - provider-prefixed diagnostic attributes such as `openai.*`, `anthropic.*`, and `google.vertex.*`. @@ -131,8 +132,15 @@ Provider-native adapters should return a `ProviderConversionResult` with: - structured warnings; - evidence containing at least `rawPayloadSha256`. -Promptfoo conversion additionally emits `eval.promptfoo.*` attributes for pass -state, score, assertion counts, failed assertion counts, and metric names. +Framework adapters additionally emit namespaced `eval.*` attributes: + +- Promptfoo: `eval.promptfoo.*` for pass state, score, assertion counts, failed + assertion counts, and metric names. +- RAGAS: `eval.ragas.*` for source metric values, metric names, and reference + fingerprints. Shared RAG quality fields are also copied into + `EvalResult.rag.metrics`. +- DeepEval: `eval.deepeval.*` for pass state, failed metric counts, normalized + metric scores, metric names, and expected-output fingerprints. ## Operational Telemetry diff --git a/python/README.md b/python/README.md index e136dd5..e4cabec 100644 --- a/python/README.md +++ b/python/README.md @@ -1,13 +1,18 @@ -# eval2otel Python Scaffold +# eval2otel Python SDK Preview -This directory is a contract-first Python SDK scaffold. It mirrors the -`eval2otel.v1` provenance, evidence, and conversion-report vocabulary used by -the TypeScript package without adding a Python OpenTelemetry dependency yet. +The Python package mirrors the TypeScript `eval2otel.v1` contract and can also +emit OpenTelemetry spans when the optional OTel extras are installed. Without +those extras, it still validates Eval2Otel payloads and returns conversion +reports. + +```bash +pip install -e ".[otel]" +``` ```python -from eval2otel import Eval2Otel +from eval2otel import instrument_all -client = Eval2Otel(service_name="eval-runner", semconv_version="1.37.0") +client = instrument_all() report = client.process_evaluation({ "id": "case-1", "timestamp": 1700000000000, @@ -15,9 +20,15 @@ report = client.process_evaluation({ "system": "openai", "operation": "chat", "request": {"model": "gpt-4o-mini"}, - "response": {}, - "usage": {}, + "response": {"model": "gpt-4o-mini"}, + "usage": {"inputTokens": 12, "outputTokens": 8}, "performance": {"duration": 0.25}, + "conversation": { + "messages": [ + {"role": "user", "content": "What shipped?"}, + {"role": "assistant", "content": "Eval2Otel Python OTLP hooks shipped."} + ] + }, "provenance": { "sourceFramework": "deepeval", "runId": "nightly", @@ -26,7 +37,50 @@ report = client.process_evaluation({ }) assert report.contract_version == "eval2otel.v1" +client.shutdown() ``` -The intended next step is a real OTLP/OpenTelemetry emitter behind this same -API, plus framework adapters for Python-native eval runners. +## Environment + +`instrument_all()` reads: + +- `OTEL_SERVICE_NAME` or `EVAL2OTEL_SERVICE_NAME` +- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` +- `OTEL_EXPORTER_OTLP_PROTOCOL` +- `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` +- `EVAL2OTEL_SAMPLE_RATE` +- `EVAL2OTEL_REDACT_PII` +- `EVAL2OTEL_PROVIDERS` + +Content capture is off by default. When +`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`, message content is +emitted as span events and sampled by `EVAL2OTEL_SAMPLE_RATE`. When +`EVAL2OTEL_REDACT_PII=true`, the built-in redactor masks common emails, +bearer tokens, secret assignments, and long number sequences before content is +emitted. + +## Provider Hooks + +`instrument_all()` returns `client.instrumentation_handles` when provider +patching is enabled. Each handle reports whether the provider package was +available, whether a compatible instrumentor was invoked, and the reason when it +could not be instrumented. + +Supported provider names: + +- `openai` +- `anthropic` +- `google-generativeai` +- `bedrock` +- `cohere` +- `huggingface` + +Set `EVAL2OTEL_PROVIDERS=openai,anthropic` to limit discovery. + +## Development + +From the repository root: + +```bash +PYTHONPATH=python python3 -m unittest discover -s python/tests +``` diff --git a/python/eval2otel/__init__.py b/python/eval2otel/__init__.py index 8976453..af8e337 100644 --- a/python/eval2otel/__init__.py +++ b/python/eval2otel/__init__.py @@ -9,8 +9,20 @@ Eval2OtelProvenance, EvalResult, build_eval2otel_attributes, + build_span_attributes, sha256_payload, ) +from .instrumentations import ( + ProviderInstrumentationHandle, + instrument_all_providers, + instrument_anthropic, + instrument_bedrock, + instrument_cohere, + instrument_google_generativeai, + instrument_huggingface, + instrument_openai, +) +from .privacy import redact_pii __all__ = [ "EVAL2OTEL_CONTRACT_VERSION", @@ -22,6 +34,16 @@ "Eval2OtelProvenance", "EvalResult", "build_eval2otel_attributes", + "build_span_attributes", "instrument_all", + "instrument_all_providers", + "instrument_anthropic", + "instrument_bedrock", + "instrument_cohere", + "instrument_google_generativeai", + "instrument_huggingface", + "instrument_openai", + "ProviderInstrumentationHandle", + "redact_pii", "sha256_payload", ] diff --git a/python/eval2otel/auto.py b/python/eval2otel/auto.py index 37025dc..95e66c1 100644 --- a/python/eval2otel/auto.py +++ b/python/eval2otel/auto.py @@ -1,22 +1,68 @@ from __future__ import annotations +import os +from typing import Callable, Iterable + from .contract import Eval2Otel +from .instrumentations import instrument_all_providers +from .privacy import redact_pii def instrument_all( - service_name: str, + service_name: str | None = None, service_version: str | None = None, semconv_version: str = "unspecified", + capture_content: bool | None = None, + sample_content_rate: float | None = None, + endpoint: str | None = None, + exporter_protocol: str | None = None, + patch_providers: bool = True, + providers: Iterable[str] | None = None, + redact: Callable[[str], str | None] | None = None, ) -> Eval2Otel: - """Create the contract-first client. - - The scaffold deliberately has no OpenTelemetry dependency yet. The API gives - Python adopters a stable construction point while the emitter grows behind - the same contract. - """ + """Create an Eval2Otel client from explicit args and OTEL/EVAL2OTEL env.""" - return Eval2Otel( - service_name=service_name, + selected_providers = tuple(providers) if providers is not None else _env_list("EVAL2OTEL_PROVIDERS") + client = Eval2Otel( + service_name=service_name or os.getenv("OTEL_SERVICE_NAME") or os.getenv("EVAL2OTEL_SERVICE_NAME") or "eval2otel-python", service_version=service_version, semconv_version=semconv_version, + capture_content=_env_bool("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", default=False) + if capture_content is None else capture_content, + sample_content_rate=_clamp_sample_rate(_env_float("EVAL2OTEL_SAMPLE_RATE", default=1.0)) + if sample_content_rate is None else sample_content_rate, + endpoint=endpoint or os.getenv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT") or os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"), + exporter_protocol=exporter_protocol or os.getenv("OTEL_EXPORTER_OTLP_PROTOCOL"), + redact=redact if redact is not None else redact_pii if _env_bool("EVAL2OTEL_REDACT_PII", default=False) else None, ) + if patch_providers: + setattr(client, "instrumentation_handles", instrument_all_providers(selected_providers or None)) + return client + + +def _env_bool(name: str, default: bool) -> bool: + value = os.getenv(name) + if value is None: + return default + return value.lower() in {"1", "true", "yes", "on"} + + +def _env_float(name: str, default: float) -> float: + value = os.getenv(name) + if value is None: + return default + try: + return float(value) + except ValueError: + return default + + +def _env_list(name: str) -> tuple[str, ...]: + value = os.getenv(name) + if not value: + return () + return tuple(part.strip() for part in value.split(",") if part.strip()) + + +def _clamp_sample_rate(value: float) -> float: + return min(1.0, max(0.0, value)) diff --git a/python/eval2otel/contract.py b/python/eval2otel/contract.py index 5056c5e..66e5756 100644 --- a/python/eval2otel/contract.py +++ b/python/eval2otel/contract.py @@ -4,7 +4,9 @@ import hashlib import json import time -from typing import Any, Mapping, MutableMapping +from typing import Any, Callable, Mapping, MutableMapping + +from .otel import resolve_tracer, shutdown_tracer_provider EVAL2OTEL_CONTRACT_VERSION = "eval2otel.v1" UNKNOWN_SEMCONV_VERSION = "unspecified" @@ -100,6 +102,7 @@ class EvalResult: usage: Mapping[str, Any] = field(default_factory=dict) performance: Mapping[str, Any] = field(default_factory=dict) system: str | None = None + conversation: Mapping[str, Any] | None = None provenance: Eval2OtelProvenance = field(default_factory=Eval2OtelProvenance) evidence: Eval2OtelEvidence = field(default_factory=Eval2OtelEvidence) @@ -121,6 +124,7 @@ def from_mapping(cls, value: Mapping[str, Any]) -> "EvalResult": usage=dict(value.get("usage") or {}), performance=performance, system=_optional_str(value, "system"), + conversation=dict(value["conversation"]) if isinstance(value.get("conversation"), Mapping) else None, provenance=Eval2OtelProvenance.from_mapping(value.get("provenance")), evidence=Eval2OtelEvidence.from_mapping(value.get("evidence")), ) @@ -148,21 +152,43 @@ def __init__( service_name: str, service_version: str | None = None, semconv_version: str = UNKNOWN_SEMCONV_VERSION, + capture_content: bool = False, + sample_content_rate: float = 1.0, + endpoint: str | None = None, + exporter_protocol: str | None = None, + resource_attributes: Mapping[str, str | int | float | bool] | None = None, + auto_configure_otel: bool = True, + tracer: Any | None = None, + redact: Callable[[str], str | None] | None = None, ) -> None: self.service_name = service_name self.service_version = service_version self.semconv_version = semconv_version + self.capture_content = capture_content + self.sample_content_rate = min(1.0, max(0.0, sample_content_rate)) + self.redact = redact + self._owns_otel_provider = tracer is None and auto_configure_otel + self.tracer = tracer if tracer is not None else resolve_tracer( + service_name=service_name, + service_version=service_version, + endpoint=endpoint, + exporter_protocol=exporter_protocol, + resource_attributes=resource_attributes, + auto_configure=auto_configure_otel, + ) def process_evaluation(self, eval_result: EvalResult | Mapping[str, Any]) -> ConversionReport: started = time.monotonic() result = eval_result if isinstance(eval_result, EvalResult) else EvalResult.from_mapping(eval_result) - attrs = build_eval2otel_attributes(result, semconv_version=self.semconv_version) + attrs = build_span_attributes(result, semconv_version=self.semconv_version) + event_count = self._emit_span(result, attrs) return ConversionReport( eval_id=result.id, success=True, contract_version=str(attrs["evalops.contract.version"]), semconv_version=str(attrs["evalops.semconv.version"]), span_name=_span_name(result.operation), + event_count=event_count, dropped_event_count=result.evidence.dropped_event_count, redacted_content_count=result.evidence.redacted_content_count, truncated_content_count=result.evidence.truncated_content_count, @@ -170,6 +196,74 @@ def process_evaluation(self, eval_result: EvalResult | Mapping[str, Any]) -> Con duration_ms=int((time.monotonic() - started) * 1000), ) + def shutdown(self) -> None: + if self._owns_otel_provider: + shutdown_tracer_provider() + + def _emit_span(self, result: EvalResult, attrs: Mapping[str, str | int | float | bool]) -> int: + if self.tracer is None: + return 0 + + span_name = _span_name(result.operation) + if hasattr(self.tracer, "start_as_current_span"): + with self.tracer.start_as_current_span(span_name, attributes=dict(attrs)) as span: + return self._emit_events(span, result) + + span = self.tracer.start_span(span_name, attributes=dict(attrs)) + try: + return self._emit_events(span, result) + finally: + end = getattr(span, "end", None) + if callable(end): + end() + + def _emit_events(self, span: Any, result: EvalResult) -> int: + if not self._should_capture_content(result): + return 0 + messages = [] + if result.conversation and isinstance(result.conversation.get("messages"), list): + messages = list(result.conversation["messages"]) + + event_count = 0 + for index, message in enumerate(messages): + if not isinstance(message, Mapping): + continue + role = str(message.get("role") or "user") + event_attrs: dict[str, str | int | bool] = { + "gen_ai.message.role": role, + "gen_ai.message.index": index, + } + if "content" in message: + event_attrs.update(self._content_attributes(message["content"])) + span.add_event(f"gen_ai.{role}.message", event_attrs) + event_count += 1 + return event_count + + def _content_attributes(self, content: Any) -> dict[str, str]: + content_type = "text" if isinstance(content, str) else "json" + text = content if isinstance(content, str) else json.dumps(content, sort_keys=True, default=str) + redacted = self.redact(text) if self.redact else text + if redacted is None: + return { + "gen_ai.message.content_type": content_type, + "evalops.content_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), + } + key = "gen_ai.message.content" if content_type == "text" else "gen_ai.message.content_json" + return { + "gen_ai.message.content_type": content_type, + key: redacted, + } + + def _should_capture_content(self, result: EvalResult) -> bool: + if not self.capture_content: + return False + if self.sample_content_rate >= 1: + return True + if self.sample_content_rate <= 0: + return False + bucket = int(hashlib.sha256(result.id.encode("utf-8")).hexdigest()[:8], 16) / 0xFFFFFFFF + return bucket <= self.sample_content_rate + def build_eval2otel_attributes( eval_result: EvalResult, @@ -202,6 +296,24 @@ def build_eval2otel_attributes( return attrs +def build_span_attributes( + eval_result: EvalResult, + semconv_version: str = UNKNOWN_SEMCONV_VERSION, +) -> MutableMapping[str, str | int | float]: + attrs: MutableMapping[str, str | int | float] = build_eval2otel_attributes(eval_result, semconv_version) + attrs["gen_ai.operation.name"] = eval_result.operation + attrs["gen_ai.system"] = eval_result.system or "unknown" + if eval_result.request.get("model") is not None: + attrs["gen_ai.request.model"] = str(eval_result.request["model"]) + if eval_result.response.get("model") is not None: + attrs["gen_ai.response.model"] = str(eval_result.response["model"]) + if eval_result.usage.get("inputTokens") is not None: + attrs["gen_ai.usage.input_tokens"] = int(eval_result.usage["inputTokens"]) + if eval_result.usage.get("outputTokens") is not None: + attrs["gen_ai.usage.output_tokens"] = int(eval_result.usage["outputTokens"]) + return attrs + + def _span_name(operation: str) -> str: if operation in {"chat", "text_completion"}: return "gen_ai.chat" diff --git a/python/eval2otel/instrumentations.py b/python/eval2otel/instrumentations.py new file mode 100644 index 0000000..97ae8e7 --- /dev/null +++ b/python/eval2otel/instrumentations.py @@ -0,0 +1,170 @@ +from __future__ import annotations + +from dataclasses import dataclass +import importlib +import importlib.util +from typing import Iterable + + +@dataclass(frozen=True) +class ProviderInstrumentationHandle: + provider: str + available: bool + instrumented: bool = False + instrumentation: str | None = None + reason: str | None = None + + +@dataclass(frozen=True) +class InstrumentationCandidate: + module: str + class_name: str + + +@dataclass(frozen=True) +class ProviderInstrumentationSpec: + package_module: str + candidates: tuple[InstrumentationCandidate, ...] + + +PROVIDER_MODULES: dict[str, ProviderInstrumentationSpec] = { + "openai": ProviderInstrumentationSpec( + package_module="openai", + candidates=( + InstrumentationCandidate("opentelemetry.instrumentation.openai", "OpenAIInstrumentor"), + InstrumentationCandidate("openinference.instrumentation.openai", "OpenAIInstrumentor"), + ), + ), + "anthropic": ProviderInstrumentationSpec( + package_module="anthropic", + candidates=( + InstrumentationCandidate("opentelemetry.instrumentation.anthropic", "AnthropicInstrumentor"), + InstrumentationCandidate("openinference.instrumentation.anthropic", "AnthropicInstrumentor"), + ), + ), + "google-generativeai": ProviderInstrumentationSpec( + package_module="google.generativeai", + candidates=( + InstrumentationCandidate("opentelemetry.instrumentation.google_generativeai", "GoogleGenerativeAIInstrumentor"), + InstrumentationCandidate("openinference.instrumentation.google_genai", "GoogleGenAIInstrumentor"), + ), + ), + "bedrock": ProviderInstrumentationSpec( + package_module="boto3", + candidates=( + InstrumentationCandidate("opentelemetry.instrumentation.bedrock", "BedrockInstrumentor"), + InstrumentationCandidate("openinference.instrumentation.bedrock", "BedrockInstrumentor"), + ), + ), + "cohere": ProviderInstrumentationSpec( + package_module="cohere", + candidates=( + InstrumentationCandidate("opentelemetry.instrumentation.cohere", "CohereInstrumentor"), + InstrumentationCandidate("openinference.instrumentation.cohere", "CohereInstrumentor"), + ), + ), + "huggingface": ProviderInstrumentationSpec( + package_module="transformers", + candidates=( + InstrumentationCandidate("opentelemetry.instrumentation.transformers", "TransformersInstrumentor"), + InstrumentationCandidate("openinference.instrumentation.transformers", "TransformersInstrumentor"), + ), + ), +} + + +def instrument_openai() -> ProviderInstrumentationHandle: + return _instrument_provider("openai") + + +def instrument_anthropic() -> ProviderInstrumentationHandle: + return _instrument_provider("anthropic") + + +def instrument_google_generativeai() -> ProviderInstrumentationHandle: + return _instrument_provider("google-generativeai") + + +def instrument_bedrock() -> ProviderInstrumentationHandle: + return _instrument_provider("bedrock") + + +def instrument_cohere() -> ProviderInstrumentationHandle: + return _instrument_provider("cohere") + + +def instrument_huggingface() -> ProviderInstrumentationHandle: + return _instrument_provider("huggingface") + + +def instrument_all_providers( + providers: Iterable[str] | None = None, + **instrumentation_options: object, +) -> tuple[ProviderInstrumentationHandle, ...]: + selected = tuple(providers or PROVIDER_MODULES.keys()) + return tuple(_instrument_provider(provider, **instrumentation_options) for provider in selected) + + +def _instrument_provider(provider: str, **instrumentation_options: object) -> ProviderInstrumentationHandle: + spec = PROVIDER_MODULES.get(provider) + if spec is None: + return ProviderInstrumentationHandle( + provider=provider, + available=False, + reason="unsupported provider", + ) + if _find_spec(spec.package_module) is None: + return ProviderInstrumentationHandle( + provider=provider, + available=False, + reason=f"{spec.package_module} is not installed", + ) + + for candidate in spec.candidates: + handle = _try_instrumentor(provider, candidate, instrumentation_options) + if handle is not None: + return handle + + return ProviderInstrumentationHandle( + provider=provider, + available=True, + reason="provider package installed; no supported instrumentation package found", + ) + + +def _try_instrumentor( + provider: str, + candidate: InstrumentationCandidate, + instrumentation_options: dict[str, object], +) -> ProviderInstrumentationHandle | None: + if _find_spec(candidate.module) is None: + return None + + try: + module = importlib.import_module(candidate.module) + instrumentor_class = getattr(module, candidate.class_name) + instrumentor = instrumentor_class() + instrument = getattr(instrumentor, "instrument") + instrument(**instrumentation_options) + except Exception as exc: + return ProviderInstrumentationHandle( + provider=provider, + available=True, + instrumented=False, + instrumentation=f"{candidate.module}.{candidate.class_name}", + reason=f"instrumentation failed: {exc}", + ) + + return ProviderInstrumentationHandle( + provider=provider, + available=True, + instrumented=True, + instrumentation=f"{candidate.module}.{candidate.class_name}", + ) + + +def _find_spec(module: str) -> object | None: + try: + return importlib.util.find_spec(module) + except (ImportError, ModuleNotFoundError): + return None diff --git a/python/eval2otel/otel.py b/python/eval2otel/otel.py new file mode 100644 index 0000000..db8703f --- /dev/null +++ b/python/eval2otel/otel.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +from typing import Any, Mapping + +_CONFIGURED_PROVIDER: Any | None = None + + +def resolve_tracer( + *, + service_name: str, + service_version: str | None, + endpoint: str | None, + exporter_protocol: str | None, + resource_attributes: Mapping[str, str | int | float | bool] | None, + auto_configure: bool, +) -> Any | None: + try: + from opentelemetry import trace + except ImportError: + return None + + if auto_configure: + _configure_sdk( + service_name=service_name, + service_version=service_version, + endpoint=endpoint, + exporter_protocol=exporter_protocol, + resource_attributes=resource_attributes, + ) + + return trace.get_tracer("eval2otel", service_version) + + +def shutdown_tracer_provider() -> None: + global _CONFIGURED_PROVIDER + + provider = _CONFIGURED_PROVIDER + if provider is None: + return + shutdown = getattr(provider, "shutdown", None) + if callable(shutdown): + shutdown() + _CONFIGURED_PROVIDER = None + + +def _configure_sdk( + *, + service_name: str, + service_version: str | None, + endpoint: str | None, + exporter_protocol: str | None, + resource_attributes: Mapping[str, str | int | float | bool] | None, +) -> None: + global _CONFIGURED_PROVIDER + + if _CONFIGURED_PROVIDER is not None: + return + + try: + from opentelemetry import trace + from opentelemetry.sdk.resources import Resource + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import BatchSpanProcessor + except ImportError: + return + + current_provider = trace.get_tracer_provider() + if _provider_is_application_owned(current_provider): + return + + attrs: dict[str, str | int | float | bool] = { + "service.name": service_name, + } + if service_version: + attrs["service.version"] = service_version + if resource_attributes: + attrs.update(resource_attributes) + + provider = TracerProvider(resource=Resource.create(attrs)) + exporter = _build_exporter(endpoint=endpoint, exporter_protocol=exporter_protocol) + if exporter is not None: + provider.add_span_processor(BatchSpanProcessor(exporter)) + + try: + trace.set_tracer_provider(provider) + except Exception: + # The application may already own the global provider. In that case the + # existing provider is the right one to use. + return + _CONFIGURED_PROVIDER = provider + + +def _build_exporter(endpoint: str | None, exporter_protocol: str | None) -> Any | None: + if not endpoint: + return None + + protocol = (exporter_protocol or "http/protobuf").lower() + try: + if protocol == "grpc": + from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter + else: + from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter + except ImportError: + return None + + return OTLPSpanExporter(endpoint=endpoint) + + +def _provider_is_application_owned(provider: Any) -> bool: + return not ( + type(provider).__module__ == "opentelemetry.trace" + and type(provider).__name__ == "ProxyTracerProvider" + ) diff --git a/python/eval2otel/privacy.py b/python/eval2otel/privacy.py new file mode 100644 index 0000000..cfdfbdf --- /dev/null +++ b/python/eval2otel/privacy.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import re + +_EMAIL_RE = re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b") +_BEARER_RE = re.compile(r"\b(Bearer\s+)[A-Za-z0-9._~+/=-]{12,}\b", re.IGNORECASE) +_SECRET_ASSIGNMENT_RE = re.compile( + r"\b(api[_-]?key|token|secret|password)\s*[:=]\s*[A-Za-z0-9._~+/=-]{8,}\b", + re.IGNORECASE, +) +_CREDIT_CARD_LIKE_RE = re.compile(r"\b(?:\d[ -]*?){13,19}\b") + + +def redact_pii(content: str) -> str: + """Best-effort local redaction used by EVAL2OTEL_REDACT_PII.""" + + redacted = _EMAIL_RE.sub("[REDACTED_EMAIL]", content) + redacted = _BEARER_RE.sub(r"\1[REDACTED_TOKEN]", redacted) + redacted = _SECRET_ASSIGNMENT_RE.sub(lambda match: f"{match.group(1)}=[REDACTED_SECRET]", redacted) + return _CREDIT_CARD_LIKE_RE.sub("[REDACTED_NUMBER]", redacted) diff --git a/python/pyproject.toml b/python/pyproject.toml index f900dd8..b2a6484 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "hatchling.build" [project] name = "eval2otel-python" version = "0.1.0" -description = "Contract-first Python scaffold for eval2otel telemetry payloads" +description = "Python SDK preview for eval2otel telemetry payloads" readme = "README.md" requires-python = ">=3.10" license = { text = "MIT" } @@ -14,5 +14,17 @@ authors = [ ] keywords = ["opentelemetry", "genai", "evaluation", "observability"] +[project.optional-dependencies] +otel = [ + "opentelemetry-api>=1.29", + "opentelemetry-sdk>=1.29", + "opentelemetry-exporter-otlp>=1.29", +] +providers = [ + "openai>=1", + "anthropic>=0.34", + "cohere>=5", +] + [tool.hatch.build.targets.wheel] packages = ["eval2otel"] diff --git a/python/tests/test_contract.py b/python/tests/test_contract.py index 070064d..408173e 100644 --- a/python/tests/test_contract.py +++ b/python/tests/test_contract.py @@ -1,11 +1,16 @@ import unittest +from unittest import mock from eval2otel import ( EVAL2OTEL_CONTRACT_VERSION, Eval2Otel, EvalResult, build_eval2otel_attributes, + build_span_attributes, instrument_all, + instrument_all_providers, + instrument_openai, + redact_pii, sha256_payload, ) @@ -61,9 +66,96 @@ def test_attributes_and_hashes_are_stable(self) -> None: self.assertEqual(attrs["evalops.eval.id"], "py-case-2") self.assertEqual(attrs["evalops.adapter.name"], "deepeval") self.assertRegex(attrs["evalops.raw_payload_sha256"], r"^[a-f0-9]{64}$") + span_attrs = build_span_attributes(result) + self.assertEqual(span_attrs["gen_ai.operation.name"], "execute_tool") + self.assertEqual(span_attrs["gen_ai.request.model"], "gpt-4o-mini") - def test_instrument_all_returns_client(self) -> None: - self.assertIsInstance(instrument_all("svc"), Eval2Otel) + def test_process_evaluation_emits_span_and_content_events_with_injected_tracer(self) -> None: + tracer = FakeTracer() + client = Eval2Otel( + service_name="pytest", + capture_content=True, + tracer=tracer, + redact=lambda content: None if "SECRET" in content else content, + ) + report = client.process_evaluation({ + "id": "py-case-otel", + "timestamp": 1700000000000, + "model": "gpt-4o-mini", + "system": "openai", + "operation": "chat", + "request": {"model": "gpt-4o-mini"}, + "response": {"model": "gpt-4o-mini"}, + "usage": {"inputTokens": 3, "outputTokens": 5}, + "performance": {"duration": 0.5}, + "conversation": { + "messages": [ + {"role": "user", "content": "hello SECRET"}, + {"role": "assistant", "content": {"answer": "ok"}}, + ] + }, + }) + + self.assertEqual(report.event_count, 2) + self.assertEqual(tracer.spans[0].name, "gen_ai.chat") + self.assertEqual(tracer.spans[0].attributes["gen_ai.system"], "openai") + self.assertEqual(tracer.spans[0].attributes["gen_ai.usage.input_tokens"], 3) + self.assertEqual(tracer.spans[0].events[0][0], "gen_ai.user.message") + self.assertIn("evalops.content_sha256", tracer.spans[0].events[0][1]) + self.assertNotIn("gen_ai.message.content", tracer.spans[0].events[0][1]) + self.assertEqual(tracer.spans[0].events[1][1]["gen_ai.message.content_json"], '{"answer": "ok"}') + + def test_instrument_all_reads_env_and_returns_provider_handles(self) -> None: + with mock.patch.dict( + "os.environ", + { + "OTEL_SERVICE_NAME": "env-service", + "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT": "true", + "EVAL2OTEL_SAMPLE_RATE": "0.25", + "EVAL2OTEL_REDACT_PII": "true", + "EVAL2OTEL_PROVIDERS": "openai,unknown", + }, + clear=False, + ): + client = instrument_all() + + self.assertIsInstance(client, Eval2Otel) + self.assertEqual(client.service_name, "env-service") + self.assertTrue(client.capture_content) + self.assertEqual(client.sample_content_rate, 0.25) + self.assertTrue(hasattr(client, "instrumentation_handles")) + self.assertEqual(len(client.instrumentation_handles), 2) + self.assertEqual(client.redact, redact_pii) + self.assertIn("[REDACTED_EMAIL]", client.redact("contact support@example.com")) + + def test_provider_instrumentation_handles_missing_packages(self) -> None: + handle = instrument_openai() + self.assertEqual(handle.provider, "openai") + self.assertIsInstance(handle.available, bool) + self.assertGreaterEqual(len(instrument_all_providers(["openai", "unknown"])), 2) + + def test_provider_instrumentation_invokes_optional_instrumentor(self) -> None: + class FakeOpenAIInstrumentor: + calls: list[dict[str, object]] = [] + + def instrument(self, **kwargs: object) -> None: + self.calls.append(kwargs) + + fake_module = type("FakeModule", (), {"OpenAIInstrumentor": FakeOpenAIInstrumentor})() + + def fake_find_spec(name: str): + if name in {"openai", "opentelemetry.instrumentation.openai"}: + return object() + return None + + with mock.patch("importlib.util.find_spec", side_effect=fake_find_spec), \ + mock.patch("importlib.import_module", return_value=fake_module): + handle = instrument_openai() + + self.assertTrue(handle.available) + self.assertTrue(handle.instrumented) + self.assertEqual(handle.instrumentation, "opentelemetry.instrumentation.openai.OpenAIInstrumentor") + self.assertEqual(FakeOpenAIInstrumentor.calls, [{}]) def test_required_contract_fields_are_validated(self) -> None: with self.assertRaisesRegex(ValueError, "performance.duration"): @@ -75,5 +167,39 @@ def test_required_contract_fields_are_validated(self) -> None: }) +class FakeSpan: + def __init__(self, name: str, attributes: dict[str, object]) -> None: + self.name = name + self.attributes = attributes + self.events: list[tuple[str, dict[str, object]]] = [] + + def add_event(self, name: str, attributes: dict[str, object]) -> None: + self.events.append((name, attributes)) + + def end(self) -> None: + pass + + +class FakeSpanContext: + def __init__(self, span: FakeSpan) -> None: + self.span = span + + def __enter__(self) -> FakeSpan: + return self.span + + def __exit__(self, exc_type, exc, tb) -> None: + self.span.end() + + +class FakeTracer: + def __init__(self) -> None: + self.spans: list[FakeSpan] = [] + + def start_as_current_span(self, name: str, attributes: dict[str, object]) -> FakeSpanContext: + span = FakeSpan(name, attributes) + self.spans.append(span) + return FakeSpanContext(span) + + if __name__ == "__main__": unittest.main() diff --git a/src/index.ts b/src/index.ts index 95e0a25..1397497 100644 --- a/src/index.ts +++ b/src/index.ts @@ -265,10 +265,18 @@ export { } from './rag'; export { detectProvider, convertProviderToEvalResult, convertProviderWithEvidence, convertAnyProvider, type ProviderMode } from './helpers'; export { + convertDeepEvalResult, + convertDeepEvalToEvalResults, convertPromptfooResult, convertPromptfooToEvalResults, + convertRagasResult, + convertRagasToEvalResults, + type DeepEvalAdapterOptions, + type DeepEvalConversionResult, type PromptfooAdapterOptions, type PromptfooConversionResult, + type RagasAdapterOptions, + type RagasConversionResult, } from './integrations'; export { Eval2OtelValidation, diff --git a/src/integrations/deepeval.ts b/src/integrations/deepeval.ts new file mode 100644 index 0000000..4a1b9b7 --- /dev/null +++ b/src/integrations/deepeval.ts @@ -0,0 +1,231 @@ +import { z } from 'zod'; +import { EVAL2OTEL_CONTRACT_VERSION, sha256 } from '../contract'; +import { ConversionWarning, EvalResult } from '../types'; + +const DeepEvalMetricSchema = z.object({ + name: z.string().optional(), + metric: z.string().optional(), + score: z.number().optional(), + threshold: z.number().optional(), + success: z.boolean().optional(), + passed: z.boolean().optional(), + reason: z.string().optional(), + explanation: z.string().optional(), +}).passthrough(); + +const DeepEvalResultSchema = z.object({ + id: z.union([z.string(), z.number()]).optional(), + name: z.string().optional(), + testCaseId: z.union([z.string(), z.number()]).optional(), + input: z.unknown().optional(), + actualOutput: z.unknown().optional(), + actual_output: z.unknown().optional(), + output: z.unknown().optional(), + expectedOutput: z.unknown().optional(), + expected_output: z.unknown().optional(), + context: z.array(z.unknown()).optional(), + retrievalContext: z.array(z.unknown()).optional(), + retrieval_context: z.array(z.unknown()).optional(), + metrics: z.union([z.array(DeepEvalMetricSchema), z.record(z.string(), z.unknown())]).optional(), + success: z.boolean().optional(), + passed: z.boolean().optional(), + latencyMs: z.number().optional(), + latency_ms: z.number().optional(), + duration: z.number().optional(), +}).passthrough(); + +interface NormalizedMetric { + name: string; + score?: number; + success?: boolean; + reason?: string; +} + +export interface DeepEvalAdapterOptions { + runId?: string; + datasetId?: string; + datasetVersion?: string; + defaultModel?: string; + defaultSystem?: string; + timestamp?: number; + includeExplanations?: boolean; +} + +export interface DeepEvalConversionResult { + evalResults: EvalResult[]; + warnings: ConversionWarning[]; +} + +export function convertDeepEvalToEvalResults( + payload: unknown, + options: DeepEvalAdapterOptions = {}, +): DeepEvalConversionResult { + const rows = extractDeepEvalRows(payload); + const warnings: ConversionWarning[] = []; + const evalResults = rows.flatMap((row, index) => { + const parsed = DeepEvalResultSchema.safeParse(row); + if (!parsed.success) { + warnings.push({ + code: 'deepeval.row_invalid', + message: `DeepEval row ${index} did not match the supported shape.`, + severity: 'warning', + }); + return []; + } + return [convertDeepEvalResult(parsed.data, index, options)]; + }); + return { evalResults, warnings }; +} + +export function convertDeepEvalResult( + row: z.infer, + index: number, + options: DeepEvalAdapterOptions = {}, +): EvalResult { + const caseId = row.testCaseId !== undefined ? String(row.testCaseId) + : row.id !== undefined ? String(row.id) + : `deepeval-${index}`; + const input = stringifyValue(row.input); + const output = stringifyValue(row.actualOutput ?? row.actual_output ?? row.output); + const expected = stringifyValue(row.expectedOutput ?? row.expected_output); + const metrics = normalizeMetrics(row.metrics); + const failedMetrics = metrics.filter(metric => metric.success === false); + const success = row.success ?? row.passed ?? failedMetrics.length === 0; + const warnings = buildDeepEvalWarnings(failedMetrics, options.includeExplanations === true); + const retrievalContexts = row.retrievalContext ?? row.retrieval_context ?? row.context ?? []; + + return { + id: `${options.runId ?? 'deepeval'}-${caseId}`, + timestamp: options.timestamp ?? Date.now(), + model: options.defaultModel ?? 'unknown', + system: options.defaultSystem ?? 'deepeval', + operation: retrievalContexts.length > 0 ? 'chat' : 'text_completion', + request: { + model: options.defaultModel ?? 'unknown', + }, + response: { + finishReasons: [success ? 'pass' : 'fail'], + choices: [{ + index: 0, + finishReason: success ? 'pass' : 'fail', + message: { + role: 'assistant', + content: output ?? '', + }, + }], + }, + usage: {}, + performance: { + duration: durationSeconds(row), + }, + conversation: input ? { + id: `deepeval-${caseId}`, + messages: [{ role: 'user', content: input }], + } : undefined, + rag: retrievalContexts.length > 0 ? { + retrievalMethod: 'hybrid', + documentsRetrieved: retrievalContexts.length, + documentsUsed: retrievalContexts.length, + chunks: retrievalContexts.map((context, contextIndex) => ({ + id: `deepeval-context-${contextIndex}`, + source: `deepeval-context-${contextIndex}`, + relevanceScore: 1, + position: contextIndex, + used: true, + evidenceSha256: sha256(context), + })), + } : undefined, + provider: { + name: 'deepeval', + attributes: { + 'eval.deepeval.success': success, + 'eval.deepeval.metric_names': metrics.map(metric => metric.name), + 'eval.deepeval.failed_metric_count': failedMetrics.length, + ...(expected ? { 'eval.deepeval.expected_sha256': sha256(expected) } : {}), + ...Object.fromEntries(metrics + .filter(metric => typeof metric.score === 'number') + .map(metric => [`eval.deepeval.${normalizeMetricName(metric.name)}`, metric.score])), + }, + }, + provenance: { + sourceFramework: 'deepeval', + runId: options.runId, + caseId, + datasetId: options.datasetId, + datasetVersion: options.datasetVersion, + adapter: 'deepeval', + adapterVersion: EVAL2OTEL_CONTRACT_VERSION, + contractVersion: EVAL2OTEL_CONTRACT_VERSION, + }, + evidence: { + rawPayloadSha256: sha256(row), + promptSha256: input ? sha256(input) : undefined, + responseSha256: output ? sha256(output) : undefined, + warningCount: warnings.length, + warnings, + }, + }; +} + +function extractDeepEvalRows(payload: unknown): unknown[] { + if (Array.isArray(payload)) return payload; + if (!payload || typeof payload !== 'object') return []; + const obj = payload as Record; + if (Array.isArray(obj.testResults)) return obj.testResults; + if (Array.isArray(obj.test_results)) return obj.test_results; + if (Array.isArray(obj.results)) return obj.results; + if (Array.isArray(obj.rows)) return obj.rows; + return [obj]; +} + +function normalizeMetrics(metrics: z.infer['metrics']): NormalizedMetric[] { + if (!metrics) return []; + if (Array.isArray(metrics)) { + return metrics.flatMap((metric, index) => { + const name = metric.name ?? metric.metric ?? `metric_${index}`; + return [{ + name, + score: metric.score, + success: metric.success ?? metric.passed, + reason: metric.reason ?? metric.explanation, + }]; + }); + } + return Object.entries(metrics).flatMap(([name, value]) => { + if (typeof value === 'number') return [{ name, score: value }]; + if (typeof value === 'boolean') return [{ name, success: value }]; + if (value && typeof value === 'object') { + const obj = value as Record; + return [{ + name, + score: typeof obj.score === 'number' ? obj.score : undefined, + success: typeof obj.success === 'boolean' ? obj.success : typeof obj.passed === 'boolean' ? obj.passed : undefined, + reason: typeof obj.reason === 'string' ? obj.reason : typeof obj.explanation === 'string' ? obj.explanation : undefined, + }]; + } + return []; + }); +} + +function buildDeepEvalWarnings(metrics: NormalizedMetric[], includeExplanations: boolean): ConversionWarning[] { + return metrics.map(metric => ({ + code: 'deepeval.metric_failed', + message: includeExplanations && metric.reason ? `${metric.name}: ${metric.reason}` : `${metric.name} failed.`, + severity: 'warning' as const, + })); +} + +function stringifyValue(value: unknown): string | undefined { + if (value === undefined || value === null) return undefined; + return typeof value === 'string' ? value : JSON.stringify(value); +} + +function normalizeMetricName(name: string): string { + return name.trim().toLowerCase().replace(/[^a-z0-9]+/g, '_').replace(/^_+|_+$/g, '') || 'metric'; +} + +function durationSeconds(row: z.infer): number { + if (typeof row.duration === 'number' && row.duration > 0) return row.duration; + const latencyMs = row.latencyMs ?? row.latency_ms; + return typeof latencyMs === 'number' && latencyMs > 0 ? latencyMs / 1000 : 0.001; +} diff --git a/src/integrations/index.ts b/src/integrations/index.ts index 92f99fa..c530d82 100644 --- a/src/integrations/index.ts +++ b/src/integrations/index.ts @@ -4,3 +4,15 @@ export { type PromptfooAdapterOptions, type PromptfooConversionResult, } from './promptfoo'; +export { + convertRagasResult, + convertRagasToEvalResults, + type RagasAdapterOptions, + type RagasConversionResult, +} from './ragas'; +export { + convertDeepEvalResult, + convertDeepEvalToEvalResults, + type DeepEvalAdapterOptions, + type DeepEvalConversionResult, +} from './deepeval'; diff --git a/src/integrations/ragas.ts b/src/integrations/ragas.ts new file mode 100644 index 0000000..99579dc --- /dev/null +++ b/src/integrations/ragas.ts @@ -0,0 +1,213 @@ +import { z } from 'zod'; +import { EVAL2OTEL_CONTRACT_VERSION, sha256 } from '../contract'; +import { ConversionWarning, EvalResult } from '../types'; + +type RagChunk = NonNullable['chunks']>[number]; + +const RagasContextSchema = z.union([ + z.string(), + z.object({ + id: z.union([z.string(), z.number()]).optional(), + source: z.string().optional(), + text: z.string().optional(), + content: z.string().optional(), + page_content: z.string().optional(), + score: z.number().optional(), + relevance_score: z.number().optional(), + citation_id: z.string().optional(), + }).passthrough(), +]); + +const RagasResultSchema = z.object({ + id: z.union([z.string(), z.number()]).optional(), + question: z.unknown().optional(), + user_input: z.unknown().optional(), + input: z.unknown().optional(), + answer: z.unknown().optional(), + response: z.unknown().optional(), + output: z.unknown().optional(), + ground_truth: z.unknown().optional(), + reference: z.unknown().optional(), + contexts: z.array(RagasContextSchema).optional(), + retrieved_contexts: z.array(RagasContextSchema).optional(), + context: z.array(RagasContextSchema).optional(), + faithfulness: z.number().optional(), + answer_relevancy: z.number().optional(), + answer_relevance: z.number().optional(), + context_precision: z.number().optional(), + context_recall: z.number().optional(), + context_entity_recall: z.number().optional(), + latencyMs: z.number().optional(), + latency_ms: z.number().optional(), + duration: z.number().optional(), +}).passthrough(); + +export interface RagasAdapterOptions { + runId?: string; + datasetId?: string; + datasetVersion?: string; + defaultModel?: string; + defaultSystem?: string; + timestamp?: number; +} + +export interface RagasConversionResult { + evalResults: EvalResult[]; + warnings: ConversionWarning[]; +} + +export function convertRagasToEvalResults( + payload: unknown, + options: RagasAdapterOptions = {}, +): RagasConversionResult { + const rows = extractRagasRows(payload); + const warnings: ConversionWarning[] = []; + const evalResults = rows.flatMap((row, index) => { + const parsed = RagasResultSchema.safeParse(row); + if (!parsed.success) { + warnings.push({ + code: 'ragas.row_invalid', + message: `RAGAS row ${index} did not match the supported shape.`, + severity: 'warning', + }); + return []; + } + return [convertRagasResult(parsed.data, index, options)]; + }); + return { evalResults, warnings }; +} + +export function convertRagasResult( + row: z.infer, + index: number, + options: RagasAdapterOptions = {}, +): EvalResult { + const caseId = row.id !== undefined ? String(row.id) : `ragas-${index}`; + const question = stringifyValue(row.user_input ?? row.question ?? row.input); + const answer = stringifyValue(row.response ?? row.answer ?? row.output); + const reference = stringifyValue(row.reference ?? row.ground_truth); + const contexts = row.retrieved_contexts ?? row.contexts ?? row.context ?? []; + const chunks = contexts.map((context, contextIndex) => toRagChunk(context, contextIndex)); + const metrics = { + contextPrecision: row.context_precision, + contextRecall: row.context_recall, + answerRelevance: row.answer_relevance ?? row.answer_relevancy, + faithfulness: row.faithfulness, + }; + const metricAttributes = Object.fromEntries( + Object.entries({ + 'eval.ragas.faithfulness': row.faithfulness, + 'eval.ragas.context_precision': row.context_precision, + 'eval.ragas.context_recall': row.context_recall, + 'eval.ragas.answer_relevance': row.answer_relevance ?? row.answer_relevancy, + 'eval.ragas.context_entity_recall': row.context_entity_recall, + }).filter(([, value]) => typeof value === 'number'), + ) as Record; + + return { + id: `${options.runId ?? 'ragas'}-${caseId}`, + timestamp: options.timestamp ?? Date.now(), + model: options.defaultModel ?? 'unknown', + system: options.defaultSystem ?? 'ragas', + operation: 'chat', + request: { + model: options.defaultModel ?? 'unknown', + }, + response: { + finishReasons: ['evaluated'], + choices: [{ + index: 0, + finishReason: 'evaluated', + message: { + role: 'assistant', + content: answer ?? '', + }, + }], + }, + usage: {}, + performance: { + duration: durationSeconds(row), + }, + conversation: question ? { + id: `ragas-${caseId}`, + messages: [{ role: 'user', content: question }], + } : undefined, + rag: { + retrievalMethod: 'hybrid', + documentsRetrieved: chunks.length, + documentsUsed: chunks.length, + chunks, + metrics, + }, + provider: { + name: 'ragas', + attributes: { + ...metricAttributes, + 'eval.ragas.metric_names': Object.keys(metricAttributes).map(key => key.replace('eval.ragas.', '')), + ...(reference ? { 'eval.ragas.reference_sha256': sha256(reference) } : {}), + }, + }, + provenance: { + sourceFramework: 'ragas', + runId: options.runId, + caseId, + datasetId: options.datasetId, + datasetVersion: options.datasetVersion, + adapter: 'ragas', + adapterVersion: EVAL2OTEL_CONTRACT_VERSION, + contractVersion: EVAL2OTEL_CONTRACT_VERSION, + }, + evidence: { + rawPayloadSha256: sha256(row), + promptSha256: question ? sha256(question) : undefined, + responseSha256: answer ? sha256(answer) : undefined, + warningCount: 0, + warnings: [], + }, + }; +} + +function extractRagasRows(payload: unknown): unknown[] { + if (Array.isArray(payload)) return payload; + if (!payload || typeof payload !== 'object') return []; + const obj = payload as Record; + if (Array.isArray(obj.scores)) return obj.scores; + if (Array.isArray(obj.results)) return obj.results; + if (Array.isArray(obj.rows)) return obj.rows; + if (Array.isArray(obj.data)) return obj.data; + return [obj]; +} + +function stringifyValue(value: unknown): string | undefined { + if (value === undefined || value === null) return undefined; + return typeof value === 'string' ? value : JSON.stringify(value); +} + +function toRagChunk(context: z.infer, index: number): RagChunk { + if (typeof context === 'string') { + return { + id: `ragas-context-${index}`, + source: `ragas-context-${index}`, + relevanceScore: 1, + position: index, + used: true, + evidenceSha256: sha256(context), + }; + } + const content = context.text ?? context.content ?? context.page_content ?? JSON.stringify(context); + return { + id: context.id !== undefined ? String(context.id) : `ragas-context-${index}`, + source: context.source ?? `ragas-context-${index}`, + relevanceScore: context.relevance_score ?? context.score ?? 1, + position: index, + used: true, + citationId: context.citation_id, + evidenceSha256: sha256(content), + }; +} + +function durationSeconds(row: z.infer): number { + if (typeof row.duration === 'number' && row.duration > 0) return row.duration; + const latencyMs = row.latencyMs ?? row.latency_ms; + return typeof latencyMs === 'number' && latencyMs > 0 ? latencyMs / 1000 : 0.001; +} diff --git a/test/index-exports.test.ts b/test/index-exports.test.ts index d25d649..aa2c039 100644 --- a/test/index-exports.test.ts +++ b/test/index-exports.test.ts @@ -31,6 +31,10 @@ describe('Package exports', () => { expect(pkg.getRagMetricValue).toBeDefined(); expect(pkg.convertPromptfooResult).toBeDefined(); expect(pkg.convertPromptfooToEvalResults).toBeDefined(); + expect(pkg.convertRagasResult).toBeDefined(); + expect(pkg.convertRagasToEvalResults).toBeDefined(); + expect(pkg.convertDeepEvalResult).toBeDefined(); + expect(pkg.convertDeepEvalToEvalResults).toBeDefined(); expect(pkg.convertOllamaToEval2Otel).toBeDefined(); expect(pkg.convertOpenAICompatibleToEval2Otel).toBeDefined(); expect(pkg.convertBedrockToEval2Otel).toBeDefined(); diff --git a/test/integrations-ragas-deepeval.test.ts b/test/integrations-ragas-deepeval.test.ts new file mode 100644 index 0000000..93e76ad --- /dev/null +++ b/test/integrations-ragas-deepeval.test.ts @@ -0,0 +1,142 @@ +import { + convertDeepEvalResult, + convertDeepEvalToEvalResults, + convertRagasResult, + convertRagasToEvalResults, +} from '../src/integrations'; + +describe('RAGAS integration adapter', () => { + it('converts RAGAS rows into EvalResult RAG metrics and chunk evidence', () => { + const converted = convertRagasToEvalResults({ + scores: [{ + id: 'ragas-case-1', + user_input: 'What shipped?', + response: 'The release added adapters.', + reference: 'Adapters shipped.', + retrieved_contexts: [ + 'release notes mention adapters', + { id: 'doc-2', source: 'contract.md', text: 'RAG metrics are tracked', score: 0.8, citation_id: 'cite-2' }, + ], + faithfulness: 0.91, + context_precision: 0.82, + context_recall: 0.74, + answer_relevancy: 0.88, + latency_ms: 250, + }], + }, { + runId: 'ragas-run', + datasetId: 'rag-evals', + datasetVersion: '2026.05', + defaultModel: 'gpt-4o-mini', + timestamp: 1700000000000, + }); + + expect(converted.warnings).toEqual([]); + expect(converted.evalResults).toHaveLength(1); + const result = converted.evalResults[0]; + expect(result.id).toBe('ragas-run-ragas-case-1'); + expect(result.provenance).toMatchObject({ + sourceFramework: 'ragas', + adapter: 'ragas', + datasetId: 'rag-evals', + }); + expect(result.performance.duration).toBe(0.25); + expect(result.rag?.documentsRetrieved).toBe(2); + expect(result.rag?.chunks?.[0].evidenceSha256).toMatch(/^[a-f0-9]{64}$/); + expect(result.rag?.chunks?.[1]).toMatchObject({ id: 'doc-2', source: 'contract.md', relevanceScore: 0.8, citationId: 'cite-2' }); + expect(result.rag?.metrics).toMatchObject({ + faithfulness: 0.91, + contextPrecision: 0.82, + contextRecall: 0.74, + answerRelevance: 0.88, + }); + expect(result.provider?.attributes).toMatchObject({ + 'eval.ragas.faithfulness': 0.91, + 'eval.ragas.context_precision': 0.82, + 'eval.ragas.answer_relevance': 0.88, + }); + expect(result.provider?.attributes?.['eval.ragas.metric_names']).toEqual(expect.arrayContaining(['faithfulness', 'answer_relevance'])); + expect(result.provider?.attributes?.['eval.ragas.reference_sha256']).toMatch(/^[a-f0-9]{64}$/); + }); + + it('converts a single RAGAS result object and reports invalid rows', () => { + const single = convertRagasResult({ + question: 'Q', + answer: 'A', + contexts: [], + duration: 1.5, + }, 0, { runId: 'run' }); + expect(single.id).toBe('run-ragas-0'); + expect(single.performance.duration).toBe(1.5); + + const converted = convertRagasToEvalResults({ rows: [null] }); + expect(converted.evalResults).toEqual([]); + expect(converted.warnings[0].code).toBe('ragas.row_invalid'); + }); +}); + +describe('DeepEval integration adapter', () => { + it('converts DeepEval test results with metric scores, warnings, and RAG context', () => { + const converted = convertDeepEvalToEvalResults({ + testResults: [{ + testCaseId: 'case-9', + input: 'Explain the release', + actualOutput: 'It added RAGAS and DeepEval adapters.', + expectedOutput: 'Adapters were added.', + retrievalContext: ['release notes', 'adapter docs'], + success: false, + latencyMs: 750, + metrics: [ + { name: 'G-Eval', score: 0.66, success: false, reason: 'Needs stronger explanation.' }, + { name: 'Answer Relevancy', score: 0.93, success: true }, + ], + }], + }, { + runId: 'deepeval-run', + includeExplanations: true, + defaultModel: 'gpt-4o-mini', + timestamp: 1700000000000, + }); + + expect(converted.warnings).toEqual([]); + expect(converted.evalResults).toHaveLength(1); + const result = converted.evalResults[0]; + expect(result.id).toBe('deepeval-run-case-9'); + expect(result.operation).toBe('chat'); + expect(result.performance.duration).toBe(0.75); + expect(result.response.finishReasons).toEqual(['fail']); + expect(result.rag?.chunks).toHaveLength(2); + expect(result.provider?.attributes).toMatchObject({ + 'eval.deepeval.success': false, + 'eval.deepeval.failed_metric_count': 1, + 'eval.deepeval.g_eval': 0.66, + 'eval.deepeval.answer_relevancy': 0.93, + }); + expect(result.evidence?.warningCount).toBe(1); + expect(result.evidence?.warnings?.[0].message).toContain('Needs stronger explanation'); + }); + + it('supports object-style metrics and invalid row warnings', () => { + const result = convertDeepEvalResult({ + id: 'case-object', + input: { prompt: 'hello' }, + output: 'world', + metrics: { + toxicity: 0.01, + faithfulness: { score: 0.8, passed: true }, + }, + duration: 2, + }, 0, { runId: 'run' }); + + expect(result.id).toBe('run-case-object'); + expect(result.provider?.attributes).toMatchObject({ + 'eval.deepeval.toxicity': 0.01, + 'eval.deepeval.faithfulness': 0.8, + }); + expect(result.conversation?.messages[0].content).toBe('{"prompt":"hello"}'); + + const converted = convertDeepEvalToEvalResults({ results: [null] }); + expect(converted.evalResults).toEqual([]); + expect(converted.warnings[0].code).toBe('deepeval.row_invalid'); + }); +});