README.md

Agent Analytics SDK

The Agent Analytics SDK is a Python package designed to enhance the observability and tracing capabilities of your LLM-based agentic applications. It integrates seamlessly with popular libraries like LangChain and OpenAI, offering robust observability and tracing features to effectively monitor your applications.

Currently, the Agent Analytics SDK extends popular LLM tracing tools like Traceloop and Langtrace, adding improved semantics and additional functionalities.

OpenTelemetry Support

Agent Analytics SDK generates traces compliant with OpenTelemetry (OTEL) standards. We're also developing semantic conventions for the traces generated by this project.

Installation

You can install Agent Analytics SDK directly from the Git repository using pip.

Install via pip from Git Repository

To install the SDK directly from the Git repository:

pip install "git+https://github.com/AgentToolkit/agent-analytics.git@main#subdirectory=sdk"

Alternative: Clone and Install Locally

If you prefer to work with the source code or contribute to the project, you can clone the repository and install it locally.

1. Clone the Repository

   git clone https://github.com/AgentToolkit/agent-analytics.git

2. Navigate to the SDK Directory

   cd agent-analytics/sdk

3. Install the Package

   • Standard Installation

     pip install .

   • Editable Installation (Recommended for Development)

     pip install -e .

Quick Start

Before using Agent Analytics SDK, ensure you have all the necessary dependencies installed. It's recommended to use a virtual environment to manage your project dependencies.

Configuration

Agent Analytics SDK offers flexible configuration options to suit various tracing backends and observability requirements.

Tracer Types

• Default: Generates a log file.

Local file Usage Example

Below is an example demonstrating how to integrate Agent Analytics SDK with your LLM application.

from langchain_openai import ChatOpenAI
from dotenv import load_dotenv
from agent_analytics.instrumentation import agent_analytics_sdk 

# Load environment variables from .env file
load_dotenv()

# Initialize observability with agent_analytics_sdk
agent_analytics_sdk.initialize_observability()

# LangGraph App code ...

Explanation

1. agent_analytics_sdk Initialization: This is the primary namespace used to initialize observability.
2. Logs: The example generates logs stored in the run file directory.

Specify the log file:

from langchain_openai import ChatOpenAI
from dotenv import load_dotenv
from agent_analytics.instrumentation import agent_analytics_sdk 
from agent_analytics.instrumentation.configs import LogExporterConfig

# Load environment variables from .env file
load_dotenv()

# Create log configuration
config = LogExporterConfig(
    logs_dir_path="path/to/your/logs",  # optional
    log_filename="my_log"  # optional
)

# Initialize observability with agent_analytics_sdk
agent_analytics_sdk.initialize_observability(config=config)

# LangGraph App code ...

Explanation

1. Logs: this example will generate logs in the specified path.

Remote Collector Example

In the example below we are configuring the sdk to work with the Jeager collector. The specific collector URL below is linked to the currently running Agent Analytics instance (At: https://dashboard-agent-analytics.agent-analytics-9ca4d14d48413d18ce61b80811ba4308-0000.us-south.containers.appdomain.cloud/)

from agent_analytics.instrumentation.configs import RemoteExporterConfig
from agent_analytics.instrumentation import agent_analytics_sdk

agent_analytics_sdk.initialize_observability(
    tracer_type = agent_analytics_sdk.SUPPORTED_TRACER_TYPES.REMOTE,
    config = RemoteExporterConfig(
            endpoint='https://jaeger-collector-otlp-agent-analytics-jaeger.agent-analytics-9ca4d14d48413d18ce61b80811ba4308-0000.us-south.containers.appdomain.cloud/v1/traces',
            app_name='my_app',
    )
)

Instana Collector Example

In the example below we are configuring the sdk to work with the Instana collector. Note that for this to run, you must have the Instana agent running locally:

from agent_analytics.instrumentation.configs import RemoteExporterConfig
from agent_analytics.instrumentation import agent_analytics_sdk

agent_analytics_sdk.initialize_observability(
    tracer_type = agent_analytics_sdk.SUPPORTED_TRACER_TYPES.REMOTE,
    config = RemoteExporterConfig(
            endpoint='http://localhost:4318/v1/traces',
            app_name='my_app',
    )
)

Explanation

1. RemoteExporterConfig: This class is used to provide the remote collector configurations to the sdk
2. Traces: With this configurations the sdk will stream the spans to the specified collector

Custom Collector Example

In the example below, we configure the SDK to use a custom exporter. This exporter extends HTTPSpanExporter and dynamically determines request headers.

from agent_analytics.instrumentation import agent_analytics_sdk
from agent_analytics.instrumentation.configs import CustomExporterConfig
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter as HTTPSpanExporter

class MyCustomExporter(HTTPSpanExporter):
    def _export(self, serialized_data: bytes):
        data = serialized_data
        # More code to handle compression...

        return self._session.post(
            url=self._endpoint,
            data=data,
            verify=self._certificate_file,
            timeout=self._timeout,
            cert=self._client_cert,
            headers=self._dynamic_headers(),
        )

    def _dynamic_headers(self):
        return {
            # Compute headers dynamically 
            "header1": "value",
        }

config = CustomExporterConfig()
agent_analytics_sdk.initialize_observability(
    tracer_type=agent_analytics_sdk.SUPPORTED_TRACER_TYPES.CUSTOM,
    config=config,
    custom_exporter=MyCustomExporter(),
)

Manual Semantic Reporting Example

The Agent Analytics SDK supports the manual reporting of semantic data such as Issues, Resources, and Data Annotations.
In the example below, we demonstrate how to manually report an Issue and a Data Annotation.

from agent_analytics.instrumentation.utils import AIEventRecorder
from agent_analytics_common.interfaces.annotations import DataAnnotation
from agent_analytics_common.interfaces.issues import Issue, IssueLevel

AIEventRecorder.record_data_annotation(
    name="my annotation",
    annotation_type=DataAnnotation.Type.THOUGHT,
    annotation_title="Agent Thought", 
    annotation_content=f"{agent_thought_content}"       
)

AIEventRecorder.record_issue(
    name="timeout",
    description="Agent response timeout reached",
    level=IssueLevel.CRITICAL,
    effect="Blocking operation",
)

Explanation

This approach allows users to manually record semantic data that is then attached to the relevant spans.
Important: Currently, the related_to_ids parameter is not supported in the recording operation.

For further and more complete examples, refer to the examples folder.

Architecture

For information on the architecture of the SDK and the key components, please review architecture.md

Agent Analytics Task Flow Analysis

The agent_analytics_sdk instrumentation enables you to generate logs and analyze them using the Agent Analytics online service. With this service, you can:

1. Generate a Flow Graph.
2. Visualize interactive hierarchical task flows.
3. Explore aggregated metrics calculated for each task.

Known Issues

1. FastAPI Instrumentation
   The SDK automatically instruments all FastAPI applications, but it requires a specific import order to take effect. To ensure proper instrumentation, you can use one of the following approaches:

   Approach 1: Import FastAPI class after SDK initialization

   from agent_analytics.instrumentation import agent_analytics_sdk
   agent_analytics_sdk.initialize_observability()
   
   from fastapi import FastAPI
   app = FastAPI()

   Approach 2: Import fastapi module (order does not matter)

   import fastapi
   
   from agent_analytics.instrumentation import agent_analytics_sdk
   agent_analytics_sdk.initialize_observability()
   
   app = fastapi.FastAPI()

   Important: Avoid the following import order, as it will prevent FastAPI instrumentation:

   from fastapi import FastAPI
   
   from agent_analytics.instrumentation import agent_analytics_sdk
   agent_analytics_sdk.initialize_observability()
   
   # app will not be instrumented
   app = FastAPI()

Support and Feedback

For any questions, feedback, or assistance, Our team is here to help ensure you get the most out of the Agent Analytics SDK.

---