OpenAgent Eval

The open-source evaluation framework for RAG systems and AI Agents.

Local-first. Framework-agnostic. Developer-friendly.

Getting Started · Documentation · Contributing

---

Why OpenAgent Eval?

Evaluating RAG systems shouldn't require a PhD or a cloud account. OpenAgent Eval brings pytest-level simplicity to AI evaluation — run from your terminal, get actionable insights, and ship with confidence.

• Local-first — No cloud services, dashboards, or authentication required
• Framework-agnostic — Works with LangChain, LlamaIndex, or any custom RAG pipeline
• 18+ metrics — Retrieval, generation, faithfulness, relevancy, performance, and cost
• Plugin-based — Extend with custom metrics, providers, and report generators
• Production-ready — Corpus auditing, failure diagnosis, and synthetic test data generation

---

Installation

pip install openagent-eval

For development:

git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval
uv sync

---

Quick Start

1. Initialize Configuration

oaeval init --interactive

2. Validate Configuration

oaeval validate config.yaml

3. Run Evaluation

oaeval run config.yaml

4. View Results

oaeval report latest

---

Features

Feature	Description
CLI + SDK	Use via command line or import as a Python library
Beautiful Reports	Terminal, Markdown, HTML, and JSON output formats
Failure Analysis	Identify why evaluations fail, not just that they failed
Corpus Health Auditor	Detect contradictions, staleness, and duplicates before evaluation
LLM-as-Judge Metrics	NLI-based scoring for faithfulness and relevancy
Component Diagnosis	Blame attribution — retrieval vs generation vs chunking
Synthetic Test Data	Auto-generate test cases from your knowledge base

---

Evaluation Metrics

Retrieval Metrics

• Context Precision & Recall
• Precision@K & Recall@K
• Hit Rate
• Mean Reciprocal Rank (MRR)
• Normalized Discounted Cumulative Gain (NDCG)

Generation Metrics

• Faithfulness (NLI-based)
• Answer Relevancy (NLI-based)
• Hallucination Detection
• Semantic Similarity
• Exact Match & F1 Score
• BLEU & ROUGE
• BERTScore
• LLM-as-Judge (custom criteria)

Performance & Cost

• Latency tracking (embedding, retrieval, LLM stages)
• Token counting (prompt, completion, total)
• Cost estimation per provider

---

Supported Providers

LLM Providers: OpenAI · Anthropic · Google Gemini · Groq · OpenRouter · Ollama

Retriever Providers: Chroma · Qdrant · Pinecone · Weaviate · FAISS · pgvector · Elasticsearch · BM25

---

CLI Reference

Command	Description
oaeval init	Create configuration file (interactive wizard)
oaeval run <config>	Run evaluation pipeline
oaeval report <id>	View evaluation reports
oaeval compare <a> <b>	Compare two experiments
oaeval list	List previous evaluations
oaeval validate <config>	Validate configuration
oaeval doctor	Check environment and dependencies
oaeval audit --corpus <path>	Audit corpus health
oaeval diagnose --report <id>	Diagnose failures and attribute blame
oaeval synth --corpus <path>	Generate synthetic test cases

---

SDK Usage

from openagent_eval.core import Engine
from openagent_eval.config import load_config

config = load_config("config.yaml")
engine = Engine(config)
report = await engine.run(dataset)

print(report.summary)

---

Project Structure

openagent-eval/
├── openagent_eval/
│   ├── cli/              # CLI commands (Typer)
│   ├── config/           # Configuration system (Pydantic)
│   ├── core/             # Core orchestration engine
│   ├── metrics/          # 18+ evaluation metrics
│   ├── providers/        # LLM & Retriever adapters
│   ├── corpus/           # Corpus Health Auditor
│   ├── diagnosis/        # Component Diagnosis
│   ├── synthesis/        # Synthetic Test Data
│   ├── reports/          # Report generators
│   └── plugins/          # Plugin system
├── tests/                # Test suite
├── docs/                 # Documentation
└── examples/             # Tutorials and examples

---

Contributing

We welcome contributions of all kinds. Whether you're fixing a bug, adding a feature, or improving documentation — we'd love your help.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

See CONTRIBUTING.md for detailed guidelines.

---

Community

• Documentation
• GitHub Issues — Bug reports and feature requests
• GitHub Discussions — Ideas and questions
• Changelog — Release history

---

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

---

Built with care by the OpenAgent community.