` | Generate synthetic test cases |
---
## SDK Usage
-Use OpenAgent Eval as a Python library:
-
```python
from openagent_eval.core import Engine
from openagent_eval.config import load_config
@@ -362,156 +171,59 @@ print(report.summary)
---
-## Evaluation Metrics
-
-### Corpus-Level (NEW)
-- Cross-document contradiction detection
-- Stale document detection
-- Divergent duplicate detection
-- Thematic coverage analysis
-
-### Retrieval
-- Context Precision
-- Context Recall
-- Precision@K
-- Recall@K
-- Hit Rate
-- Mean Reciprocal Rank (MRR)
-- Normalized Discounted Cumulative Gain (NDCG)
-
-### Generation
-- Faithfulness (NLI-based)
-- Answer Relevancy (NLI-based)
-- Hallucination Detection
-- Semantic Similarity
-- Exact Match
-- F1 Score
-- BLEU
-- ROUGE
-- BERTScore
-- LLM-as-Judge (custom criteria)
-
-### Performance
-- Latency (embedding, retrieval, LLM stages)
-
-### Cost
-- Token counting (prompt, completion, total)
-- Cost estimation per provider
-
-### Diagnosis (NEW)
-- Blame attribution (retrieval vs generation vs chunking)
-- 8 failure mode detection
-- Actionable recommendations
-
----
-
-## Supported Providers
-
-### LLM Providers
-- OpenAI
-- Anthropic
-- Google Gemini
-- Groq
-- OpenRouter
-- Ollama (local)
-
-### Retriever Providers
-- Chroma
-- Qdrant
-- Pinecone
-- Weaviate
-- FAISS
-- pgvector
-- Elasticsearch
-- BM25
-- Memory
-- HTTP
-
----
-
## Project Structure
```
openagent-eval/
-├── openagent_eval/ # Main package
-│ ├── cli/ # CLI commands (Typer)
-│ ├── config/ # Configuration system (Pydantic)
-│ ├── core/ # Core orchestration
-│ ├── datasets/ # Dataset loaders
-│ ├── metrics/ # Evaluation metrics
-│ │ ├── retrieval/ # Retrieval metrics
-│ │ ├── generation/ # Generation metrics
-│ │ ├── nli/ # NLI-based scoring (NEW)
-│ │ ├── performance/ # Performance metrics
-│ │ └── cost/ # Cost metrics
-│ ├── corpus/ # Corpus Health Auditor (NEW)
-│ ├── diagnosis/ # Component Diagnosis (NEW)
-│ ├── synthesis/ # Synthetic Test Data (NEW)
-│ ├── providers/ # LLM/Retriever adapters
-│ ├── reports/ # Report generators
-│ ├── plugins/ # Plugin system
-│ └── exceptions/ # Custom exceptions
-├── tests/ # Test suite
-├── pyproject.toml # Project configuration
-└── README.md
+├── openagent_eval/
+│ ├── cli/ # CLI commands (Typer)
+│ ├── config/ # Configuration system (Pydantic)
+│ ├── core/ # Core orchestration engine
+│ ├── metrics/ # 18+ evaluation metrics
+│ ├── providers/ # LLM & Retriever adapters
+│ ├── corpus/ # Corpus Health Auditor
+│ ├── diagnosis/ # Component Diagnosis
+│ ├── synthesis/ # Synthetic Test Data
+│ ├── reports/ # Report generators
+│ └── plugins/ # Plugin system
+├── tests/ # Test suite
+├── docs/ # Documentation
+└── examples/ # Tutorials and examples
```
---
-## Development
-
-### Setup
-
-```bash
-# Clone repository
-git clone https://github.com/openagenthq/openagent-eval.git
-cd openagent-eval
-
-# Install dependencies
-uv sync
-
-# Run tests
-uv run pytest
-
-# Run linter
-uv run ruff check .
-
-# Format code
-uv run ruff format .
-```
-
-### Running Tests
-
-```bash
-# Run all tests
-uv run pytest
+## Contributing
-# Run with coverage
-uv run pytest --cov=openagent_eval
+We welcome contributions of all kinds. Whether you're fixing a bug, adding a feature, or improving documentation — we'd love your help.
-# Run specific test file
-uv run pytest tests/unit/test_exceptions.py
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
-# Run CLI tests
-uv run pytest tests/unit/test_cli/
-```
+See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
---
-## Contributing
+## Community
-Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+- [Documentation](https://openagenthq.github.io/openagent-eval/)
+- [GitHub Issues](https://github.com/OpenAgentHQ/openagent-eval/issues) — Bug reports and feature requests
+- [GitHub Discussions](https://github.com/OpenAgentHQ/openagent-eval/discussions) — Ideas and questions
+- [Changelog](CHANGELOG.md) — Release history
---
## License
-Licensed under the Apache License, Version 2.0 - see [LICENSE](LICENSE) for details.
+Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.
---
-## Support
+
+
+**Built with care by the [OpenAgent](https://github.com/OpenAgentHQ) community.**
-- **Documentation:** [docs.openagenthq.com](https://docs.openagenthq.com)
-- **Issues:** [GitHub Issues](https://github.com/openagenthq/openagent-eval/issues)
-- **Discussions:** [GitHub Discussions](https://github.com/openagenthq/openagent-eval/discussions)
+