From 195524e0c079b27726c7b59c48209614493b3cfe Mon Sep 17 00:00:00 2001 From: Test User Date: Fri, 10 Jul 2026 18:08:58 +0530 Subject: [PATCH] docs: update documentation for CLI improvements --- CHANGELOG.md | 24 ++- README.md | 257 +++++++++++++++++++++--- docs/06_cli_spec.md | 467 ++++++++++++++++++++++++++++++++++++-------- docs/cli.md | 128 ++++++++++-- docs/quickstart.md | 60 +++++- 5 files changed, 810 insertions(+), 126 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8dd8d60..4f12720 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,13 +10,26 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), ### Added +- **CLI Improvements** + - Global error handler with friendly Rich output for `OpenAgentEvalError` subclasses + - Global flags: `--quiet`, `--json`, `--no-color`, `--verbose` + - Config auto-discovery (config.yaml/oaeval.yaml in cwd, OAEVAL_CONFIG env var) + - New `validate` command to check config without running evaluation + - Dry-run mode (`--dry-run` flag on run command) + - Shell completion support for bash, zsh, and fish + - Enhanced `doctor` command with API connectivity tests + - New `delete` command for removing old reports + - Enhanced `list` command with sorting (date/score/cost) and search filtering + - Enhanced `init` command with interactive wizard for provider/model selection + - JSON output support for all commands (`--json` flag) + - Phase 8: Documentation - Vision documentation - Problem statement - Product requirements - Architecture documentation - Project structure - - CLI specification + - CLI specification (updated with new commands and features) - Metric system documentation - Plugin system documentation - Coding guidelines @@ -37,10 +50,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), ### Changed - Improved README.md with badges and comprehensive documentation +- Updated CLI documentation with new commands and features +- Enhanced `init` command with interactive wizard +- Enhanced `run` command with dry-run mode and metrics override +- Enhanced `doctor` command with API connectivity tests +- Enhanced `list` command with sorting and filtering +- Improved error handling across all CLI commands ### Fixed -- (No fixes yet) +- Fixed error chaining in CLI commands (raise from) +- Fixed unused imports and variables --- diff --git a/README.md b/README.md index a220d3b..64c4233 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ OpenAgent Eval is a local-first, developer-friendly evaluation framework that ru - **Comprehensive Metrics** - Retrieval, generation, performance, and cost evaluation - **Beautiful Reports** - Terminal, Markdown, HTML, and JSON output formats - **Failure Analysis** - Identify why evaluations fail, not just that they failed +- **Developer Experience** - Shell completion, config auto-discovery, dry-run mode, and more --- @@ -52,38 +53,30 @@ uv sync oaeval init ``` -This creates a `config.yaml` file with default settings. +This creates a `config.yaml` file with default settings. Use the interactive wizard to select your provider, model, and metrics: -### 2. Configure Your Setup - -Edit `config.yaml`: +```bash +oaeval init --interactive +``` -```yaml -dataset: data/questions.json +### 2. Validate Configuration -retriever: - provider: chroma - settings: - collection_name: my_docs +```bash +oaeval validate config.yaml +``` -llm: - provider: openai - model: gpt-4o +Check your configuration without running the evaluation. -metrics: - - faithfulness - - answer_relevancy - - hallucination - - latency +### 3. Run Evaluation -output: terminal -output_dir: ./reports +```bash +oaeval run config.yaml ``` -### 3. Run Evaluation +Or use dry-run mode to preview the evaluation plan: ```bash -oaeval run config.yaml +oaeval run config.yaml --dry-run ``` ### 4. View Results @@ -106,14 +99,226 @@ oaeval report latest ## CLI Commands +### Core Commands + | Command | Description | |---------|-------------| -| `oaeval init` | Create configuration file | +| `oaeval init` | Create configuration file (interactive wizard) | | `oaeval run ` | Run evaluation pipeline | | `oaeval report ` | View evaluation reports | | `oaeval compare ` | Compare two experiments | | `oaeval list` | List previous evaluations | | `oaeval doctor` | Check environment and dependencies | +| `oaeval validate ` | Validate configuration | +| `oaeval delete ` | Delete evaluation reports | +| `oaeval completion ` | Generate shell completion scripts | + +### Global Flags + +| Flag | Description | +|------|-------------| +| `--quiet`, `-q` | Suppress non-essential output | +| `--json` | Output machine-readable JSON | +| `--no-color` | Disable color output | +| `--verbose`, `-v` | Enable verbose output | +| `--version`, `-V` | Show version and exit | + +### Shell Completion + +Enable tab completion for your shell: + +```bash +# Bash +oaeval completion bash >> ~/.bashrc + +# Zsh +oaeval completion zsh >> ~/.zshrc + +# Fish +oaeval completion fish > ~/.config/fish/completions/oaeval.fish +``` + +### Config Auto-Discovery + +OpenAgent Eval automatically finds your configuration file: + +1. `OAEVAL_CONFIG` environment variable +2. `config.yaml` or `config.yml` in current directory +3. `oaeval.yaml` or `oaeval.yml` in current directory +4. Parent directories up to filesystem root + +--- + +## Usage Examples + +### Validate Configuration + +```bash +oaeval validate config.yaml +``` + +Example output: + +``` +OpenAgent Eval - Configuration Validator +Config: config.yaml + +1. Checking YAML syntax... + OK YAML syntax valid + +2. Validating configuration schema... + OK Configuration schema valid + +3. Checking API keys... + OK All required API keys configured + +4. Checking dataset... + OK Dataset found: data/questions.json + Size: 12.5 KB + +5. Checking output directory... + OK Output directory exists: ./reports + +6. Checking provider configuration... + LLM: openai (gpt-4o) + Retriever: chroma + +7. Checking metrics... + Configured: 5 metrics + Retrieval: context_precision, context_recall, mrr + Generation: faithfulness, answer_relevancy + Performance: latency + Cost: token_count + +Summary: +PASSED Configuration is valid + +Ready to run: oaeval run +``` + +### Dry-Run Mode + +```bash +oaeval run config.yaml --dry-run +``` + +Example output: + +``` +OpenAgent Eval - Dry Run Mode + +Configuration Summary: + Config file: config.yaml + Dataset: data/questions.json + LLM: openai (gpt-4o) + Retriever: chroma + Output: terminal + Output dir: ./reports + +Metrics (5): + Retrieval: context_precision, context_recall, mrr + Generation: faithfulness, answer_relevancy + Performance: latency + Cost: token_count + +Dataset: + OK Loaded 500 items + + Sample item: + question: What is the capital of France? + answer: Paris is the capital of France. + ground_truth: Paris + +This was a dry run. No evaluations were performed. +Run 'oaeval run ' to execute the evaluation. +``` + +### Run with Metrics Override + +```bash +oaeval run config.yaml --metrics faithfulness,answer_relevancy,latency +``` + +### JSON Output + +```bash +oaeval run config.yaml --json +``` + +Example output: + +```json +{ + "status": "success", + "report_path": "reports/eval_2024_01_15.json", + "elapsed_seconds": 125.42, + "summary": { + "total_items": 500, + "successful_evaluations": 500, + "failed_evaluations": 0, + "metrics_summary": { + "faithfulness": 0.918, + "answer_relevancy": 0.892 + } + } +} +``` + +### List with Sorting + +```bash +oaeval list --sort score --limit 5 +``` + +### Delete Reports + +```bash +# Delete a specific report +oaeval delete report_2024_01_15 + +# Delete all reports +oaeval delete all --force +``` + +### Check Environment + +```bash +oaeval doctor --check-api +``` + +Example output: + +``` +OpenAgent Eval - Environment Check + +Environment Status + Component Status Details + Python OK v3.11.5 + openagent-eval OK v0.1.0 + typer OK CLI framework + rich OK Terminal UI + pydantic OK Data validation + +API Key Availability + Provider Environment Variable Status + OpenAI OPENAI_API_KEY Available + Gemini GEMINI_API_KEY Not set + Anthropic ANTHROPIC_API_KEY Available + +API Connectivity Tests + OK OpenAI: reachable + OK Anthropic: reachable + +Configuration: + OK Found config: config.yaml + +Summary: + OK Python version is compatible + OK Available providers: OpenAI, Anthropic + +Recommendations + - Set GEMINI_API_KEY for Gemini support +``` --- @@ -195,6 +400,9 @@ print(report.summary) openagent-eval/ ├── openagent_eval/ # Main package │ ├── cli/ # CLI commands (Typer) +│ │ ├── commands/ # Command implementations +│ │ ├── utils/ # CLI utilities +│ │ └── context.py # Global CLI context │ ├── config/ # Configuration system (Pydantic) │ ├── core/ # Core orchestration │ ├── datasets/ # Dataset loaders @@ -243,6 +451,9 @@ uv run pytest --cov=openagent_eval # Run specific test file uv run pytest tests/unit/test_exceptions.py + +# Run CLI tests +uv run pytest tests/unit/test_cli/ ``` --- diff --git a/docs/06_cli_spec.md b/docs/06_cli_spec.md index eef8c71..fd195c7 100644 --- a/docs/06_cli_spec.md +++ b/docs/06_cli_spec.md @@ -2,7 +2,7 @@ ## Overview -OpenAgent Eval provides a CLI interface via `oaeval` built with Typer and Rich. +OpenAgent Eval provides a CLI interface via `oaeval` built with Typer and Rich. The CLI offers a comprehensive set of commands for evaluating RAG systems, with features like config auto-discovery, shell completion, and machine-readable JSON output. --- @@ -20,6 +20,21 @@ uv add openagent-eval --- +## Global Flags + +These flags are available on all commands: + +| Flag | Short | Description | +|------|-------|-------------| +| `--quiet` | `-q` | Suppress non-essential output | +| `--json` | | Output machine-readable JSON | +| `--no-color` | | Disable color output | +| `--verbose` | `-v` | Enable verbose output | +| `--version` | `-V` | Show version and exit | +| `--help` | `-h` | Show help message | + +--- + ## Commands ### `oaeval init` @@ -29,21 +44,143 @@ Create a new evaluation configuration. **Usage:** ```bash -oaeval init +oaeval init [OPTIONS] ``` +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `--config`, `-c` | Path to create configuration file | config.yaml | +| `--force`, `-f` | Overwrite existing configuration file | false | +| `--interactive/--no-interactive` | Use interactive wizard or defaults | --interactive | + **Behavior:** -- Interactively prompts for configuration options +- Interactively prompts for configuration options (when `--interactive`) - Creates `config.yaml` in the current directory - Sets up default metrics and output settings **Example output:** ``` -✓ Configuration created: config.yaml -✓ Default metrics: faithfulness, answer_relevancy, hallucination -✓ Output format: terminal +OpenAgent Eval - Configuration Wizard + +Let's set up your evaluation configuration. + +1. Dataset Configuration + Path to dataset file [data/questions.json]: + +2. LLM Provider + 1. OpenAI (GPT-4, GPT-4o, etc.) + 2. Anthropic (Claude) + 3. Google Gemini + 4. Groq (fast inference) + 5. OpenRouter (multi-provider) + Select provider [1]: + +3. Model (openai) + 1. GPT-4o (recommended) + 2. GPT-4o Mini (faster, cheaper) + 3. GPT-4 Turbo + 4. GPT-3.5 Turbo (legacy) + Select model [1]: + +4. Retriever + 1. ChromaDB (local, default) + 2. Qdrant + 3. Pinecone (cloud) + 4. Weaviate + 5. FAISS (local) + 6. In-memory (testing) + Select retriever [1]: + +5. Metrics + 1. Quick (3 metrics) - Fast, basic evaluation + 2. Standard (5 metrics) - Balanced (recommended) + 3. Comprehensive (9 metrics) - Thorough, slower + Select metric preset [2]: + +6. Output Format + 1. terminal - Rich terminal output + 2. markdown - Markdown file + 3. html - HTML report + 4. json - JSON data + Select output format [1]: + +Configuration generated successfully! + +OK Configuration created: config.yaml + +Next steps: + 1. Review the configuration file + 2. Run 'oaeval validate' to check it + 3. Run 'oaeval run' to start evaluation +``` + +--- + +### `oaeval validate` + +Validate configuration without running evaluation. + +**Usage:** + +```bash +oaeval validate [CONFIG_PATH] +``` + +**Arguments:** + +| Argument | Description | Required | +|----------|-------------|----------| +| `CONFIG_PATH` | Path to configuration file | No (auto-discovered) | + +**Behavior:** + +- Validates YAML syntax +- Checks configuration schema +- Verifies API key availability +- Validates dataset file existence +- Shows warnings and recommendations + +**Example output:** + +``` +OpenAgent Eval - Configuration Validator +Config: config.yaml + +1. Checking YAML syntax... + OK YAML syntax valid + +2. Validating configuration schema... + OK Configuration schema valid + +3. Checking API keys... + OK All required API keys configured + +4. Checking dataset... + OK Dataset found: data/questions.json + Size: 12.5 KB + +5. Checking output directory... + OK Output directory exists: ./reports + +6. Checking provider configuration... + LLM: openai (gpt-4o) + Retriever: chroma + +7. Checking metrics... + Configured: 5 metrics + Retrieval: context_precision, context_recall, mrr + Generation: faithfulness, answer_relevancy + Performance: latency + Cost: token_count + +Summary: +PASSED Configuration is valid + +Ready to run: oaeval run ``` --- @@ -55,50 +192,77 @@ Run an evaluation. **Usage:** ```bash -oaeval run +oaeval run [CONFIG_PATH] [OPTIONS] ``` +**Arguments:** + +| Argument | Description | Required | +|----------|-------------|----------| +| `CONFIG_PATH` | Path to configuration file | No (auto-discovered) | + **Options:** -| Option | Description | Default | -|--------|-------------|---------| -| `--output` | Report format (terminal, markdown, html, json) | terminal | -| `--output-dir` | Output directory for reports | ./reports | -| `--metrics` | Comma-separated list of metrics | all configured | -| `--verbose` | Enable verbose output | false | -| `--parallel` | Number of parallel evaluations | 4 | +| Option | Short | Description | Default | +|--------|-------|-------------|---------| +| `--output` | `-o` | Override output format (terminal, markdown, html, json) | None | +| `--verbose` | `-v` | Enable verbose output | false | +| `--dry-run` | | Validate config and show evaluation plan without running | false | +| `--metrics` | `-m` | Comma-separated list of metrics to run (overrides config) | None | **Example:** ```bash oaeval run config.yaml --output markdown --output-dir ./reports +oaeval run config.yaml --dry-run +oaeval run config.yaml --metrics faithfulness,answer_relevancy,latency ``` -**Example output:** +**Example output (normal run):** ``` -Loading dataset: data/questions.json (500 questions) -Running evaluation... +OpenAgent Eval v0.1.0 +Configuration: config.yaml -Retrieval Metrics: - Context Precision: 0.87 - Context Recall: 0.92 - Hit Rate: 0.95 +Loaded 500 items +Running evaluation (500 items)... +████████████████████████████████ 100% 0:02:05 +Complete! -Generation Metrics: - Faithfulness: 0.918 - Answer Relevancy: 0.892 - Hallucination Rate: 0.031 - -Performance: - Average Latency: 612ms - Total Time: 5m 12s +OK Evaluation complete! +Items: 500 | Errors: 0 +Report saved to: reports/eval_2024_01_15.json +``` -Cost: - Total Tokens: 1,234,567 - Estimated Cost: $2.17 +**Example output (dry run):** -Overall Grade: A +``` +OpenAgent Eval - Dry Run Mode + +Configuration Summary: + Config file: config.yaml + Dataset: data/questions.json + LLM: openai (gpt-4o) + Retriever: chroma + Output: terminal + Output dir: ./reports + +Metrics (5): + Retrieval: context_precision, context_recall, mrr + Generation: faithfulness, answer_relevancy + Performance: latency + Cost: token_count + +Dataset: + OK Loaded 500 items + + Sample item: + question: What is the capital of France? + answer: Paris is the capital of France. + ground_truth: Paris + +This was a dry run. No evaluations were performed. +Run 'oaeval run ' to execute the evaluation. ``` --- @@ -110,21 +274,28 @@ View evaluation reports. **Usage:** ```bash -oaeval report +oaeval report [OPTIONS] ``` +**Arguments:** + +| Argument | Description | Required | +|----------|-------------|----------| +| `REPORT_ID` | Report ID or 'latest' for the most recent report | Yes | + **Options:** -| Option | Description | Default | -|--------|-------------|---------| -| `--format` | Output format (terminal, markdown, html) | terminal | -| `--save` | Save report to file | false | +| Option | Short | Description | Default | +|--------|-------|-------------|---------| +| `--output` | `-o` | Output format (terminal, markdown, html, json) | terminal | +| `--output-dir` | `-d` | Directory where reports are stored | ./reports | **Example:** ```bash oaeval report latest -oaeval report exp_2024_01_15 --format markdown +oaeval report exp_2024_01_15 --output markdown +oaeval report latest --json ``` --- @@ -136,20 +307,28 @@ Compare two or more experiments. **Usage:** ```bash -oaeval compare [...] +oaeval compare [OPTIONS] ``` +**Arguments:** + +| Argument | Description | Required | +|----------|-------------|----------| +| `EXPERIMENT_A` | First experiment ID or path | Yes | +| `EXPERIMENT_B` | Second experiment ID or path | Yes | + **Options:** -| Option | Description | Default | -|--------|-------------|---------| -| `--format` | Output format (terminal, markdown, html) | terminal | -| `--diff` | Show only differences | false | +| Option | Short | Description | Default | +|--------|-------|-------------|---------| +| `--metrics` | `-m` | Specific metrics to compare (default: all) | None | +| `--output-dir` | `-d` | Directory where reports are stored | ./reports | **Example:** ```bash -oaeval compare exp_a exp_b --diff +oaeval compare exp_a exp_b +oaeval compare exp_a exp_b --metrics faithfulness,answer_relevancy ``` **Example output:** @@ -176,20 +355,26 @@ List previous evaluations. **Usage:** ```bash -oaeval list +oaeval list [OPTIONS] ``` **Options:** -| Option | Description | Default | -|--------|-------------|---------| -| `--limit` | Number of results | 10 | -| `--sort` | Sort by (date, score, cost) | date | +| Option | Short | Description | Default | +|--------|-------|-------------|---------| +| `--limit` | `-l` | Number of evaluations to show | 10 | +| `--output` | `-o` | Filter by output format | None | +| `--output-dir` | `-d` | Directory where reports are stored | ./reports | +| `--sort` | `-s` | Sort by (date, score, cost) | date | +| `--reverse` | `-r` | Reverse sort order | false | +| `--search` | | Search reports by config path or ID | None | **Example:** ```bash oaeval list --limit 5 --sort score +oaeval list --search "data/questions" --sort date +oaeval list --output json --limit 10 ``` **Example output:** @@ -197,11 +382,58 @@ oaeval list --limit 5 --sort score ``` Recent Evaluations -| ID | Date | Score | Cost | -|---------------------|------------|-------|--------| -| exp_2024_01_15_001 | 2024-01-15 | 92.1% | $2.17 | -| exp_2024_01_14_003 | 2024-01-14 | 89.7% | $1.82 | -| exp_2024_01_13_002 | 2024-01-13 | 83.2% | $1.95 | +| ID | Date | Config | Status | +|---------------------|------------|-------------------|--------| +| exp_2024_01_15_001 | 2024-01-15 | data/questions.json | OK | +| exp_2024_01_14_003 | 2024-01-14 | data/questions.json | OK | +| exp_2024_01_13_002 | 2024-01-13 | data/custom.json | FAILED | + +Showing 3 evaluations +``` + +--- + +### `oaeval delete` + +Delete evaluation reports. + +**Usage:** + +```bash +oaeval delete [OPTIONS] +``` + +**Arguments:** + +| Argument | Description | Required | +|----------|-------------|----------| +| `REPORT_ID` | Report ID to delete, or 'all' to delete all reports | Yes | + +**Options:** + +| Option | Short | Description | Default | +|--------|-------|-------------|---------| +| `--output-dir` | `-d` | Directory where reports are stored | ./reports | +| `--force` | `-f` | Skip confirmation prompt | false | + +**Example:** + +```bash +oaeval delete exp_2024_01_15_001 +oaeval delete all --force +``` + +**Example output:** + +``` +Delete Reports + +Report to delete: exp_2024_01_15_001 +Created: 2024-01-15T10:30:00Z + +Are you sure you want to delete this report? [y/N]: y + +OK Deleted report: exp_2024_01_15_001 ``` --- @@ -213,26 +445,76 @@ Check environment and dependencies. **Usage:** ```bash -oaeval doctor +oaeval doctor [OPTIONS] ``` +**Options:** + +| Option | Short | Description | Default | +|--------|-------|-------------|---------| +| `--verbose` | `-v` | Show detailed information | false | +| `--check-api` | | Test API connectivity (requires API keys) | false | + **Example output:** ``` -Environment Check +OpenAgent Eval - Environment Check + +Environment Status + Component Status Details + Python OK v3.11.5 + openagent-eval OK v0.1.0 + typer OK CLI framework + rich OK Terminal UI + pydantic OK Data validation + +API Key Availability + Provider Environment Variable Status + OpenAI OPENAI_API_KEY Available + Gemini GEMINI_API_KEY Not set + Anthropic ANTHROPIC_API_KEY Available + Groq GROQ_API_KEY Not set + +Configuration: + OK Found config: config.yaml + +Summary: + OK Python version is compatible + OK Available providers: OpenAI, Anthropic -✓ Python 3.11.5 -✓ openagent-eval 0.1.0 -✓ PyYAML installed -✓ Pydantic v2.4.0 +Recommendations + - Set GEMINI_API_KEY for Gemini support + - Set GROQ_API_KEY for Groq support +``` -Provider Status -✓ OpenAI API key configured -✗ Gemini API key not found -✓ Anthropic API key configured +--- -Recommendations -- Set GEMINI_API_KEY for Gemini support +### `oaeval completion` + +Generate shell completion script. + +**Usage:** + +```bash +oaeval completion +``` + +**Arguments:** + +| Argument | Description | Required | +|----------|-------------|----------| +| `SHELL` | Shell to generate completion for (bash, zsh, fish) | Yes | + +**Example:** + +```bash +# Install completion +oaeval completion bash >> ~/.bashrc +oaeval completion zsh >> ~/.zshrc +oaeval completion fish > ~/.config/fish/completions/oaeval.fish + +# Or just view the script +oaeval completion bash ``` --- @@ -256,33 +538,50 @@ llm: temperature: 0.0 metrics: - - faithfulness - - answer_relevancy - - hallucination - - latency - -output: terminal -output_dir: ./reports - -parallel: 4 -timeout: 300 + retrieval: + - context_precision + - context_recall + - mrr + generation: + - faithfulness + - answer_relevancy + performance: + - latency + cost: + - token_count + +report: + output: terminal + output_dir: ./reports ``` ### Configuration Fields | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| -| `dataset` | string | Yes | - | Path to dataset file | +| `dataset.path` | string | Yes | - | Path to dataset file | | `retriever.provider` | string | Yes | - | Retriever provider name | | `retriever.settings` | object | No | {} | Provider-specific settings | | `llm.provider` | string | Yes | - | LLM provider name | | `llm.model` | string | Yes | - | Model identifier | | `llm.temperature` | float | No | 0.0 | Sampling temperature | -| `metrics` | list | No | all | List of metrics to run | -| `output` | string | No | terminal | Report format | -| `output_dir` | string | No | ./reports | Output directory | -| `parallel` | int | No | 4 | Parallel evaluations | -| `timeout` | int | No | 300 | Timeout in seconds | +| `metrics.retrieval` | list | No | all | Retrieval metrics | +| `metrics.generation` | list | No | all | Generation metrics | +| `metrics.performance` | list | No | all | Performance metrics | +| `metrics.cost` | list | No | all | Cost metrics | +| `report.output` | string | No | terminal | Report format | +| `report.output_dir` | string | No | ./reports | Output directory | + +--- + +## Config Auto-Discovery + +OpenAgent Eval automatically finds your configuration file in the following order: + +1. **Explicit path**: Command argument (e.g., `oaeval run config.yaml`) +2. **Environment variable**: `OAEVAL_CONFIG` +3. **Current directory**: `config.yaml`, `config.yml`, `oaeval.yaml`, `oaeval.yml` +4. **Parent directories**: Walks up to filesystem root --- @@ -294,6 +593,8 @@ timeout: 300 | `GEMINI_API_KEY` | Google Gemini API key | For Gemini | | `ANTHROPIC_API_KEY` | Anthropic API key | For Anthropic | | `GROQ_API_KEY` | Groq API key | For Groq | +| `OPENROUTER_API_KEY` | OpenRouter API key | For OpenRouter | +| `OAEVAL_CONFIG` | Path to configuration file | No | --- diff --git a/docs/cli.md b/docs/cli.md index e8ea27e..3747636 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -8,10 +8,16 @@ OpenAgent Eval ships a [Typer](https://typer.tiangolo.com)-based command line in | --- | --- | | `--version`, `-V` | Show the installed version and exit | | `--help` | Show help for any command | +| `--quiet`, `-q` | Suppress non-essential output | +| `--json` | Output machine-readable JSON | +| `--no-color` | Disable color output | +| `--verbose`, `-v` | Enable verbose output | ```bash oaeval --version oaeval --help +oaeval --quiet run config.yaml +oaeval --json run config.yaml ``` ## `oaeval init` @@ -21,12 +27,44 @@ Create a `config.yaml` with default settings in the current directory. ```bash oaeval init oaeval init --config eval.yaml --force +oaeval init --interactive ``` | Option | Description | | --- | --- | | `--config`, `-c` | Path to create (default `config.yaml`) | | `--force`, `-f` | Overwrite an existing file without prompting | +| `--interactive/--no-interactive` | Use interactive wizard or defaults (default: `--interactive`) | + +The interactive wizard guides you through: +1. Dataset path selection +2. LLM provider selection (OpenAI, Anthropic, Gemini, Groq, OpenRouter) +3. Model selection based on provider +4. Retriever selection (Chroma, Qdrant, Pinecone, Weaviate, FAISS, Memory) +5. Metric preset (Quick, Standard, Comprehensive) +6. Output format selection + +## `oaeval validate` + +Validate configuration without running evaluation. + +```bash +oaeval validate config.yaml +oaeval validate +``` + +| Argument | Description | +| --- | --- | +| `config_path` | Path to configuration file (optional, auto-discovered if not provided) | + +The validate command checks: +- YAML syntax validity +- Configuration schema compliance +- API key availability +- Dataset file existence +- Output directory accessibility +- Provider configuration +- Metric configuration ## `oaeval run` @@ -35,16 +73,20 @@ Run an evaluation pipeline using a configuration file. ```bash oaeval run config.yaml oaeval run config.yaml --output html --verbose +oaeval run config.yaml --dry-run +oaeval run config.yaml --metrics faithfulness,answer_relevancy,latency ``` | Argument | Description | | --- | --- | -| `config_path` | Path to the configuration file | +| `config_path` | Path to the configuration file (optional, auto-discovered if not provided) | | Option | Description | | --- | --- | | `--output`, `-o` | Override output format: `terminal`, `markdown`, `html`, `json` | | `--verbose`, `-v` | Enable verbose output | +| `--dry-run` | Validate config and show evaluation plan without running | +| `--metrics`, `-m` | Comma-separated list of metrics to run (overrides config) | ## `oaeval report` @@ -53,6 +95,7 @@ View a stored evaluation report. ```bash oaeval report latest oaeval report exp-001 --output html +oaeval report latest --json ``` | Argument | Description | @@ -89,6 +132,8 @@ List previous evaluation runs. ```bash oaeval list --limit 20 +oaeval list --sort score --limit 5 +oaeval list --search "data/questions" --sort date ``` | Option | Description | @@ -96,6 +141,27 @@ oaeval list --limit 20 | `--limit`, `-l` | Number of evaluations to show (default `10`) | | `--output`, `-o` | Filter by output format | | `--output-dir`, `-d` | Reports directory (default `./reports`) | +| `--sort`, `-s` | Sort by: `date`, `score`, or `cost` (default `date`) | +| `--reverse`, `-r` | Reverse sort order | +| `--search` | Search reports by config path or ID | + +## `oaeval delete` + +Delete evaluation reports. + +```bash +oaeval delete exp-001 +oaeval delete all --force +``` + +| Argument | Description | +| --- | --- | +| `report_id` | Report ID to delete, or `all` to delete all reports | + +| Option | Description | +| --- | --- | +| `--output-dir`, `-d` | Reports directory (default `./reports`) | +| `--force`, `-f` | Skip confirmation prompt | ## `oaeval doctor` @@ -103,36 +169,70 @@ Check the environment, installed dependencies, and API key availability. ```bash oaeval doctor --verbose +oaeval doctor --check-api ``` | Option | Description | | --- | --- | | `--verbose`, `-v` | Show detailed information | +| `--check-api` | Test API connectivity (requires API keys) | + +The doctor command checks: +- Python version compatibility +- Required dependencies +- API key availability +- API connectivity (with `--check-api`) +- Configuration file presence Use this when something looks wrong after [installation](installation.md). +## `oaeval completion` + +Generate shell completion script. + +```bash +oaeval completion bash +oaeval completion zsh +oaeval completion fish +``` + +| Argument | Description | +| --- | --- | +| `shell` | Shell to generate completion for: `bash`, `zsh`, or `fish` | + +### Installing completion + +```bash +# Bash +oaeval completion bash >> ~/.bashrc + +# Zsh +oaeval completion zsh >> ~/.zshrc + +# Fish +oaeval completion fish > ~/.config/fish/completions/oaeval.fish +``` + ## Exit codes | Code | Meaning | | --- | --- | | `0` | Success | | `1` | Runtime or configuration error | -| `2` | Invalid CLI usage | - -## Shell completion +| `2` | Invalid CLI usage or configuration error | +| `3` | Dataset error | +| `4` | Provider error | +| `5` | Metric error | -`oaeval` is built on Typer and supports shell completion. Enable it for your shell: +## Config auto-discovery -```bash -# bash -eval "$(_OAeval_COMPLETE=bash_source oaeval)" +OpenAgent Eval automatically finds your configuration file: -# zsh -eval "$(_OAeval_COMPLETE=zsh_source oaeval)" - -# fish -_oaeval_completion fish | source -``` +1. Explicit path via command argument +2. `OAEVAL_CONFIG` environment variable +3. `config.yaml` or `config.yml` in current directory +4. `oaeval.yaml` or `oaeval.yml` in current directory +5. Parent directories up to filesystem root ## Next steps diff --git a/docs/quickstart.md b/docs/quickstart.md index b019ac8..f7c5a64 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -43,6 +43,20 @@ report: output_dir: ./reports ``` +!!! tip "Use the interactive wizard" + For a guided setup, use the interactive wizard: + + ```bash + oaeval init --interactive + ``` + + The wizard will prompt you to select: + - LLM provider (OpenAI, Anthropic, Gemini, Groq, OpenRouter) + - Model based on your provider + - Retriever (Chroma, Qdrant, Pinecone, Weaviate, FAISS, Memory) + - Metric preset (Quick, Standard, Comprehensive) + - Output format + !!! tip "Legacy shorthand still works" The loader also accepts the flat, single-string form used in older examples: @@ -53,7 +67,22 @@ report: It is normalized to the canonical nested structure automatically. -## 2. Prepare a dataset +## 2. Validate your configuration + +Before running an evaluation, validate your configuration: + +```bash +oaeval validate config.yaml +``` + +This checks: +- YAML syntax +- Configuration schema +- API key availability +- Dataset file existence +- Output directory accessibility + +## 3. Prepare a dataset OpenAgent Eval loads datasets in **JSON**, **JSONL**, **CSV**, or **PDF** format. Each item needs a `question`; `ground_truth`, `context`, and `ground_truth_contexts` are optional. @@ -68,7 +97,7 @@ OpenAgent Eval loads datasets in **JSON**, **JSONL**, **CSV**, or **PDF** format ] ``` -## 3. Run the evaluation +## 4. Run the evaluation ```bash oaeval run config.yaml @@ -80,7 +109,23 @@ Override the output format from the command line: oaeval run config.yaml --output html ``` -## 4. View the report +!!! tip "Use dry-run mode" + Preview the evaluation plan without running it: + + ```bash + oaeval run config.yaml --dry-run + ``` + + This shows what would be evaluated without incurring API costs. + +!!! tip "Override metrics" + Run specific metrics instead of all configured ones: + + ```bash + oaeval run config.yaml --metrics faithfulness,answer_relevancy,latency + ``` + +## 5. View the report ```bash oaeval report latest @@ -92,11 +137,18 @@ Other report commands: # List all stored evaluations oaeval list +# List with sorting +oaeval list --sort score --limit 5 + # Compare two experiments oaeval compare exp-001 exp-002 + +# Delete old reports +oaeval delete exp-001 +oaeval delete all --force ``` -## 5. Use the Python SDK +## 6. Use the Python SDK The same pipeline is available as a library so you can embed it in `pytest`. The `Engine.run` method is `async`, so drive it with `asyncio.run`: