Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 0 additions & 66 deletions .ai/00_PROJECT.md
Original file line number Diff line number Diff line change
Expand Up @@ -622,22 +622,6 @@ Run evaluation with multiple thresholds and JSON output.

---

```bash
oaeval ui
```

Launch interactive TUI dashboard (Textual).

---

```bash
oaeval ui --config config.yaml
```

Launch TUI with specific configuration.

---

# Configuration

The framework should use YAML configuration.
Expand Down Expand Up @@ -745,7 +729,6 @@ Avoid tightly coupling metrics, providers, and report generators.
Version 1.0 (Stable Release):

* Generic LLM-as-Judge for custom criteria
* **Hybrid CLI UI** (Rich banner + Textual TUI dashboard)

Version 2.0:

Expand Down Expand Up @@ -1111,52 +1094,3 @@ result = evaluator.evaluate(...)
```

without maintaining two separate codebases. This architecture is scalable and aligns well with the long-term OpenAgentHQ ecosystem.

---

# Hybrid CLI UI (v1.0 Planned)

## Design Principle

- **Standard CLI:** All existing commands (`oaeval evaluate`, `oaeval audit`, etc.) use Rich for beautiful output
- **Interactive TUI:** `oaeval ui` launches Textual dashboard for power users
- **Zero Breaking Changes:** Existing workflows continue to work unchanged

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│ oaeval evaluate → Beautiful Rich output (no TUI) │
│ oaeval audit → Beautiful Rich output (no TUI) │
│ oaeval ui → Full Textual interactive dashboard │
└─────────────────────────────────────────────────────────────┘
```

## Tech Stack Additions

| Component | Technology | Reason |
|-----------|------------|--------|
| ASCII Art Banner | pyfiglet | Professional CLI branding |
| Interactive TUI | Textual | Full keyboard-driven dashboard |

## Dependencies

```toml
[project.optional-dependencies]
ui = ["textual>=0.40", "pyfiglet>=1.0"]
```

## UI Module Structure

```
openagent_eval/
├── cli/
│ ├── banner.py # ASCII art banner with Rich (NEW)
│ └── commands/
├── ui/ # Textual TUI application (NEW)
│ ├── __init__.py
│ ├── app.py # Main Textual App class
│ ├── screens.py # Dashboard screens (main, audit, evaluate, diagnose)
│ ├── widgets.py # Custom widgets (banner, results table, progress bars)
│ └── styles.tcss # Textual CSS for layout
```
56 changes: 3 additions & 53 deletions .ai/01_ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,19 +72,11 @@ openagent_eval/
│ │ ├── doctor.py # oaeval doctor
│ │ ├── audit.py # oaeval audit (NEW - corpus health)
│ │ ├── diagnose.py # oaeval diagnose (NEW - blame attribution)
│ │ ├── synth.py # oaeval synth (NEW - synthetic data)
│ │ └── ui.py # oaeval ui (NEW - Textual TUI)
│ │ └── synth.py # oaeval synth (NEW - synthetic data)
│ └── utils/ # CLI utilities
│ ├── __init__.py
│ └── display.py # Rich display helpers
├── ui/ # Textual TUI application (NEW)
│ ├── __init__.py
│ ├── app.py # Main Textual App class
│ ├── screens.py # Dashboard screens (main, audit, evaluate, diagnose)
│ ├── widgets.py # Custom widgets (banner, results table, progress bars)
│ └── styles.tcss # Textual CSS for layout
├── config/ # Configuration management
│ ├── __init__.py
│ ├── models.py # Pydantic configuration models
Expand Down Expand Up @@ -266,47 +258,7 @@ def run(config_path: str):
display_results(result)
```

### 2. UI Layer (`ui/`) — NEW

**Responsibility:** Interactive Textual TUI dashboard for power users.

**Key insight:** Standard CLI commands use Rich for beautiful output. `oaeval ui` launches a full Textual dashboard for interactive exploration.

**Components:**

| File | Responsibility |
|------|----------------|
| `app.py` | Main Textual App class |
| `screens.py` | Dashboard screens (main, audit, evaluate, diagnose) |
| `widgets.py` | Custom widgets (banner, results table, progress bars) |
| `styles.tcss` | Textual CSS for layout |

**Example:**
```python
from textual.app import App

class OAEvalDashboard(App):
"""Interactive evaluation dashboard."""

def compose(self):
yield Header()
yield DashboardScreen()
yield Footer()

BINDINGS = [
("1", "run_evaluation", "Run Eval"),
("2", "audit_corpus", "Audit Corpus"),
("q", "quit", "Quit"),
]
```

**Rules:**
- TUI is opt-in via `oaeval ui` command
- Standard CLI commands remain unchanged
- No business logic in UI layer
- Display-only (delegates to core modules)

### 3. Core Orchestration (`core/`)
### 2. Core Orchestration (`core/`)

**Responsibility:** Orchestrate the evaluation pipeline.

Expand Down Expand Up @@ -553,13 +505,11 @@ cli/ → core/ → datasets/
→ corpus/ (NEW)
→ diagnosis/ (NEW)
→ synthesis/ (NEW)
ui/ → core/ (same dependencies as cli/)
```

**Rules:**
1. `cli/` depends on everything
2. `ui/` depends on `core/` (same as `cli/`)
3. `core/` depends on `datasets/`, `metrics/`, `providers/`, `reports/`, `corpus/`, `diagnosis/`, `synthesis/`
2. `core/` depends on `datasets/`, `metrics/`, `providers/`, `reports/`, `corpus/`, `diagnosis/`, `synthesis/`
3. `metrics/`, `providers/`, `reports/` depend only on `utils/` and `types/`
4. `corpus/` depends on `providers/` (for LLM-as-Judge) and `utils/`
5. `diagnosis/` depends on `metrics/` (for metric results) and `utils/`
Expand Down
48 changes: 0 additions & 48 deletions .ai/02_AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -525,45 +525,6 @@ class MetricResult:

---

## Hybrid CLI Architecture (Implemented - Phase 14)

### Design Principle

- **Standard CLI:** All existing commands (`oaeval evaluate`, `oaeval audit`, etc.) use Rich for beautiful output
- **Interactive TUI:** `oaeval ui` launches Textual dashboard for power users
- **Zero Breaking Changes:** Existing workflows continue to work unchanged

### Tech Stack Additions

| Component | Technology | Reason |
|-----------|------------|--------|
| ASCII Art Banner | pyfiglet | Professional CLI branding |
| Interactive TUI | Textual | Full keyboard-driven dashboard |

### Dependencies

```toml
[project.optional-dependencies]
ui = ["textual>=0.40", "pyfiglet>=1.0"]
```

### UI Module Structure

```
openagent_eval/
├── cli/
│ ├── banner.py # ASCII art banner with Rich
│ └── commands/
├── ui/ # Textual TUI application
│ ├── __init__.py
│ ├── app.py # Main Textual App class
│ ├── screens.py # Dashboard screens (main, audit, evaluate, diagnose)
│ ├── widgets.py # Custom widgets (banner, results table, progress bars)
│ └── styles.tcss # Textual CSS for layout
```

---

## Development Phases

### Phase 1: Foundation ✅
Expand Down Expand Up @@ -657,15 +618,6 @@ openagent_eval/
- [x] Write documentation for CI/CD integration
- [x] Add GitHub Actions workflow example

### Phase 14: Hybrid CLI UI ✅
- [x] Add `pyfiglet` and `textual` to optional dependencies
- [x] Create `openagent_eval/cli/banner.py` — ASCII art banner with Rich
- [x] Create `openagent_eval/ui/` module structure
- [x] Implement Textual dashboard app
- [x] Create dashboard screens (main, audit, evaluate, diagnose)
- [x] Add custom widgets (banner, results table, progress bars)
- [x] Wire up `oaeval ui` command

---

## AI Library Dependencies
Expand Down
32 changes: 7 additions & 25 deletions .ai/03_CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@

| Field | Value |
|-------|-------|
| **Phase** | Phase 14 Complete - Hybrid CLI UI |
| **Status** | v1.0 complete; production-grade features in progress |
| **Last Updated** | 2026-07-11 |
| **Phase** | Phase 14 Complete - TUI Removed |
| **Status** | v1.0 complete; CLI-only interface |
| **Last Updated** | 2026-07-12 |
| **Next Action** | Another feature or v1.0 release |
| **Current Branch** | feature/hybrid-cli-ui |
| **Current Branch** | docs/remove-tui-references |
| **Remote** | https://github.com/OpenAgentHQ/openagent-eval.git |

---
Expand Down Expand Up @@ -224,6 +224,7 @@ SDK (openagent_eval - Core Evaluation API)
- [x] Wire up `oaeval ui` command
- [x] Add keyboard shortcuts and navigation
- [x] Test and polish (15 tests passing)
- [x] **REMOVED:** TUI removed from codebase, CLI-only interface retained

---

Expand Down Expand Up @@ -292,7 +293,7 @@ chore/{description} # Maintenance tasks
- Phase 8 is pending - Documentation
- **Phase 9-13 are NEW** — Production-grade RAG evaluation features
- **Phase 13 is COMPLETE** — CI/CD Integration (pytest plugin, threshold gating, `oaeval test` command)
- **Phase 14 is COMPLETE** — Hybrid CLI UI (Rich banner, Textual TUI dashboard, 15 tests)
- **Phase 14 is COMPLETE** — TUI removed, CLI-only interface retained
- **Phase 12 is COMPLETE** — Synthetic Test Data generator implemented (56 tests)
- CORRECTION: earlier "517+ passing / all phases complete" status was inaccurate —
the core pipeline did not actually evaluate. That gap is closed.
Expand All @@ -309,36 +310,17 @@ chore/{description} # Maintenance tasks
3. **NLI-based Scoring** — Word overlap is insufficient for production use
4. **Synthetic Test Data** — Bootstrap evaluation from your corpus
5. **CLI-first, Local-first** — No cloud services, no dashboards, no authentication
6. **Hybrid CLI UI** — Beautiful Rich output + optional Textual TUI dashboard (IMPLEMENTED)

---

## CLI UI Research Findings

**Key technologies discovered:**

| Technology | Purpose | Integration |
|------------|---------|-------------|
| pyfiglet | ASCII art banners | CLI banner generation |
| Rich Layout | Rows/columns, Panels, Tables | Standard CLI output |
| Textual | Full TUI framework | `oaeval ui` command |
| rich.live.Live | Real-time dashboards | Live updating displays |

**Architecture decision:**
- Hybrid approach: Rich for all commands, Textual for `oaeval ui`
- Zero breaking changes to existing CLI
- Opt-in interactivity for power users

---

## Change Log

| Date | Change |
|------|--------|
| 2026-07-12 | **TUI REMOVED** — Textual TUI dashboard removed, CLI-only interface retained |
| 2026-07-11 | **Phase 14 COMPLETE** — Hybrid CLI UI implemented (Rich banner, Textual TUI dashboard, 15 tests) |
| 2026-07-11 | **Phase 13 COMPLETE** — CI/CD Integration implemented (pytest plugin, threshold gating, `oaeval test` command) |
| 2026-07-11 | **Phase 12 COMPLETE** — Synthetic Test Data generator implemented (56 tests) |
| 2026-07-11 | Added Phase 14: Hybrid CLI UI (Rich banner + Textual TUI dashboard) |
| 2026-07-11 | Added CLI UI research findings to context |
| 2026-07-11 | Added Phase 9-13 (production-grade RAG eval): Corpus Auditor, LLM-as-Judge, Diagnosis, Synthetic Data, CI/CD |
| 2026-07-11 | Updated 00_PROJECT.md with production-grade vision and feature matrix |
Expand Down
29 changes: 0 additions & 29 deletions .ai/04_DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -590,35 +590,6 @@

---

### D022: Hybrid CLI UI Architecture

**Date:** 2026-07-11
**Status:** Accepted
**Context:** Need beautiful CLI output without breaking existing workflows. Users want both quick CLI evaluations and optional interactive dashboards.

**Decision:** Hybrid approach — Rich banner for all commands, Textual TUI via `oaeval ui`.

**Rationale:**
- Zero breaking changes to existing CLI workflows
- Opt-in interactivity for power users
- Beautiful ASCII art banner for professional branding
- Full Textual dashboard for interactive exploration
- Best of both worlds: simple CLI + powerful TUI

**Consequences:**
- New `ui/` module for Textual application
- New `cli/banner.py` for ASCII art banner
- New optional dependencies: `textual`, `pyfiglet`
- Standard CLI commands remain unchanged
- `oaeval ui` launches interactive dashboard

**Alternatives Considered:**
- Replace entire CLI with Textual: Rejected, breaking change, higher complexity
- Rich formatting only: Rejected, no interactive mode
- Textual for all commands: Rejected, overkill for simple evaluations

---

## Pending Decisions

None at this time.
Expand Down
Loading
Loading