diff --git a/README.md b/README.md index 2b86d1cd..24c6ad57 100644 --- a/README.md +++ b/README.md @@ -1 +1,193 @@ -Minimal README.md to please the pyproject file +# FORGE: Framework for Orchestrating Runtime Gen-AI Experiments + +FORGE is a comprehensive test harness designed for **CI-first, reproducible and observable performance & scale testing** of AI/ML workloads, specifically targeting OpenShift platforms. Developed and maintained by the **Red Hat PSAP (Performance and Scale for AI Platforms)** team. + +## ๐ŸŽฏ Purpose + +FORGE enables systematic performance and scale testing of AI/ML workloads with: + +- **Reproducible testing**: Consistent test environments and methodologies +- **Observable results**: Comprehensive metrics collection and visualization +- **CI/CD integration**: Automated testing pipelines for continuous validation +- **Scalability analysis**: Performance characteristics across different scales +- **OpenShift optimization**: Tailored for container orchestration platforms + +## ๐Ÿ—๏ธ Architecture + +FORGE works in cooperation with [Fournos](https://github.com/openshift-psap/fournos) to provide a complete testing ecosystem for AI workloads. + +### Core Components + +- **`core`**: Fundamental framework components (DSL, launcher, CI entrypoint, notifications) +- **`caliper`**: Artifact post-processing engine for parsing, visualization, and KPI analysis +- **`fournos_launcher`**: Integration with Fournos for orchestration + +## ๐Ÿš€ Quick Start + +### Installation + +```bash +# Clone the repository +git clone https://github.com/openshift-psap/forge.git +cd forge + +# Install core dependencies +pip install -e . + +# Install with optional backends +pip install -e '.[caliper]' + +# Install development dependencies +pip install -e '.[dev]' +``` + +### Container Development Environment + +FORGE provides a containerized development environment using the forge_launcher: + +```bash +# Check environment status +./bin/forge_launcher status + +# Build the FORGE container image +./bin/forge_launcher build + +# Create/recreate the development container +./bin/forge_launcher recreate + +# Enter the containerized development environment +./bin/forge_launcher enter + +# Run commands directly in the container +./bin/forge_launcher enter "python -m pytest" +``` + +### Basic Usage + +```bash +./projects/skeleton/orchestration/ci.py --help +./projects/skeleton/orchestration/ci.py prepare +./projects/skeleton/orchestration/ci.py test +``` + +```bash +./projects/skeleton/orchestration/cli.py --help +./projects/skeleton/orchestration/cli.py precleanup +./projects/skeleton/orchestration/cli.py prepare +./projects/skeleton/orchestration/cli.py test +``` + +## ๐Ÿ“Š Key Features + +### Core Framework +- **DSL (Domain Specific Language)**: Test definition and configuration +- **CI Integration**: Continuous integration entrypoints +- **Notification System**: Alert and reporting mechanisms +- **Image Management**: Container orchestration support + +### Caliper - Artifact Processing +- **Parse**: Traverse and parse test artifact trees +- **Visualize**: Generate plots and HTML reports from unified models +- **KPI Management**: Generate, import, export, and analyze key performance indicators +- **Multi-backend Support**: Export to OpenSearch, S3, and MLflow +- **AI Evaluation**: Export AI evaluation metrics in JSON format + +## ๐Ÿ“ Project Structure + +``` +forge/ +โ”œโ”€โ”€ projects/ # Main project modules +โ”‚ โ”œโ”€โ”€ caliper/ # Artifact post-processing +โ”‚ โ”œโ”€โ”€ core/ # Framework core components +โ”‚ โ”œโ”€โ”€ matrix_benchmarking/ # Performance dashboards +โ”‚ โ”œโ”€โ”€ llm_d/ # LLM deployment tools +โ”‚ โ”œโ”€โ”€ fournos_launcher/ # Fournos integration +โ”‚ โ””โ”€โ”€ skeleton/ # Project templates +โ”œโ”€โ”€ docs/ # Documentation +โ”œโ”€โ”€ specs/ # Technical specifications +โ”œโ”€โ”€ bin/ # Executable scripts +โ”œโ”€โ”€ tests/ # Test suites +โ””โ”€โ”€ vaults/ # Configuration vaults +``` + +## ๐Ÿ”ง Configuration + +### Dependencies + +**Core Requirements:** +- Python 3.12+ +- Click (CLI framework) +- PyYAML (configuration) +- JSONSchema (validation) +- Pydantic (data models) + +**Optional Backends:** +- **OpenSearch**: For KPI indexing and search (`opensearch-py`) +- **MLflow**: For experiment tracking (`mlflow`) + +**Visualization:** +- **Plotly/Dash**: Interactive dashboards +- **Pandas**: Data processing + +## ๐Ÿงช Testing + +```bash +# Run unit tests +pytest projects/core/tests/ + +# Run with coverage +pytest --cov=projects projects/core/tests/ + +# Run integration tests +pytest -m integration + +# Run performance tests (slow) +pytest -m slow +``` + +## ๐Ÿค Contributing + +### Development Setup + +```bash +# Install pre-commit hooks +pre-commit install + +# Run code formatting +ruff format projects/ +ruff check projects/ +``` + +### Code Style +- **Ruff**: An extremely fast Python linter and code formatter +- **Target**: Python 3.12+ compatibility + +## ๐Ÿ“– Documentation + +- **Specifications**: `/specs/` - Detailed technical specifications +- **Quickstart Guides**: `/specs/*/quickstart.md` - Getting started guides +- **API Documentation**: Auto-generated from docstrings +- **Documentation**: `/docs/` - Usage examples and tutorials + +## ๐Ÿ”— Related Projects + +- **[Fournos](https://github.com/openshift-psap/fournos)**: Job orchestration and execution platform + +## ๐Ÿ“„ License + +Licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details. + +## ๐Ÿ‘ฅ Team & Support + +**Maintained by**: Red Hat PSAP Team (`psap@redhat.com`) + +**Key Contributors**: +- Kevin Pouget (@kpouget) +- Alberto Perdomo (@albertoperdomo2) +- See [OWNERS](OWNERS) for the complete list + +For issues, feature requests, or contributions, please use the GitHub issue tracker. + +--- + +**Keywords**: `testing` `performance` `scale` `openshift` `ai` `mlops` `benchmarking` `ci-cd`