From afde2c78931a6bf5b5f88ce7d26fcc54abe0122b Mon Sep 17 00:00:00 2001 From: Kevin Pouget Date: Tue, 5 May 2026 15:09:17 +0200 Subject: [PATCH 1/2] README: populate it --- README.md | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 195 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2b86d1cd..fa301f33 100644 --- a/README.md +++ b/README.md @@ -1 +1,195 @@ -Minimal README.md to please the pyproject file +# FORGE: Framework for Orchestrating Runtime Gen-AI Experiments + +FORGE is a comprehensive test harness designed for **CI-first, reproducible and observable performance & scale testing** of AI/ML workloads, specifically targeting OpenShift platforms. Developed and maintained by the **Red Hat PSAP (Performance and Scale for AI Platforms)** team. + +## ๐ŸŽฏ Purpose + +FORGE enables systematic performance and scale testing of AI/ML workloads with: + +- **Reproducible testing**: Consistent test environments and methodologies +- **Observable results**: Comprehensive metrics collection and visualization +- **CI/CD integration**: Automated testing pipelines for continuous validation +- **Scalability analysis**: Performance characteristics across different scales +- **OpenShift optimization**: Tailored for container orchestration platforms + +## ๐Ÿ—๏ธ Architecture + +FORGE works in cooperation with [Fournos](https://github.com/openshift-psap/fournos) to provide a complete testing ecosystem for AI workloads. + +### Core Components + +- **`core`**: Fundamental framework components (DSL, launcher, CI entrypoint, notifications) +- **`caliper`**: Artifact post-processing engine for parsing, visualization, and KPI analysis +- **`fournos_launcher`**: Integration with Fournos for orchestration + +## ๐Ÿš€ Quick Start + +### Installation + +```bash +# Clone the repository +git clone https://github.com/openshift-psap/forge.git +cd forge + +# Install core dependencies +pip install -e . + +# Install with optional backends +pip install -e '.[caliper]' + +# Install development dependencies +pip install -e '.[dev]' +``` + +### Container Development Environment + +FORGE provides a containerized development environment using the forge_launcher: + +```bash +# Check environment status +./bin/forge_launcher status + +# Build the FORGE container image +./bin/forge_launcher build + +# Create/recreate the development container +./bin/forge_launcher recreate + +# Enter the containerized development environment +./bin/forge_launcher enter + +# Run commands directly in the container +./bin/forge_launcher enter "python -m pytest" +``` + +### Basic Usage + +```bash +./projects/skeleton/orchestration/ci.py --help +./projects/skeleton/orchestration/ci.py prepare +./projects/skeleton/orchestration/ci.py test +``` + +``` +./projects/skeleton/orchestration/cli.py --help +./projects/skeleton/orchestration/cli.py precleanup +./projects/skeleton/orchestration/cli.py prepare +./projects/skeleton/orchestration/cli.py test +``` + +## ๐Ÿ“Š Key Features + +### Core Framework +- **DSL (Domain Specific Language)**: Test definition and configuration +- **CI Integration**: Continuous integration entrypoints +- **Notification System**: Alert and reporting mechanisms +- **Image Management**: Container orchestration support + +### Caliper - Artifact Processing +- **Parse**: Traverse and parse test artifact trees +- **Visualize**: Generate plots and HTML reports from unified models +- **KPI Management**: Generate, import, export, and analyze key performance indicators +- **Multi-backend Support**: Export to OpenSearch, S3, and MLflow +- **AI Evaluation**: Export AI evaluation metrics in JSON format + +## ๐Ÿ“ Project Structure + +``` +forge/ +โ”œโ”€โ”€ projects/ # Main project modules +โ”‚ โ”œโ”€โ”€ caliper/ # Artifact post-processing +โ”‚ โ”œโ”€โ”€ core/ # Framework core components +โ”‚ โ”œโ”€โ”€ matrix_benchmarking/ # Performance dashboards +โ”‚ โ”œโ”€โ”€ llm_d/ # LLM deployment tools +โ”‚ โ”œโ”€โ”€ fournos_launcher/ # Fournos integration +โ”‚ โ””โ”€โ”€ skeleton/ # Project templates +โ”œโ”€โ”€ docs/ # Documentation +โ”œโ”€โ”€ specs/ # Technical specifications +โ”œโ”€โ”€ bin/ # Executable scripts +โ”œโ”€โ”€ tests/ # Test suites +โ””โ”€โ”€ vaults/ # Configuration vaults +``` + +## ๐Ÿ”ง Configuration + +### Dependencies + +**Core Requirements:** +- Python 3.12+ +- Click (CLI framework) +- PyYAML (configuration) +- JSONSchema (validation) +- Pydantic (data models) + +**Optional Backends:** +- **OpenSearch**: For KPI indexing and search (`opensearch-py`) +- **S3**: For artifact storage (`boto3`) +- **MLflow**: For experiment tracking (`mlflow`) + +**Visualization:** +- **Plotly/Dash**: Interactive dashboards +- **Pandas**: Data processing +- **Kaleido**: Static image generation + +## ๐Ÿงช Testing + +```bash +# Run unit tests +pytest projects/core/tests/ + +# Run with coverage +pytest --cov=projects projects/core/tests/ + +# Run integration tests +pytest -m integration + +# Run performance tests (slow) +pytest -m slow +``` + +## ๐Ÿค Contributing + +### Development Setup + +```bash +# Install pre-commit hooks +pre-commit install + +# Run code formatting +ruff format projects/ +ruff check projects/ +``` + +### Code Style +- **MyPy**: Static type checking +- **Target**: Python 3.12+ compatibility + +## ๐Ÿ“– Documentation + +- **Specifications**: `/specs/` - Detailed technical specifications +- **Quickstart Guides**: `/specs/*/quickstart.md` - Getting started guides +- **API Documentation**: Auto-generated from docstrings +- **Examples**: `/docs/` - Usage examples and tutorials + +## ๐Ÿ”— Related Projects + +- **[Fournos](https://github.com/openshift-psap/fournos)**: Job orchestration and execution platform + +## ๐Ÿ“„ License + +Licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details. + +## ๐Ÿ‘ฅ Team & Support + +**Maintained by**: Red Hat PSAP Team (`psap@redhat.com`) + +**Key Contributors**: +- Kevin Pouget (@kpouget) +- Alberto Perdomo (@albertoperdomo2) +- See [OWNERS](OWNERS) for the complete list + +For issues, feature requests, or contributions, please use the GitHub issue tracker. + +--- + +**Keywords**: `testing` `performance` `scale` `openshift` `ai` `mlops` `benchmarking` `ci-cd` From b521f57602d590a878208c15e7dc77be02f94143 Mon Sep 17 00:00:00 2001 From: Kevin Pouget Date: Tue, 5 May 2026 15:17:55 +0200 Subject: [PATCH 2/2] Update README.md --- README.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index fa301f33..24c6ad57 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ FORGE provides a containerized development environment using the forge_launcher: ./projects/skeleton/orchestration/ci.py test ``` -``` +```bash ./projects/skeleton/orchestration/cli.py --help ./projects/skeleton/orchestration/cli.py precleanup ./projects/skeleton/orchestration/cli.py prepare @@ -123,13 +123,11 @@ forge/ **Optional Backends:** - **OpenSearch**: For KPI indexing and search (`opensearch-py`) -- **S3**: For artifact storage (`boto3`) - **MLflow**: For experiment tracking (`mlflow`) **Visualization:** - **Plotly/Dash**: Interactive dashboards - **Pandas**: Data processing -- **Kaleido**: Static image generation ## ๐Ÿงช Testing @@ -161,7 +159,7 @@ ruff check projects/ ``` ### Code Style -- **MyPy**: Static type checking +- **Ruff**: An extremely fast Python linter and code formatter - **Target**: Python 3.12+ compatibility ## ๐Ÿ“– Documentation @@ -169,7 +167,7 @@ ruff check projects/ - **Specifications**: `/specs/` - Detailed technical specifications - **Quickstart Guides**: `/specs/*/quickstart.md` - Getting started guides - **API Documentation**: Auto-generated from docstrings -- **Examples**: `/docs/` - Usage examples and tutorials +- **Documentation**: `/docs/` - Usage examples and tutorials ## ๐Ÿ”— Related Projects