diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 384440b..111060c 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -1,20 +1,25 @@ -name: Deploy Documentation +name: Documentation on: push: branches: [main] + pull_request: + branches: [main, develop] permissions: contents: read pages: write id-token: write +# Keyed by ref so concurrent PR doc builds do not serialize behind each other, +# while repeated pushes to the same branch still supersede one another. concurrency: - group: "pages" + group: pages-${{ github.ref }} cancel-in-progress: false jobs: - build: + # Job id is `docs` so it satisfies the required status check of that name. + docs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 @@ -30,20 +35,24 @@ jobs: - name: Install dependencies run: uv sync --extra docs + # --strict fails on broken internal links, which is how the dangling + # agent-page links were caught rather than shipped. - name: Build docs - run: uv run mkdocs build + run: uv run mkdocs build --strict - name: Upload artifact + if: github.event_name == 'push' && github.ref == 'refs/heads/main' uses: actions/upload-pages-artifact@v3 with: path: ./site deploy: + if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} runs-on: ubuntu-latest - needs: build + needs: docs steps: - name: Deploy to GitHub Pages id: deployment diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 2415b66..f3e1c3b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -31,3 +31,32 @@ jobs: - name: Run type checking run: uv run mypy src/whet/ tests/ --strict + + archetypes: + # Renders every archetype and verifies the *generated* project is usable. + # Without this, a template can ship code that never imports, crashes on the + # first step, or fails its own lint gate — none of which the unit tests notice. + name: Archetypes render and pass their own gates + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v4 + with: + version: "latest" + + - name: Set up Python + run: uv python install 3.11 + + - name: Install dependencies + run: uv sync --all-extras + + - name: Render each archetype and validate the generated project + run: uv run pytest tests/test_archetype_templates.py -v + + - name: Validate archetype skill composition + run: uv run pytest tests/test_archetypes.py -v + + - name: Enforce skill-authoring rules + run: uv run pytest tests/test_skill_hygiene.py -v diff --git a/.gitignore b/.gitignore index 57b6c11..bf0c9c5 100644 --- a/.gitignore +++ b/.gitignore @@ -14,6 +14,10 @@ venv/ env/ .pixi/ +# Ruff / mypy caches +.ruff_cache/ +.mypy_cache/ + # IDE .vscode/ .idea/ @@ -42,6 +46,10 @@ checkpoints/ # Data data/ +# ...but archetype templates ship data/ scaffolding that must be committed, +# otherwise a scaffolded project is missing directories its README documents. +!archetypes/*/template/**/data/ +!archetypes/*/template/**/data/** *.h5 *.hdf5 *.tfrecord diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..a17c945 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,99 @@ +# Changelog + +All notable changes to this project are documented in this file. + +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this +project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). While the +version is below 1.0.0, breaking changes are released in a minor bump. + +## [0.3.0] - 2026-07-24 + +A library-wide audit of all three pillars — skills, agents, and archetypes — measured +against Anthropic's Agent Skills guidance. Always-resident skill context is down 73%, +the agents concept is gone, every archetype generates a project that runs, and the +authoring rules are now enforced in CI rather than by discipline. + +### Removed + +- **The `agents/` directory and the agent concept.** The six "agents" were never Claude + Code subagents: four had no YAML frontmatter (so they installed with their H1 as the + entire description and were effectively untriggerable), all six installed into + `.claude/skills/` rather than `.claude/agents/`, `agent.toml` was never parsed by any + code path, and both "blocking" `action.yml` files ran `pixi` against a repo with no + `pixi.toml`. Roughly 77% of their content duplicated existing skills, often diverging + from them. GSD owns agency; whet owns domain knowledge, and knowledge is a skill. +- **`whet install --agents-only` and `--skills-only` flags**, along with + `discover_agents()` and the `agents_dir` config field. +- **Skills `dvc` and `cv-model-selection`.** Guidance that referenced DVC as a generic + concept was rewritten tool-neutral rather than left pointing at a deleted skill. + +### Added + +- **`model-evaluation`** — detection mAP/IoU via `supervision`, confusion matrices, + per-class and per-size breakdowns, deployment-threshold selection, per-slice failure + analysis, and eval-as-CI regression gates. +- **`pydantic-ai`** — typed LLM/VLM structured outputs and VLM-in-the-loop auto-labeling. +- **`data-pipelines`** — storage-format selection, group-aware leakage-preventing dataset + splitting, schema evolution, and data-quality validation. +- **`whet install --prune`** — removes skills that no longer exist upstream. Scoped by a + `.whet-manifest.json` recording what whet installed, so skills placed in the same + directory by other tools are never touched. +- **`whet install --include-extras`** and a `tier` field in `skill.toml`. Skills outside + the flagship path are opt-in: `aws-sagemaker`, `github-repo-setup`, `gradio`, + `huggingface`, `kubernetes`, `mlflow`, `vscode`. +- **`whet init --with-recommended` / `--no-skills`.** +- **Progressive disclosure** across the library: 29 of 32 skills are now a thin index + plus a `references/` directory loaded on demand — 155 reference files in total. +- **CI enforcement.** A new `archetypes` job renders every archetype and runs the + toolchain against the generated project, plus guards for skill-authoring rules + (line ceiling, reference-link integrity, orphaned references, one-level-deep + references, tables of contents, trigger-first and third-person descriptions, + listing budget) and for archetype composition and documented directory trees. + +### Changed + +- **`whet init` now installs the archetype's skills** (with their `references/`) instead + of printing a `whet add ...` hint. An archetype's value over a folder copy is the skill + set it composes, so leaving that as homework made `[skills]` advisory. +- **Skill `pydantic-strict` renamed to `pydantic`.** +- **All skill descriptions rewritten** as third-person, trigger-first "use this skill + when…" statements with explicit disambiguation between adjacent skills. They were topic + summaries, which under-trigger. +- **Every archetype template now generates a project that runs.** Previously four of six + generated only a four-file stub, and the two with templates produced projects that could + not import their own entry point, crashed on the first training step, or shipped no ONNX + despite being named for it. Template files went from 19 to 125. +- **pixi is the canonical environment manager** for generated projects; manifests migrated + from the deprecated `[project]` table to `[workspace]`. +- **Archetype skill composition** now reflects what each template actually ships — every + archetype requires `pixi` and `code-quality`, and `cv-inference-service` finally + requires `fastapi`. + +### Fixed + +- **The published wheel shipped no skills.** `[tool.hatch.build.targets.wheel]` packaged + only `src/whet`, and the bundled directories were resolved relative to the repository + root — so `uv tool install whet` produced a CLI that reported "No skills found" outside + a source checkout. The wheel now force-includes `skills/`, `archetypes/`, and + `settings/` under `whet/_data/`, and a single resolver prefers the packaged copy while + falling back to the repo layout for development. +- Template files under a `data/` path were silently excluded by the repository's own + `.gitignore`, so a clean checkout produced projects missing `data/raw`, `data/processed`, + and an entire Hydra `data` config group. +- `.ipynb` files were not treated as text by the scaffold engine, so `${package_name}` + inside a notebook was never substituted. +- Adapters now install a skill's `references/` directory (Claude, Antigravity) or inline + it (Cursor, Copilot), so deep-dive links resolve on every platform. +- Documented directory trees in archetype READMEs and docs pages are regenerated from the + templates; 78 documented-but-absent files were removed. + +## [0.2.1] - 2026-07-23 + +Published to PyPI outside this repository's tag-based release workflow; no corresponding +git tag exists. Recorded here for continuity. + +## [0.1.0] - 2026-02-09 + +Initial whet CLI implementation. + +[0.3.0]: https://github.com/ortizeg/whet/releases/tag/v0.3.0 diff --git a/CLAUDE.md b/CLAUDE.md index 363c24a..8114438 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,8 +13,7 @@ Distribution: `uvx whet` (one-shot) / `uv tool install whet` (permanent) ### What whet provides -- **30 Skills** — Best-practice knowledge modules (PyTorch Lightning, Pydantic, Docker, FastAPI, Hugging Face, AWS SageMaker, Gradio, Kubernetes, etc.) -- **6 Agents** — Pre-configured behavioral profiles (Expert Coder, ML Engineer, DevOps/Infra, Data Engineer, Code Review, Test Engineer) +- **32 Skills** — Best-practice knowledge modules (PyTorch Lightning, Pydantic, Docker, FastAPI, Hugging Face, AWS SageMaker, Gradio, Kubernetes, Model Evaluation, etc.) - **6+ Archetypes** — Complete project templates for common CV/ML project types - **CLI** — `whet add`, `whet install`, `whet list`, `whet search`, `whet doctor` - **Multi-platform** — Claude Code, Google Antigravity, Cursor, GitHub Copilot @@ -29,14 +28,13 @@ whet/ │ ├── core/ # Domain models (Pydantic: skill, config) │ └── registry/ # Skill discovery, search, dependency resolution ├── skills/ # 25+ skill definitions (SKILL.md + README.md + skill.toml) -├── agents/ # 4+ agent definitions (SKILL.md + README.md + agent.toml) ├── archetypes/ # 6+ project templates (README.md + archetype.toml + template/) ├── settings/ # Pre-built settings templates (claude.json, etc.) ├── docs/ # MkDocs Material documentation ├── tests/ # Self-tests (dynamic discovery, no hardcoded counts) ├── .github/workflows/ # CI/CD (test, lint, docs) — uses uv ├── pyproject.toml # uv-managed, publishable to PyPI -├── justfile # Task runner (replaces pixi tasks) +├── justfile # Task runner (wraps uv commands) └── mkdocs.yml # Documentation site config ``` @@ -63,22 +61,99 @@ uv run mypy src/whet/ tests/ --strict ## How to Add a New Skill -1. Create `skills//SKILL.md` — Must start with YAML frontmatter (`name`, `description`), be >500 chars, include code examples and headers -2. Create `skills//skill.toml` — Machine-readable metadata (category, tags, deps, compatibility) +1. Create `skills//SKILL.md` — YAML frontmatter (`name`, `description`), >500 chars, code examples and headers +2. Create `skills//skill.toml` — Machine-readable metadata (category, tier, tags, deps, compatibility) 3. Create `skills//README.md` — Must explain purpose (include words like "when", "use", "purpose") 4. Create `docs/skills/.md` — Documentation page (Purpose, When to Use, Key Patterns, Anti-Patterns) 5. Add nav entry to `mkdocs.yml` under Skills section Tests discover skills dynamically — no hardcoded lists to update. -## How to Add a New Agent +### Writing the `description` (this is the trigger) -1. Create `agents//SKILL.md` — Must be >500 chars -2. Create `agents//agent.toml` — Agent metadata (type: advisory/blocking, tags) -3. Create `agents//README.md` — Agent overview -4. If blocking agent: create `agents//action.yml` -5. Create `docs/agents/.md` — Documentation page -6. Add nav entry to `mkdocs.yml` under Agents section +Only the frontmatter is preloaded; the `description` is the entire discovery mechanism. +Write it third-person and trigger-first, and be a little "pushy": + +``` +description: > + Use this skill when . Reach for it any time you would otherwise + , even if the user doesn't say "" explicitly. Not for + (see ). +``` + +Always disambiguate against adjacent skills so two skills never compete for the same trigger. + +### Progressive disclosure (keep SKILL.md thin) + +A SKILL.md loads fully into context when triggered and **stays there for the session**, so +every line is a recurring token cost. Structure a skill as an index plus on-demand detail: + +``` +skills// +├── SKILL.md # ~120-200 line index: core patterns, conventions, anti-patterns +└── references/ # topic files, loaded only when needed + ├── .md + └── .md +``` + +Rules: +- Keep `SKILL.md` **under 500 lines** (target 120–200 for a split skill). +- Keep the 80%-case patterns, conventions, and anti-patterns **inline**; move depth to `references/`. +- End `SKILL.md` with a `## Deep dives` list giving each reference a *"read this when…"* trigger. +- **One level deep only** — a reference file must never link to another reference file. +- Reference files over 100 lines start with a table of contents; name them descriptively. + +The installer copies `references/` for directory-based platforms (Claude, Antigravity) and +inlines them for flat-file platforms (Cursor, Copilot), so no content is lost either way. + +### Scaffolding a project (`whet init`) + +`whet init ` renders the template **and installs the archetype's required +skills** into the new project, so it is usable immediately: + +```bash +whet init pytorch-training-project +whet init pytorch-training-project --with-recommended # + recommended skills +whet init pytorch-training-project --no-skills # scaffold files only +``` + +An archetype's value over a plain folder copy is the skill set it composes, so leaving +that as a printed hint made the `[skills]` list advisory. A name in `archetype.toml` that +no longer resolves is reported rather than silently skipped. + +### Keeping an install in sync (`--prune`) + +`whet install` copies skills but does not remove ones that were deleted upstream, so a +stale skill can linger in `.claude/skills/` long after it left the repo. `--prune` fixes +that: + +```bash +whet install --prune +``` + +Prune is **manifest-scoped**. Each install writes `.whet-manifest.json` into the target +directory recording which skills whet owns there. Prune only ever removes names from that +manifest, so skills installed by other tools (GSD, `interface-design`, hand-written ones) +that share the same directory are never touched. A skill merely filtered out of a run +(`--category`, or an `extra` skill without `--include-extras`) is not an orphan either — +only skills that no longer exist in `skills/` are removed. + +Installs predating the manifest are invisible to prune; clear those once with +`whet remove `. + +### Core vs extra tier + +`tier = "core"` (default) installs with `whet install`. `tier = "extra"` marks a skill as +opt-in — it is skipped unless the user passes `--include-extras`. Use `extra` for skills +that are real but outside the flagship path, so they don't dilute the default trigger surface. + +## Companion skills (not shipped by whet) + +whet does not ship a general product-UI ruleset — `gradio` covers ML demos only. For +application/dashboard UI work, use the external **`interface-design`** skill alongside whet +(upstream: https://github.com/Dammyjay93/interface-design). Its `SKILL.md` is self-contained; +`references/` only holds design-system templates. It also provides `/design-review` and +`/design-deslop`. ## How to Add a New Archetype @@ -109,7 +184,8 @@ description: > [skill] name = "skill-name" version = "1.0.0" -category = "cv-ml" # core | cv-ml | infra | experiment-tracking +category = "cv-ml" # core | cv-ml | infra | cloud | experiment-tracking +tier = "core" # core (default, installed) | extra (opt-in via --include-extras) tags = ["tag1", "tag2"] [dependencies] diff --git a/README.md b/README.md index 567ead1..26c99c8 100644 --- a/README.md +++ b/README.md @@ -31,8 +31,10 @@ whet install --global # Add specific skills to your project whet add pytorch-lightning wandb hydra-config -# Scaffold a new project +# Scaffold a new project (also installs the archetype's required skills) whet init pytorch-training-project +whet init pytorch-training-project --with-recommended # + recommended skills +whet init pytorch-training-project --no-skills # files only ``` ## What is whet? @@ -41,8 +43,7 @@ whet is a CLI tool that installs curated expert skill definitions into AI coding ### Flagship Collection: CV/ML -- **30 Skills** — PyTorch Lightning, Pydantic, Docker, ONNX, TensorRT, OpenCV, FastAPI, Hugging Face, AWS SageMaker, Gradio, Kubernetes, and more -- **6 Agents** — Expert Coder, ML Engineer, DevOps/Infra, Data Engineer, Code Review, Test Engineer +- **32 Skills** — PyTorch Lightning, Pydantic, Docker, ONNX, TensorRT, OpenCV, FastAPI, Hugging Face, AWS SageMaker, Gradio, Kubernetes, Model Evaluation, and more - **6+ Archetypes** — Training pipelines, inference services, notebooks, packages ### Multi-Platform @@ -102,7 +103,7 @@ just typecheck ## Contributing -Contributions welcome! See [CLAUDE.md](CLAUDE.md) for how to add new skills, agents, or archetypes. +Contributions welcome! See [CLAUDE.md](CLAUDE.md) for how to add new skills or archetypes. ## License diff --git a/agents/README.md b/agents/README.md deleted file mode 100644 index e103154..0000000 --- a/agents/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# Agents - -This directory contains 4 specialized agents for AI/CV projects. - -## Agent Overview - -| Agent | Role | Strictness | Deployment | -|-------|------|------------|------------| -| **Expert Coder** | Primary coding assistant | Advisory | Claude Code | -| **ML Engineer** | ML architecture and training guidance | Advisory | Claude Code | -| **Code Review** | Automated quality enforcement | Blocking | GitHub Action | -| **Test Engineer** | Testing and coverage enforcement | Blocking | GitHub Action | - -## Advisory vs Blocking - -- **Advisory** agents guide and suggest but do not prevent actions -- **Blocking** agents run in CI and must pass before merging - -## Agent Files - -Each agent contains: -- `SKILL.md` — Instructions for Claude Code -- `README.md` — Human-readable documentation -- `action.yml` — GitHub Action definition (blocking agents only) -- `examples/` — Example usage sessions diff --git a/agents/code-review/README.md b/agents/code-review/README.md deleted file mode 100644 index c684182..0000000 --- a/agents/code-review/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# Code Review Agent - -Automated code quality enforcement via GitHub Actions. - -## Strictness Level - -**BLOCKING** — Must pass to merge - -## What It Checks - -1. **Formatting** — Ruff format -2. **Linting** — Ruff lint with full rule set -3. **Types** — MyPy strict mode -4. **Security** — Bandit rules via Ruff -5. **Imports** — Proper sorting - -## Setup - -Add to `.github/workflows/code-review.yml` - -## Local Usage - -```bash -pixi run lint -pixi run format -pixi run typecheck -``` diff --git a/agents/code-review/SKILL.md b/agents/code-review/SKILL.md deleted file mode 100644 index 03435ee..0000000 --- a/agents/code-review/SKILL.md +++ /dev/null @@ -1,335 +0,0 @@ -# Code Review Agent - -You are a Code Review Agent that enforces code quality standards across the project. You act as an automated gatekeeper ensuring all code meets formatting, linting, type safety, and security requirements before it can be merged. - -## Purpose - -This agent automates code quality enforcement through a combination of tools: -- **Ruff** for formatting and linting -- **MyPy** for static type checking in strict mode -- **Bandit rules** (via Ruff) for security scanning - -All checks are **blocking** -- code that fails any check cannot be merged. - -## Checks Performed - -### 1. Formatting (Ruff Format) - -All code must be formatted with `ruff format`. This ensures consistent style across the project with zero configuration debates. - -```bash -# Check formatting (CI mode -- reports but does not fix) -ruff format --check . - -# Fix formatting (local development) -ruff format . -``` - -**What it enforces:** -- Consistent indentation (4 spaces) -- Line length (88 characters default, configurable) -- Trailing commas -- Quote style -- Blank line conventions -- Parenthesization - -### 2. Linting (Ruff Lint) - -Ruff lint checks are run with a comprehensive rule set covering code quality, correctness, and best practices. - -```bash -# Check linting (CI mode) -ruff check . - -# Fix auto-fixable issues -ruff check --fix . -``` - -**Rule sets enabled:** -```toml -# pyproject.toml -[tool.ruff.lint] -select = [ - "E", # pycodestyle errors - "W", # pycodestyle warnings - "F", # pyflakes - "I", # isort (import sorting) - "N", # pep8-naming - "UP", # pyupgrade - "B", # flake8-bugbear - "A", # flake8-builtins - "C4", # flake8-comprehensions - "DTZ", # flake8-datetimez - "T20", # flake8-print (no print statements) - "SIM", # flake8-simplify - "TCH", # flake8-type-checking - "ARG", # flake8-unused-arguments - "PTH", # flake8-use-pathlib - "ERA", # eradicate (commented-out code) - "RUF", # Ruff-specific rules - "S", # flake8-bandit (security) - "D", # pydocstyle (docstrings) -] -``` - -### 3. Type Checking (MyPy Strict) - -MyPy runs in strict mode, enforcing comprehensive type safety. - -```bash -# Run type checking -mypy src/ -``` - -**MyPy configuration:** -```toml -# pyproject.toml -[tool.mypy] -python_version = "3.11" -strict = true -warn_return_any = true -warn_unused_configs = true -disallow_untyped_defs = true -disallow_any_generics = true -check_untyped_defs = true -no_implicit_optional = true -warn_redundant_casts = true -warn_unused_ignores = true - -[[tool.mypy.overrides]] -module = "tests.*" -disallow_untyped_defs = false -``` - -### 4. Import Sorting - -Imports must be sorted according to isort conventions, enforced via Ruff. - -```bash -# Check import sorting -ruff check --select I . - -# Fix import sorting -ruff check --select I --fix . -``` - -**Expected import order:** -```python -# 1. Standard library -from __future__ import annotations - -import os -from pathlib import Path - -# 2. Third-party -import numpy as np -import torch -from pydantic import BaseModel - -# 3. Local -from myproject.models import Detector -from myproject.utils import setup_logging -``` - -### 5. Security Checks (Bandit via Ruff) - -Security-focused linting rules catch common vulnerabilities. - -```bash -# Run security checks -ruff check --select S . -``` - -**What it catches:** -- Hardcoded passwords or secrets -- Use of `eval()` or `exec()` -- Insecure temp file creation -- Weak cryptographic choices -- SQL injection patterns -- Unsafe YAML loading -- Subprocess with `shell=True` - -## Enforcement Rules - -### Blocking (Must Pass) -These checks **must pass** before code can be merged: -- Ruff format check -- Ruff lint (all enabled rules) -- MyPy strict mode -- Security checks -- Import sorting - -### Warnings (Non-Blocking) -These are logged but do not block merging: -- Code complexity warnings (McCabe) -- TODO/FIXME comments (tracked but allowed) -- Documentation coverage (encouraged, not enforced per-line) - -## Local Development Commands - -Run these commands locally before pushing to avoid CI failures: - -```bash -# Run all checks (recommended before pushing) -pixi run lint - -# Format code -pixi run format - -# Run type checking -pixi run typecheck - -# Fix all auto-fixable issues -pixi run fix - -# Run specific check -pixi run ruff check --select S . # Security only -pixi run ruff check --select I . # Imports only -pixi run ruff check --select D . # Docstrings only -``` - -## Pre-Commit Integration - -The code review checks can also run as pre-commit hooks for immediate feedback. - -```yaml -# .pre-commit-config.yaml -repos: - - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.8.0 - hooks: - - id: ruff - args: [--fix] - - id: ruff-format - - - repo: https://github.com/pre-commit/mirrors-mypy - rev: v1.13.0 - hooks: - - id: mypy - additional_dependencies: - - pydantic - - torch - - types-all -``` - -**Setup:** -```bash -# Install pre-commit hooks -pixi run pre-commit install - -# Run on all files -pixi run pre-commit run --all-files -``` - -## CI Integration - -The code review agent runs automatically on every pull request via GitHub Actions. - -### Workflow Configuration -```yaml -# .github/workflows/code-review.yml -name: Code Review - -on: - pull_request: - branches: [main, develop] - push: - branches: [main] - -jobs: - code-review: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: prefix-dev/setup-pixi@v0.8.1 - - run: pixi run ruff format --check . - - run: pixi run ruff check . - - run: pixi run mypy src/ -``` - -### Status Checks -Configure these as **required status checks** on the `main` branch: -- `code-review / ruff-format` -- `code-review / ruff-lint` -- `code-review / mypy` -- `code-review / security` - -## Common Failures and Fixes - -### Missing Type Hints -```python -# FAILS mypy strict -def process(data): - return data - -# PASSES -def process(data: np.ndarray) -> np.ndarray: - return data -``` - -### Print Statements -```python -# FAILS T20 -print("debug output") - -# PASSES: use logging -import logging -logger = logging.getLogger(__name__) -logger.info("debug output") -``` - -### Unsorted Imports -```python -# FAILS I001 -import torch -import os -from pathlib import Path - -# PASSES -import os -from pathlib import Path - -import torch -``` - -### Missing Docstrings -```python -# FAILS D100, D103 -def process(data: np.ndarray) -> np.ndarray: - return data - -# PASSES -def process(data: np.ndarray) -> np.ndarray: - """Process input array and return result.""" - return data -``` - -### Security Issues -```python -# FAILS S603 -import subprocess -subprocess.call(cmd, shell=True) - -# PASSES -subprocess.run(cmd, shell=False, check=True) # noqa: S603 (if needed) -``` - -## Suppressing Rules - -In rare cases, rules can be suppressed with inline comments: - -```python -# Suppress a specific rule on one line -result = eval(expression) # noqa: S307 -- validated input from trusted source - -# Suppress in pyproject.toml for specific files -# [tool.ruff.lint.per-file-ignores] -# "tests/**" = ["S101"] # Allow assert in tests -# "scripts/**" = ["T20"] # Allow print in scripts -``` - -**Rules for suppression:** -- Always include a justification comment -- Prefer fixing over suppressing -- Track all suppressions in code review -- Never suppress security rules without team approval diff --git a/agents/code-review/action.yml b/agents/code-review/action.yml deleted file mode 100644 index 76baa08..0000000 --- a/agents/code-review/action.yml +++ /dev/null @@ -1,39 +0,0 @@ -name: 'Code Review Agent' -description: 'Enforce code quality standards' - -inputs: - python-version: - description: 'Python version' - required: false - default: '3.11' - -runs: - using: 'composite' - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Install pixi - uses: prefix-dev/setup-pixi@v0.8.1 - with: - pixi-version: 'latest' - - - name: Run ruff format check - shell: bash - run: pixi run ruff format --check . - - - name: Run ruff lint - shell: bash - run: pixi run ruff check . - - - name: Run mypy - shell: bash - run: pixi run mypy src/ - - - name: Check import sorting - shell: bash - run: pixi run ruff check --select I . - - - name: Security checks - shell: bash - run: pixi run ruff check --select S . diff --git a/agents/code-review/agent.toml b/agents/code-review/agent.toml deleted file mode 100644 index fb08876..0000000 --- a/agents/code-review/agent.toml +++ /dev/null @@ -1,8 +0,0 @@ -[agent] -name = "code-review" -version = "1.0.0" -type = "blocking" -tags = ["code-review", "quality", "ci-cd", "enforcement"] - -[ci] -has_action = true diff --git a/agents/data-engineer/README.md b/agents/data-engineer/README.md deleted file mode 100644 index a642530..0000000 --- a/agents/data-engineer/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Data Engineer Agent - -Advisory agent specialized in data pipeline architecture, data quality, and data lifecycle management for ML/CV projects. - -## Purpose - -This agent guides the design and implementation of robust data pipelines for computer vision and machine learning workflows. It covers ETL/ELT patterns, data validation with Pydantic and Great Expectations, dataset versioning with DVC, splitting strategies that prevent data leakage, augmentation pipeline design, storage format selection (Parquet, Arrow, LMDB, TFRecord, WebDataset), data quality monitoring, large-scale processing, and schema evolution. - -## Strictness Level - -**ADVISORY** — This agent suggests best practices and warns about anti-patterns but does not block commits or CI pipelines. - -## When to Use - -- Building data ingestion or ETL pipelines for image datasets. -- Designing data quality checks and validation suites with Great Expectations. -- Choosing storage formats (Parquet vs. LMDB vs. WebDataset vs. TFRecord). -- Setting up dataset versioning with DVC and cloud remotes. -- Splitting datasets with stratification and group-aware leakage prevention. -- Designing data augmentation pipelines with Albumentations. -- Processing large-scale datasets that exceed memory (streaming, chunking). -- Evolving dataset schemas across project versions with migration paths. -- Reviewing data pipeline code for correctness, performance, and reproducibility. - -## Example Session - -``` -You: "I have 500K medical images from 2,000 patients. How should I organize the data pipeline?" - -Data Engineer Agent: "For medical imaging at this scale, I recommend: -1. Store images in LMDB for fast random access during training -2. Build a Parquet manifest with file hashes, dimensions, patient IDs, and labels -3. Use GROUP-AWARE splitting on patient_id — all images from one patient - must be in the same split to prevent data leakage -4. Version everything with DVC using an S3 remote -5. Add Great Expectations validation: no nulls, no duplicate hashes, - valid dimensions, balanced class distribution per split -6. Define augmentation configs with Pydantic, scaled by a strength parameter" -``` - -## Related Skills - -- `dvc` — Dataset versioning, pipeline reproducibility, and remote storage -- `pydantic-strict` — Validated configuration objects and schema definitions -- `testing` — Unit tests for transforms, integration tests for pipeline stages -- `pytorch-lightning` — DataModule integration for feeding validated data into training diff --git a/agents/data-engineer/SKILL.md b/agents/data-engineer/SKILL.md deleted file mode 100644 index c6512fe..0000000 --- a/agents/data-engineer/SKILL.md +++ /dev/null @@ -1,940 +0,0 @@ ---- -name: data-engineer -description: > - Data pipeline architecture and data quality advisory agent for ML/CV projects. - Guides ETL/ELT design, data validation, versioning with DVC, dataset splitting - strategies, augmentation pipeline design, storage format selection, data quality - monitoring, large-scale processing patterns, and schema evolution. ---- - -# Data Engineer Agent - -You are a Data Engineer Agent specializing in data pipeline architecture, data quality, and data lifecycle management for computer vision and machine learning projects. You provide expert guidance on building robust, reproducible, and scalable data pipelines that feed into ML training and inference workflows. - -## Core Principles - -1. **Data Quality First:** No model can compensate for bad data. Validate early, validate often, and treat data quality as a first-class concern alongside code quality. -2. **Reproducibility:** Every dataset version, transformation, and split must be traceable. Use content-addressable storage and deterministic pipelines. -3. **Schema as Contract:** Data schemas define the contract between pipeline stages. Schema changes require explicit migrations, never silent mutations. -4. **Immutable Datasets:** Published dataset versions are never modified in place. New versions are created with clear lineage back to their source. -5. **Fail Fast, Fail Loud:** Data pipelines must raise errors immediately on quality violations rather than silently propagating corrupt data downstream. - -## Data Pipeline Architecture - -### Decision Framework - -``` -Building a data pipeline? -├── Small dataset (< 10 GB) → Pandas / Polars + DVC -├── Medium dataset (10-500 GB) → Polars / DuckDB + DVC + Parquet -├── Large dataset (500 GB - 10 TB) → Apache Arrow / Spark + cloud storage -├── Streaming data → Kafka / Flink + Delta Lake -└── Annotation pipeline → Label Studio / CVAT + validation hooks -``` - -### Pipeline Structure with Pydantic - -```python -"""Data pipeline configuration and orchestration.""" - -from __future__ import annotations - -from enum import Enum -from pathlib import Path - -from pydantic import BaseModel, Field, field_validator -from loguru import logger - - -class StorageFormat(str, Enum): - """Supported storage formats for CV datasets.""" - PARQUET = "parquet" - ARROW = "arrow" - LMDB = "lmdb" - TFRECORD = "tfrecord" - WEBDATASET = "webdataset" - - -class PipelineConfig(BaseModel): - """Top-level data pipeline configuration.""" - name: str = Field(min_length=1, description="Pipeline name") - source_dir: Path - output_dir: Path - storage_format: StorageFormat = StorageFormat.PARQUET - num_workers: int = Field(ge=1, le=64, default=8) - chunk_size: int = Field(ge=100, default=10_000) - validate_on_write: bool = True - schema_version: str = Field(pattern=r"^\d+\.\d+\.\d+$", default="1.0.0") - - @field_validator("output_dir") - @classmethod - def output_must_differ_from_source(cls, v: Path, info) -> Path: # noqa: N805 - """Ensure output is not the same as source to prevent data loss.""" - if "source_dir" in info.data and v == info.data["source_dir"]: - msg = "output_dir must differ from source_dir" - raise ValueError(msg) - return v - - -# CORRECT: Pipeline with explicit stages -class DataPipeline: - """Orchestrates data processing stages.""" - - def __init__(self, config: PipelineConfig) -> None: - self.config = config - logger.info("Initializing pipeline: {}", config.name) - - def run(self) -> None: - """Execute all pipeline stages in order.""" - logger.info("Starting pipeline run for {}", self.config.name) - self.extract() - self.validate_raw() - self.transform() - self.validate_transformed() - self.load() - logger.info("Pipeline run complete") - - def extract(self) -> None: - """Extract raw data from source.""" - logger.info("Extracting from {}", self.config.source_dir) - ... - - def validate_raw(self) -> None: - """Validate raw data before transformation.""" - ... - - def transform(self) -> None: - """Apply transformations.""" - ... - - def validate_transformed(self) -> None: - """Validate transformed data before loading.""" - ... - - def load(self) -> None: - """Write to output in the configured format.""" - logger.info( - "Loading to {} as {}", - self.config.output_dir, - self.config.storage_format.value, - ) - ... - - -# WRONG: No config, no validation, no logging -def process_data(input_path, output_path): - import glob - files = glob.glob(f"{input_path}/*.jpg") - for f in files: - img = cv2.imread(f) - cv2.imwrite(f"{output_path}/{os.path.basename(f)}", img) -``` - -## ETL/ELT Patterns - -### ETL for Image Datasets - -```python -"""ETL pipeline for image classification datasets.""" - -from __future__ import annotations - -import hashlib -from pathlib import Path - -import polars as pl -from PIL import Image -from pydantic import BaseModel, Field -from loguru import logger - - -class ImageRecord(BaseModel): - """Validated image metadata record.""" - file_hash: str = Field(min_length=64, max_length=64, description="SHA-256 hash") - relative_path: str - width: int = Field(ge=1) - height: int = Field(ge=1) - channels: int = Field(ge=1, le=4) - label: str - split: str = Field(pattern=r"^(train|val|test)$") - file_size_bytes: int = Field(ge=1) - - -def compute_file_hash(path: Path) -> str: - """Compute SHA-256 hash for content-addressable storage.""" - hasher = hashlib.sha256() - with open(path, "rb") as f: - for chunk in iter(lambda: f.read(8192), b""): - hasher.update(chunk) - return hasher.hexdigest() - - -def extract_image_metadata(image_path: Path, label: str, split: str) -> ImageRecord: - """Extract and validate metadata from a single image.""" - img = Image.open(image_path) - width, height = img.size - channels = len(img.getbands()) - - return ImageRecord( - file_hash=compute_file_hash(image_path), - relative_path=str(image_path), - width=width, - height=height, - channels=channels, - label=label, - split=split, - file_size_bytes=image_path.stat().st_size, - ) - - -def build_manifest(data_dir: Path, output_path: Path) -> None: - """Build a validated dataset manifest as Parquet.""" - records: list[dict] = [] - - for split_dir in sorted(data_dir.iterdir()): - if not split_dir.is_dir(): - continue - split_name = split_dir.name - for class_dir in sorted(split_dir.iterdir()): - if not class_dir.is_dir(): - continue - label = class_dir.name - for img_path in sorted(class_dir.glob("*.jpg")): - record = extract_image_metadata(img_path, label, split_name) - records.append(record.model_dump()) - - df = pl.DataFrame(records) - df.write_parquet(output_path) - logger.info("Manifest written: {} records to {}", len(records), output_path) - - -# WRONG: No validation, no hashing, CSV instead of Parquet -# with open("manifest.csv", "w") as f: -# for img in glob.glob("data/**/*.jpg"): -# f.write(f"{img},{os.path.getsize(img)}\n") -``` - -## Data Validation with Great Expectations - -### Expectation Suites for CV Datasets - -```python -"""Data validation using Great Expectations and Pydantic.""" - -from __future__ import annotations - -import great_expectations as gx -from loguru import logger - - -def create_image_dataset_expectations(context: gx.DataContext) -> None: - """Define expectations for an image classification dataset.""" - suite = context.add_expectation_suite("image_classification_suite") - - # Schema expectations - suite.add_expectation( - gx.expectations.ExpectTableColumnsToMatchOrderedList( - column_list=[ - "file_hash", "relative_path", "width", "height", - "channels", "label", "split", "file_size_bytes", - ] - ) - ) - - # No null values in critical columns - for col in ["file_hash", "relative_path", "label", "split"]: - suite.add_expectation( - gx.expectations.ExpectColumnValuesToNotBeNull(column=col) - ) - - # Value range checks - suite.add_expectation( - gx.expectations.ExpectColumnValuesToBeBetween( - column="width", min_value=32, max_value=8192, - ) - ) - suite.add_expectation( - gx.expectations.ExpectColumnValuesToBeBetween( - column="height", min_value=32, max_value=8192, - ) - ) - - # Split values must be valid - suite.add_expectation( - gx.expectations.ExpectColumnValuesToBeInSet( - column="split", value_set=["train", "val", "test"], - ) - ) - - # No duplicate file hashes (detect duplicate images) - suite.add_expectation( - gx.expectations.ExpectColumnValuesToBeUnique(column="file_hash") - ) - - logger.info("Created expectation suite with {} expectations", len(suite.expectations)) - - -def run_validation(context: gx.DataContext, batch: gx.dataset.Dataset) -> bool: - """Run validation and log results.""" - result = context.run_validation_operator( - "action_list_operator", - assets_to_validate=[batch], - ) - success = result["success"] - if not success: - logger.error("Data validation FAILED — check Great Expectations report") - else: - logger.info("Data validation PASSED") - return success -``` - -## Data Versioning with DVC - -### DVC Pipeline Configuration - -```yaml -# dvc.yaml — Reproducible data pipeline -stages: - download: - cmd: python -m src.data.download --config configs/data.yaml - deps: - - src/data/download.py - - configs/data.yaml - outs: - - data/raw/ - - preprocess: - cmd: python -m src.data.preprocess --config configs/data.yaml - deps: - - src/data/preprocess.py - - configs/data.yaml - - data/raw/ - params: - - preprocess.image_size - - preprocess.normalize - outs: - - data/processed/ - - split: - cmd: python -m src.data.split --config configs/data.yaml - deps: - - src/data/split.py - - data/processed/ - params: - - split.train_ratio - - split.val_ratio - - split.test_ratio - - split.seed - outs: - - data/splits/train/ - - data/splits/val/ - - data/splits/test/ - metrics: - - data/splits/statistics.json: - cache: false - - validate: - cmd: python -m src.data.validate --config configs/data.yaml - deps: - - src/data/validate.py - - data/splits/ - metrics: - - data/validation_report.json: - cache: false -``` - -### DVC Remote Storage Setup - -```bash -# CORRECT: Configure DVC with cloud remote -dvc init -dvc remote add -d myremote s3://my-bucket/dvc-store -dvc remote modify myremote region us-east-1 - -# Track large data files -dvc add data/raw/images/ -git add data/raw/images.dvc data/raw/.gitignore -git commit -m "Track raw images with DVC" -dvc push - -# WRONG: Committing large data directly to git -# git add data/raw/images/ # Never do this! -# git lfs track "*.jpg" # DVC is better for ML datasets -``` - -## Dataset Splitting Strategies - -### Stratified Splitting for CV - -```python -"""Dataset splitting with stratification and leak prevention.""" - -from __future__ import annotations - -from pathlib import Path - -import polars as pl -from pydantic import BaseModel, Field, model_validator -from loguru import logger - - -class SplitConfig(BaseModel): - """Configuration for dataset splitting.""" - train_ratio: float = Field(gt=0.0, lt=1.0, default=0.7) - val_ratio: float = Field(gt=0.0, lt=1.0, default=0.15) - test_ratio: float = Field(gt=0.0, lt=1.0, default=0.15) - seed: int = 42 - stratify_column: str = "label" - group_column: str | None = None # Prevent data leakage across groups - - @model_validator(mode="after") - def ratios_must_sum_to_one(self) -> SplitConfig: - """Ensure split ratios sum to 1.0.""" - total = self.train_ratio + self.val_ratio + self.test_ratio - if abs(total - 1.0) > 1e-6: - msg = f"Split ratios must sum to 1.0, got {total}" - raise ValueError(msg) - return self - - -def split_dataset( - manifest: pl.DataFrame, - config: SplitConfig, -) -> dict[str, pl.DataFrame]: - """Split dataset with stratification and optional grouping. - - When group_column is set, all records sharing the same group value - (e.g., patient_id, video_id, scene_id) go into the same split. - This prevents data leakage from correlated samples. - """ - if config.group_column is not None: - return _group_aware_split(manifest, config) - return _stratified_split(manifest, config) - - -def _stratified_split( - df: pl.DataFrame, - config: SplitConfig, -) -> dict[str, pl.DataFrame]: - """Per-class stratified random split.""" - train_parts, val_parts, test_parts = [], [], [] - - for label in sorted(df[config.stratify_column].unique().to_list()): - subset = df.filter(pl.col(config.stratify_column) == label).sample( - fraction=1.0, seed=config.seed, shuffle=True, - ) - n = len(subset) - n_train = int(n * config.train_ratio) - n_val = int(n * config.val_ratio) - - train_parts.append(subset[:n_train]) - val_parts.append(subset[n_train : n_train + n_val]) - test_parts.append(subset[n_train + n_val :]) - - splits = { - "train": pl.concat(train_parts), - "val": pl.concat(val_parts), - "test": pl.concat(test_parts), - } - - for name, split_df in splits.items(): - logger.info("Split '{}': {} records", name, len(split_df)) - - return splits - - -def _group_aware_split( - df: pl.DataFrame, - config: SplitConfig, -) -> dict[str, pl.DataFrame]: - """Split by group to prevent data leakage (e.g., same patient in train+test).""" - groups = df[config.group_column].unique().sample( - fraction=1.0, seed=config.seed, shuffle=True, - ) - n = len(groups) - n_train = int(n * config.train_ratio) - n_val = int(n * config.val_ratio) - - train_groups = set(groups[:n_train].to_list()) - val_groups = set(groups[n_train : n_train + n_val].to_list()) - test_groups = set(groups[n_train + n_val :].to_list()) - - splits = { - "train": df.filter(pl.col(config.group_column).is_in(train_groups)), - "val": df.filter(pl.col(config.group_column).is_in(val_groups)), - "test": df.filter(pl.col(config.group_column).is_in(test_groups)), - } - - # Verify no leakage - assert train_groups.isdisjoint(val_groups), "Train/val group overlap!" - assert train_groups.isdisjoint(test_groups), "Train/test group overlap!" - assert val_groups.isdisjoint(test_groups), "Val/test group overlap!" - - for name, split_df in splits.items(): - logger.info("Split '{}': {} records ({} groups)", name, len(split_df), - split_df[config.group_column].n_unique()) - - return splits - - -# WRONG: Random split without stratification or group awareness -# train, val, test = np.split(df.sample(frac=1), [int(.7*len(df)), int(.85*len(df))]) -``` - -## Data Augmentation Pipeline Design - -### Albumentations Pipeline with Config - -```python -"""Data augmentation pipeline with Pydantic configuration.""" - -from __future__ import annotations - -import albumentations as A -from albumentations.pytorch import ToTensorV2 -from pydantic import BaseModel, Field -from loguru import logger - - -class AugmentationConfig(BaseModel): - """Augmentation pipeline configuration.""" - image_size: int = Field(ge=32, default=640) - strength: float = Field(ge=0.0, le=1.0, default=0.5) - normalize_mean: tuple[float, float, float] = (0.485, 0.456, 0.406) - normalize_std: tuple[float, float, float] = (0.229, 0.224, 0.225) - mosaic_probability: float = Field(ge=0.0, le=1.0, default=0.5) - mixup_alpha: float = Field(ge=0.0, default=0.2) - - -def build_train_transforms(config: AugmentationConfig) -> A.Compose: - """Build training augmentation pipeline scaled by strength.""" - s = config.strength - transforms = A.Compose([ - A.RandomResizedCrop( - height=config.image_size, - width=config.image_size, - scale=(0.5 + 0.3 * (1 - s), 1.0), - ), - A.HorizontalFlip(p=0.5), - A.VerticalFlip(p=0.1 * s), - A.ColorJitter( - brightness=0.2 * s, - contrast=0.2 * s, - saturation=0.2 * s, - hue=0.05 * s, - p=0.8, - ), - A.GaussNoise(p=0.3 * s), - A.GaussianBlur(blur_limit=(3, 7), p=0.2 * s), - A.CoarseDropout( - max_holes=int(8 * s), - max_height=int(config.image_size * 0.1), - max_width=int(config.image_size * 0.1), - p=0.3 * s, - ), - A.Normalize(mean=config.normalize_mean, std=config.normalize_std), - ToTensorV2(), - ], bbox_params=A.BboxParams(format="pascal_voc", label_fields=["class_labels"])) - - logger.info("Built training transforms (strength={}): {} ops", s, len(transforms)) - return transforms - - -def build_val_transforms(config: AugmentationConfig) -> A.Compose: - """Build deterministic validation transforms (no randomness).""" - return A.Compose([ - A.Resize(height=config.image_size, width=config.image_size), - A.Normalize(mean=config.normalize_mean, std=config.normalize_std), - ToTensorV2(), - ]) - - -# WRONG: Hard-coded transforms, no config, train augmentations on val set -# transform = transforms.Compose([ -# transforms.RandomHorizontalFlip(), # Applied to val too! -# transforms.ToTensor(), -# ]) -``` - -## Storage Format Selection - -### Format Comparison and Decision Guide - -``` -Choosing a storage format? -├── Tabular metadata (labels, splits, paths) -│ └── Parquet — columnar, compressed, fast filtering with Polars/DuckDB -├── Streaming large image datasets -│ ├── WebDataset (.tar shards) — best for distributed training -│ └── TFRecord — TensorFlow ecosystem, sequential reads -├── Random-access image datasets -│ └── LMDB — memory-mapped, zero-copy reads, fast random access -├── In-memory analytics / interchange -│ └── Apache Arrow IPC — zero-copy, language-agnostic -└── Small datasets (< 1 GB) - └── Image folders + Parquet manifest — simplest, debuggable -``` - -### Writing Parquet with Polars - -```python -"""Storage utilities for writing datasets in optimized formats.""" - -from __future__ import annotations - -from pathlib import Path - -import polars as pl -from loguru import logger - - -def write_parquet_partitioned( - df: pl.DataFrame, - output_dir: Path, - partition_by: str = "split", -) -> None: - """Write Parquet partitioned by split for efficient filtering.""" - for partition_value in df[partition_by].unique().sort().to_list(): - partition_df = df.filter(pl.col(partition_by) == partition_value) - partition_path = output_dir / f"{partition_by}={partition_value}" / "data.parquet" - partition_path.parent.mkdir(parents=True, exist_ok=True) - partition_df.write_parquet( - partition_path, - compression="zstd", - row_group_size=10_000, - ) - logger.info( - "Wrote partition {}={}: {} rows", - partition_by, partition_value, len(partition_df), - ) -``` - -### LMDB for Random-Access Image Datasets - -```python -"""LMDB dataset for fast random-access image loading.""" - -from __future__ import annotations - -import io -import pickle -from pathlib import Path - -import lmdb -from PIL import Image -from loguru import logger - - -def build_lmdb_dataset(image_paths: list[Path], output_path: Path) -> None: - """Pack images into an LMDB database for fast random access.""" - map_size = 1024 * 1024 * 1024 * 50 # 50 GB max - env = lmdb.open(str(output_path), map_size=map_size) - - with env.begin(write=True) as txn: - for idx, img_path in enumerate(image_paths): - img_bytes = img_path.read_bytes() - txn.put(f"{idx:08d}".encode(), img_bytes) - - txn.put(b"__len__", str(len(image_paths)).encode()) - - logger.info("Built LMDB dataset: {} images at {}", len(image_paths), output_path) - env.close() - - -def read_lmdb_image(env: lmdb.Environment, index: int) -> Image.Image: - """Read a single image from LMDB by index.""" - with env.begin(buffers=True) as txn: - img_bytes = txn.get(f"{index:08d}".encode()) - if img_bytes is None: - msg = f"Image at index {index} not found in LMDB" - raise KeyError(msg) - return Image.open(io.BytesIO(img_bytes)) -``` - -## Data Quality Monitoring - -### Continuous Quality Checks - -```python -"""Data quality monitoring for production pipelines.""" - -from __future__ import annotations - -from pydantic import BaseModel, Field -from loguru import logger -import polars as pl - - -class QualityReport(BaseModel): - """Summary of data quality checks.""" - total_records: int = Field(ge=0) - null_count: dict[str, int] - duplicate_count: int = Field(ge=0) - class_distribution: dict[str, int] - min_image_size: tuple[int, int] - max_image_size: tuple[int, int] - corrupted_files: list[str] - passed: bool - - -def run_quality_checks(df: pl.DataFrame) -> QualityReport: - """Run comprehensive quality checks on a dataset manifest.""" - null_counts = {col: df[col].null_count() for col in df.columns} - duplicates = df["file_hash"].is_duplicated().sum() - class_dist = dict( - df.group_by("label").len().sort("label") - .select(["label", "len"]) - .iter_rows() - ) - corrupted = df.filter( - (pl.col("width") < 32) | (pl.col("height") < 32) - )["relative_path"].to_list() - - report = QualityReport( - total_records=len(df), - null_count=null_counts, - duplicate_count=duplicates, - class_distribution=class_dist, - min_image_size=(df["width"].min(), df["height"].min()), - max_image_size=(df["width"].max(), df["height"].max()), - corrupted_files=corrupted, - passed=duplicates == 0 and len(corrupted) == 0 and all(v == 0 for v in null_counts.values()), - ) - - if report.passed: - logger.info("Quality checks PASSED: {} records", report.total_records) - else: - logger.error("Quality checks FAILED: {} duplicates, {} corrupted", - report.duplicate_count, len(report.corrupted_files)) - - return report -``` - -## Large-Scale Data Processing Patterns - -### Chunked Processing with Progress - -```python -"""Large-scale data processing utilities.""" - -from __future__ import annotations - -from pathlib import Path -from concurrent.futures import ProcessPoolExecutor, as_completed - -from pydantic import BaseModel, Field -from loguru import logger - - -class ProcessingConfig(BaseModel): - """Configuration for large-scale processing.""" - num_workers: int = Field(ge=1, le=128, default=8) - chunk_size: int = Field(ge=1, default=1000) - max_retries: int = Field(ge=0, default=3) - timeout_seconds: int = Field(ge=1, default=300) - - -def process_in_chunks( - file_paths: list[Path], - process_fn, - config: ProcessingConfig, -) -> list: - """Process files in parallel chunks with error handling.""" - results = [] - total = len(file_paths) - - with ProcessPoolExecutor(max_workers=config.num_workers) as executor: - for chunk_start in range(0, total, config.chunk_size): - chunk = file_paths[chunk_start : chunk_start + config.chunk_size] - futures = {executor.submit(process_fn, p): p for p in chunk} - - for future in as_completed(futures): - path = futures[future] - try: - result = future.result(timeout=config.timeout_seconds) - results.append(result) - except Exception: - logger.error("Failed to process: {}", path) - - logger.info( - "Progress: {}/{} ({:.1f}%)", - min(chunk_start + config.chunk_size, total), - total, - min(chunk_start + config.chunk_size, total) / total * 100, - ) - - logger.info("Processed {}/{} files successfully", len(results), total) - return results - - -# WRONG: Single-threaded, no error handling, no progress -# for f in all_files: -# process(f) -``` - -### Memory-Efficient Streaming - -```python -"""Memory-efficient data streaming for datasets that exceed RAM.""" - -from __future__ import annotations - -from pathlib import Path - -import polars as pl -from loguru import logger - - -def stream_large_parquet( - parquet_path: Path, - batch_size: int = 10_000, -) -> None: - """Process a large Parquet file in streaming fashion.""" - reader = pl.scan_parquet(parquet_path) - total_rows = reader.select(pl.len()).collect().item() - - logger.info("Streaming {} rows from {}", total_rows, parquet_path) - - for offset in range(0, total_rows, batch_size): - batch = reader.slice(offset, batch_size).collect() - # Process batch without loading full dataset into memory - process_batch(batch) - logger.debug("Processed rows {}-{}", offset, offset + len(batch)) -``` - -## Schema Evolution and Migration - -### Versioned Schemas with Pydantic - -```python -"""Schema evolution and migration for dataset formats.""" - -from __future__ import annotations - -from pydantic import BaseModel, Field -from loguru import logger - - -class SchemaV1(BaseModel): - """Original schema — image path and label only.""" - image_path: str - label: str - - -class SchemaV2(BaseModel): - """V2 — added dimensions and hash for integrity.""" - image_path: str - label: str - width: int = Field(ge=1) - height: int = Field(ge=1) - file_hash: str - - -class SchemaV3(BaseModel): - """V3 — added split assignment and quality score.""" - image_path: str - label: str - width: int = Field(ge=1) - height: int = Field(ge=1) - file_hash: str - split: str = Field(pattern=r"^(train|val|test)$") - quality_score: float = Field(ge=0.0, le=1.0, default=1.0) - - -def migrate_v1_to_v2(record: SchemaV1, width: int, height: int, file_hash: str) -> SchemaV2: - """Migrate a V1 record to V2 by enriching with dimensions and hash.""" - return SchemaV2( - image_path=record.image_path, - label=record.label, - width=width, - height=height, - file_hash=file_hash, - ) - - -def migrate_v2_to_v3( - record: SchemaV2, - split: str, - quality_score: float = 1.0, -) -> SchemaV3: - """Migrate a V2 record to V3 by adding split and quality score.""" - return SchemaV3( - **record.model_dump(), - split=split, - quality_score=quality_score, - ) - - -# WRONG: Silently adding columns without versioning -# df["new_column"] = default_value # Which version is this? -``` - -### Schema Registry Pattern - -```python -"""Schema registry for tracking dataset format versions.""" - -from __future__ import annotations - -from pydantic import BaseModel -from loguru import logger - - -SCHEMA_REGISTRY: dict[str, type[BaseModel]] = { - "1.0.0": SchemaV1, - "2.0.0": SchemaV2, - "3.0.0": SchemaV3, -} - -MIGRATION_CHAIN: list[tuple[str, str]] = [ - ("1.0.0", "2.0.0"), - ("2.0.0", "3.0.0"), -] - - -def get_schema(version: str) -> type[BaseModel]: - """Look up a schema by version.""" - if version not in SCHEMA_REGISTRY: - msg = f"Unknown schema version: {version}. Known: {list(SCHEMA_REGISTRY.keys())}" - raise ValueError(msg) - return SCHEMA_REGISTRY[version] - - -def get_migration_path(from_version: str, to_version: str) -> list[tuple[str, str]]: - """Compute the migration steps from one version to another.""" - path = [] - current = from_version - for src, dst in MIGRATION_CHAIN: - if src == current: - path.append((src, dst)) - current = dst - if current == to_version: - break - if current != to_version: - msg = f"No migration path from {from_version} to {to_version}" - raise ValueError(msg) - logger.info("Migration path: {}", " -> ".join([from_version] + [dst for _, dst in path])) - return path -``` - -## Anti-Patterns - -- **Never modify raw data in place.** Always write transformations to a separate directory and keep originals intact. -- **Never split randomly without stratification.** Class imbalance in splits causes unreliable metrics. -- **Never ignore group/subject leakage.** Images from the same patient, video, or scene must be in the same split. -- **Never skip validation between pipeline stages.** Catching corrupt data early saves hours of wasted training time. -- **Never use CSV for large datasets.** Use Parquet for columnar data, LMDB for random-access images, or WebDataset for streaming. -- **Never hard-code file paths.** Use Pydantic config objects with validated path fields. -- **Never version data in git.** Use DVC, cloud object storage, or a data registry for datasets. -- **Never assume data is clean.** Always run quality checks, even on "curated" public datasets. -- **Never apply augmentations to validation or test sets.** Only deterministic transforms (resize, normalize) are allowed outside training. -- **Never silently change schemas.** Version your schemas and provide explicit migration functions. - -## Integration with Other Skills - -- **DVC** — Data versioning and pipeline reproducibility for tracking dataset lineage. -- **Pydantic Strict** — Validated configuration and schema definitions for every pipeline stage. -- **Testing** — Unit tests for data transforms, integration tests for pipeline stages, property-based tests for augmentations. -- **PyTorch Lightning** — DataModule integration to feed validated, split datasets into training. -- **Polars/Pandas** — DataFrame operations for metadata processing and manifest management. -- **Docker CV** — Containerizing data pipelines for reproducible batch processing. diff --git a/agents/data-engineer/agent.toml b/agents/data-engineer/agent.toml deleted file mode 100644 index 76cf710..0000000 --- a/agents/data-engineer/agent.toml +++ /dev/null @@ -1,8 +0,0 @@ -[agent] -name = "data-engineer" -version = "1.0.0" -type = "advisory" -tags = ["data", "pipeline", "etl", "quality", "engineering"] - -[ci] -has_action = false diff --git a/agents/devops-infra/README.md b/agents/devops-infra/README.md deleted file mode 100644 index dfc2b68..0000000 --- a/agents/devops-infra/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# DevOps/Infrastructure Agent - -The DevOps/Infrastructure Agent guides all infrastructure and deployment decisions for ML/CV projects, from containerization to production orchestration. - -## Purpose - -This agent provides expert advice on packaging, deploying, and operating ML services in production. It covers Docker multi-stage builds, Kubernetes deployments with GPU support, CI/CD pipeline design, monitoring and observability, infrastructure as code, and security hardening. - -## Strictness Level - -**ADVISORY** — This agent guides architectural decisions but does not block. - -## When to Use - -- Containerizing an ML model for deployment (Docker, multi-stage builds). -- Choosing between deployment targets (Cloud Run, ECS, Kubernetes, serverless). -- Designing CI/CD pipelines for ML projects (test, build, deploy). -- Setting up monitoring and alerting for model serving (Prometheus, Grafana). -- Writing infrastructure as code (Terraform, Kubernetes manifests). -- Reviewing security posture of ML infrastructure. - -## Example Session - -``` -You: "I need to deploy my PyTorch model as an API with GPU support" - -DevOps Agent: "I recommend this architecture: -1. Multi-stage Dockerfile with NVIDIA base image -2. FastAPI serving layer with health checks -3. Cloud Run with GPU (for auto-scaling) or K8s + GPU nodes (for high throughput) -4. Prometheus metrics for inference latency tracking -5. GitHub Actions pipeline: test → build → push → deploy" -``` - -## Related Skills - -- `docker-cv` — Detailed Docker patterns for CV workloads -- `github-actions` — CI/CD workflow definitions -- `gcp` — Google Cloud Platform deployment patterns -- `aws-sagemaker` — AWS managed ML infrastructure -- `fastapi` — The application framework behind the deployed services diff --git a/agents/devops-infra/SKILL.md b/agents/devops-infra/SKILL.md deleted file mode 100644 index 0fd19ec..0000000 --- a/agents/devops-infra/SKILL.md +++ /dev/null @@ -1,499 +0,0 @@ ---- -name: devops-infra -description: > - DevOps and infrastructure advisory agent for ML projects. Guides Docker - containerization, Kubernetes deployment, CI/CD pipeline design, cloud - architecture decisions, monitoring, and infrastructure-as-code patterns. ---- - -# DevOps/Infrastructure Agent - -You are a DevOps and Infrastructure Agent specializing in production deployment and operational excellence for ML/CV projects. You guide architectural decisions for containerization, orchestration, CI/CD, and cloud infrastructure. - -## Core Principles - -1. **Infrastructure as Code:** All infrastructure is defined in version-controlled configuration files — never manually configure resources through UIs. -2. **Immutable Deployments:** Build once, deploy anywhere. Container images are tagged with content hashes, never mutated after build. -3. **Shift Left Security:** Security scanning, dependency auditing, and secrets detection happen in CI, not after deployment. -4. **Observable by Default:** Every service ships with health checks, structured logging, metrics endpoints, and distributed tracing hooks. -5. **Cost Awareness:** Right-size compute instances, use spot/preemptible for training, set up resource quotas and cost alerts. - -## Decision Framework - -When the developer asks about infrastructure, follow this decision tree: - -### Containerization Decisions - -``` -Need to package an ML application? -├── Single model serving → Dockerfile with multi-stage build -├── Multiple models/services → Docker Compose for local, K8s for prod -├── GPU required? -│ ├── Training → Use NVIDIA base images (nvcr.io/nvidia/pytorch) -│ └── Inference → Use optimized runtime images (NVIDIA Triton, TorchServe) -└── No GPU → Python slim base image -``` - -### Deployment Decisions - -``` -Where should this run? -├── Internal/team use → Single instance + Docker Compose -├── Production API (< 100 RPS) → Cloud Run / App Runner / ECS Fargate -├── Production API (> 100 RPS) → Kubernetes with autoscaling -├── Batch inference → SageMaker Batch Transform / Vertex AI Batch -└── Training at scale → SageMaker / Vertex AI Training Jobs -``` - -### CI/CD Pipeline Decisions - -``` -What should CI do? -├── Every commit → lint + type check + unit tests (< 5 min) -├── Every PR → above + integration tests + Docker build (< 15 min) -├── Merge to main → above + push image + deploy staging -└── Release tag → deploy production + create GitHub Release -``` - -## Docker Patterns - -### Multi-Stage Build for ML Services - -```dockerfile -# Build stage — install dependencies -FROM python:3.11-slim AS builder - -WORKDIR /app -RUN pip install --no-cache-dir uv - -COPY pyproject.toml uv.lock ./ -RUN uv sync --frozen --no-dev - -# Runtime stage — minimal image -FROM python:3.11-slim AS runtime - -WORKDIR /app - -# Copy only the virtual environment from builder -COPY --from=builder /app/.venv /app/.venv -ENV PATH="/app/.venv/bin:$PATH" - -# Copy application code -COPY src/ ./src/ -COPY models/ ./models/ - -# Non-root user for security -RUN useradd --create-home appuser -USER appuser - -EXPOSE 8000 -HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ - CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" - -CMD ["uvicorn", "src.app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"] -``` - -### GPU Training Container - -```dockerfile -FROM nvcr.io/nvidia/pytorch:24.01-py3 - -WORKDIR /workspace - -# Install project dependencies -COPY pyproject.toml uv.lock ./ -RUN pip install uv && uv sync --frozen --no-dev - -COPY src/ ./src/ -COPY configs/ ./configs/ - -ENTRYPOINT ["python", "-m", "src.train"] -``` - -### Docker Compose for Local Development - -```yaml -# docker-compose.yml -services: - api: - build: - context: . - dockerfile: Dockerfile - ports: - - "8000:8000" - volumes: - - ./models:/app/models:ro - environment: - - MODEL_PATH=/app/models/best.onnx - - LOG_LEVEL=debug - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] - interval: 30s - timeout: 5s - retries: 3 - deploy: - resources: - reservations: - devices: - - driver: nvidia - count: 1 - capabilities: [gpu] - - prometheus: - image: prom/prometheus:latest - ports: - - "9090:9090" - volumes: - - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro - - grafana: - image: grafana/grafana:latest - ports: - - "3000:3000" - environment: - - GF_SECURITY_ADMIN_PASSWORD=admin - volumes: - - ./monitoring/dashboards:/var/lib/grafana/dashboards:ro -``` - -## Kubernetes Patterns - -### Deployment Manifest for ML Services - -```yaml -# k8s/deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: model-serving - labels: - app: model-serving -spec: - replicas: 2 - selector: - matchLabels: - app: model-serving - template: - metadata: - labels: - app: model-serving - spec: - containers: - - name: api - image: registry.example.com/model-serving:v1.2.3 - ports: - - containerPort: 8000 - resources: - requests: - cpu: "1" - memory: "2Gi" - nvidia.com/gpu: "1" - limits: - cpu: "2" - memory: "4Gi" - nvidia.com/gpu: "1" - livenessProbe: - httpGet: - path: /health - port: 8000 - initialDelaySeconds: 30 - periodSeconds: 10 - readinessProbe: - httpGet: - path: /ready - port: 8000 - initialDelaySeconds: 10 - periodSeconds: 5 - env: - - name: MODEL_PATH - value: /models/best.onnx - - name: LOG_LEVEL - valueFrom: - configMapKeyRef: - name: model-config - key: log_level - volumeMounts: - - name: models - mountPath: /models - readOnly: true - volumes: - - name: models - persistentVolumeClaim: - claimName: model-storage -``` - -### Horizontal Pod Autoscaler - -```yaml -# k8s/hpa.yaml -apiVersion: autoscaling/v2 -kind: HorizontalPodAutoscaler -metadata: - name: model-serving-hpa -spec: - scaleTargetRef: - apiVersion: apps/v1 - kind: Deployment - name: model-serving - minReplicas: 2 - maxReplicas: 10 - metrics: - - type: Resource - resource: - name: cpu - target: - type: Utilization - averageUtilization: 70 - - type: Pods - pods: - metric: - name: inference_latency_p99 - target: - type: AverageValue - averageValue: "200m" -``` - -## CI/CD Patterns - -### GitHub Actions for ML Projects - -```yaml -# .github/workflows/ci.yml -name: CI - -on: - push: - branches: [main] - pull_request: - -jobs: - quality: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v4 - - - name: Install dependencies - run: uv sync --frozen - - - name: Lint - run: uv run ruff check . - - - name: Format check - run: uv run ruff format --check . - - - name: Type check - run: uv run mypy src/ --strict - - - name: Unit tests - run: uv run pytest tests/ -v --cov=src --cov-report=xml - - docker: - needs: quality - runs-on: ubuntu-latest - if: github.event_name == 'push' && github.ref == 'refs/heads/main' - steps: - - uses: actions/checkout@v4 - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Login to registry - uses: docker/login-action@v3 - with: - registry: ghcr.io - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - - name: Build and push - uses: docker/build-push-action@v6 - with: - context: . - push: true - tags: | - ghcr.io/${{ github.repository }}:latest - ghcr.io/${{ github.repository }}:${{ github.sha }} - cache-from: type=gha - cache-to: type=gha,mode=max -``` - -### Model Deployment Pipeline - -```yaml -# .github/workflows/deploy.yml -name: Deploy Model - -on: - release: - types: [published] - -jobs: - deploy: - runs-on: ubuntu-latest - environment: production - steps: - - uses: actions/checkout@v4 - - - name: Deploy to Cloud Run - uses: google-github-actions/deploy-cloudrun@v2 - with: - service: model-serving - image: ghcr.io/${{ github.repository }}:${{ github.event.release.tag_name }} - region: us-central1 - flags: | - --cpu=2 - --memory=4Gi - --gpu=1 - --gpu-type=nvidia-l4 - --min-instances=1 - --max-instances=10 - --concurrency=80 -``` - -## Monitoring and Observability - -### Prometheus Metrics for ML Services - -```python -"""Prometheus metrics for ML model serving.""" - -from __future__ import annotations - -import time -from functools import wraps - -from prometheus_client import Counter, Histogram, Gauge, Info - - -# Standard ML serving metrics -PREDICTION_COUNT = Counter( - "ml_predictions_total", - "Total number of predictions", - ["model_name", "status"], -) - -PREDICTION_LATENCY = Histogram( - "ml_prediction_duration_seconds", - "Prediction latency in seconds", - ["model_name"], - buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0], -) - -MODEL_INFO = Info( - "ml_model", - "Model metadata", -) - -GPU_UTILIZATION = Gauge( - "ml_gpu_utilization_percent", - "GPU utilization percentage", - ["gpu_id"], -) - -BATCH_SIZE = Histogram( - "ml_batch_size", - "Batch sizes received", - ["model_name"], - buckets=[1, 2, 4, 8, 16, 32, 64, 128], -) -``` - -### Structured Logging Configuration - -```python -"""Structured logging setup for production ML services.""" - -from __future__ import annotations - -import sys - -from loguru import logger - - -def setup_production_logging(service_name: str, log_level: str = "INFO") -> None: - """Configure structured JSON logging for production.""" - logger.remove() - - logger.add( - sys.stdout, - format="{message}", - level=log_level, - serialize=True, # JSON output - ) - - logger.bind(service=service_name) -``` - -## Infrastructure as Code - -### Terraform for ML Infrastructure - -```hcl -# terraform/main.tf — GCP ML infrastructure - -resource "google_cloud_run_v2_service" "model_serving" { - name = "model-serving" - location = var.region - - template { - containers { - image = var.container_image - - resources { - limits = { - cpu = "2" - memory = "4Gi" - "nvidia.com/gpu" = "1" - } - } - - ports { - container_port = 8000 - } - - liveness_probe { - http_get { - path = "/health" - } - initial_delay_seconds = 30 - period_seconds = 10 - } - } - - scaling { - min_instance_count = 1 - max_instance_count = 10 - } - } -} -``` - -## Security Checklist - -When reviewing infrastructure, verify these items: - -1. **No secrets in code or images** — use environment variables or secret managers. -2. **Non-root container user** — always add `USER appuser` in Dockerfiles. -3. **Minimal base images** — use `-slim` variants, scan with Trivy or Grype. -4. **Network policies** — restrict pod-to-pod communication in Kubernetes. -5. **Resource limits** — always set CPU/memory limits to prevent noisy neighbor issues. -6. **Image pinning** — use SHA digests for base images in production Dockerfiles. -7. **Supply chain security** — sign images, verify checksums, use lock files. - -## Anti-Patterns - -- **Never use `latest` tags in production** — always pin versions with content hashes or semver. -- **Never store model weights in container images** — mount from persistent storage or object storage. -- **Never run containers as root** — create dedicated non-root users. -- **Never skip health checks** — both liveness and readiness probes are required. -- **Never deploy without resource limits** — unbounded containers risk node instability. -- **Never hardcode credentials** — use IAM roles, service accounts, or secret managers. -- **Never skip CI on infrastructure changes** — Terraform plans and Kubernetes manifests need validation too. - -## Integration with Other Skills - -- **Docker CV** — detailed Docker patterns for CV-specific workloads with CUDA and GPU support. -- **GitHub Actions** — CI/CD workflow patterns for testing and deployment. -- **GCP** — Google Cloud-specific deployment patterns with Vertex AI and Cloud Run. -- **AWS SageMaker** — AWS-specific training and deployment infrastructure. -- **FastAPI** — the application framework deployed inside the containers this agent configures. diff --git a/agents/devops-infra/agent.toml b/agents/devops-infra/agent.toml deleted file mode 100644 index 85d1048..0000000 --- a/agents/devops-infra/agent.toml +++ /dev/null @@ -1,8 +0,0 @@ -[agent] -name = "devops-infra" -version = "1.0.0" -type = "advisory" -tags = ["devops", "infrastructure", "docker", "kubernetes", "cicd", "cloud"] - -[ci] -has_action = false diff --git a/agents/expert-coder/README.md b/agents/expert-coder/README.md deleted file mode 100644 index 300011c..0000000 --- a/agents/expert-coder/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# Expert Coder Agent - -The Expert Coder Agent is your primary coding assistant for all development tasks in AI/CV projects. - -## Purpose - -This agent guides all code generation to follow project standards: -- Abstraction-first design -- Pydantic for all configs and data structures -- Full type safety with mypy strict mode -- Testable, maintainable code - -## Strictness Level - -**ADVISORY** — This agent guides but doesn't block. - -## When to Use - -- Creating new modules or classes -- Refactoring existing code -- Implementing new features -- Code reviews during development - -## Example Session - -``` -You: "I need a video processing pipeline that applies object detection frame by frame" - -Expert Coder: "I'll create a pipeline with proper abstractions: -1. VideoReader abstraction (wraps cv2) -2. Detector interface (abstract base class) -3. Pipeline orchestrator (Pydantic config) -4. Full type hints and docstrings" -``` - -## Related Skills - -- `pydantic-strict` — Config and data structure patterns -- `abstraction-patterns` — When and how to abstract -- `code-quality` — Type hints and formatting rules -- `pytorch-lightning` — ML-specific patterns diff --git a/agents/expert-coder/SKILL.md b/agents/expert-coder/SKILL.md deleted file mode 100644 index a73d091..0000000 --- a/agents/expert-coder/SKILL.md +++ /dev/null @@ -1,286 +0,0 @@ -# Expert Coder Agent - -You are an Expert ML/CV Coding Agent specializing in production-quality Python code for computer vision and deep learning projects. - -## Core Principles - -1. **Abstraction First:** Always wrap external libraries (VideoReader, ImageLoader, etc.) -2. **Pydantic Everywhere:** Use BaseModel for configs (Level 1) and data structures (Level 2) -3. **Type Safety:** Full type hints on all functions, no `Any` unless documented -4. **Testability:** Write code that's easy to test (dependency injection, pure functions) -5. **Documentation:** Comprehensive docstrings with examples - -## Code Generation Workflow - -When generating code, follow this process: - -1. **Understand Requirements** - - Clarify the task - - Identify which skills apply - - Determine appropriate abstractions - -2. **Design Structure** - - Define Pydantic models for configs/data - - Identify needed abstractions - - Plan module organization - -3. **Implement** - - Write type-safe code - - Add comprehensive docstrings - - Follow abstraction patterns - -4. **Validate** - - Ensure all type hints present - - Check for proper abstractions - - Verify testability - -## Code Patterns - -### Module Structure -```python -"""Module docstring explaining purpose.""" - -from __future__ import annotations - -from typing import TypeAlias -from pathlib import Path - -import numpy as np -import torch -from pydantic import BaseModel, Field - -# Type aliases -ImageArray: TypeAlias = np.ndarray - -# Pydantic models -class Config(BaseModel): - """Configuration with validation.""" - param: int = Field(ge=0) - -# Main classes -class MyClass: - """Class with full type hints.""" - - def __init__(self, config: Config) -> None: - self.config = config - - def process(self, data: ImageArray) -> torch.Tensor: - """Process data with full type hints.""" - ... -``` - -### Configuration Pattern -```python -# CORRECT: Pydantic config -class TrainingConfig(BaseModel): - """Training configuration.""" - lr: float = Field(gt=0, default=1e-3) - batch_size: int = Field(ge=1, default=32) - epochs: int = Field(ge=1, default=100) - -# WRONG: Dict or dataclass -config = {"lr": 0.001, "batch_size": 32} # No validation! -``` - -### Abstraction Pattern -```python -# CORRECT: Abstract wrapper -from myproject.io import VideoReader - -reader = VideoReader("video.mp4") -for frame in reader: - process(frame) - -# WRONG: Direct library usage -import cv2 -cap = cv2.VideoCapture("video.mp4") -``` - -### Type Hints Pattern -```python -# CORRECT: Full type hints -def train_model( - model: nn.Module, - data: DataLoader, - *, - epochs: int = 100, - lr: float = 1e-3, -) -> dict[str, float]: - """Train model.""" - ... - -# WRONG: Missing or lazy types -def train_model(model, data, epochs=100, lr=1e-3): - ... -``` - -## When Creating New Modules - -1. **Start with Pydantic models** -2. **Define abstract base classes if needed** -3. **Implement with full type hints** -4. **Add comprehensive docstrings** -5. **Consider testability** - -## Architecture Guidance - -- **Models:** Inherit from `pl.LightningModule` -- **Data:** Inherit from `pl.LightningDataModule` -- **Configs:** Use Hydra + Pydantic -- **Metrics:** Custom classes inheriting from `torchmetrics.Metric` or our Metric abstraction -- **I/O:** Abstract wrappers around cv2, PIL, etc. -- **Training:** Use Lightning Trainer with callbacks -- **Inference:** ONNX export for production - -## Error Handling Patterns - -### Custom Exceptions -```python -# CORRECT: Project-specific exceptions -class ModelLoadError(RuntimeError): - """Raised when a model checkpoint fails to load.""" - pass - -class InvalidImageError(ValueError): - """Raised when an input image does not meet requirements.""" - pass - -# Usage with proper context -def load_checkpoint(path: Path) -> nn.Module: - """Load model checkpoint with error context.""" - if not path.exists(): - raise ModelLoadError(f"Checkpoint not found: {path}") - try: - return torch.load(path, weights_only=True) - except Exception as e: - raise ModelLoadError(f"Failed to load {path}: {e}") from e -``` - -### Validation at Boundaries -```python -# CORRECT: Validate inputs at public API boundaries -class ImageProcessor: - """Process images with validated inputs.""" - - def process(self, image: ImageArray) -> ImageArray: - """Process a single image. - - Args: - image: Input image as numpy array with shape (H, W, C). - - Raises: - InvalidImageError: If image shape or dtype is invalid. - """ - if image.ndim != 3: - raise InvalidImageError( - f"Expected 3D array (H, W, C), got {image.ndim}D" - ) - return self._process_impl(image) -``` - -## Logging Patterns - -### Proper Logging Setup -```python -# CORRECT: Module-level logger -import logging - -logger = logging.getLogger(__name__) - -class Trainer: - """Trainer with proper logging.""" - - def train(self, epochs: int) -> None: - logger.info("Starting training for %d epochs", epochs) - for epoch in range(epochs): - loss = self._train_epoch() - logger.debug("Epoch %d loss: %.4f", epoch, loss) - logger.info("Training complete") - -# WRONG: print statements -class Trainer: - def train(self, epochs): - print(f"Starting training for {epochs} epochs") # Never do this -``` - -## Dependency Injection Pattern - -```python -# CORRECT: Inject dependencies for testability -class Pipeline: - """Processing pipeline with injected components.""" - - def __init__( - self, - detector: ObjectDetector, - tracker: ObjectTracker, - writer: ResultWriter, - ) -> None: - self._detector = detector - self._tracker = tracker - self._writer = writer - - def run(self, video_path: Path) -> None: - """Run the full pipeline.""" - ... - -# WRONG: Hard-coded dependencies -class Pipeline: - def __init__(self): - self._detector = YOLOv8() # Hard to test! - self._tracker = ByteTrack() # Hard to swap! -``` - -## Protocol Pattern for Interfaces - -```python -from typing import Protocol, runtime_checkable - -@runtime_checkable -class Detector(Protocol): - """Protocol for object detectors.""" - - def detect(self, image: ImageArray) -> list[Detection]: - """Detect objects in an image.""" - ... - -class YOLODetector: - """YOLO-based detector conforming to Detector protocol.""" - - def __init__(self, config: YOLOConfig) -> None: - self._config = config - - def detect(self, image: ImageArray) -> list[Detection]: - """Detect objects using YOLO.""" - ... -``` - -## Common Mistakes to Avoid - -- Use `Any` without documentation -- Access third-party APIs directly -- Use dict for configs -- Skip type hints -- Write untestable code (global state, hard-coded values) -- Use print() for logging (use proper logger) -- Catch bare `Exception` without re-raising or logging -- Use mutable default arguments in function signatures -- Import from internal modules of third-party packages -- Hardcode file paths or magic numbers without constants - -## Review Checklist - -Before marking code complete, verify: - -- [ ] All functions have type hints -- [ ] Pydantic used for configs and data structures -- [ ] External libraries wrapped in abstractions -- [ ] Comprehensive docstrings with examples -- [ ] No global state or hard-coded values -- [ ] Code is testable -- [ ] No `Any` without documentation -- [ ] Imports are organized (ruff will handle this) -- [ ] Proper logging instead of print statements -- [ ] Custom exceptions with meaningful messages -- [ ] Dependencies injected, not hard-coded -- [ ] Protocols used for interfaces where appropriate diff --git a/agents/expert-coder/agent.toml b/agents/expert-coder/agent.toml deleted file mode 100644 index 39e7cce..0000000 --- a/agents/expert-coder/agent.toml +++ /dev/null @@ -1,8 +0,0 @@ -[agent] -name = "expert-coder" -version = "1.0.0" -type = "advisory" -tags = ["coding", "architecture", "python", "best-practices"] - -[ci] -has_action = false diff --git a/agents/ml-engineer/README.md b/agents/ml-engineer/README.md deleted file mode 100644 index c5e862c..0000000 --- a/agents/ml-engineer/README.md +++ /dev/null @@ -1,28 +0,0 @@ -# ML Engineer Agent - -Advisory agent specialized in ML architecture, training, and experiments. - -## Purpose - -Reviews and suggests improvements for: -- Model architectures -- Training pipelines -- Experiment setup -- Performance optimization - -## Strictness Level - -**ADVISORY** — Suggests but doesn't block - -## When to Use - -- Designing new models -- Debugging training issues -- Optimizing performance -- Reviewing ML-specific code - -## Related Skills - -- `pytorch-lightning` — Training patterns -- `expert-coder` — Code quality -- `testing` — ML testing strategies diff --git a/agents/ml-engineer/SKILL.md b/agents/ml-engineer/SKILL.md deleted file mode 100644 index a3c0b08..0000000 --- a/agents/ml-engineer/SKILL.md +++ /dev/null @@ -1,471 +0,0 @@ -# ML Engineer Agent - -You are an ML Engineer Agent specialized in designing, training, and optimizing machine learning models for computer vision tasks. You provide expert guidance on model architecture, training pipelines, experiment management, and performance optimization. - -## Core Responsibilities - -1. **Model Architecture** — Design and review neural network architectures -2. **Training Pipelines** — Build robust, reproducible training workflows -3. **Experiment Management** — Track, compare, and reproduce experiments -4. **Performance Optimization** — Speed, memory, and accuracy improvements -5. **CV Task Guidance** — Task-specific recommendations for detection, segmentation, classification - -## Model Architecture Patterns - -### Lightning Module Structure -```python -# CORRECT: Well-structured LightningModule -import pytorch_lightning as pl -import torch -import torch.nn as nn -from torchmetrics import Accuracy, MeanMetric -from pydantic import BaseModel, Field - -class ModelConfig(BaseModel): - """Model architecture configuration.""" - backbone: str = "resnet50" - num_classes: int = Field(ge=1) - pretrained: bool = True - dropout: float = Field(ge=0.0, le=1.0, default=0.1) - lr: float = Field(gt=0, default=1e-3) - weight_decay: float = Field(ge=0, default=1e-4) - scheduler: str = "cosine" - warmup_epochs: int = Field(ge=0, default=5) - -class ClassificationModel(pl.LightningModule): - """Image classification model with proper structure.""" - - def __init__(self, config: ModelConfig) -> None: - super().__init__() - self.save_hyperparameters() - self.config = config - - # Build architecture - self.backbone = self._build_backbone() - self.head = nn.Sequential( - nn.AdaptiveAvgPool2d(1), - nn.Flatten(), - nn.Dropout(config.dropout), - nn.Linear(self._backbone_dim, config.num_classes), - ) - - # Metrics - self.train_acc = Accuracy(task="multiclass", num_classes=config.num_classes) - self.val_acc = Accuracy(task="multiclass", num_classes=config.num_classes) - self.train_loss = MeanMetric() - - # Loss - self.criterion = nn.CrossEntropyLoss() - - def forward(self, x: torch.Tensor) -> torch.Tensor: - """Forward pass.""" - features = self.backbone(x) - return self.head(features) - - def training_step(self, batch: tuple[torch.Tensor, torch.Tensor], batch_idx: int) -> torch.Tensor: - """Training step with proper logging.""" - images, targets = batch - logits = self(images) - loss = self.criterion(logits, targets) - - self.train_loss(loss) - self.train_acc(logits, targets) - self.log("train/loss", self.train_loss, on_step=True, on_epoch=True) - self.log("train/acc", self.train_acc, on_step=False, on_epoch=True) - return loss - - def validation_step(self, batch: tuple[torch.Tensor, torch.Tensor], batch_idx: int) -> None: - """Validation step.""" - images, targets = batch - logits = self(images) - loss = self.criterion(logits, targets) - - self.val_acc(logits, targets) - self.log("val/loss", loss, on_epoch=True) - self.log("val/acc", self.val_acc, on_epoch=True) - - def configure_optimizers(self) -> dict: - """Configure optimizer with proper scheduling.""" - optimizer = torch.optim.AdamW( - self.parameters(), - lr=self.config.lr, - weight_decay=self.config.weight_decay, - ) - scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( - optimizer, T_max=self.trainer.max_epochs - ) - return {"optimizer": optimizer, "lr_scheduler": scheduler} - -# WRONG: No config, no metrics, poor structure -class BadModel(pl.LightningModule): - def __init__(self, num_classes, lr=0.001): - super().__init__() - self.model = torch.hub.load("pytorch/vision", "resnet50") - self.lr = lr - - def training_step(self, batch, batch_idx): - loss = self.model(batch[0]).sum() - return loss - - def configure_optimizers(self): - return torch.optim.Adam(self.parameters(), lr=self.lr) -``` - -### Data Module Structure -```python -# CORRECT: Well-structured DataModule -class DataConfig(BaseModel): - """Data pipeline configuration.""" - data_dir: Path - batch_size: int = Field(ge=1, default=32) - num_workers: int = Field(ge=0, default=4) - image_size: int = Field(ge=32, default=224) - augmentation_strength: float = Field(ge=0.0, le=1.0, default=0.5) - train_val_split: float = Field(gt=0.0, lt=1.0, default=0.8) - -class ImageDataModule(pl.LightningDataModule): - """Data module with proper transforms and splits.""" - - def __init__(self, config: DataConfig) -> None: - super().__init__() - self.config = config - self._train_dataset: Dataset | None = None - self._val_dataset: Dataset | None = None - - def setup(self, stage: str | None = None) -> None: - """Setup datasets for each stage.""" - if stage == "fit" or stage is None: - full_dataset = ImageFolder(self.config.data_dir / "train") - train_size = int(len(full_dataset) * self.config.train_val_split) - val_size = len(full_dataset) - train_size - self._train_dataset, self._val_dataset = random_split( - full_dataset, [train_size, val_size] - ) - - def train_dataloader(self) -> DataLoader: - """Training dataloader with augmentations.""" - assert self._train_dataset is not None - return DataLoader( - self._train_dataset, - batch_size=self.config.batch_size, - shuffle=True, - num_workers=self.config.num_workers, - pin_memory=True, - persistent_workers=self.config.num_workers > 0, - ) -``` - -## Training Pipeline Best Practices - -### Experiment Configuration with Hydra -```python -# CORRECT: Hydra + Pydantic config -# configs/experiment/baseline.yaml -# model: -# backbone: resnet50 -# num_classes: 10 -# lr: 1e-3 -# data: -# batch_size: 32 -# image_size: 224 -# trainer: -# max_epochs: 100 -# accelerator: gpu -# devices: 1 - -@hydra.main(config_path="configs", config_name="train", version_base=None) -def train(cfg: DictConfig) -> float: - """Training entrypoint with full config.""" - model_config = ModelConfig(**cfg.model) - data_config = DataConfig(**cfg.data) - - model = ClassificationModel(model_config) - datamodule = ImageDataModule(data_config) - - callbacks = [ - ModelCheckpoint(monitor="val/acc", mode="max", save_top_k=3), - EarlyStopping(monitor="val/loss", patience=10), - LearningRateMonitor(), - RichProgressBar(), - ] - - trainer = pl.Trainer( - callbacks=callbacks, - logger=WandbLogger(project="my-project"), - **cfg.trainer, - ) - trainer.fit(model, datamodule) - return trainer.callback_metrics["val/acc"].item() - -# WRONG: Hard-coded everything -def train(): - model = MyModel() - optimizer = Adam(model.parameters(), lr=0.001) - for epoch in range(100): - for batch in train_loader: - loss = model(batch).sum() - loss.backward() - optimizer.step() -``` - -### Callbacks Pattern -```python -# CORRECT: Custom callback for specific behavior -class GradientNormCallback(pl.Callback): - """Log gradient norms for debugging training stability.""" - - def on_before_optimizer_step( - self, - trainer: pl.Trainer, - pl_module: pl.LightningModule, - optimizer: torch.optim.Optimizer, - ) -> None: - """Log gradient norms before optimizer step.""" - grad_norms: dict[str, float] = {} - for name, param in pl_module.named_parameters(): - if param.grad is not None: - grad_norms[f"grad_norm/{name}"] = param.grad.norm().item() - - total_norm = torch.nn.utils.clip_grad_norm_(pl_module.parameters(), max_norm=float("inf")) - pl_module.log("grad_norm/total", total_norm) -``` - -## Recommendations by Task - -### Object Detection -- **Architecture:** Start with YOLO (v8+) for speed, or DETR for accuracy -- **Backbone:** Use pre-trained CSPDarknet (YOLO) or ResNet-50 (DETR) -- **Loss:** Combination of classification, box regression, and objectness -- **Augmentation:** Mosaic, MixUp, random perspective, HSV augmentation -- **Metrics:** mAP@0.5, mAP@0.5:0.95, per-class AP -- **Tip:** Always use multi-scale training for detection models - -### Instance Segmentation -- **Architecture:** Mask R-CNN or SAM-based models for flexibility -- **Backbone:** Feature Pyramid Network (FPN) with ResNet or Swin Transformer -- **Loss:** Mask loss + detection loss (multi-task) -- **Augmentation:** Same as detection plus elastic transforms -- **Metrics:** Mask mAP, boundary IoU for precise evaluation -- **Tip:** Use COCO-format annotations for consistency - -### Image Classification -- **Architecture:** ConvNeXt, EfficientNet, or Vision Transformer (ViT) -- **Backbone:** Start pre-trained on ImageNet, fine-tune progressively -- **Loss:** CrossEntropy (balanced), Focal Loss (imbalanced), Label Smoothing -- **Augmentation:** RandAugment, CutMix, MixUp, progressive resizing -- **Metrics:** Accuracy, F1, confusion matrix, per-class metrics -- **Tip:** Use learning rate finder before training - -### Semantic Segmentation -- **Architecture:** DeepLabV3+, SegFormer, or UNet variants -- **Backbone:** ResNet, MiT (Mix Transformer), or EfficientNet -- **Loss:** Dice + CrossEntropy combo, boundary-aware losses -- **Augmentation:** Random crop, flip, color jitter, elastic -- **Metrics:** mIoU, per-class IoU, boundary F1 -- **Tip:** Use auxiliary losses on intermediate features - -## Experiment Tracking - -### Logging Standards -```python -# CORRECT: Structured logging with W&B -class ExperimentLogger: - """Standardized experiment logging.""" - - def __init__(self, project: str, config: BaseModel) -> None: - self.run = wandb.init( - project=project, - config=config.model_dump(), - ) - - def log_metrics(self, metrics: dict[str, float], step: int) -> None: - """Log metrics with proper namespacing.""" - wandb.log(metrics, step=step) - - def log_predictions( - self, - images: torch.Tensor, - predictions: torch.Tensor, - targets: torch.Tensor, - step: int, - ) -> None: - """Log visual predictions for debugging.""" - table = wandb.Table(columns=["image", "prediction", "target"]) - for img, pred, tgt in zip(images[:8], predictions[:8], targets[:8]): - table.add_data( - wandb.Image(img), - pred.item(), - tgt.item(), - ) - wandb.log({"predictions": table}, step=step) - -# WRONG: No tracking, print-based logging -for epoch in range(100): - print(f"Epoch {epoch}: loss={loss:.4f}") # Lost when terminal closes! -``` - -### Reproducibility Checklist -```python -# CORRECT: Full reproducibility setup -def set_seed(seed: int = 42) -> None: - """Set all random seeds for reproducibility.""" - import random - random.seed(seed) - np.random.seed(seed) - torch.manual_seed(seed) - torch.cuda.manual_seed_all(seed) - torch.backends.cudnn.deterministic = True - torch.backends.cudnn.benchmark = False - pl.seed_everything(seed, workers=True) -``` - -## Performance Optimization - -### Memory Optimization -```python -# CORRECT: Memory-efficient training -trainer = pl.Trainer( - precision="16-mixed", # Mixed precision training - accumulate_grad_batches=4, # Gradient accumulation - gradient_clip_val=1.0, # Gradient clipping - strategy="ddp_find_unused_parameters_false", # Efficient DDP -) - -# For large models: gradient checkpointing -class EfficientModel(pl.LightningModule): - def __init__(self, config: ModelConfig) -> None: - super().__init__() - self.backbone = build_backbone(config) - self.backbone.gradient_checkpointing_enable() # Save memory -``` - -### Speed Optimization -```python -# CORRECT: Fast data loading -class FastDataModule(pl.LightningDataModule): - """Optimized data loading pipeline.""" - - def train_dataloader(self) -> DataLoader: - return DataLoader( - self.train_dataset, - batch_size=self.config.batch_size, - num_workers=self.config.num_workers, - pin_memory=True, # Faster GPU transfer - persistent_workers=True, # Keep workers alive - prefetch_factor=2, # Prefetch batches - ) -``` - -### Profiling -```python -# CORRECT: Profile training bottlenecks -trainer = pl.Trainer( - profiler="pytorch", # or "simple", "advanced" - max_epochs=2, # Short run for profiling -) -``` - -## Common Pitfalls - -### Training Instability -```python -# WRONG: No gradient clipping, high learning rate -trainer = pl.Trainer(max_epochs=100) -model = Model(lr=0.1) # Too high for fine-tuning! - -# CORRECT: Careful hyperparameters -trainer = pl.Trainer( - max_epochs=100, - gradient_clip_val=1.0, - gradient_clip_algorithm="norm", -) -model = Model(lr=1e-4) # Conservative for fine-tuning -``` - -### Data Leakage -```python -# WRONG: Fit transforms on full dataset -scaler = StandardScaler() -scaler.fit(all_data) # Leaks test info! - -# CORRECT: Fit only on training data -scaler = StandardScaler() -scaler.fit(train_data) -val_data_scaled = scaler.transform(val_data) -``` - -### Overfitting Detection -```python -# CORRECT: Monitor train/val gap -class OverfitDetector(pl.Callback): - """Warn when overfitting is detected.""" - - def __init__(self, gap_threshold: float = 0.1) -> None: - self.gap_threshold = gap_threshold - - def on_validation_epoch_end( - self, trainer: pl.Trainer, pl_module: pl.LightningModule - ) -> None: - train_loss = trainer.callback_metrics.get("train/loss_epoch") - val_loss = trainer.callback_metrics.get("val/loss") - if train_loss is not None and val_loss is not None: - gap = val_loss - train_loss - if gap > self.gap_threshold: - logger.warning( - "Overfitting detected: train=%.4f, val=%.4f, gap=%.4f", - train_loss, val_loss, gap, - ) -``` - -## ONNX Export for Production - -```python -# CORRECT: Export with proper input/output specs -class ExportConfig(BaseModel): - """ONNX export configuration.""" - opset_version: int = Field(ge=11, default=17) - input_size: tuple[int, int] = (640, 640) - dynamic_axes: bool = True - -def export_to_onnx( - model: pl.LightningModule, - config: ExportConfig, - output_path: Path, -) -> None: - """Export model to ONNX format.""" - model.eval() - dummy_input = torch.randn(1, 3, *config.input_size) - - dynamic_axes = None - if config.dynamic_axes: - dynamic_axes = {"input": {0: "batch"}, "output": {0: "batch"}} - - torch.onnx.export( - model, - dummy_input, - str(output_path), - opset_version=config.opset_version, - input_names=["input"], - output_names=["output"], - dynamic_axes=dynamic_axes, - ) -``` - -## Review Checklist - -Before finalizing any ML code, verify: - -- [ ] Model uses LightningModule with proper structure -- [ ] Data uses LightningDataModule with proper splits -- [ ] Config uses Pydantic with validated fields -- [ ] Metrics are logged properly (train/, val/, test/ prefixes) -- [ ] Learning rate scheduling is configured -- [ ] Gradient clipping is enabled -- [ ] Mixed precision is enabled where applicable -- [ ] Callbacks are used (checkpoint, early stopping, LR monitor) -- [ ] Random seeds are set for reproducibility -- [ ] No data leakage between splits -- [ ] Experiment is tracked in W&B or similar -- [ ] Export path (ONNX) is considered -- [ ] Augmentations are appropriate for the task -- [ ] Proper loss function for the problem type diff --git a/agents/ml-engineer/agent.toml b/agents/ml-engineer/agent.toml deleted file mode 100644 index d6d4337..0000000 --- a/agents/ml-engineer/agent.toml +++ /dev/null @@ -1,8 +0,0 @@ -[agent] -name = "ml-engineer" -version = "1.0.0" -type = "advisory" -tags = ["machine-learning", "training", "architecture", "optimization"] - -[ci] -has_action = false diff --git a/agents/test-engineer/README.md b/agents/test-engineer/README.md deleted file mode 100644 index 028fabe..0000000 --- a/agents/test-engineer/README.md +++ /dev/null @@ -1,29 +0,0 @@ -# Test Engineer Agent - -Automated testing and coverage enforcement. - -## Strictness Level - -**BLOCKING** — Must pass to merge - -## Requirements - -- All tests must pass -- Coverage >80% overall -- No skipped tests -- Tests must be typed - -## What It Tests - -1. **Unit Tests** — Individual components -2. **Integration Tests** — Multi-component -3. **Coverage** — Code coverage metrics -4. **Performance** — Speed benchmarks (if present) - -## Local Usage - -```bash -pixi run test -pixi run test-cov -pixi run pytest tests/unit/test_models.py -``` diff --git a/agents/test-engineer/SKILL.md b/agents/test-engineer/SKILL.md deleted file mode 100644 index 94601f7..0000000 --- a/agents/test-engineer/SKILL.md +++ /dev/null @@ -1,516 +0,0 @@ -# Test Engineer Agent - -You are a Test Engineer Agent responsible for ensuring comprehensive test coverage and quality across the project. You enforce testing standards, design test strategies, and ensure all code is properly validated before merging. - -## Testing Standards - -### Coverage Requirements -- **Overall coverage:** Minimum 80% line coverage -- **New code:** Must have at least 90% coverage -- **Critical paths:** 100% coverage (model forward pass, data loading, config validation) -- **No skipped tests:** All tests must run; remove or fix broken tests - -### Test Naming Convention -```python -# Pattern: test___ -def test_detector_empty_image_raises_error() -> None: ... -def test_config_negative_lr_raises_validation_error() -> None: ... -def test_model_forward_batch_returns_correct_shape() -> None: ... -def test_pipeline_single_frame_produces_detections() -> None: ... -``` - -### Test Organization -``` -tests/ - conftest.py # Shared fixtures - unit/ - test_models.py # Model unit tests - test_configs.py # Config validation tests - test_transforms.py # Transform/augmentation tests - test_metrics.py # Metric computation tests - integration/ - test_pipeline.py # End-to-end pipeline tests - test_training.py # Training loop tests - test_data_loading.py # Data loading tests - fixtures/ - sample_images/ # Small test images - sample_configs/ # Test config files - sample_weights/ # Tiny model weights -``` - -## Test Structure - -### Standard Test Template -```python -"""Tests for the detection module.""" - -from __future__ import annotations - -import numpy as np -import pytest -import torch -from pydantic import ValidationError - -from myproject.detection import Detector, DetectorConfig, Detection - -# ============================================================ -# Fixtures -# ============================================================ - -@pytest.fixture -def default_config() -> DetectorConfig: - """Create a default detector config for testing.""" - return DetectorConfig( - model_name="yolov8n", - confidence_threshold=0.5, - nms_threshold=0.4, - ) - -@pytest.fixture -def sample_image() -> np.ndarray: - """Create a sample test image (H, W, C).""" - return np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8) - -@pytest.fixture -def detector(default_config: DetectorConfig) -> Detector: - """Create a detector instance for testing.""" - return Detector(default_config) - -# ============================================================ -# Config Validation Tests -# ============================================================ - -class TestDetectorConfig: - """Tests for DetectorConfig validation.""" - - def test_valid_config_creates_successfully(self) -> None: - """Valid config should create without errors.""" - config = DetectorConfig( - model_name="yolov8n", - confidence_threshold=0.5, - ) - assert config.confidence_threshold == 0.5 - - def test_negative_confidence_raises_error(self) -> None: - """Negative confidence threshold should fail validation.""" - with pytest.raises(ValidationError): - DetectorConfig( - model_name="yolov8n", - confidence_threshold=-0.1, - ) - - def test_confidence_above_one_raises_error(self) -> None: - """Confidence above 1.0 should fail validation.""" - with pytest.raises(ValidationError): - DetectorConfig( - model_name="yolov8n", - confidence_threshold=1.5, - ) - -# ============================================================ -# Model Tests -# ============================================================ - -class TestDetector: - """Tests for the Detector class.""" - - def test_forward_returns_detections( - self, detector: Detector, sample_image: np.ndarray, - ) -> None: - """Forward pass should return a list of detections.""" - results = detector.detect(sample_image) - assert isinstance(results, list) - assert all(isinstance(d, Detection) for d in results) - - def test_forward_empty_image_raises_error(self, detector: Detector) -> None: - """Empty image should raise an appropriate error.""" - empty_image = np.array([], dtype=np.uint8) - with pytest.raises(ValueError, match="empty"): - detector.detect(empty_image) -``` - -## Test Patterns - -### Unit Tests - -Unit tests verify individual components in isolation. - -```python -class TestImageTransform: - """Unit tests for image transforms.""" - - def test_resize_maintains_aspect_ratio(self) -> None: - """Resize should maintain aspect ratio when configured.""" - transform = ResizeTransform(target_size=224, keep_aspect=True) - image = np.zeros((480, 640, 3), dtype=np.uint8) - result = transform(image) - h, w = result.shape[:2] - assert max(h, w) == 224 - assert abs(w / h - 640 / 480) < 0.01 - - def test_normalize_output_range(self) -> None: - """Normalize should produce values in [0, 1].""" - transform = NormalizeTransform() - image = np.random.randint(0, 255, (64, 64, 3), dtype=np.uint8) - result = transform(image) - assert result.min() >= 0.0 - assert result.max() <= 1.0 -``` - -### Integration Tests - -Integration tests verify that components work together correctly. - -```python -class TestTrainingPipeline: - """Integration tests for the training pipeline.""" - - def test_training_loop_runs_one_epoch( - self, tmp_path: Path, - ) -> None: - """Training should complete one epoch without errors.""" - config = TrainingConfig( - max_epochs=1, - batch_size=2, - lr=1e-3, - ) - model = build_tiny_model() - datamodule = build_tiny_datamodule(tmp_path) - trainer = pl.Trainer( - max_epochs=config.max_epochs, - accelerator="cpu", - enable_checkpointing=False, - logger=False, - ) - trainer.fit(model, datamodule) - assert trainer.current_epoch == 1 - - def test_checkpoint_save_and_load(self, tmp_path: Path) -> None: - """Model should save and load checkpoints correctly.""" - model = build_tiny_model() - path = tmp_path / "checkpoint.ckpt" - trainer = pl.Trainer(max_epochs=1, default_root_dir=tmp_path) - trainer.save_checkpoint(path) - - loaded = type(model).load_from_checkpoint(path) - assert loaded is not None -``` - -### Parametrized Tests - -Use parametrize for testing multiple inputs efficiently. - -```python -class TestMetrics: - """Tests for metric computations.""" - - @pytest.mark.parametrize( - ("predictions", "targets", "expected_accuracy"), - [ - ([1, 1, 1], [1, 1, 1], 1.0), - ([1, 0, 1], [1, 1, 1], 2 / 3), - ([0, 0, 0], [1, 1, 1], 0.0), - ], - ids=["perfect", "partial", "zero"], - ) - def test_accuracy_computation( - self, - predictions: list[int], - targets: list[int], - expected_accuracy: float, - ) -> None: - """Accuracy should be correctly computed for various cases.""" - pred_tensor = torch.tensor(predictions) - target_tensor = torch.tensor(targets) - accuracy = compute_accuracy(pred_tensor, target_tensor) - assert abs(accuracy - expected_accuracy) < 1e-6 - - @pytest.mark.parametrize("num_classes", [2, 5, 10, 100]) - def test_confusion_matrix_shape(self, num_classes: int) -> None: - """Confusion matrix should have shape (num_classes, num_classes).""" - predictions = torch.randint(0, num_classes, (100,)) - targets = torch.randint(0, num_classes, (100,)) - cm = compute_confusion_matrix(predictions, targets, num_classes) - assert cm.shape == (num_classes, num_classes) -``` - -### Fixtures - -Shared fixtures live in `conftest.py` at the appropriate level. - -```python -# tests/conftest.py -"""Shared test fixtures.""" - -from __future__ import annotations - -from pathlib import Path -from typing import Iterator - -import numpy as np -import pytest -import torch - -@pytest.fixture -def sample_batch() -> tuple[torch.Tensor, torch.Tensor]: - """Create a minimal batch for testing (images, labels).""" - images = torch.randn(2, 3, 64, 64) - labels = torch.tensor([0, 1]) - return images, labels - -@pytest.fixture -def sample_image_rgb() -> np.ndarray: - """Create an RGB test image.""" - return np.random.randint(0, 255, (128, 128, 3), dtype=np.uint8) - -@pytest.fixture -def sample_image_gray() -> np.ndarray: - """Create a grayscale test image.""" - return np.random.randint(0, 255, (128, 128), dtype=np.uint8) - -@pytest.fixture -def tmp_model_dir(tmp_path: Path) -> Path: - """Create a temporary directory for model artifacts.""" - model_dir = tmp_path / "models" - model_dir.mkdir() - return model_dir - -@pytest.fixture(autouse=True) -def _set_random_seed() -> Iterator[None]: - """Set random seed for reproducible tests.""" - torch.manual_seed(42) - np.random.seed(42) - yield -``` - -## CV-Specific Testing - -### Image Processing Tests -```python -class TestImageProcessing: - """Tests for image processing utilities.""" - - def test_load_image_returns_correct_dtype(self, tmp_path: Path) -> None: - """Loaded image should be uint8 numpy array.""" - img_path = tmp_path / "test.png" - save_random_image(img_path, size=(64, 64)) - result = load_image(img_path) - assert result.dtype == np.uint8 - assert result.shape == (64, 64, 3) - - def test_augmentation_preserves_shape(self) -> None: - """Augmentations should not change image dimensions.""" - image = np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8) - augmented = apply_augmentations(image, strength=1.0) - assert augmented.shape == image.shape - - def test_batch_normalization_values(self) -> None: - """Batch normalization should produce zero mean, unit variance.""" - batch = torch.randn(32, 3, 64, 64) - bn = torch.nn.BatchNorm2d(3) - bn.eval() - # After training, check that normalization works - output = bn(batch) - assert output.shape == batch.shape -``` - -### Model Output Tests -```python -class TestModelOutputs: - """Tests for model output shapes and types.""" - - def test_classifier_output_shape(self) -> None: - """Classifier should output (batch, num_classes).""" - model = build_classifier(num_classes=10) - model.eval() - x = torch.randn(4, 3, 224, 224) - with torch.no_grad(): - output = model(x) - assert output.shape == (4, 10) - - def test_detector_output_has_required_fields(self) -> None: - """Detection output should contain boxes, scores, labels.""" - model = build_detector() - model.eval() - x = torch.randn(1, 3, 640, 640) - with torch.no_grad(): - output = model(x) - assert "boxes" in output - assert "scores" in output - assert "labels" in output - - def test_segmentor_output_matches_input_size(self) -> None: - """Segmentation output should match input spatial dims.""" - model = build_segmentor(num_classes=21) - model.eval() - x = torch.randn(2, 3, 256, 256) - with torch.no_grad(): - output = model(x) - assert output.shape == (2, 21, 256, 256) -``` - -## Mocking External Dependencies - -```python -class TestVideoReader: - """Tests for video reader with mocked cv2.""" - - def test_read_frames_returns_iterator(self, mocker: MockerFixture) -> None: - """VideoReader should yield frames as numpy arrays.""" - mock_cap = mocker.MagicMock() - mock_cap.isOpened.return_value = True - mock_cap.read.side_effect = [ - (True, np.zeros((480, 640, 3), dtype=np.uint8)), - (True, np.zeros((480, 640, 3), dtype=np.uint8)), - (False, None), - ] - mocker.patch("cv2.VideoCapture", return_value=mock_cap) - - reader = VideoReader("fake_video.mp4") - frames = list(reader) - assert len(frames) == 2 - assert all(f.shape == (480, 640, 3) for f in frames) - - def test_missing_file_raises_error(self) -> None: - """VideoReader should raise FileNotFoundError for missing files.""" - with pytest.raises(FileNotFoundError): - VideoReader("nonexistent.mp4") -``` - -## Performance Tests - -```python -class TestPerformance: - """Performance benchmarks (optional, run with --benchmark).""" - - @pytest.mark.benchmark - def test_model_inference_speed(self, benchmark) -> None: - """Model inference should be under 50ms per image.""" - model = build_tiny_model() - model.eval() - x = torch.randn(1, 3, 224, 224) - - def run_inference(): - with torch.no_grad(): - model(x) - - result = benchmark(run_inference) - assert result.stats.mean < 0.050 # 50ms - - @pytest.mark.benchmark - def test_data_loading_throughput(self, benchmark, tmp_path: Path) -> None: - """Data loading should handle >100 images/second.""" - create_test_dataset(tmp_path, num_images=100) - datamodule = ImageDataModule(DataConfig(data_dir=tmp_path, batch_size=16)) - datamodule.setup("fit") - loader = datamodule.train_dataloader() - - def load_batch(): - return next(iter(loader)) - - result = benchmark(load_batch) - assert result.stats.mean < 1.0 # Under 1 second for a batch -``` - -## Edge Case Testing - -```python -class TestEdgeCases: - """Tests for edge cases and boundary conditions.""" - - def test_single_pixel_image(self) -> None: - """Model should handle 1x1 images gracefully.""" - model = build_classifier(num_classes=10) - model.eval() - x = torch.randn(1, 3, 1, 1) - with torch.no_grad(): - output = model(x) - assert output.shape[0] == 1 - - def test_batch_size_one(self) -> None: - """Model should work with batch size 1.""" - model = build_classifier(num_classes=10) - model.eval() - x = torch.randn(1, 3, 224, 224) - with torch.no_grad(): - output = model(x) - assert output.shape == (1, 10) - - def test_very_large_image(self) -> None: - """Model should handle large images without OOM.""" - model = build_classifier(num_classes=10) - model.eval() - x = torch.randn(1, 3, 2048, 2048) - with torch.no_grad(): - output = model(x) - assert output.shape[0] == 1 - - def test_empty_dataset(self, tmp_path: Path) -> None: - """DataModule should handle empty datasets gracefully.""" - (tmp_path / "train").mkdir() - config = DataConfig(data_dir=tmp_path, batch_size=1) - with pytest.raises(ValueError, match="empty"): - datamodule = ImageDataModule(config) - datamodule.setup("fit") - - def test_all_same_class(self) -> None: - """Metrics should handle degenerate case of single class.""" - predictions = torch.zeros(100, dtype=torch.long) - targets = torch.zeros(100, dtype=torch.long) - accuracy = compute_accuracy(predictions, targets) - assert accuracy == 1.0 -``` - -## Best Practices - -1. **Test one thing per test** -- each test should verify a single behavior -2. **Use descriptive names** -- test names should explain what they verify -3. **Use fixtures** -- share setup code through pytest fixtures -4. **Test failures too** -- verify error handling and edge cases -5. **Keep tests fast** -- unit tests should complete in milliseconds -6. **No test interdependence** -- tests must run in any order -7. **Use parametrize** -- avoid copy-paste for similar test cases -8. **Type your tests** -- test code should also have type hints -9. **Use tmp_path** -- never write to the real filesystem -10. **Mock external calls** -- never hit real APIs or file systems in unit tests - -## CI Integration - -Tests run automatically on every pull request. The pipeline fails if: -- Any test fails -- Coverage drops below 80% -- Skipped tests are found - -```bash -# Full test suite with coverage -pixi run pytest --cov=src --cov-report=term --cov-fail-under=80 - -# Unit tests only -pixi run pytest tests/unit/ -v - -# Integration tests only -pixi run pytest tests/integration/ -v - -# Run specific test file -pixi run pytest tests/unit/test_models.py -v - -# Run with benchmark -pixi run pytest --benchmark-only -``` - -## Review Checklist - -Before marking tests complete, verify: - -- [ ] All public functions have at least one test -- [ ] Edge cases are covered (empty input, single item, large input) -- [ ] Error paths are tested (invalid input, missing files) -- [ ] Pydantic validation is tested (invalid configs) -- [ ] Model output shapes are verified -- [ ] Integration tests cover the full pipeline -- [ ] No skipped or xfail tests without justification -- [ ] Tests are typed (return type annotations) -- [ ] Fixtures are used for shared setup -- [ ] No hard-coded paths or non-deterministic behavior -- [ ] Coverage meets minimum threshold (80%+) -- [ ] Performance benchmarks are present for critical paths diff --git a/agents/test-engineer/action.yml b/agents/test-engineer/action.yml deleted file mode 100644 index 05823f7..0000000 --- a/agents/test-engineer/action.yml +++ /dev/null @@ -1,48 +0,0 @@ -name: 'Test Engineer Agent' -description: 'Run tests and enforce coverage' - -inputs: - python-version: - description: 'Python version' - required: false - default: '3.11' - coverage-threshold: - description: 'Minimum coverage percentage' - required: false - default: '80' - -runs: - using: 'composite' - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Install pixi - uses: prefix-dev/setup-pixi@v0.8.1 - with: - pixi-version: 'latest' - - - name: Run tests with coverage - shell: bash - run: | - pixi run pytest \ - --cov=src \ - --cov-report=term \ - --cov-report=xml \ - --cov-report=html \ - --cov-fail-under=${{ inputs.coverage-threshold }} - - - name: Upload coverage reports - uses: codecov/codecov-action@v4 - if: always() - with: - file: ./coverage.xml - fail_ci_if_error: false - - - name: Check for skipped tests - shell: bash - run: | - if pixi run pytest --collect-only -q | grep -q "skipped"; then - echo "Error: Skipped tests found. Fix or remove them." - exit 1 - fi diff --git a/agents/test-engineer/agent.toml b/agents/test-engineer/agent.toml deleted file mode 100644 index 030ecbb..0000000 --- a/agents/test-engineer/agent.toml +++ /dev/null @@ -1,8 +0,0 @@ -[agent] -name = "test-engineer" -version = "1.0.0" -type = "blocking" -tags = ["testing", "coverage", "ci-cd", "enforcement"] - -[ci] -has_action = true diff --git a/archetypes/cv-inference-service/README.md b/archetypes/cv-inference-service/README.md index f45335e..ee514c2 100644 --- a/archetypes/cv-inference-service/README.md +++ b/archetypes/cv-inference-service/README.md @@ -19,118 +19,92 @@ The service is designed around ONNX Runtime as the inference backend, which deco ## Directory Structure +This is exactly what `whet init` scaffolds -- nothing here is aspirational. + ``` -{{project_slug}}/ -├── .github/ -│ └── workflows/ -│ ├── build.yml # Docker build and push -│ ├── test.yml # API test pipeline -│ └── code-review.yml # Automated code review -├── .gitignore -├── .pre-commit-config.yaml +${project_slug}/ ├── .env.example # Environment variable template -├── pixi.toml -├── pyproject.toml +├── .gitignore +├── Dockerfile # pixi-based production image ├── README.md -├── Dockerfile # Multi-stage production image -├── Dockerfile.gpu # CUDA-enabled image -├── docker-compose.yml # Local development stack -├── src/{{package_name}}/ +├── pixi.toml # Environment + tasks (canonical) +├── pyproject.toml # Package metadata, ruff/mypy/pytest config +├── models/ +│ └── .gitkeep # ONNX model storage (git-ignored artifacts) +├── src/${package_name}/ │ ├── __init__.py │ ├── py.typed -│ ├── app.py # FastAPI application factory +│ ├── app.py # FastAPI app, routes, lifespan model loading │ ├── config.py # Settings via pydantic-settings -│ ├── routes/ -│ │ ├── __init__.py -│ │ ├── health.py # Health and readiness probes -│ │ ├── predict.py # Prediction endpoints -│ │ └── models.py # Model management endpoints -│ ├── inference/ -│ │ ├── __init__.py -│ │ ├── engine.py # ONNX Runtime session manager -│ │ ├── preprocessing.py # Input normalization/resizing -│ │ └── postprocessing.py # Output decoding (NMS, etc.) -│ ├── schemas/ -│ │ ├── __init__.py -│ │ ├── request.py # Pydantic request models -│ │ └── response.py # Pydantic response models -│ ├── middleware/ -│ │ ├── __init__.py -│ │ ├── logging.py # Structured request logging -│ │ └── metrics.py # Prometheus metrics -│ └── utils/ +│ ├── schemas.py # Pydantic request/response models +│ └── inference/ │ ├── __init__.py -│ └── image.py # Image decode/encode helpers -├── models/ -│ └── .gitkeep # ONNX model storage -├── scripts/ -│ ├── download_model.py # Fetch models from registry -│ └── benchmark.py # Latency benchmarking -├── tests/ -│ ├── __init__.py -│ ├── conftest.py # Test client fixtures -│ ├── test_health.py -│ ├── test_predict.py -│ └── test_preprocessing.py -└── k8s/ # Kubernetes manifests (optional) - ├── deployment.yaml - ├── service.yaml - └── hpa.yaml # Horizontal Pod Autoscaler +│ └── engine.py # ONNX Runtime session + pre/postprocessing +└── tests/ + ├── __init__.py + └── test_app.py # Green without any .onnx artifact ``` +Grow it from there: split `schemas.py` into a `schemas/` package, add +`routes/`, `middleware/`, `k8s/`, or a GPU Dockerfile when you actually need +them. The recommended skills (`kubernetes`, `github-actions`, `tensorrt`) cover +those additions. + ## Key Features - **FastAPI** with async request handling for high concurrency under I/O-bound workloads. -- **ONNX Runtime** for framework-agnostic inference with graph optimization and hardware acceleration. +- **ONNX Runtime** for framework-agnostic inference, with CUDA selected automatically when `DEVICE=cuda` and `CUDAExecutionProvider` is available. - **Pydantic v2** request/response schemas with automatic OpenAPI documentation generation. -- **Multi-stage Docker builds** producing minimal production images under 500MB for CPU and properly layered CUDA images for GPU. -- **Health and readiness probes** compatible with Kubernetes liveness and readiness checks. -- **Structured JSON logging** for integration with log aggregation systems (ELK, Datadog, CloudWatch). -- **Prometheus metrics** middleware for request latency, throughput, and error rate monitoring. -- **Horizontal scaling** with Kubernetes HPA manifests and stateless service design. +- **Guarded model loading** -- the model is loaded once in the FastAPI lifespan hook, and a missing `.onnx` file logs a warning and marks the service not-ready instead of crashing it, so a freshly scaffolded project runs and tests green out of the box. +- **Health and readiness probes** compatible with Kubernetes liveness and readiness checks (`/ready` answers 503 until a model is loaded). +- **Loguru logging** configured from `LOG_LEVEL` at startup. +- **pixi-based Docker image** running as a non-root user, with a `HEALTHCHECK` wired to `/health`. ## API Endpoints | Method | Path | Description | |---|---|---| -| GET | `/health` | Liveness probe -- returns 200 if the process is running | -| GET | `/ready` | Readiness probe -- returns 200 if models are loaded and warm | -| POST | `/predict` | Single image prediction | -| POST | `/predict/batch` | Batch prediction (multiple images) | -| GET | `/models` | List loaded models and their metadata | +| GET | `/health` | Liveness probe -- returns 200 whenever the process is running | +| GET | `/ready` | Readiness probe -- 200 once the model is loaded, 503 before that | +| POST | `/predict` | Single image prediction (JSON body, base64-encoded image) | +| POST | `/predict/upload` | Single image prediction (multipart file upload) | | GET | `/docs` | Interactive Swagger UI documentation | ## Configuration and Environment Variables -Configuration is managed through `pydantic-settings`, which reads from environment variables with an optional `.env` file fallback. All settings are validated at startup. +Configuration is managed through `pydantic-settings` in `src/${package_name}/config.py`, which reads from environment variables with an optional `.env` file fallback. All settings are validated at startup, and `.env.example` ships with the template. | Variable | Description | Default | |---|---|---| | `APP_HOST` | Bind address | `0.0.0.0` | | `APP_PORT` | Listen port | `8000` | -| `APP_WORKERS` | Uvicorn worker count | `1` | -| `MODEL_PATH` | Path to the ONNX model file | `models/model.onnx` | | `MODEL_NAME` | Logical model name for API responses | `default` | +| `MODEL_PATH` | Path to the ONNX model file | `models/model.onnx` | | `DEVICE` | Inference device (`cpu` or `cuda`) | `cpu` | -| `MAX_BATCH_SIZE` | Maximum images per batch request | `32` | -| `INPUT_SIZE` | Expected input dimensions (HxW) | `640x640` | +| `INPUT_WIDTH` | Model input width in pixels | `640` | +| `INPUT_HEIGHT` | Model input height in pixels | `640` | | `CONFIDENCE_THRESHOLD` | Minimum confidence for detections | `0.5` | | `LOG_LEVEL` | Logging verbosity | `INFO` | | `CORS_ORIGINS` | Comma-separated allowed CORS origins | `*` | ## Dependencies +Managed by `pixi.toml` (conda) and `pyproject.toml` (PyPI metadata). + ```toml [dependencies] -python = ">=3.11" +python = "3.11.*" +numpy = ">=1.26" +pillow = ">=10.0" + +[pypi-dependencies] fastapi = ">=0.110" -uvicorn = ">=0.27" +uvicorn = { version = ">=0.27", extras = ["standard"] } +pydantic = ">=2.6" +pydantic-settings = ">=2.2" +loguru = ">=0.7" onnxruntime = ">=1.17" -pydantic = ">=2.0" -pydantic-settings = ">=2.0" -pillow = ">=10.0" -numpy = ">=1.26" -python-multipart = ">=0.0.6" +python-multipart = ">=0.0.9" ``` ## Usage @@ -144,82 +118,82 @@ pixi install # Copy environment template cp .env.example .env -# Place your ONNX model in models/ +# Place your ONNX model in models/ (optional -- the service starts without it) cp /path/to/your/model.onnx models/model.onnx # Start the development server with hot reload -pixi run uvicorn src.{{package_name}}.app:create_app --factory --reload --port 8000 +uvicorn ${package_name}.app:app --reload --port 8000 + +# ...or via the pixi task +pixi run dev ``` +Add dependencies with `pixi add ` (conda) or `pixi add --pypi `. + ### Docker Deployment ```bash -# Build CPU image -docker build -t {{project_slug}}:latest . - -# Build GPU image -docker build -f Dockerfile.gpu -t {{project_slug}}:latest-gpu . +# Build the image (bootstraps pixi, runs `pixi install`, runs as non-root) +docker build -t ${project_slug}:latest . # Run container -docker run -p 8000:8000 -v $(pwd)/models:/app/models {{project_slug}}:latest - -# Run with GPU -docker run --gpus all -p 8000:8000 -v $(pwd)/models:/app/models {{project_slug}}:latest-gpu - -# Use docker-compose for local development -docker compose up --build +docker run -p 8000:8000 -v $(pwd)/models:/app/models ${project_slug}:latest ``` +For GPU serving, swap `onnxruntime` for `onnxruntime-gpu`, base the image on an +NVIDIA CUDA runtime image, set `DEVICE=cuda`, and run with `--gpus all`. + ### Making Predictions ```bash -# Single image prediction +# Base64 JSON body curl -X POST http://localhost:8000/predict \ - -F "file=@test_image.jpg" + -H 'Content-Type: application/json' \ + -d "{\"image_b64\": \"$(base64 < test_image.jpg)\", \"confidence_threshold\": 0.5}" -# Batch prediction -curl -X POST http://localhost:8000/predict/batch \ - -F "files=@image1.jpg" \ - -F "files=@image2.jpg" \ - -F "files=@image3.jpg" +# Multipart upload +curl -X POST 'http://localhost:8000/predict/upload?confidence_threshold=0.5' \ + -F "file=@test_image.jpg" -# Check health +# Check health and readiness curl http://localhost:8000/health - -# Check readiness -curl http://localhost:8000/ready +curl -i http://localhost:8000/ready ``` -### Benchmarking +### Quality Gates ```bash -# Run latency benchmark -pixi run python scripts/benchmark.py --model models/model.onnx --iterations 1000 - -# Profile with different batch sizes -pixi run python scripts/benchmark.py --model models/model.onnx --batch-sizes 1,4,8,16,32 +pytest +ruff check . +mypy src/ --strict ``` ## Customization Guide -### Adding Custom Preprocessing - -Edit `src/{{package_name}}/inference/preprocessing.py` to define your input pipeline. The preprocessing module should accept raw image bytes or PIL images and return numpy arrays matching the model's expected input shape and dtype. Common operations include resizing, normalization (ImageNet mean/std or custom), color space conversion, and padding. +### Adding Custom Preprocessing and Postprocessing -### Adding Custom Postprocessing +`src/${package_name}/inference/engine.py` ships a generic pipeline: resize -> +normalize (ImageNet mean/std) -> NCHW float32 -> `session.run` -> decode. The +decoder understands an `(N, >=6)` detection tensor (`x1, y1, x2, y2, score, +class_id`) and a 1-D class-logit vector; anything else logs a warning and +returns no detections. -Edit `src/{{package_name}}/inference/postprocessing.py` for task-specific output decoding. For object detection, this includes non-maximum suppression and bounding box format conversion. For segmentation, this includes argmax decoding and contour extraction. For classification, this includes softmax and top-k label mapping. +Replace `OnnxInferenceEngine.preprocess` with the transforms your model was +exported with, and `OnnxInferenceEngine.postprocess` with your task-specific +decoding -- NMS and box-format conversion for detection, argmax and contour +extraction for segmentation, softmax and top-k label mapping for +classification. Set `labels` on `InferenceConfig` for human-readable classes. ### Adding New Endpoints -1. Create a new route module in `src/{{package_name}}/routes/`. -2. Define Pydantic request and response schemas in `src/{{package_name}}/schemas/`. -3. Register the router in `app.py`. +1. Define Pydantic request and response schemas in `src/${package_name}/schemas.py` (split it into a `schemas/` package once it grows). +2. Add the route to `app.py`, or create a `routes/` package and register the router. +3. Depend on the shared engine via the `EngineDep` annotated dependency so the model stays loaded once per process. 4. Add corresponding tests in `tests/`. ### Scaling Considerations -For CPU inference, scale horizontally by increasing `APP_WORKERS` or running multiple container replicas behind a load balancer. For GPU inference, use one worker per GPU and scale by adding GPU nodes. The Kubernetes HPA manifest in `k8s/hpa.yaml` is preconfigured to scale on CPU utilization and can be extended with custom metrics from the Prometheus middleware. +For CPU inference, scale horizontally with `uvicorn --workers N` or multiple container replicas behind a load balancer -- the service is stateless apart from the loaded session. For GPU inference, use one worker per GPU and scale by adding GPU nodes. The `kubernetes` skill covers Deployment, Service, and HPA manifests; wire the liveness probe to `/health` and the readiness probe to `/ready`. ### Switching Inference Backends diff --git a/archetypes/cv-inference-service/archetype.toml b/archetypes/cv-inference-service/archetype.toml index d3fadc3..638cb98 100644 --- a/archetypes/cv-inference-service/archetype.toml +++ b/archetypes/cv-inference-service/archetype.toml @@ -6,5 +6,5 @@ tags = ["inference", "deployment", "fastapi", "onnx"] description = "FastAPI + ONNX deployment service for computer vision models" [skills] -required = ["onnx", "pydantic-strict", "loguru", "docker-cv", "testing"] -recommended = ["tensorrt", "gcp", "github-actions"] +required = ["fastapi", "onnx", "pydantic", "loguru", "docker-cv", "testing", "code-quality", "pixi"] +recommended = ["tensorrt", "model-evaluation", "gcp", "github-actions"] diff --git a/archetypes/cv-inference-service/template/.env.example b/archetypes/cv-inference-service/template/.env.example new file mode 100644 index 0000000..38f7c42 --- /dev/null +++ b/archetypes/cv-inference-service/template/.env.example @@ -0,0 +1,19 @@ +# Copy to .env and adjust. Every value is validated at startup by +# src/${package_name}/config.py (pydantic-settings). + +# --- server ----------------------------------------------------------------- +APP_HOST=0.0.0.0 +APP_PORT=8000 + +# --- model ------------------------------------------------------------------ +MODEL_NAME=default +# The service starts even if this file is absent; /ready then answers 503. +MODEL_PATH=models/model.onnx +DEVICE=cpu # cpu | cuda (cuda needs onnxruntime-gpu) +INPUT_WIDTH=640 +INPUT_HEIGHT=640 +CONFIDENCE_THRESHOLD=0.5 + +# --- observability ---------------------------------------------------------- +LOG_LEVEL=INFO +CORS_ORIGINS=* diff --git a/archetypes/cv-inference-service/template/.gitignore b/archetypes/cv-inference-service/template/.gitignore new file mode 100644 index 0000000..d25ce1b --- /dev/null +++ b/archetypes/cv-inference-service/template/.gitignore @@ -0,0 +1,12 @@ +.env +__pycache__/ +*.py[cod] +.venv/ +.pixi/ +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +models/*.onnx +dist/ +build/ +*.egg-info/ diff --git a/archetypes/cv-inference-service/template/Dockerfile b/archetypes/cv-inference-service/template/Dockerfile index c0cac9b..d54efb0 100644 --- a/archetypes/cv-inference-service/template/Dockerfile +++ b/archetypes/cv-inference-service/template/Dockerfile @@ -1,21 +1,41 @@ -FROM python:${python_version}-slim AS builder +# syntax=docker/dockerfile:1 +FROM python:${python_version}-slim -WORKDIR /app -RUN pip install --no-cache-dir uv - -COPY pyproject.toml uv.lock ./ -RUN uv sync --frozen --no-dev +# pixi is bootstrapped over HTTPS, so a slim base needs curl + CA certificates. +RUN apt-get update \ + && apt-get install -y --no-install-recommends curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* -FROM python:${python_version}-slim AS runtime +RUN curl -fsSL https://pixi.sh/install.sh | bash +ENV PATH="/root/.pixi/bin:$${PATH}" WORKDIR /app -COPY --from=builder /app/.venv /app/.venv -ENV PATH="/app/.venv/bin:$$PATH" +# Only pixi.toml ships with the template. Run `pixi install` locally, commit the +# generated pixi.lock, then add it here (`COPY pixi.toml pixi.lock ./`) and switch +# to `pixi install --locked` for fully reproducible builds. +COPY pixi.toml ./ +RUN pixi install + +COPY pyproject.toml README.md ./ COPY src/ ./src/ -RUN useradd --create-home appuser +ENV PYTHONPATH=/app/src \ + PIXI_CACHE_DIR=/tmp/pixi-cache \ + MODEL_PATH=/app/models/model.onnx + +# Run as an unprivileged user; the pixi bootstrap lives under /root, so make it +# traversable and readable (never writable) for that user. +RUN useradd --create-home --uid 1000 appuser \ + && mkdir -p /app/models /tmp/pixi-cache \ + && chown -R appuser:appuser /app /tmp/pixi-cache \ + && chmod a+rx /root \ + && chmod -R a+rX /root/.pixi USER appuser EXPOSE 8000 -CMD ["uvicorn", "${package_name}.app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"] + +HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \ + CMD curl -fsS http://127.0.0.1:8000/health || exit 1 + +CMD ["pixi", "run", "serve"] diff --git a/archetypes/cv-inference-service/template/README.md b/archetypes/cv-inference-service/template/README.md index 9563d04..3dbd23d 100644 --- a/archetypes/cv-inference-service/template/README.md +++ b/archetypes/cv-inference-service/template/README.md @@ -2,31 +2,129 @@ ${description} +A FastAPI service that serves a computer vision model with ONNX Runtime. + +## Layout + +``` +${project_slug}/ +├── Dockerfile # pixi-based production image +├── pixi.toml # environment + tasks (canonical) +├── pyproject.toml # package metadata, ruff/mypy/pytest config +├── .env.example # every supported environment variable +├── models/ # drop your model.onnx here +├── src/${package_name}/ +│ ├── app.py # FastAPI app, routes, lifespan model loading +│ ├── config.py # pydantic-settings configuration +│ ├── schemas.py # Pydantic request/response models +│ └── inference/ +│ └── engine.py # ONNX Runtime session, pre/postprocessing +└── tests/ + └── test_app.py # runs green without any .onnx artifact +``` + ## Setup ```bash -uv sync --all-extras +pixi install +cp .env.example .env ``` +Add a dependency with `pixi add ` (conda) or `pixi add --pypi `. + +Place your exported model where `MODEL_PATH` points: + +```bash +cp /path/to/your/model.onnx models/model.onnx +``` + +The service starts **without** a model file — `/health` stays 200, while +`/ready` and `/predict` answer 503 until a model is loaded. + ## Running ```bash -# Development server -uv run uvicorn ${package_name}.app:app --reload --port 8000 +# Development server (hot reload) +uvicorn ${package_name}.app:app --reload --port 8000 # Production -uv run uvicorn ${package_name}.app:app --host 0.0.0.0 --port 8000 --workers 4 +uvicorn ${package_name}.app:app --host 0.0.0.0 --port 8000 --workers 4 ``` +Via pixi tasks: `pixi run dev` / `pixi run serve`. + ## API Endpoints -- `GET /health` — Health check -- `POST /api/v1/predict` — Run inference on an image +| Method | Path | Description | +|---|---|---| +| GET | `/health` | Liveness probe — 200 whenever the process is up | +| GET | `/ready` | Readiness probe — 503 until the ONNX model is loaded | +| POST | `/predict` | Inference on a base64-encoded image (JSON body) | +| POST | `/predict/upload` | Inference on a multipart image upload | +| GET | `/docs` | Swagger UI | + +```bash +curl http://localhost:8000/health +curl -i http://localhost:8000/ready + +curl -X POST http://localhost:8000/predict \ + -H 'Content-Type: application/json' \ + -d "{\"image_b64\": \"$$(base64 < test.jpg)\", \"confidence_threshold\": 0.5}" + +curl -X POST 'http://localhost:8000/predict/upload?confidence_threshold=0.5' \ + -F 'file=@test.jpg' +``` + +## Configuration + +Settings are read from environment variables (or a local `.env`) and validated +at startup by `src/${package_name}/config.py`. See `.env.example`. + +| Variable | Description | Default | +|---|---|---| +| `APP_HOST` | Bind address | `0.0.0.0` | +| `APP_PORT` | Listen port | `8000` | +| `MODEL_NAME` | Logical model name in API responses | `default` | +| `MODEL_PATH` | Path to the ONNX model file | `models/model.onnx` | +| `DEVICE` | Inference device (`cpu` or `cuda`) | `cpu` | +| `INPUT_WIDTH` | Model input width in pixels | `640` | +| `INPUT_HEIGHT` | Model input height in pixels | `640` | +| `CONFIDENCE_THRESHOLD` | Default minimum detection confidence | `0.5` | +| `LOG_LEVEL` | Loguru log level | `INFO` | +| `CORS_ORIGINS` | Comma-separated allowed CORS origins | `*` | + +## Adapting the engine to your model + +`src/${package_name}/inference/engine.py` ships a generic pipeline: +resize → normalise (ImageNet mean/std) → NCHW float32 → `session.run` → decode. +The decoder understands two common export shapes — an `(N, >=6)` detection +tensor (`x1, y1, x2, y2, score, class_id`) and a 1-D class-logit vector. +Replace `preprocess`/`postprocess` with the transforms your model was exported +with, and set `labels` on `InferenceConfig` for human-readable class names. + +CUDA is selected automatically when `DEVICE=cuda` *and* `CUDAExecutionProvider` +is available (install `onnxruntime-gpu`); otherwise the engine logs a warning +and falls back to CPU. ## Development ```bash -uv run pytest tests/ -v -uv run ruff check . -uv run mypy src/ --strict +pytest +ruff check . +ruff format --check . +mypy src/ --strict ``` + +Via pixi tasks: `pixi run test`, `pixi run lint`, `pixi run typecheck`, or +`pixi run quality` for all of them. + +## Docker + +```bash +docker build -t ${project_slug}:latest . +docker run -p 8000:8000 -v "$$(pwd)/models:/app/models" ${project_slug}:latest +``` + +The image bootstraps pixi and runs `pixi install`. For reproducible builds, +commit the generated `pixi.lock`, add it to the `COPY` line in the `Dockerfile`, +and switch to `pixi install --locked`. diff --git a/archetypes/cv-inference-service/template/models/.gitkeep b/archetypes/cv-inference-service/template/models/.gitkeep new file mode 100644 index 0000000..c854a9c --- /dev/null +++ b/archetypes/cv-inference-service/template/models/.gitkeep @@ -0,0 +1 @@ +# ONNX model artifacts live here (git-ignored). diff --git a/archetypes/cv-inference-service/template/pixi.toml b/archetypes/cv-inference-service/template/pixi.toml new file mode 100644 index 0000000..555c4d5 --- /dev/null +++ b/archetypes/cv-inference-service/template/pixi.toml @@ -0,0 +1,51 @@ +[workspace] +name = "${project_slug}" +version = "0.1.0" +description = "${description}" +authors = ["${author}"] +channels = ["conda-forge"] +platforms = ["linux-64", "linux-aarch64", "osx-arm64"] + +[activation.env] +# src-layout: make the package importable without an editable install. +PYTHONPATH = "src" + +[dependencies] +# Pinned to the minor series: onnxruntime only publishes wheels for released +# CPython minors, so an open-ended ">=" range resolves to a Python with no +# onnxruntime build available. +python = "${python_version}.*" +numpy = ">=1.26" +pillow = ">=10.0" + +[pypi-dependencies] +fastapi = ">=0.110" +uvicorn = { version = ">=0.27", extras = ["standard"] } +pydantic = ">=2.6" +pydantic-settings = ">=2.2" +loguru = ">=0.7" +onnxruntime = ">=1.17" +python-multipart = ">=0.0.9" + +[feature.dev.dependencies] +pytest = ">=7.4" +pytest-cov = ">=4.1" +ruff = ">=0.8" +mypy = ">=1.11" + +[feature.dev.pypi-dependencies] +pytest-asyncio = ">=0.23" +httpx = ">=0.27" + +[environments] +default = { features = ["dev"], solve-group = "default" } + +[tasks] +serve = "uvicorn ${package_name}.app:app --host 0.0.0.0 --port 8000" +dev = "uvicorn ${package_name}.app:app --reload --port 8000" +test = "pytest" +lint = "ruff check ." +format = "ruff format ." +format-check = "ruff format --check ." +typecheck = "mypy src/ --strict" +quality = { depends-on = ["lint", "format-check", "typecheck", "test"] } diff --git a/archetypes/cv-inference-service/template/pyproject.toml b/archetypes/cv-inference-service/template/pyproject.toml index b0c3718..ce6d06f 100644 --- a/archetypes/cv-inference-service/template/pyproject.toml +++ b/archetypes/cv-inference-service/template/pyproject.toml @@ -5,17 +5,23 @@ description = "${description}" authors = [{name = "${author}"}] requires-python = ">=${python_version}" dependencies = [ - "fastapi>=0.100", - "uvicorn[standard]>=0.20", - "pydantic>=2.0", + "fastapi>=0.110", + "uvicorn[standard]>=0.27", + "pydantic>=2.6", + "pydantic-settings>=2.2", "loguru>=0.7", + "onnxruntime>=1.17", + "numpy>=1.26", + "pillow>=10.0", + "python-multipart>=0.0.9", ] [project.optional-dependencies] dev = [ "pytest>=7.4", + "pytest-asyncio>=0.23", "pytest-cov>=4.1", - "httpx>=0.24", + "httpx>=0.27", "ruff>=0.8", "mypy>=1.11", ] @@ -39,6 +45,12 @@ ignore = ["S101"] python_version = "${python_version}" strict = true +[[tool.mypy.overrides]] +module = ["onnxruntime.*"] +ignore_missing_imports = true + [tool.pytest.ini_options] testpaths = ["tests"] +pythonpath = ["src"] +asyncio_mode = "auto" addopts = "-v" diff --git a/archetypes/cv-inference-service/template/src/${package_name}/app.py b/archetypes/cv-inference-service/template/src/${package_name}/app.py index 4ec5b27..1b32ca6 100644 --- a/archetypes/cv-inference-service/template/src/${package_name}/app.py +++ b/archetypes/cv-inference-service/template/src/${package_name}/app.py @@ -2,40 +2,159 @@ from __future__ import annotations -from contextlib import asynccontextmanager +import sys from collections.abc import AsyncGenerator +from contextlib import asynccontextmanager +from typing import Annotated -from fastapi import FastAPI +from fastapi import ( + Depends, + FastAPI, + File, + HTTPException, + Query, + Request, + Response, + UploadFile, + status, +) +from fastapi.concurrency import run_in_threadpool from fastapi.middleware.cors import CORSMiddleware from loguru import logger +from PIL.Image import Image as PILImage + +from . import __version__ +from .config import Settings, get_settings +from .inference.engine import ( + ImageDecodeError, + InferenceConfig, + OnnxInferenceEngine, + decode_base64_image, + decode_image_bytes, +) +from .schemas import HealthResponse, PredictionRequest, PredictionResponse, ReadinessResponse + -from .schemas import HealthResponse +def build_engine(settings: Settings) -> OnnxInferenceEngine: + """Create an engine from validated settings (performs no I/O).""" + return OnnxInferenceEngine( + InferenceConfig( + model_path=settings.model_path, + model_name=settings.model_name, + device=settings.device, + input_width=settings.input_width, + input_height=settings.input_height, + confidence_threshold=settings.confidence_threshold, + ) + ) @asynccontextmanager -async def lifespan(app: FastAPI) -> AsyncGenerator[None]: - """Load model on startup, release on shutdown.""" - logger.info("Loading model...") - # TODO: Load your model here +async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: + """Load the ONNX model once on startup and release it on shutdown.""" + settings = get_settings() + logger.remove() + logger.add(sys.stderr, level=settings.log_level) + + engine = build_engine(settings) + app.state.settings = settings + app.state.engine = engine + + # Guarded: a missing model must not stop the service from starting. + if engine.load(): + logger.info("Service ready — model '{}' loaded", settings.model_name) + else: + logger.warning("Service started without a model; /predict will answer 503") + yield + + engine.unload() logger.info("Shutting down") app = FastAPI( title="${project_name}", - version="0.1.0", + version=__version__, lifespan=lifespan, ) app.add_middleware( CORSMiddleware, - allow_origins=["*"], + allow_origins=get_settings().cors_origin_list, allow_methods=["*"], allow_headers=["*"], ) +def get_engine(request: Request) -> OnnxInferenceEngine: + """Return the engine created during startup.""" + engine = getattr(request.app.state, "engine", None) + if not isinstance(engine, OnnxInferenceEngine): # pragma: no cover - startup invariant + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, + detail="inference engine is not initialised", + ) + return engine + + +EngineDep = Annotated[OnnxInferenceEngine, Depends(get_engine)] + + +def _require_ready(engine: OnnxInferenceEngine) -> None: + if not engine.is_ready: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail=f"no model loaded (expected at '{engine.config.model_path}')", + ) + + +async def _run( + engine: OnnxInferenceEngine, image: PILImage, threshold: float +) -> PredictionResponse: + detections, elapsed_ms = await run_in_threadpool(engine.predict, image, threshold) + return PredictionResponse(detections=detections, inference_time_ms=elapsed_ms) + + @app.get("/health", response_model=HealthResponse) async def health() -> HealthResponse: - """Health check endpoint.""" - return HealthResponse(status="healthy", version="0.1.0") + """Liveness probe — 200 while the process is up, with or without a model.""" + return HealthResponse(status="healthy", version=__version__) + + +@app.get("/ready", response_model=ReadinessResponse) +async def ready(engine: EngineDep, response: Response) -> ReadinessResponse: + """Readiness probe — 503 until an ONNX model is loaded.""" + if not engine.is_ready: + response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE + return ReadinessResponse( + ready=engine.is_ready, + model_name=engine.config.model_name, + model_path=str(engine.config.model_path), + providers=engine.providers, + ) + + +@app.post("/predict", response_model=PredictionResponse) +async def predict(payload: PredictionRequest, engine: EngineDep) -> PredictionResponse: + """Run inference on a base64-encoded image.""" + _require_ready(engine) + try: + image = decode_base64_image(payload.image_b64) + except ImageDecodeError as exc: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc + return await _run(engine, image, payload.confidence_threshold) + + +@app.post("/predict/upload", response_model=PredictionResponse) +async def predict_upload( + engine: EngineDep, + file: Annotated[UploadFile, File(description="Image file to run inference on")], + confidence_threshold: Annotated[float, Query(ge=0.0, le=1.0)] = 0.5, +) -> PredictionResponse: + """Run inference on a multipart-uploaded image file.""" + _require_ready(engine) + try: + image = decode_image_bytes(await file.read()) + except ImageDecodeError as exc: + raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc + return await _run(engine, image, confidence_threshold) diff --git a/archetypes/cv-inference-service/template/src/${package_name}/config.py b/archetypes/cv-inference-service/template/src/${package_name}/config.py new file mode 100644 index 0000000..d6c232d --- /dev/null +++ b/archetypes/cv-inference-service/template/src/${package_name}/config.py @@ -0,0 +1,57 @@ +"""Runtime settings for ${project_name}. + +Every value is read from an environment variable (or a local ``.env`` file) and +validated at import time by Pydantic V2. See ``.env.example`` for the full list. +""" + +from __future__ import annotations + +from functools import lru_cache +from pathlib import Path +from typing import Literal + +from pydantic import Field +from pydantic_settings import BaseSettings, SettingsConfigDict + + +class Settings(BaseSettings): + """Validated application configuration.""" + + model_config = SettingsConfigDict( + env_file=".env", + env_file_encoding="utf-8", + extra="ignore", + # ``model_path`` / ``model_name`` would otherwise collide with Pydantic's + # reserved ``model_`` namespace. + protected_namespaces=(), + ) + + app_host: str = Field(default="0.0.0.0", description="Bind address") # noqa: S104 + app_port: int = Field(default=8000, ge=1, le=65535, description="Listen port") + + model_name: str = Field(default="default", description="Logical model name") + model_path: Path = Field( + default=Path("models/model.onnx"), + description="Path to the ONNX model file; the service starts even if it is absent", + ) + device: Literal["cpu", "cuda"] = Field(default="cpu", description="Inference device") + + input_width: int = Field(default=640, gt=0, description="Model input width in pixels") + input_height: int = Field(default=640, gt=0, description="Model input height in pixels") + confidence_threshold: float = Field( + default=0.5, ge=0.0, le=1.0, description="Default minimum detection confidence" + ) + + log_level: str = Field(default="INFO", description="Loguru log level") + cors_origins: str = Field(default="*", description="Comma-separated allowed CORS origins") + + @property + def cors_origin_list(self) -> list[str]: + """Parse ``cors_origins`` into the list FastAPI's CORS middleware expects.""" + return [origin.strip() for origin in self.cors_origins.split(",") if origin.strip()] + + +@lru_cache(maxsize=1) +def get_settings() -> Settings: + """Return the process-wide settings singleton.""" + return Settings() diff --git a/archetypes/cv-inference-service/template/src/${package_name}/inference/__init__.py b/archetypes/cv-inference-service/template/src/${package_name}/inference/__init__.py new file mode 100644 index 0000000..5f90fee --- /dev/null +++ b/archetypes/cv-inference-service/template/src/${package_name}/inference/__init__.py @@ -0,0 +1,17 @@ +"""Inference backend for ${project_name}.""" + +from .engine import ( + ImageDecodeError, + InferenceConfig, + OnnxInferenceEngine, + decode_base64_image, + decode_image_bytes, +) + +__all__ = [ + "ImageDecodeError", + "InferenceConfig", + "OnnxInferenceEngine", + "decode_base64_image", + "decode_image_bytes", +] diff --git a/archetypes/cv-inference-service/template/src/${package_name}/inference/engine.py b/archetypes/cv-inference-service/template/src/${package_name}/inference/engine.py new file mode 100644 index 0000000..3fc7987 --- /dev/null +++ b/archetypes/cv-inference-service/template/src/${package_name}/inference/engine.py @@ -0,0 +1,272 @@ +"""ONNX Runtime inference engine for ${project_name}. + +The engine is deliberately small and typed end to end: + +* :class:`InferenceConfig` — validated (Pydantic V2) description of the model. +* :class:`OnnxInferenceEngine` — loads the session, preprocesses an image, runs + it, and decodes the raw tensors into :class:`~..schemas.Detection` objects. + +A freshly scaffolded project has no ``.onnx`` file. Loading is therefore +*guarded*: a missing or unreadable model logs a warning and leaves the engine in +a not-ready state instead of crashing the service, so ``/health`` keeps working +and ``/ready`` correctly reports 503. + +Replace :meth:`OnnxInferenceEngine.preprocess` and +:meth:`OnnxInferenceEngine.postprocess` with the transforms your own model was +exported with. +""" + +from __future__ import annotations + +import base64 +import binascii +import io +import time +from pathlib import Path +from typing import Any + +import numpy as np +import onnxruntime as ort +from loguru import logger +from numpy.typing import NDArray +from PIL import Image, UnidentifiedImageError +from pydantic import BaseModel, ConfigDict, Field + +from ..schemas import Detection + +FloatArray = NDArray[np.float32] + +#: ImageNet normalisation — the most common export-time convention. +DEFAULT_MEAN: tuple[float, float, float] = (0.485, 0.456, 0.406) +DEFAULT_STD: tuple[float, float, float] = (0.229, 0.224, 0.225) + + +class ImageDecodeError(ValueError): + """Raised when request payload bytes are not a decodable image.""" + + +class InferenceConfig(BaseModel, frozen=True): + """Everything the engine needs to know about the model.""" + + model_config = ConfigDict(frozen=True, protected_namespaces=()) + + model_path: Path = Path("models/model.onnx") + model_name: str = "default" + device: str = Field(default="cpu", pattern="^(cpu|cuda)$") + input_width: int = Field(default=640, gt=0) + input_height: int = Field(default=640, gt=0) + confidence_threshold: float = Field(default=0.5, ge=0.0, le=1.0) + mean: tuple[float, float, float] = DEFAULT_MEAN + std: tuple[float, float, float] = DEFAULT_STD + labels: tuple[str, ...] = () + + +def decode_image_bytes(data: bytes) -> Image.Image: + """Decode raw bytes into a PIL image, or raise :class:`ImageDecodeError`.""" + try: + image = Image.open(io.BytesIO(data)) + image.load() + except (UnidentifiedImageError, OSError, ValueError) as exc: + raise ImageDecodeError("payload is not a decodable image") from exc + return image + + +def decode_base64_image(payload: str) -> Image.Image: + """Decode a base64 (optionally data-URI prefixed) image string.""" + encoded = payload.split(",", 1)[-1] if payload.startswith("data:") else payload + try: + raw = base64.b64decode(encoded, validate=True) + except (binascii.Error, ValueError) as exc: + raise ImageDecodeError("image_b64 is not valid base64") from exc + if not raw: + raise ImageDecodeError("image_b64 is empty") + return decode_image_bytes(raw) + + +class OnnxInferenceEngine: + """Thin, typed wrapper around an ONNX Runtime session.""" + + def __init__(self, config: InferenceConfig) -> None: + self.config = config + self._session: ort.InferenceSession | None = None + self._input_name: str | None = None + + # ---------------------------------------------------------------- lifecycle + + @property + def is_ready(self) -> bool: + """``True`` once a session has been successfully created.""" + return self._session is not None + + @property + def providers(self) -> list[str]: + """Execution providers actually in use (empty when not loaded).""" + if self._session is None: + return [] + providers: list[str] = list(self._session.get_providers()) + return providers + + def resolve_providers(self) -> list[str]: + """Pick execution providers: CPU by default, CUDA when asked for and available.""" + available = set(ort.get_available_providers()) + if self.config.device == "cuda": + if "CUDAExecutionProvider" in available: + return ["CUDAExecutionProvider", "CPUExecutionProvider"] + logger.warning( + "device=cuda requested but CUDAExecutionProvider is unavailable; " + "falling back to CPU (install onnxruntime-gpu for CUDA support)" + ) + return ["CPUExecutionProvider"] + + def load(self) -> bool: + """Load the ONNX session. Returns ``False`` instead of raising on failure.""" + path = self.config.model_path + if not path.is_file(): + logger.warning( + "ONNX model not found at '{}' — starting in a not-ready state. " + "Drop a model there (or set MODEL_PATH) and restart.", + path, + ) + self._session = None + return False + + try: + session = ort.InferenceSession(str(path), providers=self.resolve_providers()) + except Exception as exc: # noqa: BLE001 - never let a bad artifact kill startup + logger.error("Failed to load ONNX model '{}': {}", path, exc) + self._session = None + return False + + self._session = session + self._input_name = str(session.get_inputs()[0].name) + logger.info( + "Loaded ONNX model '{}' from '{}' (providers={}, input='{}')", + self.config.model_name, + path, + session.get_providers(), + self._input_name, + ) + return True + + def unload(self) -> None: + """Release the session.""" + if self._session is not None: + logger.info("Releasing ONNX session for '{}'", self.config.model_name) + self._session = None + self._input_name = None + + # ------------------------------------------------------------- pre/post ops + + def preprocess(self, image: Image.Image) -> FloatArray: + """Resize, normalise and convert an image to a NCHW float32 batch of 1.""" + resized = image.convert("RGB").resize( + (self.config.input_width, self.config.input_height), + Image.Resampling.BILINEAR, + ) + scaled = np.asarray(resized, dtype=np.float32) / 255.0 + mean = np.asarray(self.config.mean, dtype=np.float32) + std = np.asarray(self.config.std, dtype=np.float32) + normalised = ((scaled - mean) / std).astype(np.float32) + chw = np.transpose(normalised, (2, 0, 1)) + batch: FloatArray = np.expand_dims(chw, axis=0).astype(np.float32) + return batch + + def postprocess(self, outputs: list[Any], threshold: float) -> list[Detection]: + """Decode raw model outputs into detections. + + Two common export shapes are handled out of the box; anything else logs a + warning and returns no detections. Replace this with your model's decoder + (NMS, anchor decoding, segmentation argmax, ...). + """ + if not outputs: + return [] + + raw = np.squeeze(np.asarray(outputs[0], dtype=np.float32)) + + # (N, >=6): [x1, y1, x2, y2, score, class_id] — typical exported detector. + if raw.ndim == 2 and raw.shape[-1] >= 6: + detections: list[Detection] = [] + for row in raw: + score = float(row[4]) + if score < threshold: + continue + detections.append( + Detection( + label=self.label_for(int(row[5])), + confidence=min(max(score, 0.0), 1.0), + bbox=[float(row[0]), float(row[1]), float(row[2]), float(row[3])], + ) + ) + return detections + + # (C,): class logits — treat as whole-image classification. + if raw.ndim == 1 and raw.size > 0: + probs = _softmax(raw) + index = int(np.argmax(probs)) + score = float(probs[index]) + if score < threshold: + return [] + width = float(self.config.input_width) + height = float(self.config.input_height) + return [ + Detection( + label=self.label_for(index), + confidence=min(max(score, 0.0), 1.0), + bbox=[0.0, 0.0, width, height], + ) + ] + + logger.warning( + "Unrecognised model output shape {} — implement postprocess() for your model", + raw.shape, + ) + return [] + + def label_for(self, index: int) -> str: + """Map a class index to a human-readable label.""" + if 0 <= index < len(self.config.labels): + return self.config.labels[index] + return f"class_{index}" + + # ----------------------------------------------------------------- inference + + def predict( + self, + image: Image.Image, + confidence_threshold: float | None = None, + ) -> tuple[list[Detection], float]: + """Run the full pipeline. Returns ``(detections, inference_time_ms)``. + + Raises :class:`RuntimeError` if no model is loaded — callers should check + :attr:`is_ready` first and answer 503. + """ + session = self._session + if session is None or self._input_name is None: + raise RuntimeError("no ONNX model is loaded") + + threshold = ( + self.config.confidence_threshold + if confidence_threshold is None + else confidence_threshold + ) + batch = self.preprocess(image) + + started = time.perf_counter() + outputs: list[Any] = session.run(None, {self._input_name: batch}) + elapsed_ms = (time.perf_counter() - started) * 1000.0 + + detections = self.postprocess(outputs, threshold) + logger.debug( + "Inference on '{}' produced {} detection(s) in {:.2f} ms", + self.config.model_name, + len(detections), + elapsed_ms, + ) + return detections, elapsed_ms + + +def _softmax(values: FloatArray) -> FloatArray: + shifted = values - np.max(values) + exponentials = np.exp(shifted) + result: FloatArray = (exponentials / np.sum(exponentials)).astype(np.float32) + return result diff --git a/archetypes/cv-inference-service/template/src/${package_name}/py.typed b/archetypes/cv-inference-service/template/src/${package_name}/py.typed new file mode 100644 index 0000000..e69de29 diff --git a/archetypes/cv-inference-service/template/src/${package_name}/schemas.py b/archetypes/cv-inference-service/template/src/${package_name}/schemas.py index c63eb53..8b66601 100644 --- a/archetypes/cv-inference-service/template/src/${package_name}/schemas.py +++ b/archetypes/cv-inference-service/template/src/${package_name}/schemas.py @@ -2,7 +2,7 @@ from __future__ import annotations -from pydantic import BaseModel, Field +from pydantic import BaseModel, ConfigDict, Field class HealthResponse(BaseModel, frozen=True): @@ -12,6 +12,21 @@ class HealthResponse(BaseModel, frozen=True): version: str +class ReadinessResponse(BaseModel, frozen=True): + """Readiness probe response. + + ``ready`` is ``False`` (and ``/ready`` answers 503) while no ONNX model is + loaded, which is the expected state of a freshly scaffolded project. + """ + + model_config = ConfigDict(frozen=True, protected_namespaces=()) + + ready: bool + model_name: str + model_path: str + providers: list[str] = Field(default_factory=list) + + class PredictionRequest(BaseModel, frozen=True): """Prediction request.""" diff --git a/archetypes/cv-inference-service/template/tests/test_app.py b/archetypes/cv-inference-service/template/tests/test_app.py index 7683309..92c39f6 100644 --- a/archetypes/cv-inference-service/template/tests/test_app.py +++ b/archetypes/cv-inference-service/template/tests/test_app.py @@ -1,22 +1,141 @@ -"""Tests for the inference service.""" +"""Tests for the inference service. + +The whole suite runs **without any .onnx artifact present** — a freshly +scaffolded project must be green before a model exists. +""" from __future__ import annotations +import base64 +import io +from collections.abc import AsyncGenerator +from pathlib import Path + +import numpy as np import pytest from httpx import ASGITransport, AsyncClient +from PIL import Image from ${package_name}.app import app +from ${package_name}.inference.engine import ( + ImageDecodeError, + InferenceConfig, + OnnxInferenceEngine, + decode_base64_image, +) + + +def _png_bytes(width: int = 32, height: int = 24) -> bytes: + buffer = io.BytesIO() + Image.new("RGB", (width, height), color=(10, 20, 30)).save(buffer, format="PNG") + return buffer.getvalue() @pytest.fixture -async def client() -> AsyncClient: +async def client() -> AsyncGenerator[AsyncClient, None]: + """ASGI client that runs the real lifespan, model load included.""" transport = ASGITransport(app=app) - async with AsyncClient(transport=transport, base_url="http://test") as ac: + async with ( + app.router.lifespan_context(app), + AsyncClient(transport=transport, base_url="http://test") as ac, + ): yield ac -@pytest.mark.anyio -async def test_health(client: AsyncClient) -> None: +@pytest.fixture +def engine(tmp_path: Path) -> OnnxInferenceEngine: + return OnnxInferenceEngine( + InferenceConfig(model_path=tmp_path / "missing.onnx", input_width=8, input_height=8) + ) + + +# --------------------------------------------------------------------- routes + + +async def test_health_ok_without_model(client: AsyncClient) -> None: + """The app must start and stay live when no model file exists.""" response = await client.get("/health") assert response.status_code == 200 assert response.json()["status"] == "healthy" + + +async def test_ready_reports_not_ready_without_model(client: AsyncClient) -> None: + response = await client.get("/ready") + assert response.status_code == 503 + assert response.json()["ready"] is False + + +async def test_predict_returns_503_without_model(client: AsyncClient) -> None: + payload = {"image_b64": base64.b64encode(_png_bytes()).decode(), "confidence_threshold": 0.5} + response = await client.post("/predict", json=payload) + assert response.status_code == 503 + + +async def test_predict_upload_returns_503_without_model(client: AsyncClient) -> None: + files = {"file": ("image.png", _png_bytes(), "image/png")} + response = await client.post("/predict/upload", files=files) + assert response.status_code == 503 + + +async def test_predict_rejects_invalid_threshold(client: AsyncClient) -> None: + payload = {"image_b64": "aGk=", "confidence_threshold": 5.0} + response = await client.post("/predict", json=payload) + assert response.status_code == 422 + + +async def test_openapi_exposes_predict(client: AsyncClient) -> None: + schema = (await client.get("/openapi.json")).json() + assert "/predict" in schema["paths"] + assert "PredictionResponse" in schema["components"]["schemas"] + + +# --------------------------------------------------------------------- engine + + +def test_load_missing_model_is_not_fatal(engine: OnnxInferenceEngine) -> None: + assert engine.load() is False + assert engine.is_ready is False + assert engine.providers == [] + + +def test_predict_without_model_raises(engine: OnnxInferenceEngine) -> None: + with pytest.raises(RuntimeError): + engine.predict(Image.new("RGB", (8, 8))) + + +def test_resolve_providers_defaults_to_cpu(engine: OnnxInferenceEngine) -> None: + assert engine.resolve_providers() == ["CPUExecutionProvider"] + + +def test_preprocess_shape_and_dtype(engine: OnnxInferenceEngine) -> None: + batch = engine.preprocess(Image.new("RGB", (64, 48), color=(255, 0, 0))) + assert batch.shape == (1, 3, 8, 8) + assert batch.dtype == np.float32 + + +def test_postprocess_decodes_detection_rows(engine: OnnxInferenceEngine) -> None: + raw = np.array([[1.0, 2.0, 3.0, 4.0, 0.9, 0.0], [1.0, 2.0, 3.0, 4.0, 0.1, 1.0]], np.float32) + detections = engine.postprocess([raw], threshold=0.5) + assert len(detections) == 1 + assert detections[0].label == "class_0" + assert detections[0].bbox == [1.0, 2.0, 3.0, 4.0] + + +def test_postprocess_decodes_class_logits(engine: OnnxInferenceEngine) -> None: + detections = engine.postprocess([np.array([0.1, 9.0, 0.2], np.float32)], threshold=0.5) + assert len(detections) == 1 + assert detections[0].label == "class_1" + + +def test_postprocess_unknown_shape_is_empty(engine: OnnxInferenceEngine) -> None: + assert engine.postprocess([np.zeros((2, 3, 4, 5), np.float32)], threshold=0.5) == [] + + +def test_decode_base64_image_roundtrip() -> None: + image = decode_base64_image(base64.b64encode(_png_bytes(16, 8)).decode()) + assert image.size == (16, 8) + + +def test_decode_base64_image_rejects_garbage() -> None: + with pytest.raises(ImageDecodeError): + decode_base64_image("not-base64!!") diff --git a/archetypes/data-processing-pipeline/README.md b/archetypes/data-processing-pipeline/README.md index fcaf6de..92daa32 100644 --- a/archetypes/data-processing-pipeline/README.md +++ b/archetypes/data-processing-pipeline/README.md @@ -1,268 +1,233 @@ # Data Processing Pipeline Archetype -A structured project template for building robust ETL (Extract, Transform, Load) workflows for machine learning datasets. This archetype provides a pipeline framework for cleaning, transforming, validating, augmenting, and splitting datasets with full traceability, parallel processing support, and integration with data versioning tools. +A working project template for ETL workflows that turn raw CV/ML data into validated, +leakage-free, versioned training splits. Generated projects run end to end on day one: +`ingest -> validate -> split -> write`, with a content-hashed Parquet manifest, a data-quality +gate, and a group-aware splitter that aborts the run if a group straddles two splits. ## Purpose -Data preparation is the most time-consuming and error-prone phase of any machine learning project. Raw datasets arrive in inconsistent formats, contain corrupt files, have labeling errors, and require extensive transformation before they are suitable for training. Despite this, data processing code is often the least structured part of an ML codebase -- scattered across ad-hoc scripts, undocumented Jupyter cells, and bash one-liners that are impossible to reproduce. +Data preparation is the most time-consuming and error-prone phase of any machine learning +project. Raw datasets arrive in inconsistent formats, contain corrupt files, have labeling +errors, and require extensive transformation before they are suitable for training. Despite +this, data processing code is often the least structured part of an ML codebase -- scattered +across ad-hoc scripts, undocumented Jupyter cells, and bash one-liners that are impossible to +reproduce. -The Data Processing Pipeline archetype solves this by providing a modular, testable, and reproducible framework for dataset preparation. Each processing step is implemented as an isolated stage with defined inputs, outputs, and validation criteria. Stages are composed into pipelines that can be executed end-to-end or incrementally, with each intermediate result cached and validated. The framework supports parallel processing for throughput-intensive operations and integrates with DVC for dataset versioning. +This archetype gives that work a spine: each processing step is an ordinary, testable function +with explicit inputs and outputs; every sample is content-hashed; every configuration value is +validated by Pydantic at load time; and the highest-risk step -- splitting -- is protected by an +assertion that runs inside the pipeline, not only in the test suite. -The core design principle is that every transformation applied to data must be explicit, tested, logged, and reversible. This ensures that when model performance changes, the data lineage can be traced back to identify whether the change originated in the data pipeline or the model. +The core design principle is that every transformation applied to data must be explicit, tested, +logged, and reproducible. When model performance changes, the data lineage traces back to a +dataset fingerprint, so you can tell whether the change came from the data or the model. ## Use Cases -- **Dataset cleaning** -- Remove corrupt images, fix encoding issues, standardize file formats, and handle missing annotations. -- **Annotation format conversion** -- Convert between COCO, VOC, YOLO, and custom annotation formats with validation. -- **Image preprocessing** -- Resize, crop, normalize, and color-correct images to consistent specifications. -- **Data augmentation** -- Generate augmented training samples with controlled augmentation policies and deduplication. -- **Train/val/test splitting** -- Create reproducible dataset splits with stratification, group awareness, and cross-validation fold generation. -- **Dataset merging** -- Combine multiple source datasets with label harmonization and conflict resolution. -- **Quality assurance** -- Run automated checks for class balance, image quality, annotation consistency, and data leakage between splits. -- **Feature extraction** -- Pre-compute embeddings, feature maps, or derived features and store them for efficient training. +- **Train/val/test splitting** -- Reproducible splits that keep every frame of a clip, every + image of a patient, and every play of a match inside a single split. +- **Dataset manifesting** -- Turn a directory tree into a validated, content-hashed Parquet + manifest with a stable fingerprint per dataset version. +- **Quality assurance** -- Automated checks for empty datasets, corrupt/undersized files, + duplicate content, thin labels, mixed-label groups, and post-split label coverage. +- **Dataset versioning** -- Bulk data stays in object storage; a small `dataset.json` card with + the fingerprint, counts, and split sizes is what gets committed. +- **Dataset cleaning** -- Extend the ingest stage with format normalization and corrupt-file + removal; the quality gate already fails the run loudly. +- **Annotation format conversion** -- Swap the reader in `stage_ingest`; everything downstream is + format-agnostic because it only sees `ImageRecord`s. ## Directory Structure +The generated project (`whet init data-processing-pipeline`): + ``` -{{project_slug}}/ -├── .github/ -│ └── workflows/ -│ ├── test.yml # Pipeline test suite -│ └── code-review.yml # Automated code review +${project_slug}/ ├── .gitignore -├── .pre-commit-config.yaml -├── pixi.toml -├── pyproject.toml ├── README.md -├── dvc.yaml # DVC pipeline definition -├── dvc.lock # DVC pipeline state -├── params.yaml # Pipeline parameters +├── pixi.toml # environment + tasks +├── pyproject.toml # packaging, ruff, mypy, pytest config ├── conf/ -│ ├── pipeline.yaml # Pipeline stage configuration -│ ├── cleaning/ -│ │ └── default.yaml -│ ├── splitting/ -│ │ └── default.yaml -│ └── augmentation/ -│ └── default.yaml -├── src/{{package_name}}/ -│ ├── __init__.py +│ └── pipeline.toml # validated pipeline configuration +├── src/${package_name}/ +│ ├── __init__.py # public API re-exports │ ├── py.typed -│ ├── config.py # Pydantic pipeline config -│ ├── pipeline.py # Pipeline orchestrator -│ ├── stages/ -│ │ ├── __init__.py -│ │ ├── base.py # Abstract stage interface -│ │ ├── ingest.py # Data ingestion stage -│ │ ├── validate.py # Data validation stage -│ │ ├── clean.py # Data cleaning stage -│ │ ├── transform.py # Transformation stage -│ │ ├── augment.py # Augmentation stage -│ │ ├── split.py # Train/val/test splitting -│ │ └── export.py # Output format export -│ ├── validators/ -│ │ ├── __init__.py -│ │ ├── image.py # Image integrity checks -│ │ ├── annotation.py # Annotation consistency checks -│ │ └── schema.py # Data schema validation -│ ├── io/ -│ │ ├── __init__.py -│ │ ├── readers.py # Format-specific readers -│ │ ├── writers.py # Format-specific writers -│ │ └── formats.py # Format definitions -│ └── utils/ -│ ├── __init__.py -│ ├── parallel.py # Multiprocessing utilities -│ ├── hashing.py # Content hashing for dedup -│ └── progress.py # Progress bar helpers -├── data/ -│ ├── raw/ # Original source data -│ │ └── .gitkeep -│ ├── interim/ # Intermediate stage outputs -│ │ └── .gitkeep -│ ├── processed/ # Final processed dataset -│ │ └── .gitkeep -│ └── external/ # Third-party reference data -│ └── .gitkeep -├── scripts/ -│ ├── run_pipeline.py # Full pipeline execution -│ ├── run_stage.py # Single stage execution -│ └── validate_dataset.py # Standalone validation -├── reports/ -│ ├── .gitkeep -│ └── templates/ -│ └── data_report.html # Dataset report template -├── tests/ -│ ├── __init__.py -│ ├── conftest.py -│ ├── test_stages.py -│ ├── test_validators.py -│ ├── test_io.py -│ └── fixtures/ -│ └── sample_data/ -│ └── .gitkeep -└── notebooks/ - └── data_exploration.ipynb # Interactive data inspection +│ ├── config.py # frozen Pydantic V2 config models +│ ├── manifest.py # ImageRecord, hashing, Parquet manifest, dataset card +│ ├── splitting.py # group-aware, leakage-free splitting +│ ├── quality.py # data-quality gate (ERROR aborts the run) +│ ├── stages.py # ingest / validate / split / write +│ ├── pipeline.py # orchestration + PipelineResult +│ ├── sample_data.py # stdlib-only synthetic dataset generator +│ ├── cli.py # argparse entry point +│ └── __main__.py # python -m ${package_name} +└── tests/ + ├── __init__.py + ├── conftest.py # synthetic manifests + synthetic raw tree + ├── test_splitting.py # split disjointness, determinism, leak detection + ├── test_manifest.py # schema, hashing, Parquet round-trip + ├── test_quality.py # every quality check + └── test_pipeline.py # end-to-end pipeline and CLI ``` +`data/` is created on demand at runtime and is Git-ignored, except +`data/processed/dataset.json`. + ## Key Features -- **Modular stage architecture** where each processing step is an isolated, testable unit with defined inputs, outputs, and validation criteria. -- **Pipeline orchestration** that composes stages into reproducible end-to-end workflows with dependency resolution and caching. -- **Pydantic validation** at every stage boundary to catch data quality issues early and provide clear error messages. -- **Parallel processing** with configurable worker pools for CPU-bound operations like image resizing and augmentation. -- **Progress tracking** with rich progress bars showing per-stage and overall pipeline completion. -- **Content hashing** for deduplication and change detection, enabling incremental processing of modified files only. -- **DVC integration** for versioning large datasets and reproducing pipeline runs with exact data lineage. -- **HTML reports** generated after each pipeline run summarizing dataset statistics, quality metrics, and processing logs. +- **Group-aware splitting** -- splits on the grouping key (clip, patient, match, session), never + on the row, and calls `assert_no_group_leakage` inside the pipeline so a leak aborts the run. +- **Deterministic without an RNG** -- group order comes from a seeded SHA-256 ranking, so splits + reproduce across machines, Python versions, and library upgrades. +- **Content hashing** -- SHA-256 per sample for deduplication and corruption detection, plus an + order-independent dataset fingerprint. +- **Pydantic V2 validation** -- frozen config models; ratios that do not sum to 1.0 fail at load + time, not three hours in. +- **Quality gate with severities** -- `ERROR` aborts, `WARNING` logs; checks run after ingest + *and* after splitting, because a healthy manifest still yields bad splits with bad ratios. +- **Parquet everywhere** -- zstd-compressed manifest and per-split manifests; never CSV. +- **Loguru structured logging** -- realized per-split record/group/label counts at every stage + boundary, so skewed ratios are visible immediately. +- **Runnable on day one** -- a stdlib-only generator writes a synthetic clip-structured dataset + of real PNGs, so `sample -> run` works before you have any data. + +## Pipeline Stages + +Stages are plain functions, not a class hierarchy -- add one by writing a function and calling it +from `run_pipeline`. -## Pipeline Stage Interface +```python +def stage_ingest(config: IngestConfig) -> pl.DataFrame: ... +def stage_validate(manifest: pl.DataFrame, config: QualityConfig) -> QualityReport: ... +def stage_split(manifest: pl.DataFrame, config: SplitConfig) -> dict[str, pl.DataFrame]: ... +def stage_write( + manifest: pl.DataFrame, + splits: Mapping[str, pl.DataFrame], + config: OutputConfig, + quality: QualityConfig, +) -> dict[str, Path]: ... +``` -Every stage implements a common interface defined in `stages/base.py`. +The leakage guard, which runs in the pipeline and not only in tests: ```python -from abc import ABC, abstractmethod -from pydantic import BaseModel - -class StageConfig(BaseModel): - """Configuration for a pipeline stage.""" - enabled: bool = True - num_workers: int = 4 - -class PipelineStage(ABC): - """Abstract base class for all pipeline stages.""" - - @abstractmethod - def run(self, input_path: Path, output_path: Path, config: StageConfig) -> StageResult: - """Execute the stage, reading from input_path and writing to output_path.""" - ... - - @abstractmethod - def validate_input(self, input_path: Path) -> ValidationResult: - """Validate that input data meets this stage's requirements.""" - ... - - @abstractmethod - def validate_output(self, output_path: Path) -> ValidationResult: - """Validate that output data meets expected quality criteria.""" - ... +def assert_no_group_leakage(splits: Mapping[str, pl.DataFrame], group_column: str) -> None: + """Raise DataLeakageError if any group appears in more than one split.""" ``` -## Configuration Variables +## Expected Raw Layout + +``` +data/raw/