From c2917e6d2fe794668517c51e3bede204dce25b8c Mon Sep 17 00:00:00 2001 From: Daniele Nerini Date: Tue, 17 Mar 2026 11:35:05 +0100 Subject: [PATCH 1/2] Add AGENTS guide for evalml repo --- AGENTS.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..c237b68 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,41 @@ +# AGENTS.md + +**Purpose** +EvalML runs evaluation pipelines for data-driven weather models (Anemoi). The CLI `evalml` orchestrates Snakemake workflows in `workflow/` using YAML experiment configs. + +**Repo Layout** +- `src/` Python packages (`evalml`, `verification`, `data_input`). +- `workflow/` Snakemake pipeline (`Snakefile`, `rules/`, `scripts/`, `envs/`, `tools/`). +- `config/` Example experiment configs. +- `tests/` Unit and integration tests. +- `output/` Default workflow output location (often a symlink to scratch). + +**Setup** +- Install `uv`: `curl -LsSf https://astral.sh/uv/install.sh | sh`. +- Install dependencies (including dev tools like pre-commit) with `uv sync --dev`. +- Activate the venv with `source .venv/bin/activate`. +- Some experiments require credentials; coordinate with maintainers to obtain access. + +**Common Commands** +- Run an experiment: `evalml experiment path/to/config.yaml --report`. +- Validate configs against schema: use `workflow/tools/config.schema.json` in your YAML editor. +- EvalML is a thin wrapper over Snakemake; pass Snakemake options after `--` (e.g. `evalml experiment config.yaml -- --dry-run -j 1`). + +**Testing** +- Run unit tests: `pytest tests/unit`. +- Run integration tests: `pytest tests/integration`. +- Skip long tests: `pytest -m "not longtest"`. +- For full workflow tests, use a minimal config to keep runs fast: +- Copy a sample config from `config/` to a new file (e.g. `config/minimal-test.yaml`). +- Reduce `dates` to a few reference times (e.g. 1–2 days). +- Reduce `runs` to 1–2 models. +- Reduce the forecast range/steps to a few lead times. +- Run the workflow with that minimal config. + +**Formatting and QA** +- If editing Snakemake files, run `snakefmt workflow`. +- If pre-commit hooks are installed, run `pre-commit run --all-files` before large changes. + +**Data and Outputs** +- Workflow outputs default to `output/`. Avoid committing generated data. +- Prefer using a scratch-backed symlink for `output/` when running large jobs. From ba4e539ebde73f4000e5dd2986072d3b4e90c4c5 Mon Sep 17 00:00:00 2001 From: Louis-Frey Date: Tue, 17 Mar 2026 11:53:05 +0100 Subject: [PATCH 2/2] Consolidate CLAUDE.md into AGENTS.md Merged content from the newly created CLAUDE.md into AGENTS.md, adding feature overview, detailed repo layout, configuration field descriptions, and pre-commit setup step. Removed CLAUDE.md. Co-Authored-By: Claude Sonnet 4.6 --- AGENTS.md | 60 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 38 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c237b68..80776f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,40 +1,56 @@ # AGENTS.md **Purpose** -EvalML runs evaluation pipelines for data-driven weather models (Anemoi). The CLI `evalml` orchestrates Snakemake workflows in `workflow/` using YAML experiment configs. +EvalML runs evaluation pipelines for data-driven weather models (Anemoi). Features: +- **Experiments**: compare model performance via standard and diagnostic verification +- **Showcasing**: produce visual material for specific weather events +- **Sandboxing**: generate isolated inference development environments + +The CLI `evalml` orchestrates Snakemake workflows in `workflow/` using YAML experiment configs. **Repo Layout** -- `src/` Python packages (`evalml`, `verification`, `data_input`). -- `workflow/` Snakemake pipeline (`Snakefile`, `rules/`, `scripts/`, `envs/`, `tools/`). -- `config/` Example experiment configs. -- `tests/` Unit and integration tests. -- `output/` Default workflow output location (often a symlink to scratch). +- `src/evalml/` — CLI (`cli.py`), config models (`config.py`), helpers +- `src/verification/` — metrics and verification logic (`spatial.py`) +- `src/data_input/` — data loading and ingestion +- `src/plotting/` — visualization and colormap handling +- `workflow/` — Snakemake pipeline (`Snakefile`, `rules/`, `scripts/`, `envs/`, `tools/`) +- `config/` — example experiment configs +- `tests/` — unit and integration tests +- `output/` — default workflow output location (often a symlink to scratch) **Setup** -- Install `uv`: `curl -LsSf https://astral.sh/uv/install.sh | sh`. -- Install dependencies (including dev tools like pre-commit) with `uv sync --dev`. -- Activate the venv with `source .venv/bin/activate`. +- Install `uv`: `curl -LsSf https://astral.sh/uv/install.sh | sh` +- Install dependencies (including dev tools): `uv sync --dev` +- Activate the venv: `source .venv/bin/activate` +- Install pre-commit hooks: `pre-commit install` - Some experiments require credentials; coordinate with maintainers to obtain access. **Common Commands** -- Run an experiment: `evalml experiment path/to/config.yaml --report`. -- Validate configs against schema: use `workflow/tools/config.schema.json` in your YAML editor. -- EvalML is a thin wrapper over Snakemake; pass Snakemake options after `--` (e.g. `evalml experiment config.yaml -- --dry-run -j 1`). +- Run an experiment: `evalml experiment path/to/config.yaml --report` +- Validate configs against schema: use `workflow/tools/config.schema.json` in your YAML editor +- EvalML is a thin wrapper over Snakemake; pass Snakemake options after `--` (e.g. `evalml experiment config.yaml -- --dry-run -j 1`) + +**Configuration** +Experiment YAML files are validated by Pydantic. Key fields: +- `dates` — date range or explicit list of reference times +- `runs` — ML model runs referenced by MLflow ID +- `baselines` — reference forecasts for comparison +- `truth` — ground truth dataset +- `locations` — output paths and MLflow URIs +- `profile` — executor config (e.g. SLURM) **Testing** -- Run unit tests: `pytest tests/unit`. -- Run integration tests: `pytest tests/integration`. -- Skip long tests: `pytest -m "not longtest"`. +- Run unit tests: `pytest tests/unit` +- Run integration tests: `pytest tests/integration` +- Skip long tests: `pytest -m "not longtest"` - For full workflow tests, use a minimal config to keep runs fast: -- Copy a sample config from `config/` to a new file (e.g. `config/minimal-test.yaml`). -- Reduce `dates` to a few reference times (e.g. 1–2 days). -- Reduce `runs` to 1–2 models. -- Reduce the forecast range/steps to a few lead times. -- Run the workflow with that minimal config. + - Copy a sample config from `config/` (e.g. `config/minimal-test.yaml`) + - Reduce `dates` to 1–2 reference times, `runs` to 1–2 models, and steps to a few lead times + - Run the workflow with that minimal config **Formatting and QA** -- If editing Snakemake files, run `snakefmt workflow`. -- If pre-commit hooks are installed, run `pre-commit run --all-files` before large changes. +- If editing Snakemake files, run `snakefmt workflow` +- Run `pre-commit run --all-files` before large changes (checks ruff, snakefmt, schema validation) **Data and Outputs** - Workflow outputs default to `output/`. Avoid committing generated data.