Add ECMWF AIFS deterministic forecast dataset by mrshll · Pull Request #449 · dynamical-org/reformatters

mrshll · 2026-02-25T05:20:01Z

Summary

Add ECMWF AIFS deterministic forecast dataset integration (15-day, 0.25 degree)
Address all review feedback from initial submission

Review feedback addressed

Rename ecmwf/aifs to ecmwf/aifs_deterministic, update dataset_id to ecmwf-aifs-deterministic-forecast-15-day-0-25-degree
Use shared ecmwf_grib_index.py instead of custom byte-range parser; generalize it to support deterministic (non-ensemble) data
Set max_vars_per_download_group=10 and handle multi-band GRIB reads via rasterio metadata matching
Use group_by() from common.iterating for source_groups
Remove skin_temperature_surface variable, rename total_column_water_vapour_atmosphere to precipitable_water_atmosphere
Add replica storage config (EcmwfAifsIcechunkAwsOpenDataDatasetStorageConfig)
Update k8s resources to match GFS (cpu=3.5, memory=7G, shared_memory=1.5G, ephemeral=20G)
Update cron schedule to 25 */6 * * * with 30-minute deadline
Add comprehensive region_job unit tests (source_groups, generate_source_file_coords, download_file, read_data single/multi-band, operational_update_jobs)
Add snapshot value assertions in integration test
Add lat/lon spacing assertions in template_config test
Add cross-dataset consistency exceptions for ECMWF parameter naming
Add dataset integration guide sections to prevent common review issues

Test plan

uv run pytest tests/ecmwf/aifs_deterministic/ — all pass
uv run pytest tests/ecmwf/test_ecmwf_grib_index.py — all pass
uv run pytest tests/common/datasets_cf_compliance_test.py — all pass (excluding pre-existing NASA SMAP failure)
uv run pytest tests/common/common_template_config_subclasses_test.py — all pass (excluding pre-existing NASA SMAP failure)
uv run pytest -m slow tests/ecmwf/aifs_deterministic/ — integration test passes with snapshot values
uv run ruff format && uv run ruff check — clean
uv run ty check — all checks passed

Implements the ECMWF AIFS (Artificial Intelligence Forecasting System) deterministic forecast product as a new dataset with 16 variables, 0.25° global grid, 6-hourly inits, 0-360h lead times. Handles the S3 path change (aifs/ -> aifs-single/) on 2025-02-26 and GRIB master table version differences for precipitation variables between early and current data.

aldenks

lol got nerd sniped into reviewing this. I do think we should do AIFS ENS first but after these comments its basically there

src/reformatters/ecmwf/aifs/forecast/dynamical_dataset.py

src/reformatters/ecmwf/aifs/forecast/region_job.py

src/reformatters/__main__.py

tests/common/datasets_cf_compliance_test.py

tests/ecmwf/aifs/forecast/dynamical_dataset_test.py

tests/ecmwf/aifs/forecast/region_job_test.py

tests/ecmwf/aifs_deterministic/forecast/template_config_test.py

- Rename ecmwf/aifs to ecmwf/aifs_deterministic, update dataset_id - Use shared ecmwf_grib_index.py instead of custom byte-range parser - Support deterministic (non-ensemble) data in shared GRIB index parser - Set max_vars_per_download_group=10, handle multi-band GRIB reads - Use group_by() from common.iterating for source_groups - Remove skin_temperature_surface, rename to precipitable_water_atmosphere - Add replica storage config, update k8s resources to match GFS - Add comprehensive region_job unit tests and snapshot value assertions - Add cross-dataset consistency exceptions for ECMWF naming

Add guidance on: finding a reference dataset before coding, cross-dataset variable naming consistency, running the chunk/shard tool instead of estimating, shared provider utilities, minimum RegionJob test coverage, copying k8s resource values from similar datasets, replica storage setup, snapshot value assertions, and pre-submission cross-dataset checks.

mrshll · 2026-02-26T18:32:40Z

@aldenks robot addressed your suggestions. lmk how you want to proceed? I'm not precious about the PR if you want to redo or start with ENS only. I'd lean towards doing the deterministic one in addition to ENS

aldenks · 2026-02-27T17:14:19Z

tests/common/datasets_cf_compliance_test.py

+    # ECMWF AIFS uses ECMWF parameter names per CLAUDE.md conventions,
+    # while NOAA datasets use NOAA parameter names.
+    (
+        "precipitable_water_atmosphere",
+        "short_name",
+        "ecmwf-aifs-deterministic-forecast-15-day-0-25-degree",
+    ),
+    (
+        "precipitable_water_atmosphere",
+        "long_name",
+        "ecmwf-aifs-deterministic-forecast-15-day-0-25-degree",
+    ),


we do want these consistent across datasets, not excluded from the check

aldenks · 2026-02-27T17:16:01Z

pyproject.toml

+"scripts/**/*.py" = [
+    "INP001",  # not a package
+    "T201",    # print is expected in CLI scripts
+    "S108",    # /tmp usage is fine for staging
+]


don't think we need this addition, this PR isn't modifying scripts

aldenks · 2026-02-27T17:16:21Z

pyproject.toml

+    "boto3>=1.42.49",
    "cf-xarray>=0.10.10",
+    "ecmwf-api-client>=1.6.5",


remove, no need to add permanently

aldenks · 2026-02-27T17:18:13Z

tests/ecmwf/aifs_deterministic/forecast/dynamical_dataset_test.py

+
+    t2m_updated = updated_ds.sel(latitude=0, longitude=0).temperature_2m
+    assert t2m_updated.shape == (2, 2)  # 2 init_times x 2 lead_times
+    assert not np.all(np.isnan(t2m_updated.values))


also snapshot complete temperature and precip values at the test point after the update. we should see the initial values from the backfill unchanged along with new values

aldenks · 2026-02-27T17:18:39Z

tests/ecmwf/aifs_deterministic/forecast/dynamical_dataset_test.py

+    assert float(point_6h["precipitation_rate_surface"]) == pytest.approx(
+        0.00164794921875
+    )


use numpy testing assert_allclose following conventions in other tests

docs/dataset_integration_guide.md

src/reformatters/ecmwf/aifs_deterministic/forecast/template_config.py

aldenks · 2026-02-27T17:42:31Z

src/reformatters/ecmwf/aifs_deterministic/forecast/template_config.py

+        # ~4MB uncompressed, ~0.8MB compressed
+        var_chunks: dict[Dim, int] = {
+            "init_time": 4,  # 1 day of 6-hourly data
+            "lead_time": 61,  # All lead times
+            "latitude": 64,  # 12 chunks over 721 pixels
+            "longitude": 64,  # 23 chunks over 1440 pixels
+        }


this is too small and has too many init times. read the docs/chunk_shard_size_tool.md and use that tool to find a chunk+shard size set that meets the documented targets

make spatial chunks larger

exactly 1 init time per chunk and per shard

If stuck, see GFS forecast template config if you need examples, it has the same resolution and init frequency

aldenks · 2026-02-27T18:03:17Z

@mrshll to your larger question -- i think we should do this before aifs because it's basically there!

…nfig.py Co-authored-by: Alden Keefe Sampson <alden@dynamical.org>

Co-authored-by: Alden Keefe Sampson <alden@dynamical.org>

…nfig.py Co-authored-by: Alden Keefe Sampson <alden@dynamical.org>

- Fix chunk/shard sizes: 1 init_time per chunk/shard, larger spatial chunks (240x240) per chunk_shard_size tool output - Make precipitable_water_atmosphere metadata consistent across datasets (short_name=pwat, long_name=Precipitable water) and remove stale CF compliance exceptions - Restructure dataset_integration_guide.md: remove ECMWF-specific examples, revert chunk/shard docs to simple wording, simplify storage config section, remove pre-submission checks (duplicate of AGENTS.md) - Use np.testing.assert_allclose in integration test and add post-update snapshot values for temperature_2m and precipitation_rate_surface - Remove ecmwf-api-client dev dependency and scripts ruff ignore rules - Regenerate zarr template https://claude.ai/code/session_01C5o4hepUqwYHeEeYk5rqjy

aldenks requested changes Feb 26, 2026

View reviewed changes

mrshll added 2 commits February 26, 2026 12:22

mrshll self-assigned this Feb 26, 2026

mrshll requested a review from aldenks February 26, 2026 18:31

mrshll marked this pull request as ready for review February 26, 2026 18:32

mrshll mentioned this pull request Feb 26, 2026

ECMWF AIFS (deterministic) #448

Open

aldenks requested changes Feb 27, 2026

View reviewed changes

mrshll and others added 4 commits February 27, 2026 12:16

Update src/reformatters/ecmwf/aifs_deterministic/forecast/template_co…

3c94115

…nfig.py Co-authored-by: Alden Keefe Sampson <alden@dynamical.org>

Update docs/dataset_integration_guide.md

7a8cf82

Co-authored-by: Alden Keefe Sampson <alden@dynamical.org>

Update src/reformatters/ecmwf/aifs_deterministic/forecast/template_co…

368683d

…nfig.py Co-authored-by: Alden Keefe Sampson <alden@dynamical.org>

Update src/reformatters/ecmwf/aifs_deterministic/forecast/template_co…

dab3935

…nfig.py Co-authored-by: Alden Keefe Sampson <alden@dynamical.org>

aldenks linked an issue Feb 27, 2026 that may be closed by this pull request

ECMWF AIFS (deterministic) #448

Open

Conversation

mrshll commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review feedback addressed

Test plan

Uh oh!

aldenks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrshll commented Feb 26, 2026

Uh oh!

aldenks Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aldenks Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mrshll commented Feb 25, 2026 •

edited

Loading