Skip to content

Add zarr target#716

Merged
sandorkertesz merged 10 commits into
developfrom
feature/zarr-target
Jun 10, 2025
Merged

Add zarr target#716
sandorkertesz merged 10 commits into
developfrom
feature/zarr-target

Conversation

@sandorkertesz
Copy link
Copy Markdown
Collaborator

@sandorkertesz sandorkertesz commented May 29, 2025

This PR adds the "zarr" target. It has a limited scope: GRIB data first converted to Xarray then the Xarray.to_zarr() method is used to generate the target Zarr store. E.g.

import earthkit.data as ekd
ds = ekd.from_source("sample", "pl.grib")

# with these options each field will be a separate chunk
ds.to_target("zarr", 
             earthkit_to_xarray_kwargs={"chunks": {"forecast_reference_time": 1, 
                                                   "step": 1, 
                                                   "level": 1}},
             xarray_to_zarr_kwargs={"store": "_pl.zarr", "mode": "w"})


import zarr 
root = zarr.group("_pl.zarr")
root.tree()
/
├── forecast_reference_time (4,) int64
├── latitude (19,) float64
├── level (2,) int64
├── longitude (36,) float64
├── r (4, 2, 2, 19, 36) float64
├── step (2,) int64
└── t (4, 2, 2, 19, 36) float64

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 29, 2025

Codecov Report

❌ Patch coverage is 45.83333% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.70%. Comparing base (387ed5a) to head (9a1e918).
⚠️ Report is 322 commits behind head on develop.

Files with missing lines Patch % Lines
tests/targets/test_target_zarr.py 43.47% 13 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #716      +/-   ##
===========================================
- Coverage    90.77%   90.70%   -0.08%     
===========================================
  Files          169      170       +1     
  Lines        12916    12936      +20     
  Branches       615      616       +1     
===========================================
+ Hits         11724    11733       +9     
- Misses        1003     1015      +12     
+ Partials       189      188       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sandorkertesz sandorkertesz changed the title WIP: add zarr target Add zarr target Jun 10, 2025
@sandorkertesz sandorkertesz marked this pull request as ready for review June 10, 2025 14:56
@sandorkertesz sandorkertesz merged commit 17ed541 into develop Jun 10, 2025
125 of 126 checks passed
@sandorkertesz sandorkertesz deleted the feature/zarr-target branch June 10, 2025 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants