Skip to content

Improve chunking for Xarray generated with the earthkit engine#700

Merged
sandorkertesz merged 5 commits into
developfrom
feature/fieldlist-serialisation
May 15, 2025
Merged

Improve chunking for Xarray generated with the earthkit engine#700
sandorkertesz merged 5 commits into
developfrom
feature/fieldlist-serialisation

Conversation

@sandorkertesz
Copy link
Copy Markdown
Collaborator

@sandorkertesz sandorkertesz commented May 14, 2025

This PR adds the following changes:

  • improves the serialisation of GRIB fieldlists used when Xarray is generated with chunks
  • a new config option grib-file-serialisation-policy was added to control how GRIB data on disk is pickled. The options are "path" and "memory". The default is "path". Previously, only "memory" was implemented.
  • serialisation was added to GRIB fields (both on disk and in-memory)
  • serialisation was added to the FieldListTensor
  • TensorBackendArray, which implements the lazy loading of DataArrays in the Xarray engine, now uses a dask.utils.SerializableLock for data loading

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 14, 2025

Codecov Report

Attention: Patch coverage is 97.53086% with 2 lines in your changes missing coverage. Please review.

Project coverage is 90.88%. Comparing base (1361f58) to head (610ddaa).
Report is 2 commits behind head on develop.

Files with missing lines Patch % Lines
tests/xr_engine/test_xr_builder.py 97.01% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #700      +/-   ##
===========================================
+ Coverage    90.80%   90.88%   +0.08%     
===========================================
  Files          164      165       +1     
  Lines        12453    12533      +80     
  Branches       609      612       +3     
===========================================
+ Hits         11308    11391      +83     
+ Misses         964      960       -4     
- Partials       181      182       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sandorkertesz sandorkertesz changed the title Improve fieldlist serialisation Improve fieldlist serialisation for Xarray chunking May 15, 2025
@sandorkertesz sandorkertesz changed the title Improve fieldlist serialisation for Xarray chunking Improve chunking for Xarray generated with the earthkit engine May 15, 2025
@sandorkertesz sandorkertesz merged commit 7519b38 into develop May 15, 2025
78 of 79 checks passed
@sandorkertesz sandorkertesz deleted the feature/fieldlist-serialisation branch May 15, 2025 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants