Base implementation of Parquet file writing by VeckoTheGecko · Pull Request #2583 · Parcels-code/Parcels

VeckoTheGecko · 2026-04-20T09:48:33Z

Description

This PR introduces Parquet file writing to Parcels.

I still need to work on:

How to work with cftime output in the Parquet (how does this work with our internal model of time in Parcels? How should it work?)
- Scoped out in Consistent time handling #2586 , and now implemented in this PR. Closes Consistent time handling #2586. This PR also includes a function parcels.read_particlefile(...) to streamline ingestion into cftime objects.
Reviewing the test_particlefile.py file - are there tests that are no longer needed? What would be the best testing approach here?
- Will leave for a future PR
Update documentation
- Will leave for a future PR

Posting as draft for initial feedback

Checklist

Contributes to Migrate particlefile output to tabular (via Parquet) #2582
Tests added
This PR targets the correct branch (main for normal development, v3-support for v3 support)

AI Disclosure

This PR contains AI-generated content.
- I have tested any AI-generated content in my PR.
- I take responsibility for any AI-generated content in my PR.
- Describe how you used it (e.g., by pasting your prompt): Just to help with learning the PyArrow API (i.e., used it to create an example script - which I then used as an entry to exploring the docs on pyarrow)

VeckoTheGecko · 2026-04-20T12:17:39Z

@erikvansebille let's table some of these questions for our meeting tomorrow (mainly around datetime serialization in the Parquet file)

Covered by test_write_dtypes_pfile

for more information, see https://pre-commit.ci

This mark was only introduced during refactoring

Remove temporary test_cftime.py file

This function is now independent of the time_interval as time is now stored as float

Remove nested key - save on root instead

VeckoTheGecko · 2026-04-23T12:59:44Z

@erikvansebille I moved read_particlefile to the global scope (i.e.,parcels.read_particlefile()) as well as added a docstring.

VeckoTheGecko · 2026-04-23T13:02:49Z

(we'll likely get doc failures on this PR - which is to be expected since they haven't been updated)

erikvansebille

Very nice work. And as I'm going through updating all the documentation, it seems to work smoothly too!

A few small comments below

erikvansebille · 2026-04-23T16:24:06Z

@@ -187,7 +154,7 @@ def test_variable_written_once():
 @pytest.mark.skip(reason="Pending ParticleFile refactor; see issue #2386")


Does this test work again? If not out of the box, perhaps we should discuss what we actually expect from output of a looped pset.execute...

This test was introduced in 4b28ddb . I'm ok cutting tests that are no longer valuable (/tested specific implementation bugs from the old codebase). I don't really understand this test - do you think its still valuable?

perhaps we should discuss what we actually expect from output of a looped pset.execute...

Agreed. Should we open a tracking issue? Whatever we decide on that front we can add separate tests

Yes, I think it is a valuable test (as it tests what happens in the particle file when pset.execute() is run in a for-loop). But perhaps better for a later PR

And perhaps rename from test_pset_repeated_release_delayed_adding_deleting - since I find that quite confusing

for more information, see https://pre-commit.ci

Parcels-code#2583 (comment)

* Update .gitignore * Disable zarr writing * Fix parquet writing * Remove test_vriable_write_double * Fix all "uses_old_zarr" tests * Remove test_variable_write_double Covered by test_write_dtypes_pfile * Fixing tests * More test fixing * Fix last tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove old fixtures * Fix pre-commit errors * Cleanup This mark was only introduced during refactoring * Add pandas and pyarrow as explicit dependencies * Add assert_cftime_like_particlefile Remove temporary test_cftime.py file * MAINT: Cleanup create_particle_data This function is now independent of the time_interval as time is now stored as float * Add cftime metadata serialization * Add np.timedelta64 support * Fix assert_cftime_like_particlefile Remove nested key - save on root instead * Move imports * Fixing tests * Fix test_time_is_age test * Refactor assert_cftime_like_particlefile * Self-review feedback * Fix test_particle_schema * Make read_particlefile public * Add docstring to read_particlefile * Updarting Argo tutorial to use parquet * Updating tutorial_nemo to use parquet output * Update tutorial_diffusion to use parquet * Update tutorial_output to use parquet * Review feedback * Update migration guide * Remove obs_written * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update migration guide * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert from extra_metadata to metadata * Fix test_pfile_array_remove_particles * Fix numpy warning #2583 (comment) * Using polars in tutorial_output * Update explanation_concepts.md * update tutorial_croco to use parquet * using polars for read_particlefile * Update tutorial_quickstart to use parquet * Update tutorial_output to use polars * Update v4-migration.md * Fix using polars in tutorial_output * Fixing read_parquet to use polars * Update tutorial_delaystart to use parquet * Update tutorial_dt_integrators to use parquet * Fixing parcels.read_particlefile for timedelta time * Update tutorial_interaction to use parquet * Update tutorial_manipulating_field_data to use parquet * Update tutorial_mitgcm to use parquet * Updsate tutorial_nestedgrids to use parquet * Update tutorial_sampling to use parquet (and remove to_write="once" section) * Removing old attributes from particlefile.repr * Using more intuitive variable names for polars subsetting * Fixing repr of particleset * Update tutorial_Argofloats.ipynb * Update tutorial_quickstart to use parquet * Update tutorial_croco_3D.ipynb * Using polars in tutorial_diffusion * Use polars in tutorial_nemo * Use parquet in explanation_kernelloop * Update policies.md * Fixing unit tests to use polars in parcels.read_particlefile * Doc fix: docs/user_guide/examples/tutorial_dt_integrators.ipynb * Fixing plotting in tutorial_nestedgrids * Fixing colour range of nemo 3D plot Now that each trajectory is plotted by itself, colour range has to be set for all * Using start location for last plot in quickstart * Fixing assert statement in tutorial_quickstart * Adding comment on cftime * Update tutorial_quickstart.md * Update docs/getting_started/tutorial_quickstart.md Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com> --------- Co-authored-by: Vecko <36369090+VeckoTheGecko@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

github-project-automation Bot added this to Parcels development Apr 20, 2026

github-project-automation Bot moved this to Backlog in Parcels development Apr 20, 2026

VeckoTheGecko changed the title ~~Add parquet file writing~~ Base implementation of Parquet file writing Apr 20, 2026

VeckoTheGecko force-pushed the push-zlxyoyvlpoqm branch from 1d26f4a to 91ce496 Compare April 20, 2026 09:54

VeckoTheGecko and others added 12 commits April 20, 2026 14:41

Update .gitignore

ffe0ebf

Disable zarr writing

9ebb653

Fix parquet writing

0218c28

Remove test_vriable_write_double

4e7de3e

Fix all "uses_old_zarr" tests

bc653f1

Remove test_variable_write_double

5daec3b

Covered by test_write_dtypes_pfile

Fixing tests

2a07ced

More test fixing

b8c5477

Fix last tests

2d438c5

[pre-commit.ci] auto fixes from pre-commit.com hooks

32a82fa

for more information, see https://pre-commit.ci

Remove old fixtures

19fbd8d

Fix pre-commit errors

e74672f

VeckoTheGecko mentioned this pull request Apr 21, 2026

Consistent time handling #2586

Closed

VeckoTheGecko force-pushed the push-zlxyoyvlpoqm branch from e06a618 to 60ceef6 Compare April 22, 2026 14:23

VeckoTheGecko added 11 commits April 23, 2026 13:11

Cleanup

db9f983

This mark was only introduced during refactoring

Add pandas and pyarrow as explicit dependencies

ac2a830

Add assert_cftime_like_particlefile

de464e5

Remove temporary test_cftime.py file

MAINT: Cleanup create_particle_data

57ccf6f

This function is now independent of the time_interval as time is now stored as float

Add cftime metadata serialization

b2bde50

Add np.timedelta64 support

55493a9

Fix assert_cftime_like_particlefile

bab4d5d

Remove nested key - save on root instead

Move imports

b28665c

Fixing tests

7184e1f

Fix test_time_is_age test

e7e37ef

Refactor assert_cftime_like_particlefile

54c829a

Fix test_particle_schema

3693329

VeckoTheGecko marked this pull request as ready for review April 23, 2026 12:48

VeckoTheGecko added 2 commits April 23, 2026 14:52

Make read_particlefile public

81f127b

Add docstring to read_particlefile

9fcb5bf

VeckoTheGecko requested a review from erikvansebille April 23, 2026 12:58

erikvansebille reviewed Apr 23, 2026

View reviewed changes

erikvansebille reviewed Apr 24, 2026

View reviewed changes

Comment thread src/parcels/_core/particle.py

VeckoTheGecko mentioned this pull request Apr 24, 2026

Rename trajectory variable in particle file output #2592

Closed

VeckoTheGecko added 2 commits April 24, 2026 11:12

Review feedback

04d8676

Remove obs_written

b4e2214

VeckoTheGecko mentioned this pull request Apr 24, 2026

ParticleFile writing behaviour when there's an existing file #2593

Open

VeckoTheGecko force-pushed the push-zlxyoyvlpoqm branch from 61430c9 to 6695187 Compare April 24, 2026 09:15

erikvansebille reviewed Apr 24, 2026

View reviewed changes

Comment thread src/parcels/_core/particlefile.py

VeckoTheGecko and others added 4 commits April 24, 2026 13:06

Update migration guide

663f80c

[pre-commit.ci] auto fixes from pre-commit.com hooks

5be22aa

for more information, see https://pre-commit.ci

Revert from extra_metadata to metadata

002b8f2

Fix test_pfile_array_remove_particles

3c52647

VeckoTheGecko force-pushed the push-zlxyoyvlpoqm branch from a5bdd31 to 3c52647 Compare April 24, 2026 11:27

VeckoTheGecko added 2 commits April 24, 2026 13:39

Merge branch 'main' into push-zlxyoyvlpoqm

39ab122

Fix numpy warning

ae375b3

Parcels-code#2583 (comment)

VeckoTheGecko requested a review from erikvansebille April 24, 2026 11:47

erikvansebille approved these changes Apr 24, 2026

View reviewed changes

VeckoTheGecko merged commit 439ea29 into Parcels-code:main Apr 24, 2026
14 of 15 checks passed

github-project-automation Bot moved this from Backlog to Done in Parcels development Apr 24, 2026

VeckoTheGecko deleted the push-zlxyoyvlpoqm branch April 24, 2026 13:06

erikvansebille mentioned this pull request Apr 28, 2026

Explore and expose use of writing particles in a Kernel #2599

Open

		@@ -187,7 +154,7 @@ def test_variable_written_once():
		@pytest.mark.skip(reason="Pending ParticleFile refactor; see issue #2386")

Conversation

VeckoTheGecko commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

AI Disclosure

Uh oh!

VeckoTheGecko commented Apr 20, 2026

Uh oh!

VeckoTheGecko commented Apr 23, 2026

Uh oh!

VeckoTheGecko commented Apr 23, 2026

Uh oh!

erikvansebille left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erikvansebille Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

VeckoTheGecko Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

erikvansebille Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

VeckoTheGecko Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

VeckoTheGecko commented Apr 20, 2026 •

edited

Loading