953 stochastic noise by maxwhitemet · Pull Request #2275 · metoppv/improver

maxwhitemet · 2026-01-08T12:09:22Z

Addresses #953.

This PR adds the plugin, CLI, and tests for stochastic noise generation.

The acceptance tests show the impact, with data available here.

Testing:

Ran tests and they passed OK
Added new tests for the new feature(s)

cpelley · 2026-01-12T08:44:28Z

@maxwhitemet, could you merge/rebase master into your branch. The failing "CI Tests / Test-Coverage (pull_request)" should hopefully be addressed by the now merged #2273

There does appear to be a new issue potentially with the improver documentation building now though... (different issue)

cpelley · 2026-01-13T11:41:26Z

Ignore the test coverage failure. Fixed in #2282

gavinevans

Thanks @maxwhitemet 👍

I've added some comments below.

maxwhitemet

Thank you for the review @gavinevans. I have now made the requested changes.

gavinevans

I've added some minor comments.

maxwhitemet

Thanks @gavinevans. I have implemented your feedback

bayliffe

Thanks Max, a few comments. The thrust of some of my comments it to make this less precipitation specific, but I've not highlighted everywhere this needs to happen, so think about variable names (e.g. dry_mask) and comments to make it less precip specific, using precip as an example if that's instructive.

I'm also interested in you making changes to better handle the expected shape of cubes and the coordinate order, rather than assuming realization is first and simply indexing it with that assumption.

bayliffe · 2026-02-17T10:45:28Z

+        n_realiz = template.coord("realization").points.size
+        tasks = []
+        for k in range(n_realiz):
+            realiz_data = template_dB.data[k].astype(np.float32)


There is an implicit assumption that the first index of the cube here corresponds to realization but I don't think that's been written anywhere, checked, or enforeced.

I have now changed the looping to use the iris slices_over method with the realization dimension.

One more question on this. Does the slice passed to do_fft need to be 2-dimensional? Would a time, realization, y, x cube work with this code?

If it does need to be 2D then you could either set a limitation in the code that raises an error if there are more than realization, y, x coordinates, or you could create y, x slices requiring that realization be one of the remaining dimensions, and reconstruct a cube back to the original shape at the end. We do this in a few places and Gavin could give you a steer as to whether this will ever be needed or just to go for the exception (again, assuming it has to be a 2D slice passed in here).

Yes, for our usage, the input to do_fft needs to be 2-dimensional. I don't think we have to support the presence of a time coordinate, so perhaps just checking that the input cube has the expected dimension coordinates, and if not raising an exception, would be fine.

bayliffe · 2026-02-17T11:42:18Z

I forgot to say that this code is currently within the precipitation directory. If you do make it less precip-specific then it should probably move.

maxwhitemet

Thank you for the review @bayliffe. I believe I have addressed all requested changes.

Regarding your comment on where the plugin sits: stochastic_noise and its associated tests now sit under the calibration directory.

maxwhitemet · 2026-02-26T17:18:00Z

+        n_realiz = template.coord("realization").points.size
+        tasks = []
+        for k in range(n_realiz):
+            realiz_data = template_dB.data[k].astype(np.float32)


I have now changed the looping to use the iris slices_over method with the realization dimension.

bayliffe

One more question, but this looks good. Thanks for all the changes.

maxwhitemet · 2026-03-10T17:47:36Z

One more question, but this looks good. Thanks for all the changes.

Thank you for your review @bayliffe. I have added a broad utility to handle the dimensionality concern here: #2327. As this has created a merge dependency, I have set the ticket as blocked for now.

cpelley

Please remember to expose the plugin via improver/api/init.py

bayliffe

Stuff to fix, but basically there.

bayliffe · 2026-03-19T11:33:23Z

+  - pysteps
+>>>>>>> 897e0cb9 (Add pytest to environments)
+=======
+>>>>>>> 08001d9b (Undo environemnt changes)


You've left unresolved conflicts from merging in these files which is causing the tests to fail.

bayliffe · 2026-03-19T11:33:39Z

+# estimate-emos-coefficients, nowcast-accumulate, nowcast-extrapolate, 
+# nowcast-optical-flow-from-winds, and stochastic-noise.
+# See doc/source/Dependencies.rst for more information.
+>>>>>>> 08001d9b (Undo environemnt changes)


Further unresolved merging issues.

maxwhitemet · 2026-03-19T12:14:09Z

I have pulled in the latest changes from upstream here. I am however not sure if this looks correct

Nope. You can look at the current upstream master to see what's there:

https://github.com/metoppv/improver/blob/master/envs/environment_b.yml

Unless you intended to delete the temporal-interpolate and forecast-trajectory-gap-filler entries then this is incorrect.

…her PR

maxwhitemet

Thanks @bayliffe. I fixed the problems highlighted.

Requested change made.

* master: Add attribute to record secondary forecast sources when clustering realizations (#2332) 953 stochastic noise (#2275) Add configurable grid spacing relative tolerance (#2308)

…lustering * upstream/master: Add attribute to record secondary forecast sources when clustering realizations (metoppv#2332) 953 stochastic noise (metoppv#2275) Add configurable grid spacing relative tolerance (metoppv#2308) Cube dim validation utility (metoppv#2327)

maxwhitemet mentioned this pull request Jan 8, 2026

Add acceptance test data for stochastic noise metoppv/improver_test_data#119

Merged

maxwhitemet force-pushed the 953_stochastic_noise branch from 84d3c1b to 897e0cb Compare January 12, 2026 09:56

gavinevans requested changes Jan 16, 2026

View reviewed changes

maxwhitemet commented Jan 21, 2026

View reviewed changes

gavinevans requested changes Jan 27, 2026

View reviewed changes

maxwhitemet commented Jan 28, 2026

View reviewed changes

Comment thread improver/precipitation/stochastic_noise.py Outdated

Comment thread improver/calibration/stochastic_noise.py

Comment thread improver/precipitation/stochastic_noise.py Outdated

Comment thread improver_tests/calibration/stochastic_noise/test_StochasticNoise.py

gavinevans previously approved these changes Jan 28, 2026

View reviewed changes

bayliffe requested changes Feb 17, 2026

View reviewed changes

maxwhitemet dismissed gavinevans’s stale review via 70dd999 February 26, 2026 17:52

maxwhitemet commented Feb 26, 2026

View reviewed changes

bayliffe reviewed Mar 9, 2026

View reviewed changes

maxwhitemet force-pushed the 953_stochastic_noise branch from e911f7c to 2549cef Compare March 10, 2026 17:43

cpelley previously requested changes Mar 12, 2026

View reviewed changes

maxwhitemet force-pushed the 953_stochastic_noise branch from 24cf6c2 to 3d041d1 Compare March 12, 2026 16:16

bayliffe requested changes Mar 19, 2026

View reviewed changes

maxwhitemet added 11 commits March 19, 2026 12:08

Initial creation of plugin

9b09c86

Remove redundant unit conversion

f8a95ba

Add stochastic noise finalisations, including tests

2a4b2cc

Fix reStructuredText issue with Sphinx

c07bacc

Add pysteps to Sphinx's mock imports

46246ed

Integrate changes to environment

91c87a1

Undo environemnt changes

bd07d69

Implement review feedback

703de53

Add pytest importskip for pysteps

0a81f38

Implement review feedback

06e349c

Resolve ModuleNotFoundError

1704770

maxwhitemet added 5 commits March 19, 2026 12:08

Address reviewer feedback

28f846d

Move stochastic noise plugin

ced691c

Use validate_cube_dimensions function

60ccb1c

expose plugin

205895a

Fix merge error from environment_b

0f9b432

maxwhitemet force-pushed the 953_stochastic_noise branch from bd73a41 to 0f9b432 Compare March 19, 2026 12:09

Fix conda-forge failures

7261a5a

maxwhitemet commented Mar 19, 2026

View reviewed changes

maxwhitemet added 2 commits March 19, 2026 12:48

Fix test failure following modification of used functionality in anot…

7060bbe

…her PR

Move import to top of file

4d7a31d

maxwhitemet commented Mar 19, 2026

View reviewed changes

Comment thread improver/calibration/stochastic_noise.py Outdated

maxwhitemet added 2 commits March 19, 2026 15:34

Align environment_b.yml with upstream/master equivalent

133511d

Align environment_b.yml with upstream/master equivalent

3a99f77

bayliffe approved these changes Mar 19, 2026

View reviewed changes

bayliffe assigned maxwhitemet Mar 19, 2026

maxwhitemet merged commit 195d349 into metoppv:master Mar 19, 2026
7 checks passed

Conversation

maxwhitemet commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpelley commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpelley commented Jan 13, 2026

Uh oh!

gavinevans left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gavinevans left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bayliffe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bayliffe commented Feb 17, 2026

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxwhitemet commented Jan 8, 2026 •

edited

Loading

cpelley commented Jan 12, 2026 •

edited

Loading