Skip to content

API for state saving and restoring (NGWPC-8996)#85

Merged
idtodd merged 135 commits intodevelopmentfrom
philmiller-8996-state-saving-api
Mar 13, 2026
Merged

API for state saving and restoring (NGWPC-8996)#85
idtodd merged 135 commits intodevelopmentfrom
philmiller-8996-state-saving-api

Conversation

@PhilMiller
Copy link

@PhilMiller PhilMiller commented Nov 25, 2025

This adds the structure for save stating and compatibility with hindcasting (saving at the end of a run, then loading that state at the beginning of the next).

Save stating is controlled with the realization config JSON using the state_saving property. The optional property is an array of objects with properties related to how and when state is supposed to be saved or loaded:

{
  "label": text representation of the state, mainly used for logging
  "direction": "save" or "load" based on whether state is being saved out or loaded in
  "path": folder where flat files representing the save state will be written to; this folder will not be created by NGEN
  "type": "FilePerUnit" is the only option right now, representing saving each catchment to its own file
  "when": "EndOfRun" or "StartOfRun" for the two hindcasting cases
}

An example for saving state would be

{
  "label": "hindcasting end state",
  "direction": "save",
  "path": "./ngen_states/20260310",
  "type": "FilePerUnit",
  "when": "EndOfRun"
}

An example of loading that state for the next state would be

{
  "label": "hindcasting reload",
  "direction": "load",
  "path": "./ngen_states/20260310",
  "type": "FilePerUnit",
  "when": "StartOfRun"
}

Additions

  • Classes for controlling how and when state is saved.
  • Class functions on NgenSimulation, Layer, and the BMI formulation classes for controlling saving and loading state.
  • Bmi_Py_Adapter::set_value_unchecked function for sending arbitrarily large data to a python BMI. This is needed to allow sending serialized state data without the BMI interfaces checking that the receiving "variable" is the appropriate size.

Removals

Changes

Testing

Notes

  • Cf NGWPC-8989 for details about more things that may need to be handled here.
  • Because of the unchecked size for sending data to python, python BMIs should be designed to account for incoming data being a different size than what is currently stored.

Todos

  • Bmi_Multi_Formulation
  • Iteration over catchment features
  • x

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

@PhilMiller PhilMiller force-pushed the philmiller-8996-state-saving-api branch from 1e36556 to 6817947 Compare November 26, 2025 01:09
@PhilMiller PhilMiller force-pushed the philmiller-8996-state-saving-api branch from e48e098 to 6817947 Compare December 9, 2025 01:54
@PhilMiller PhilMiller force-pushed the philmiller-8996-state-saving-api branch from ee15910 to d97e4a7 Compare December 30, 2025 23:41
Copy link

@cmaynard-ngwpc cmaynard-ngwpc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested running will all modules but Evapotranspiration (PET), since I don't have the capability to test a formulation with that module. We can go ahead with this merge and test PET once it has been added by Mohammed with the ewts updates.

@idtodd idtodd marked this pull request as ready for review March 13, 2026 15:20
@idtodd idtodd merged commit 844c5f6 into development Mar 13, 2026
3 checks passed
@cmaynard-ngwpc cmaynard-ngwpc deleted the philmiller-8996-state-saving-api branch March 18, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants