Add Multiprocessing to Forecasts#43
Conversation
|
Rebased |
kyle-larkin
left a comment
There was a problem hiding this comment.
The partition_config_basename_prefix wasn't set when I tested, so the output file was _partition_config.json (just the suffix).
How is the prefix set? I may have just missed a step.
|
It was intentional on my part to keep the file name generic (no prefix). If more information should be added to the partition config file I can make that change. |
I'm ok with no prefix, but that makes it a little strange that there is a partition_config_basename_prefix variable, which the logger uses to produce the following:
I think if the variable exists, then it should be either optional, have a default value, or be set each time. |
… instead of empty string
I pushed a change that results in the partition config files being named generically "forecast_partition_config.json" instead of "_partition_config.json", and the log msg now reads like "Creating partition file for forecast" |
In support of adding multiprocessing capability to the forecast workflow, this adjusts the step of building the forecast realization, to call the partition generator and create a new partition configuration file, placed in the forecast realization folder.
To support this, provide a ParallelConfig instance in the InputConfig instance when using RealizationBuilder.
This was tested in conjunction with the following PRs:
Forcing: NGWPC/ngen-forcing#120
Forecast Manager: NGWPC/nwm-fcst-mgr#10
RTE: NGWPC/nwm-rte#42