Skip to content

Add Multiprocessing to Forecasts#43

Merged
mxkpp merged 3 commits intodevelopmentfrom
maxkipp-fcst-n-2
Mar 27, 2026
Merged

Add Multiprocessing to Forecasts#43
mxkpp merged 3 commits intodevelopmentfrom
maxkipp-fcst-n-2

Conversation

@mxkpp
Copy link
Copy Markdown

@mxkpp mxkpp commented Mar 17, 2026

In support of adding multiprocessing capability to the forecast workflow, this adjusts the step of building the forecast realization, to call the partition generator and create a new partition configuration file, placed in the forecast realization folder.

To support this, provide a ParallelConfig instance in the InputConfig instance when using RealizationBuilder.

This was tested in conjunction with the following PRs:

Forcing: NGWPC/ngen-forcing#120

Forecast Manager: NGWPC/nwm-fcst-mgr#10

RTE: NGWPC/nwm-rte#42

@mxkpp mxkpp force-pushed the maxkipp-fcst-n-2 branch from 876447f to 9af4387 Compare March 26, 2026 14:27
@mxkpp
Copy link
Copy Markdown
Author

mxkpp commented Mar 26, 2026

Rebased

Copy link
Copy Markdown

@kyle-larkin kyle-larkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The partition_config_basename_prefix wasn't set when I tested, so the output file was _partition_config.json (just the suffix).

How is the prefix set? I may have just missed a step.

@mxkpp
Copy link
Copy Markdown
Author

mxkpp commented Mar 27, 2026

It was intentional on my part to keep the file name generic (no prefix). If more information should be added to the partition config file I can make that change.

@kyle-larkin
Copy link
Copy Markdown

kyle-larkin commented Mar 27, 2026

It was intentional on my part to keep the file name generic (no prefix). If more information should be added to the partition config file I can make that change.

I'm ok with no prefix, but that makes it a little strange that there is a partition_config_basename_prefix variable, which the logger uses to produce the following:

2026-03-27T12:29:19.640 MSWM INFO Creating partition file for

I think if the variable exists, then it should be either optional, have a default value, or be set each time.

@mxkpp
Copy link
Copy Markdown
Author

mxkpp commented Mar 27, 2026

It was intentional on my part to keep the file name generic (no prefix). If more information should be added to the partition config file I can make that change.

I'm ok with no prefix, but that makes it a little strange that there is a partition_config_basename_prefix variable, which the logger uses to produce the following:

2026-03-27T12:29:19.640 MSWM INFO Creating partition file for

I think if the variable exists, then it should be either optional, have a default value, or be set each time.

I pushed a change that results in the partition config files being named generically "forecast_partition_config.json" instead of "_partition_config.json", and the log msg now reads like "Creating partition file for forecast"

Copy link
Copy Markdown

@kyle-larkin kyle-larkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are good.

@mxkpp mxkpp merged commit 2a40e84 into development Mar 27, 2026
4 checks passed
@mxkpp mxkpp deleted the maxkipp-fcst-n-2 branch March 27, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants