Move more inputs to `inputs.h5` and add compatibility with `forecast.py` by patrickbrown4 · Pull Request #139 · ReEDS-Model/ReEDS

patrickbrown4 · 2026-07-09T15:32:50Z

Summary

This PR builds on #61 by moving more inputs to inputs.h5, removing another ~10% (~600 lines) of b_inputs.gms.

Technical details

Implementation notes

Most of the changes are just normalizing headers in input .csv files and changing .to_csv() to reeds.io.write_input_to_h5(). Outputs of the following input-processing scripts are moved to inputs.h5:

calc_financial_inputs.py: complete
transmission.py: complete
copy_files.py: a smattering (not complete)

More substantive changes are:

forecast.py is adapted to work for parameters in inputs.h5
- forecast.py is also moved later in the input-processing pipeline; in its old location it was missing some parameters from transmission.py. The new parameters are added to the bottom of futurefiles.csv.
The CO2 storage parameter processing is moved from b_inputs.gms to get_co2_site_char() in transmission.py
- The t index is removed from cost_co2_stor_bec because it doesn't vary with time
The application of GSw_TransCostMult is moved upstream to transmission.py
inputs/waterclimate files are reshaped from wide to long
overwrite in reeds.io.write_input_to_h5() is turned on by default, making the input-processing scripts easier to debug

Additional changes

reedsplots.plot_diff_maps(): A level parameter is added, and when used in compare_cases.py, state resolution is used when runs with different spatial resolution are compared (thanks @kodiobika for the idea and initial implementation)
plots.rainbowmapper(): colormap moved to keyword argument

Validation, testing, and comparison report(s)

Only rounding differences for the Pacific test case (expected when moving from .csv to .h5):
results-v20260708_mainM0_Pacific,v20260708_inputsM2_Pacific.pptx

Same for USA_defaults: results-v20260708_mainK0_USA_defaults,v20260708_inputsK1_USA_defaults.pptx

Checklist for author

r2x (either write trancap_init_energy to .csv or set up r2x to read from inputs.h5)

Details to double-check

Charge code provided to reviewers
Included comparison reports for appropriate test cases
Code formatting standardized
Reusable functions used where possible instead of copy/pasted code

General information to guide review

Zero impact on results of default case
No large data file(s) added/modified
No substantive impact on runtime for full-US reference case
No substantive impact on folder size for full-US reference case
No change to process flow (runreeds.py, reeds/core/solve/solve.py)
No change to code organization
No change to package requirements (environment.yml or Project.toml)

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

No

…rs to inputs.h5

… some more inputs to inputs.h5

… table into individual parameters

…es.py when resolutions differ

…rts(); io.py: clean up gamsify_header()

patrickbrown4 · 2026-07-09T15:34:29Z

 x.csv,.csv,1,None,9999,None,9999,0,constant,None,None,new,
 yearafter.csv,.csv,1,None,9999,None,9999,0,constant,None,None,new,
 years_until_endogenous.csv,.csv,1,None,9999,None,9999,0,constant,None,None,done,constant
+allh,inputs.h5,1,,,,9999,,,,,,


I'm looking forward to removing all the entries that are ignored, but let's save that for a followup (and discuss how to make sure new t-dependent parameters are handled if we aren't enforcing (mostly empty) entries for all parameters)

patrickbrown4 · 2026-07-09T15:35:24Z

 GSw_Region,cendiv/Pacific,,country/USA,country/USA,country/USA,,st/ID.WY.NE.IA.IL,st/MA,,,,,,,st/WY,interconnect/western,transreg/PJM,st/NY.VT,st/OR,st/NE.NY.PA,st/NE.NY.PA,st/ID.WY.NE.IA.IL,st/KS,country/USA,country/USA,,,,st/MA.RI.CT.NY.NJ.PA.OH,
 endyear,2032,,2050,2050,2050,2029,2060,2026,,,,,,,,,,,2035,2030,2030,2060,2035,2050,2050,,,,,
-yearset,,,,,,,2010..2060..10,,,,,,,,,,,,,2010..2050..5,2010..2050..5,2010..2060..10,,,2010_2025_2050,,,,,
+yearset,,,,,,,2010..2060..10,,,,,,,,,,,,,2010..2050..5,2010..2050..5,2010..2060..10,,2010..2050..5,2010_2025_2050,,,,,


USA_fast originally used 5-year steps; I'm not sure when that was dropped but this just changes it back

patrickbrown4 · 2026-07-09T15:35:53Z

                    + '\n> '.join([c for c in df_cases.columns])
                )
-                raise KeyError(err)
+                raise ValueError(err)


Prints a prettier message this way

patrickbrown4 · 2026-07-09T15:36:33Z

        sw['financials_sys_suffix'], inflation_df, modeled_years, 
        years, year_map, sw['sys_eval_years'], scen_settings, scalars['co2_capture_incentive_length'],scalars['h2_ptc_length'])
-    financials_sys.to_csv(os.path.join(inputs_case,'financials_sys.csv'),index=False)
+    financials_sys.to_csv(os.path.join(inputs_case,'financials_sys_full.csv'),index=False)


Avoids errors when restarting this script (since before it was overwriting a file in inputs_case)

Co-authored-by: kodiobika <35176195+kodiobika@users.noreply.github.com>

kodiobika

LGTM

…_gen_frac

patrickbrown4 added 21 commits June 17, 2026 15:46

use float32 for all parameters in inputs.h5; move more input paramete…

2e3d144

…rs to inputs.h5

add inputs.h5 support to copy_files.write_region_indexed_file(); move…

2060be5

… some more inputs to inputs.h5

io.write_to_h5(): turn overwrite on by default

0e612ee

plots.rainbowmapper(): move cmap for large number of traces to kwarg

c1c66a2

Merge branch 'main' into pb/inputs

7413571

reshape some waterclimate inputs to long

c69b109

move transmission.py outputs to inputs.h5; disaggregate co2_site_char…

7c8c3bf

… table into individual parameters

fix co2_site_char index

471ae98

move outputs of calc_financial_inputs.py to inputs.h5

61ada42

Merge branch 'main' into pb/inputs

8eed5a3

adapt forecast.py for inputs.h5

ba47c99

fix a few futurefiles.csv entries

2b9dbb0

Merge branch 'main' into pb/inputs

5610ac1

reedsplots.plot_map_diff(): add level kwarg and use it in compare_cas…

ef4bf05

…es.py when resolutions differ

cases_test.csv: change yearset back to 5-year steps for USA_fast

58db8ad

calc_financial_inputs.py: t -> allt; reedsplots.py: fix plot_max_impo…

7849a14

…rts(); io.py: clean up gamsify_header()

calc_financial_inputs.py: fix a few more t/allt

05156e1

h5_to_gdx.py: add v to special_keys

066d8fe

finish moving outputs of calc_financial_inputs.py to inputs.h5

85fdbc2

adjust parameter names in futurefiles.csv

6b937e7

more futurefiles.csv

b644a6d

github-actions Bot added data_changes model_changes postprocessing labels Jul 9, 2026

patrickbrown4 commented Jul 9, 2026

View reviewed changes

kodiobika requested review from kodiobika and removed request for kodiobika July 9, 2026 18:48

patrickbrown4 added 2 commits July 10, 2026 15:59

write trancap_init_energy for r2x

4884398

inputs.h5: include units in attrs

916654d

patrickbrown4 requested a review from kodiobika July 10, 2026 22:27

This was referenced Jul 12, 2026

Add structure for moving report.gms to Python #142

Open

Move input and output processing from GAMS to Python #143

Open

bsergi mentioned this pull request Jul 13, 2026

MGA updates & employment outputs #21

Merged

23 tasks

patrickbrown4 mentioned this pull request Jul 15, 2026

Add regional interconnection costs #138

Draft

15 tasks

kodiobika reviewed Jul 23, 2026

View reviewed changes

Comment thread reeds/plots.py

Comment thread reeds/input_processing/runfiles.csv

Comment thread inputs/userinput/futurefiles.csv

Comment thread reeds/input_processing/calc_financial_inputs.py Outdated

Comment thread reeds/input_processing/writecapdat.py Outdated

patrickbrown4 and others added 5 commits July 23, 2026 14:57

trans_cost_cap_fin_mult_noITC: Fix typo

70a2c68

Co-authored-by: kodiobika <35176195+kodiobika@users.noreply.github.com>

writecapdat.py: 2010->startyear

0d98abe

Co-authored-by: kodiobika <35176195+kodiobika@users.noreply.github.com>

runfiles.csv: change 'comment' to 'GAMScomment'

2d91c94

Merge branch 'main' into pb/inputs

fb0b095

forecast.py: write params read from inputs.h5 back to inputs.h5

f3a3530

kodiobika approved these changes Jul 23, 2026

View reviewed changes

patrickbrown4 added 4 commits July 23, 2026 17:16

futurefiles.csv: fill in forecasts for cost_hurdle_rate* and national…

9065e68

…_gen_frac

Merge branch 'main' into pb/inputs

6df28f7

mcs_sampler.py: remove removed overwrite in get_regions_and_agglevel()

f1ba970

fix MARICTNYNJPAOH_Offshore: change trancap_fut t index to allt

9249534

patrickbrown4 added a commit that referenced this pull request Jul 27, 2026

minor cleanups from #139

c96dff9

patrickbrown4 added 2 commits July 27, 2026 11:22

Merge branch 'main' into pb/inputs

78d3012

transmission.py: fix trancap_fut (move t->allt to end)

1a4a711

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Move more inputs to `inputs.h5` and add compatibility with `forecast.py` - #139

Move more inputs to `inputs.h5` and add compatibility with `forecast.py`#139
patrickbrown4 wants to merge 34 commits into
mainfrom
pb/inputs

patrickbrown4 commented Jul 9, 2026 •

edited

Loading

Uh oh!

patrickbrown4 Jul 9, 2026

Uh oh!

patrickbrown4 Jul 9, 2026

Uh oh!

patrickbrown4 Jul 9, 2026

Uh oh!

patrickbrown4 Jul 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kodiobika left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

patrickbrown4 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Technical details

Implementation notes

Additional changes

Validation, testing, and comparison report(s)

Checklist for author

Details to double-check

General information to guide review

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

Uh oh!

patrickbrown4 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

patrickbrown4 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

patrickbrown4 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

patrickbrown4 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kodiobika left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

patrickbrown4 commented Jul 9, 2026 •

edited

Loading