Adding technology classes by atpham88 · Pull Request #12 · ReEDS-Model/ReEDS

atpham88 · 2026-04-15T21:50:06Z

Summary

This PR:

Removes startyear switch and instead establishes startyear based on yearset
Adds algorithm to assign pv, wind, and geothermal tech classes to NEMS units
Fixes a few issues that cause missing existing capacities in main
Adds an option to specify a flexible start year, instead of 2010
- Although this option is added in this PR, enabling and testing for start year later than 2010 is in a separate PR.

Technical details

These changes are made in this PR:

process_unitdata.py: Read unitdata.csv from inputs_case and process it to assign sc_point_pids, capacity factors/mean temps to pv, wind, and geothermal units. This file is run immediately after copy_files.py. This allows us to remove all the mapping of NEMS units to sc_point_gids in the ReEDS_input_processing repo.
WriteHintage.py: Allow start year to be flexible instead of 2010.
writecapdat.py: Add mappings to solar, wind, geothermal resource class are based on the resource quality (capacity factors for wind and solar and temperature for geothermal) of the technology as it comes from reV.
c_supplymodel.gms
- pcat set is removed from the model: The equations eq_force_prescription_power and eq_force_prescription_energy, along with the variables EXTRA_PRESCRIP and EXTRA_PRESCRIP_ENERGY, are currently indexed by pcat. Since the prescribed generator classes and vintages are now defined in writecap.py, the pcat index is no longer needed in these equations or variables. Accordingly, the equations are revised so that the investment (INV) of prescribed generators—indexed by (i, newv, r, t)—is equal to the sum of the prescribed build values and the additional prescribed investments (EXTRA_PRESCRIP) for those same (i, newv, r, t) combinations.
- Integrate prescribed retirements when the prescribed facilities retires before their life time: In the NEMS database, the retirement year of some facilities occurs before the end of their expected lifetimes. To accurately capture these early retirements for prescribed facilities, the prescribed retirement values are subtracted from total investments when calculating facility capacities in eq_cap_new_noret and eq_cap_energy_new_noret. Although this implementation follows the same logic as m_capacity_exog, it explicitly captures prescribed generator builds in the INV variables in the current version of the model.
hourlize:
- Remove scripts that generate exog_cap_{tech}.csv and those are associated with existing_sites files as they are no longer needed.
geothermal_classification.csv is not generated in hourlize but taken from https://pages.github.nrel.gov/ReEDS/ReEDS-2.0/model_documentation.html#technical-resource-potential.
Update checks on prescribed builds and existing capacity to ensure feasibility with resource constraints.

Issues resolved

Corrected prescribed_build to capture all capacity in unitdata.csv. Currently some prescribed coal capacities are missing due to coalolduns is missing from prescribed_build categories https://github.com/ReEDS-Model/ReEDS/blob/main/reeds/input_processing/writecapdat.py#L79
Corrected missing capacity in capnonrsc. This is due to due to the comment='#' option when read in unitdata.csv, which ignores all the data after unit ID of the rows where unit ID has # character. So when capnonrsc is calculated by group summing from unitdata.csv, some of the categories to sum over are nan and those rows are omitted from the sum.

Validation, testing, and comparison report(s)

Total prescribed builds and exogenous caps are generally the same with main, but spatial distribution is slightly different:
Comparison of prescribed builds (prescribed_build) vs. main (noncumulative_prescriptions) in 2010 for UPV - Total UPV capacity matches and matches with the ReEDS-NEMS plant database but there's some difference in spatial distribution.

Comparison of prescribed builds (prescribed_build ) vs. main (noncumulative_prescriptions) in 2015 for UPV - Total UPV capacity in this branch is slightly lower but it matches exactly with NEMS.

Comparison of prescribed builds (prescribed_build ) vs. main in 2010 for wind-ons - Wind-ons capacity matches exactly:

Comparison of existing cap (exog_upv) vs. main for UPV - Existing UPV capacity is slightly higher in this branch but matches exactly with NEMS:

Comparison of existing cap (exog_wind-ons) vs. main for wind-ons - Existing wind-ons capacity matches exactly:

Comparison report (full here)

Increase in coal capacity due to capturing coalolduns prescribed capacity not captured in main
Slight decrease in solar due to lower prescribed pv capacity in this branch compared to main, but this branch's prescribed pv capacity matches NEMS (see above)

Increase in csp generation since main does not have any csp generation (fails to convert upv generation to csp generation)

New annual capacity

Annual retirement

Checklist for author

Details to double-check

General information to guide review

Zero impact on results of default case
No large data file(s) added/modified
No substantive impact on runtime for full-US reference case
No substantive impact on folder size for full-US reference case
No change to process flow (runbatch.py, d_solve_iterate.py)
No change to code organization
No change to package requirements (environment.yml or Project.toml)

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

No

Tag points of contact here if you would like additional review of the relevant parts of the model

Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

patrickbrown4

I haven't gone through the whole thing yet and will take a closer look next week, but some initial thoughts in the meantime. Happy to discuss if helpful (including whether to try some of the suggested changes here or in a followup PR). Thanks!

…tdata.py

…ce class

patrickbrown4

Thanks An, lots of work in this PR! I think all of my comments are pretty minor and related to style/readability/error-handling rather than functionality, so it seems really close.

To Brian's point, can you add a comparison between a USA run starting in 2010 and one starting in 2020? If the differences are bigger than expected, and you'd like to wait for a followup PR to look into it more (since there are already lots of changes here), then you could add a temporary NotImplementedError if startyear != 2010 here to give more time to work out issues while moving forward with the rest of the changes implemented here.
@wesleyjcole I'm comfortable with the python changes, but I never really wrapped my head around all the prescriptive/exogenous capacity handling in b_inputs.gms and c_model.gms, so I'd prefer to defer to you on those parts.

patrickbrown4 · 2026-07-23T19:10:47Z

+    prescribed_nonRSC['tech'] = prescribed_nonRSC['tech'].str.lower()
+    ### assign vintage based on start year of the unit
+    prescribed_nonRSC= pd.merge(prescribed_nonRSC, ivt_df, how='left', left_on='tech', right_on='Unnamed: 0')
+    prescribed_nonRSC['vin'] = prescribed_nonRSC.apply(lambda row: f"new{row[str(row['StartYear'])]}", axis=1)


I like the idea of including the year in the vintage name but it seems inconsistent with the contents of v (below). So it looks like vin is just temporary - maybe note that in a comment?

ReEDS/reeds/input_processing/WriteHintage.py

Lines 616 to 618 in 5102c8c

initv = [f'init-{i}' for i in range(1, numhintage+1)]

newv = [f'new{i}' for i in range(1, int(sw.numclass)+1)]

v = initv + newv

patrickbrown4 · 2026-07-23T19:16:59Z

+                    $m_required_prescriptions(i,newv,r,t)
+                    $sum{tt$[m_required_prescriptions(i,newv,r,tt) 
+                          $(yeart(tt)<=yeart(t))], ivt(i,newv,tt) }$(not ban(i))] = yes ;


Could you take a look at the inputs.gdx diff for valcap and add it to the compare report? I expect there will be changes given the coal bugfix but it'd be nice to confirm everything else is as expected.

An: the new valcap should be tighter because only classes with prescribed builds will be included in early years. Before every class would be included if there was a prescribed build.

patrickbrown4 · 2026-07-23T19:32:23Z

 cf_adj_t(i,v,t)$[(rsc_i(i) or hydro(i))$sum{r, valcap(i,v,r,t) }] = 1 ;

-* Existing wind uses 2010 cf adjustment
+* Existing wind uses startyear cf adjustment


Interesting - so if we use a later startyear, all the existing wind gets a higher CF?
A fix for that will be a pretty big effort. In the meantime (assuming I'm understanding correctly), could you add a caveat to faq.md, here?

- Wind capacity existing in the first modeled year (the first year specified by the `yearset` switch) is treated as if it were built in the first modeled year, using the ATB capacity factor multiplier for that year. For example, if 2020 is the first modeled year, all wind capacity existing in 2020 uses the ATB capacity factor multiplier for 2020. This approach neglects variation in capacity factor across different existing wind vintages.

That's correct for two reasons: 1) we don't have older vintages with older CF values, and 2) we have not applied degradation for older plants. The first is relatively easy to address, as we can just update our CF adjustment parameters (it's harder to make it dynamic, but we can easily make it "correct" for a different startyear). The second is more effort, but Anna S. was going to look into it to see what we can do there.

Thanks Wesley. I was mostly concerned about the difference between a startyear of 2010 and, for example, 2025. According to the ATB file, the 2010 CF multiplier is 0.61 and the 2025 CF multiplier is 0.96 (57% higher). Using the 2025 CF for all ≤2025 wind capacity seems like a bigger issue than using the 2010 CF for all ≤2010 capacity, and could lead to bigger differences than we want between start years (if I'm understanding correctly).

It will help to look at the results for a run with a >2010 startyear, but if the differences are too large, it might be worth considering whether to disallow >2010 startyears until we update the handling of post-2010 existing wind.

patrickbrown4 · 2026-07-23T19:33:58Z

+*beginning year value of 2010 is zero (i.e., no elasticity)
+cd_beta(cendiv,t)$[t.val>2010] = cd_beta0_allsector(cendiv) ;

-nat_beta(t)$(not tfirst(t)) = nat_beta_energy ;
+nat_beta(t)$[t.val>2010] = nat_beta_energy ;

 $else.gassector

-*beginning year value is zero (i.e., no elasticity)
-cd_beta(cendiv,t)$[not tfirst(t)] = cd_beta0(cendiv) ;
+*beginning year value of 2010 is zero (i.e., no elasticity)
+cd_beta(cendiv,t)$[t.val>2010] = cd_beta0(cendiv) ;

 *see documentation for how value is calculated
-nat_beta(t)$(not tfirst(t)) =  nat_beta_nonenergy ;
+nat_beta(t)$[t.val>2010] =  nat_beta_nonenergy ;


Why does 2010 need to be hardcoded here? If it's a property of the gas elasticity data, could you move it to scalars.csv?

Due to hold the model used to work, we did not include elasticity information in the supply curves for the 2010 year (i.e., beta = 0 for 2010). That has persisted.

wesleyjcole

@bsergi - reeds_to_rev.py uses "existing_capacity". It's possible that the changes here might break that. Just flagging for awareness.

The changes all look good to me. I'll just want to check out an updated comparison report before approving.

wesleyjcole · 2026-07-23T21:12:51Z

-
    if min_cap > 0:
        #Remove sites with less than minimum capacity threshold, but keep sites that have existing capacity
        df = df[(df['capacity'] >= min_cap) | (df['existing_capacity'] > 0)].copy()


Will this reference to existing_capacity cause an issue now that the other instances have been removed?

wesleyjcole · 2026-07-23T21:13:04Z

@@ -776,46 +681,6 @@ def save_sc_outputs(
    df_sc.to_csv(os.path.join(outpath, 'results', tech + '_supply_curve_raw.csv'), index=False)
    #Round now to prevent infeasibility in model because existing (pre-2010 + prescribed) capacity is slightly higher than supply curve capacity
    df_sc[['capacity','existing_capacity']] = df_sc[['capacity','existing_capacity']].round(decimals)


Same question here. Will this reference to existing_capacity cause an issue now that the other instances have been removed?

wesleyjcole · 2026-07-23T21:25:43Z

+                    $m_required_prescriptions(i,newv,r,t)
+                    $sum{tt$[m_required_prescriptions(i,newv,r,tt) 
+                          $(yeart(tt)<=yeart(t))], ivt(i,newv,tt) }$(not ban(i))] = yes ;


An: the new valcap should be tighter because only classes with prescribed builds will be included in early years. Before every class would be included if there was a prescribed build.

wesleyjcole · 2026-07-23T21:28:26Z

 cf_adj_t(i,v,t)$[(rsc_i(i) or hydro(i))$sum{r, valcap(i,v,r,t) }] = 1 ;

-* Existing wind uses 2010 cf adjustment
+* Existing wind uses startyear cf adjustment


That's correct for two reasons: 1) we don't have older vintages with older CF values, and 2) we have not applied degradation for older plants. The first is relatively easy to address, as we can just update our CF adjustment parameters (it's harder to make it dynamic, but we can easily make it "correct" for a different startyear). The second is more effort, but Anna S. was going to look into it to see what we can do there.

wesleyjcole · 2026-07-23T21:29:57Z

+*beginning year value of 2010 is zero (i.e., no elasticity)
+cd_beta(cendiv,t)$[t.val>2010] = cd_beta0_allsector(cendiv) ;

-nat_beta(t)$(not tfirst(t)) = nat_beta_energy ;
+nat_beta(t)$[t.val>2010] = nat_beta_energy ;

 $else.gassector

-*beginning year value is zero (i.e., no elasticity)
-cd_beta(cendiv,t)$[not tfirst(t)] = cd_beta0(cendiv) ;
+*beginning year value of 2010 is zero (i.e., no elasticity)
+cd_beta(cendiv,t)$[t.val>2010] = cd_beta0(cendiv) ;

 *see documentation for how value is calculated
-nat_beta(t)$(not tfirst(t)) =  nat_beta_nonenergy ;
+nat_beta(t)$[t.val>2010] =  nat_beta_nonenergy ;


Due to hold the model used to work, we did not include elasticity information in the supply curves for the 2010 year (i.e., beta = 0 for 2010). That has persisted.

Apply Patrick's suggestions Co-authored-by: Patrick Brown <25125211+patrickbrown4@users.noreply.github.com>

…eEDS into ap_flex-start-year

Apply more suggested changes Co-authored-by: Patrick Brown <25125211+patrickbrown4@users.noreply.github.com> Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

patrickbrown4 · 2026-07-27T16:30:55Z

-                os.path.join(resultspath, f'{tech}_prescribed_builds.csv'),
-                os.path.join(
-                    inputspath, 'capacity_exogenous',
-                    f'prescribed_builds_{techlabel}_{access_case}.csv',


Sorry, I missed this before, but I'm pretty sure these files and functions can be removed too, since they're no longer used:

inputs/capacity_exogenous/prescribed_builds_*.csv

Associated entries in futurefiles.csv and runfiles.csv

reeds.io.assemble_prescribed_builds()

And the one place it's called in copy_files.py

I think I removed all the prescribed_builds, exog_cap, and their associated files/functions now. If I still miss anything, I can clean up the rest in the follow up PR

…eEDS into ap_flex-start-year

atpham88 added 6 commits April 15, 2026 15:43

moved to github.com

35b2339

Update ReEDS_generator_database_final_EIA-NEMS.csv

8a17399

put climate folder back

04d2355

update upload-artifact to v4 to pass github tests

b8779fa

Merge branch 'main' into ap_flex-start-year

2c1fec0

fix issue with wind-ons supply curves

64ea661

atpham88 requested a review from wesleyjcole April 16, 2026 22:32

Merge branch 'main' into ap_flex-start-year

bacf02a

wesleyjcole reviewed Apr 16, 2026

View reviewed changes

Update sources.csv

e760e49

Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

atpham88 requested review from bsergi and patrickbrown4 April 17, 2026 00:01

atpham88 added 4 commits April 16, 2026 23:21

remove redundant statements

ad71748

remove exog_cap and exog_prescribe files from hourlize

1d989fc

Update sources.csv

2afc17b

add more description for tech classification files

3dda333

patrickbrown4 reviewed Apr 17, 2026

View reviewed changes

atpham88 added 13 commits April 17, 2026 18:08

Update e_report.gms

2848ff5

Merge branch 'main' into ap_flex-start-year

eb1f1ce

only use max cf for class cut off to avoid gaps

2f78001

keep rounding to 4 decimals

a880c2a

add function to get missing_class_resource

bb0282f

clean up

13079d8

clean up

c7d90f6

remove sc_point_gid and geothermal temp from NEMS

c208046

remove unitdata.csv from runfiles.csv

91cec02

Create generate_unitdata.py

74ae71c

remove reading NEMS data and its processing from copy_files.py

496246c

Add generate_unitdata.py script

10fe8ef

move all NEMS data processing from copy_files and add to generate_uni…

8e160ac

…tdata.py

atpham88 added 3 commits July 22, 2026 16:31

reverse changes to config iles in metadata folder

9676496

Add check to filter out rsc tech classes in unitdata without a resour…

ed89fc1

…ce class

clean up

5102c8c

kodiobika mentioned this pull request Jul 23, 2026

Move more inputs to inputs.h5 and add compatibility with forecast.py #139

Open

12 tasks

patrickbrown4 reviewed Jul 23, 2026

View reviewed changes

wesleyjcole reviewed Jul 23, 2026

View reviewed changes

atpham88 and others added 6 commits July 23, 2026 15:44

Apply suggestions from code review

10f17a6

Apply Patrick's suggestions Co-authored-by: Patrick Brown <25125211+patrickbrown4@users.noreply.github.com>

clean up and fix typos

f91b518

Merge branch 'ap_flex-start-year' of https://github.com/ReEDS-Model/R…

84a3464

…eEDS into ap_flex-start-year

Apply suggestions from code review

7a76c0d

Apply more suggested changes Co-authored-by: Patrick Brown <25125211+patrickbrown4@users.noreply.github.com> Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

fix indentation and remove ggregate_regions.py related statement

dd6ff26

Merge branch 'main' into ap_flex-start-year

8fc3727

atpham88 mentioned this pull request Jul 24, 2026

follow up PR: update doc to reflect all the change in PR#12 #158

Open

atpham88 added 6 commits July 23, 2026 23:13

add fixed inputs, clean up, resolve conflict

835aef2

Update writecapdat.py

fd177ea

resolve issue with updated conda

af80ba5

Update process_unitdata.py

12b9d79

if wind-ofs cf matches to both fixed and floating, assign it to fixed

c24ea76

resolve github runner issue

247eeb5

patrickbrown4 reviewed Jul 27, 2026

View reviewed changes

Comment thread runreeds.py

atpham88 changed the title ~~Flexible start year option & tech classes addition~~ Adding technology classes Jul 27, 2026

patrickbrown4 reviewed Jul 27, 2026

View reviewed changes

atpham88 and others added 8 commits July 28, 2026 12:41

Merge branch 'main' into ap_flex-start-year

25778d1

collapse rsc_pv terms and terminate if startyear is not 2010

7b81215

Merge branch 'main' into ap_flex-start-year

4893eec

clean up

b381ae8

remove unnecessary files and functions

d508ca7

Delete uncertainty_plots_all_cases.csv

73cda95

clean up and revert some changes

24e58df

Merge branch 'ap_flex-start-year' of https://github.com/ReEDS-Model/R…

47f8023

…eEDS into ap_flex-start-year

	initv = [f'init-{i}' for i in range(1, numhintage+1)]
	newv = [f'new{i}' for i in range(1, int(sw.numclass)+1)]
	v = initv + newv

Uh oh!

Conversation

atpham88 commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Technical details

Issues resolved

Validation, testing, and comparison report(s)

Checklist for author

Details to double-check

General information to guide review

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

Tag points of contact here if you would like additional review of the relevant parts of the model

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickbrown4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickbrown4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wesleyjcole left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

patrickbrown4 Jul 27, 2026 • edited by atpham88 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

atpham88 commented Apr 15, 2026 •

edited

Loading

patrickbrown4 Jul 27, 2026 •

edited by atpham88

Loading