Move input and output processing from GAMS to Python

[patrickbrown4]

Summary

To support the long-term goal of a free software base, we'd like to move the upstream and downstream parts of the code from GAMS into Python. That will facilitate the testing (and hopefully eventual mainstream use) of free/open-source LP and solver approaches for the core optimization.

Parallel component 1: Inputs

It'd probably be worth discussing the sequencing, but my initial inclination for a rough order of operations, to be applied in separate PRs, is below. Let's use this as a workspace to track progress and plans (so feel free to modify it as work evolves).

• Finish moving most of the rest of the parameters that are read in directly from .csv files (basically every $include *.csv) to the associated input_processing python script by replacing .to_csv() with reeds.io.write_to_inputs_h5()
  • Move the units and comment from the point of definition in b_inputs.gmsto where they are written in the associated input_processing script so they show up in inputs.h5 (see Move more inputs to inputs.h5 and add compatibility with forecast.py #139 for examples)
    • If they are handled programmatically in copy_files.py, add the comment to runfiles.csv
  • Parameters defined via table will need to be reshaped from wide to long format for inclusion in inputs.h5, with associated changes to runfiles.csv and futurefiles.csv
  • Wherever possible, apply switch-based conditionals and multipliers upstream when the parameter is written, rather than modifying the parameter in b_inputs.gms
• Move some of the simpler set definitions to a new input_processing/create_sets.py (or something like that) script:
  • hierarchy() and related r_{level}() and routes() sets
  • ban() and bannew()
  • Other sets and parameters defined before valcap or that don't depend on valcap
• It'd be great to address Try to simplify cooling water #40 before moving the water-cooling parts to Python, as it could simplify a lot of processing that depends on the i set.
• valcap (probably the biggest effort). Should be calculated in its own function. (We might need to subdivide create_sets.py if it gets too long.)
• All the rest of the parameters that depend on valcap : valinv, valgen, and derivatives
• Once b_inputs.gms is done, move to the core/solve scripts:
  • 2_financials.gms
  • 2_temporal_params.gms
  • 3_solve_oneyear.gms
  • 4_post_solve_adjustments.gms
  • 6_data_dump.gms

Parallel component 2: Outputs

• This part is much more straightforward; just apply the approach in Add structure for moving report.gms to Python #142 to more blocks of output parameters.