diff --git a/CHANGELOG.md b/CHANGELOG.md index 716c27ca..d0ca61de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,9 @@ ## [4.18.17RC] - 2026-06-13 Unreleased in PyPI - [CHANGED] fix some errors in writing H2 and heat network output results -- [ADDED] Mode C post-build hot-swap re-solve (`openTEPES_ProblemSolvingResolve.py`). `resolve(OptModel, SolverName, overlays)` re-solves an already-built model once per parameter overlay without rebuilding it, so a sweep, sensitivity study or Monte-Carlo reuses the one slow build; `overlay_scaled` builds a scale-by-a-factor overlay. Overlays are applied relative to the baseline, not cumulatively, and the baseline is restored at the end. Only `mutable=True` Params can be hot-swapped, so the operational set is promoted to mutable: `pDemandElec`, `pENSCost`, `pLinearVarCost`, `pEFOR`, `pReserveMargin`, `pRESEnergy`. Structural and topology Params stay immutable because they are read in build-time `if` statements, where a mutable Param raises instead of returning its value (RFC Modes A / B). The six promoted Params are now read by call (`pFoo[idx]()`) in the build-time guards and in the cost and demand sums; reads inside constraints and the objective stay symbolic so the swap reaches the solver. No behaviour change: case 9n solves bit-identical (164.382043867 MEUR, `PYTHONHASHSEED=0`) and the solve suite passes with the same locked costs. New test `test_mode_c_resolve_demand_hot_swap` builds 9n once, then checks an identity overlay reproduces the cost, a +10 % demand overlay raises it, and restoring the baseline returns it. +- [FIXED] `setup_solver` no longer crashes on Windows when an earlier in-process solve still holds the stale log file open. +- [FIXED] deprecated `datetime.utcnow()` replaced with timezone-aware UTC; emitted timestamp unchanged. +- [ADDED] Mode C post-build hot-swap re-solve (`openTEPES_ProblemSolvingResolve.py`): `resolve(OptModel, SolverName, overlays)` re-solves a built model once per parameter overlay without rebuilding, so a sweep reuses the single slow build; `overlay_scaled` makes a scale-factor overlay. Overlays apply relative to the baseline, which is restored at the end. Only `mutable=True` Params hot-swap, so the operational set (`pDemandElec`, `pENSCost`, `pLinearVarCost`, `pEFOR`, `pReserveMargin`, `pRESEnergy`) is promoted to mutable and read by call in the build-time guards; structural and topology Params stay immutable. No behaviour change: 9n solves bit-identical (164.382043867 MEUR, `PYTHONHASHSEED=0`). New test `test_mode_c_resolve_demand_hot_swap`. - [CHANGED] split the ~2900-line `openTEPES_InputData.py` into three modules at the package root, one per model-build step — the input-side counterpart of the earlier Formulation, Solver and Output splits, and the last of the three mega-file splits. `openTEPES_InputData.py` keeps `InputData`, which reads the raw sets and parameters from the case source (a CSV directory or a DuckDB file). The new `openTEPES_DataConfiguration.py` holds `DataConfiguration`, which builds the derived and instrumental sets and the flag-driven branches (hydro topology, hydrogen, heat, PTDF). The new `openTEPES_SettingUpVariables.py` holds `SettingUpVariables`, which creates the decision variables and their bounds, fixes the generators' commitment, relaxes or forbids investment conditions, zeroes out epsilon values, and screens for infeasibilities. `openTEPES.py` imports each function from its own module, and `__init__.py` re-exports all three, so a user who imported `DataConfiguration` or `SettingUpVariables` from `openTEPES.openTEPES_InputData` moves to the matching module (e.g. `...DataConfiguration`). This is a pure move — the three functions are byte-identical to before, checked function by function, and none of them calls another — so results do not change: the whole solve test suite passes with the same locked costs. The two new modules import only pandas and Pyomo (no sibling `openTEPES` modules), so they need no relative-import guard; `tests/test_direct_run.py` still runs them as scripts like every other module. This finishes the Layer-3 model-build split from the architecture RFC. Separating set construction from parameter construction inside `InputData` (groundwork for the planned overlay and mutable-parameter work) is function surgery rather than a pure move and is left for its own later change. - [CHANGED] document every bundled case on the Download & Installation page (`doc/md/Download.md`). The list now covers all ten cases under `openTEPES/cases/`, not just seven: the three 9-node variants that were missing are added — `9n_PTDF` (the 9-node case solved with the PTDF network formulation instead of the angle-based DC power flow), `9n_heat` (the 9-node case coupled with a heat network), and `9n_H2` (the 9-node case coupled with a hydrogen network) — and the `sSEP` entry now notes it also includes a hydrogen network. Each entry says in one line what makes the case different, so a new user can pick the right starting point. - [CHANGED] split the ~1770-line `openTEPES_ModelFormulation.py` into six per-concern modules at the package root — the formulation-side counterpart of the earlier Input, Solver and Output splits. The two cross-sector concerns are `openTEPES_ModelFormulationObjective.py` (the total-cost objective and the per-stage operation-cost accumulation) and `openTEPES_ModelFormulationInvestment.py` (the four investment builders plus the installed-capacity, adequacy-reserve-margin and emission / RES-energy limits). Each energy sector then gets its own module: `openTEPES_ModelFormulationElectricity.py` (demand balance, operating reserves and inertia, storage, unit commitment, ramping, line switching, DC network operation and the cycle constraints), `openTEPES_ModelFormulationHydro.py` (reservoir water balance), `openTEPES_ModelFormulationHydrogen.py` (H2 network) and `openTEPES_ModelFormulationHeat.py` (heat network). The old `openTEPES_ModelFormulation.py` is removed: `openTEPES.py` imports the objective and investment builders from their modules, `openTEPES_ProblemSolvingStageIter.py` imports the per-stage operation builders, and `__init__.py` re-exports all six (so `from openTEPES.openTEPES_ModelFormulation import ...` users move to the concern module — e.g. `...Investment`). This is a pure move — the 19 builder functions are byte-identical to before, checked function by function — so results do not change: the whole solve test suite passes with the same locked costs. Keeping each sector in one file, with its existing granular per-constraint functions inside, is the groundwork for selectable formulations later (for example a DC vs AC network build, or unit commitment on or off) chosen by which functions the stage driver calls. The new modules use the descriptive-docstring convention of the other split modules; the standalone test `tests/test_heat_investment_typo.py` now imports `InvestmentHeatModelFormulation` from `...Investment`. diff --git a/openTEPES/openTEPES.py b/openTEPES/openTEPES.py index af2b03c0..798fc620 100644 --- a/openTEPES/openTEPES.py +++ b/openTEPES/openTEPES.py @@ -138,7 +138,7 @@ def openTEPES_run(DirName, CaseName, SolverName, pIndOutputResults, pIndLogConso """ InitialTime = time.time() - _RunStartedUtc = datetime.datetime.utcnow().isoformat(timespec="seconds") + "Z" + _RunStartedUtc = datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None).isoformat(timespec="seconds") + "Z" # If the caller pointed at a .duckdb file (CaseName carries the # suffix), open it as a DuckDBSource; InputData reads tables from @@ -387,7 +387,7 @@ def openTEPES_run(DirName, CaseName, SolverName, pIndOutputResults, pIndLogConso "backend": getattr(mTEPES, "pOutputBackend", "csv"), "opentepes_version": "4.18.17RC", "run_started_utc": _RunStartedUtc, - "run_finished_utc": datetime.datetime.utcnow().isoformat(timespec="seconds") + "Z", + "run_finished_utc": datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None).isoformat(timespec="seconds") + "Z", "outputs_enabled": [k for k, v in _flags.items() if v], "gzip_patterns": list(gzip_patterns) if gzip_patterns else None, "gzip_files": _GzipFiles, diff --git a/openTEPES/openTEPES_ProblemSolvingPersistent.py b/openTEPES/openTEPES_ProblemSolvingPersistent.py index 67cca4a7..90fc0120 100644 --- a/openTEPES/openTEPES_ProblemSolvingPersistent.py +++ b/openTEPES/openTEPES_ProblemSolvingPersistent.py @@ -12,6 +12,7 @@ """ from __future__ import annotations +import gc import os import pyomo.environ as pyo @@ -28,7 +29,14 @@ def setup_solver(OptModel, SolverName: str, FileName: str, ncall: int, mTEPES): if SolverName != "appsi_gurobi" and SolverName != "gurobi_persistent": Solver = SolverFactory(SolverName) if os.path.exists(FileName): - os.remove(FileName) + # A prior in-process solve may still hold this log open (the HiGHS interface keeps it open until + # GC frees the highspy object); on Windows an open file cannot be deleted. Collect first, then + # remove best-effort: a still-locked log is left in place, since the solver truncates it on re-open. + gc.collect() + try: + os.remove(FileName) + except OSError: + pass return Solver if SolverName == "appsi_gurobi": diff --git a/openTEPES/openTEPES_ProblemSolvingResolve.py b/openTEPES/openTEPES_ProblemSolvingResolve.py index c9b846fa..8ad4cb7c 100644 --- a/openTEPES/openTEPES_ProblemSolvingResolve.py +++ b/openTEPES/openTEPES_ProblemSolvingResolve.py @@ -3,9 +3,8 @@ openTEPES.openTEPES_ProblemSolvingResolve — Mode C post-build hot-swap re-solve. -Re-solve an already-built model once per parameter overlay without rebuilding it, so a -sweep, sensitivity study or Monte-Carlo reuses the one slow build. Only ``mutable=True`` -Params can be hot-swapped: the operational set ``pDemandElec``, ``pENSCost``, +Re-solve a built model once per parameter overlay without rebuilding, so a sweep reuses the +one slow build. Only ``mutable=True`` Params hot-swap: ``pDemandElec``, ``pENSCost``, ``pLinearVarCost``, ``pEFOR``, ``pReserveMargin``, ``pRESEnergy``. See ``resolve`` for the overlay format. """ @@ -16,9 +15,9 @@ def overlay_scaled(OptModel, param_name: str, factor: float) -> dict: - """Return an overlay that scales every current value of ``param_name`` by ``factor``. + """Return an overlay scaling every current value of ``param_name`` by ``factor``. - Example: ``overlay_scaled(model, "pDemandElec", 1.10)`` for a 10 % demand uplift. + E.g. ``overlay_scaled(model, "pDemandElec", 1.10)`` for a 10 % demand uplift. """ param = getattr(OptModel, param_name) return {param_name: {idx: val * factor for idx, val in param.extract_values().items()}} @@ -27,18 +26,16 @@ def overlay_scaled(OptModel, param_name: str, factor: float) -> dict: def resolve(OptModel, SolverName: str, overlays, *, restore: bool = True, tee: bool = False): """Re-solve ``OptModel`` once per overlay, swapping mutable Param values in place. - ``overlays`` is a list of dicts, each mapping a mutable Param name to new values (a - dict ``index -> value`` for an indexed Param, or a scalar). Each overlay is applied - relative to the baseline (values at call time), not cumulatively; ``{}`` re-solves the - baseline. With ``restore`` (default) the baseline is restored at the end. ``SolverName`` - must be a non-persistent solver. Returns one dict per overlay with the termination + ``overlays`` is a list of dicts mapping a mutable Param name to new values (``index -> + value``, or a scalar). Each overlay applies relative to the baseline, not cumulatively; + ``{}`` re-solves the baseline. ``restore`` (default) restores the baseline at the end. + ``SolverName`` must be non-persistent. Returns one dict per overlay with the termination condition and total system cost (None if not optimal). """ overlays = list(overlays) - # Snapshot the baseline values of every Param any overlay touches. The snapshot serves - # two purposes: resetting the touched Params before each overlay (so overlays are - # independent, not cumulative) and the optional restore at the end of the sweep. + # Snapshot the baseline of every touched Param: used to reset before each overlay (so + # overlays are independent, not cumulative) and for the optional restore at the end. touched = set() for ov in overlays: touched.update(ov.keys())