Test run didn't go through

[BlurakLP]

Version Checks (indicate both or one)

• I have confirmed this bug exists on the lastest release of PyPSA-DE.

• I have confirmed this bug exists on the current master branch of PyPSA-DE.

Issue Description

I just freshly set-up the model (clone git, pixi shell in the directory) on Ubuntu and tried to do a test-run with the provided config example (I did nothing else with the files). But I get an error that I can't fix (see below). I tried to use a newer snakemake version (similar erros seem to occur to snakemake users and it should be fixed with newer versions) but that comes with new errors (the new error is "no rule to produce this file").
Doing the example config.electricity.yaml at the pypsa-eur works on the same system.
This works: snakemake build_scenarios -f
This doesn't work: snakemake ariadne_all it outputs:

WorkflowError:
MissingInputException: Missing input files for rule retrieve_egon_data:
    output: data/egon/demandregio_spatial_2018.json, data/egon/mapping_technologies.json
    affected files:
        https://api.opendata.ffe.de/demandregio/demandregio_spatial?id_spatial=5&year=2018 (in storage)
        https://api.opendata.ffe.de/demandregio/demandregio_spatial_description?id_spatial=5 (in storage)
MissingInputException: Missing input files for rule retrieve_egon_data:
    output: data/egon/demandregio_spatial_2018.json, data/egon/mapping_technologies.json
    affected files:
        https://api.opendata.ffe.de/demandregio/demandregio_spatial?id_spatial=5&year=2018 (in storage)
        https://api.opendata.ffe.de/demandregio/demandregio_spatial_description?id_spatial=5 (in storage)

I'm not sure if I missed some step in the setup, but doing the same steps with pypsa-eur works.

Reproducible Example

snakemake -call results/test-elec/networks/base_s_6_elec_.nc --configfile config/config.de.yaml -n

Result:
(pypsa-de) ubuntu@pypsa-initialtest:~/pypsa-de$ snakemake -call results/test-elec/networks/base_s_6_elec_.nc --configfile config/config.de.yaml -n
-traceback
Restricted license - for non-production use only - expires 2027-11-29
Config file config/config.default.yaml is extended by additional config specified via the command line.
Config file config/plotting.default.yaml is extended by additional config specified via the command line.
Config file config/config.de.yaml is extended by additional config specified via the command line.
WARNING! Scenario management enabled but scenarios file does not exist.
host: pypsa-initialtest
Building DAG of jobs...
results/test-elec/networks/base_s_6_elec_.nc
<class 'str'>
Traceback (most recent call last):

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/cli.py", line 2193, in args_to_api
    dag_api.execute_workflow(

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/api.py", line 634, in execute_workflow
    workflow.execute(

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/workflow.py", line 1277, in execute
    self._build_dag()

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/workflow.py", line 1220, in _build_dag
    async_run(self.dag.init())

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/common/__init__.py", line 104, in async_run
    return asyncio.run(coroutine)
           ^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/dag.py", line 246, in init
    await self.file2jobs(file),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/dag.py", line 2609, in file2jobs
    raise MissingRuleException(targetfile)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/exceptions.py", line 423, in __init__
    f"No rule to produce {fmt_iofile(file)} (if you use input "
                          ^^^^^^^^^^^^^^^^

  File "/home/ubuntu/pypsa-de/.pixi/envs/default/lib/python3.12/site-packages/snakemake/io/fmt.py", line 9, in fmt_iofile
    if f.is_storage:
       ^^^^^^^^^^^^

AttributeError: 'str' object has no attribute 'is_storage'
'''

Expected Behavior

It should go through just as it does with the example on pypsa-eur

Installed Versions

Details

[dependencies] pycountry = ">=24.6.1" pyam = ">=2.0" "ruamel.yaml" = ">=0.18.10" # common packages atlite = ">=0.3" bokeh = ">=3.8.0" cartopy = ">=0.25.0" copernicusmarine = ">=2.2.4" country_converter = ">=1.3.2" dask = ">=2025.10.0" descartes = ">=1.1.0" entsoe-py = ">=0.7.8" fiona = ">=1.10.1" folium = ">=0.20.0" geojson = ">=3.2.0" geopandas = ">=1" geopy = ">=2.4.1" glpk = ">=5.0" graphviz = ">=12.2.1" gurobi = ">=12.0.3" highspy = ">=1.12.0" ipython = ">=9.7.0" jpype1 = ">=1.6.0" jupyter = ">=1.1.1" libgdal-netcdf = ">=3.10.3" linopy = ">=0.4.4" lxml = ">=6.0.2" matplotlib = ">=3.10.7" memory_profiler = ">=0.61.0" netcdf4 = ">=1.7.2" networkx = ">=3.5" numpy = ">=1.26.4" openpyxl = ">=3.1.5" pandas = ">=2.1" plotly = ">=6.4.0" powerplantmatching = ">=0.5.15" pre-commit = ">=4.3.0" proj = ">=9.6.2" pylint = ">=4.0.2" pydeck = ">0.6" # pypsa fails to import with pydeck <0.6, lower bound was only added from pypsa 1.0.6 pypsa = ">=0.35.2" pyscipopt = ">=5.6.0" pytables = ">=3.10.2" python = ">=3.10" pytz = ">=2025.2" pyxlsb = ">=1.0.10" rasterio = ">=1.4.3" rioxarray = ">=0.20.0" ruff = ">=0.14.3" scipy = ">=1.16.3" seaborn = ">=0.13.2" shapely = ">=2.0" snakemake-executor-plugin-cluster-generic = ">=1.0.9" snakemake-executor-plugin-slurm = ">=1.9.2" snakemake-minimal = ">=9" snakemake-storage-plugin-http = ">=0.3" tenacity = ">=9.1.2" tqdm = ">=4.67.1" tsam = ">=2.3.1" xarray = ">=2024.3.0,<2025.7.0" xlrd = ">=2.0.2" yaml = ">=0.2.5" snakemake-storage-plugin-cached-http = ">=0.1.0"

[feature.doc.tasks.build-docs]
args = ["dir", {"arg" = "output", "default" = "html"}]
cmd = "dot -c && sphinx-build -T -b {{ output }} doc {{ dir }}/{{ output }} "

[feature.doc.dependencies]
atlite = ">=0.2.9"
cartopy = ">=0.25.0"
dask = ">=2025.10.0"
descartes = ">=1.1.0"
fiona = ">=1.10.1"
graphviz = ">=13.1.2"
matplotlib = ">3.5.1"
memory_profiler = ">=0.61.0"
myst-parser = ">=4.0.1"
powerplantmatching = ">=0.5.5"
pydot = ">=4.0.1,<5"
pypsa = ">=0.35.2"
pytables = ">=3.10.1"
python = ">=3.10.19"
pyyaml = ">=6.0.3"
requests = ">=2.32.5"
scikit-learn = ">=1.7.2"
seaborn = ">=0.13.2"
sphinx = ">=8.1.3"
sphinx-book-theme = ">=1.1.4"
sphinxcontrib-bibtex = ">=2.6.5"
tabula-py = ">=2.7.0"
tenacity = ">=9.1.2"
tsam = ">=2.3.1"

[pworschischek-aggmag]

Hi @BlurakLP, thanks for raising this. I run into the same error in the CI test pipelines over at PyPSA-AT. I didn't investigate yet, so I can't say much more at this point. I'll keep you posted here if I find out more.

I use an adapted up-to-date PyPSA-DE main branch.

edit: cannot reproduce this for PyPSA-AT productive runs using config.at.yaml without the --configfile config/test/config.at10.yaml extensions, and on my devcontainer. So it might be related to the test configs

[lkstrp]

Hey both,

this is a data retrieval problem which is one of the main pain points of all the PyPSA-X workflows. We fixed that in PyPSA-Eur by adding a mirror for all primary data sources and a data dependency layer.

I hope to have this also merged in PyPSA-DE soon.

[pworschischek-aggmag]

Hi @lkstrp, data retrieval is a pain sometimes, indeed. I analysed in the meantime and found that the cached-http plugin is_valid_query denies accessing the egon dataset URL api.opendata.ffe.de.

# snakemake_storage_plugin_cached_http/monkeypatch.py
def is_pypsa_or_zenodo_url(url: str) -> bool:
    parsed = urlparse(url)
    return parsed.netloc in (
        "zenodo.org",
        "sandbox.zenodo.org",
        "data.pypsa.org",
        "storage.googleapis.com",
    ) and parsed.scheme in (
        "http",
        "https",
    )

I rolled back to vanilla retrieval: https://github.com/AGGM-AG/pypsa-at/pull/46/changes#diff-47959dfd378b3cd1d39b5515418ee8e4444ab7a6036d5197a5bea82814f928a3R452

# Snakefile 
rule retrieve_egon_data:
    params:
        url=EGON["url"],
    output:
        spatial="data/egon/demandregio_spatial_2018.json",
        mapping="data/egon/mapping_technologies.json",
    shell:
        """
        mkdir -p data/egon
        curl -o {output.spatial} "{params.url}?id_spatial=5&year=2018"
        curl -o {output.mapping} "{params.url}_description?id_spatial=5"
        """

And the pipeline works again. However, I'd prefer using the plugin for improved workflow robustness. Any suggestions on how to tackle this cleanly?

[pworschischek-aggmag]

Hi @BlurakLP, please try again with up-to-date main branch. It should be solved after todays merge.

[lkstrp]

You are right @pworschischek-aggmag. We had that on some other datasets as well, that cached-http is blocked while curl or a different header works

[BlurakLP]

First of all, thank you very much for looking into it!
I haven't worked very often in collaborative git environments, so to be sure I did it correctly:
git branch -> main
git pull origin main -> pulls the changes

I repeated the testing steps in my initial post and still get the same output errors (unfortunately)

[lkstrp]

@BlurakLP In PyPSA-DE or PyPSA-AT? I think @pworschischek-aggmag was referring to PyPSA-AT. In DE it still needs to be merged, see #159

[BlurakLP]

Ah, I'm talking about PyPSA-DE, sorry.

[lkstrp]

Closed via #158

So now the fix is also in the master of PyPSA-DE @BlurakLP

[BlurakLP]

This works now: snakemake build_scenarios -f
This: snakemake ariadne_all returns out-of-memory (but I guess thats something on my side)

This still fails:
(pypsa-de) ubuntu@pypsa-initialtest:~/pypsa-de$ snakemake -call results/test-elec/networks/base_s_6_elec_.nc --configfile config/config.de.yaml -n
Output:
Using workflow specific profile profiles/default for setting default command line arguments. Config file config/config.default.yaml is extended by additional config specified via the command line. Config file config/plotting.default.yaml is extended by additional config specified via the command line. Config file config/config.de.yaml is extended by additional config specified via the command line. host: pypsa-initialtest Building DAG of jobs... MissingRuleException: No rule to produce results/test-elec/networks/base_s_6_elec_.nc (if you use input functions make sure that they don't raise unexpected exceptions).