I have identified two issues, whose fixes are closely related and could potentially be subsumed and addressed in a single PR.
-
Currently, we use the XDG cache specification via platformdirs.user_cache_dir() to store our xbuildenvs if a persistent xbuildenv_path is not specified by the user. This results in multiple stray pyodide-build xbuildenv folders that accumulate in the cache directory and must be manually cleaned up later. The naming structure for these folders is .pyodide-xbuildenv-{pyodide-version}. It would be better to create a single folder to store all xbuildenvs per pyodide-build installation, matching the behaviour of tools such as cibuildwheel, black, conda, and virtualenv, and several other Python software packages. By organising it this way, we would have a dedicated pyodide-build folder for each installation, which would house all the xbuildenvs that it downloads (or has downloaded). I develop and use pyodide-build from multiple locations on my machine, including directly from the repository, which uses VCS versioning and generates .pyodide-xbuildenv folders. Additionally, when using pyodide-recipes, that also installs pyodide-build separately, from the submodule. If I also install pyodide-build from PyPI for use in a downstream project, it creates yet another folder.
-
The situation described in the first point also introduces a slight (narrow, but valid) security concern. For example, if I were to download pyodide-build 0.34.5 from PyPI in one virtual environment and then do the same in another virtual environment for a different project, both environments would reuse the same 0.34.5-suffixed folder, which ideally should be individual to each installation. To address this, I suggest appending a UUID to the folder name, in conjunction with the first point. This way, the xbuildenvs would be stored in a location of the form Users/username/Caches/pyodide-build-xbuildenvs-<UUID>/314.0.0a2 (or 0.29.4, etc.). However, I am unsure how the subsequent Python session from the same pyodide-build installation would recognise that specific UUID and not others from different installations. Is such a thing possible if we use UUIDs? Or would we need to embed the path to the pyodide-build installation in a marker file within its dedicated cache folder and read from there?
I have identified two issues, whose fixes are closely related and could potentially be subsumed and addressed in a single PR.
Currently, we use the XDG cache specification via
platformdirs.user_cache_dir()to store our xbuildenvs if a persistentxbuildenv_pathis not specified by the user. This results in multiple stray pyodide-build xbuildenv folders that accumulate in the cache directory and must be manually cleaned up later. The naming structure for these folders is.pyodide-xbuildenv-{pyodide-version}. It would be better to create a single folder to store all xbuildenvs per pyodide-build installation, matching the behaviour of tools such as cibuildwheel, black, conda, and virtualenv, and several other Python software packages. By organising it this way, we would have a dedicated pyodide-build folder for each installation, which would house all the xbuildenvs that it downloads (or has downloaded). I develop and use pyodide-build from multiple locations on my machine, including directly from the repository, which uses VCS versioning and generates.pyodide-xbuildenvfolders. Additionally, when using pyodide-recipes, that also installs pyodide-build separately, from the submodule. If I also install pyodide-build from PyPI for use in a downstream project, it creates yet another folder.The situation described in the first point also introduces a slight (narrow, but valid) security concern. For example, if I were to download pyodide-build 0.34.5 from PyPI in one virtual environment and then do the same in another virtual environment for a different project, both environments would reuse the same 0.34.5-suffixed folder, which ideally should be individual to each installation. To address this, I suggest appending a UUID to the folder name, in conjunction with the first point. This way, the xbuildenvs would be stored in a location of the form
Users/username/Caches/pyodide-build-xbuildenvs-<UUID>/314.0.0a2(or0.29.4, etc.). However, I am unsure how the subsequent Python session from the same pyodide-build installation would recognise that specific UUID and not others from different installations. Is such a thing possible if we use UUIDs? Or would we need to embed the path to the pyodide-build installation in a marker file within its dedicated cache folder and read from there?