feat(python): Update x-plat lib so that python bindings can be published to pypi and more easily built/installed locally#665
Open
finger563 wants to merge 1 commit into
Open
Conversation
…hed to pypi and more easily built/installed locally
|
✅Static analysis result - no issues found! ✅ |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+19
to
+50
| name: Build wheels on ${{ matrix.os }} | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: | ||
| - os: ubuntu-latest # linux x86_64 | ||
| - os: ubuntu-24.04-arm # linux aarch64 | ||
| - os: macos-13 # macos x86_64 | ||
| - os: macos-latest # macos arm64 | ||
| - os: windows-latest # windows amd64 | ||
|
|
||
| steps: | ||
| - name: Checkout repo | ||
| uses: actions/checkout@v7 | ||
| with: | ||
| submodules: 'recursive' | ||
| # full history + tags so setuptools-scm can compute the version | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Build wheels | ||
| run: pipx run cibuildwheel | ||
| # configuration (python versions, test command, etc.) lives in | ||
| # [tool.cibuildwheel] in pyproject.toml | ||
|
|
||
| - name: Upload wheels | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: wheels-${{ matrix.os }} | ||
| path: wheelhouse/*.whl | ||
|
|
||
| build_sdist: |
Comment on lines
+51
to
+70
| name: Build source distribution | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout repo | ||
| uses: actions/checkout@v7 | ||
| with: | ||
| submodules: 'recursive' | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Build sdist | ||
| run: pipx run build --sdist | ||
|
|
||
| - name: Upload sdist | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: sdist | ||
| path: dist/*.tar.gz | ||
|
|
||
| publish_pypi: |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR restructures the repository’s Python packaging/build so the cross-platform espp bindings can be installed via pip (from PyPI or directly from Git) and built more reliably locally, while also addressing Python deadlocks by releasing the GIL around blocking binding calls.
Changes:
- Introduces repo-root
pyproject.toml(scikit-build-core + setuptools-scm) and makesesppa proper Python package with a compiledespp._esppextension that’s re-exported fromespp/__init__.py. - Updates/cleans Python example scripts to
import esppdirectly (removessupport_loader.pyshim). - Adds automated wheel/sdist build + release publishing workflow, and updates bindings generation/checked-in bindings to add GIL-release guards on blocking methods.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents pip install espp / pip install git+… availability for x-plat subset. |
| python/udp_server.py | Switches examples from support_loader to import espp. |
| python/udp_client.py | Switches examples from support_loader to import espp. |
| python/timer.py | Switches examples from support_loader to import espp. |
| python/task.py | Switches examples from support_loader to import espp. |
| python/support_loader.py | Removes sys.path shim now that package is pip-installable. |
| python/rtsp_server_multitrack.py | Switches examples from support_loader to import espp. |
| python/rtsp_client_multitrack.py | Switches examples from support_loader to import espp. |
| python/rtsp_api_test.py | Switches examples from support_loader to import espp. |
| python/rtps_subscriber.py | Switches examples from support_loader to import espp. |
| python/rtps_pubsub.py | Switches examples from support_loader to import espp. |
| python/rtps_publisher.py | Switches examples from support_loader to import espp. |
| python/README.md | Updates setup instructions for PyPI + editable installs. |
| python/cobs_demo.py | Switches example from support_loader to import espp. |
| pyproject.toml | Adds root packaging config for scikit-build-core, cibuildwheel, setuptools-scm. |
| lib/README.md | Documents pip install paths + editable installs + wheel CI. |
| lib/README_PYPI.md | Adds PyPI-focused README for package description/usage. |
| lib/python_bindings/pybind_espp.cpp | Adds py::gil_scoped_release call guards to blocking defs. |
| lib/python_bindings/module.cpp | Renames extension module to _espp (for espp._espp). |
| lib/python_bindings/espp/init.py | Re-exports compiled extension; adds package docstring. |
| lib/pyproject.toml | Removes obsolete lib/pyproject.toml that produced broken wheels. |
| lib/espp.cmake | Refactors python binding target creation/installation for package layout + version embedding. |
| lib/CMakeLists.txt | Adds SKBUILD-aware flow and prefers installed pybind11 with CPM fallback. |
| lib/bind.cpp | Removes dead legacy bindings source. |
| lib/autogenerate_bindings.py | Adds postprocess pass to auto-append GIL-release guards; lazy-imports litgen. |
| .github/workflows/build_wheels.yml | Adds wheel/sdist build workflow and PyPI publishing on releases. |
Comment on lines
+8
to
+9
| from espp._espp import * # type: ignore # noqa: F403 | ||
| from espp._espp import __version__ # noqa: F401 |
Comment on lines
+48
to
+50
| espp_add_python_module() | ||
| install(TARGETS _espp LIBRARY DESTINATION espp) | ||
| else() |
Comment on lines
+146
to
+147
| install(TARGETS _espp | ||
| LIBRARY DESTINATION ${FOLDER}/espp/) |
Comment on lines
+9
to
+12
| readme = "lib/README_PYPI.md" | ||
| license = "MIT" | ||
| license-files = ["LICENSE"] | ||
| authors = [{ name = "William Emfinger", email = "waemfinger@gmail.com" }] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Restructure the python packaging so
pip install espp(PyPI) andpip install git+https://github.com/esp-cpp/espp.gitboth work:refactor(python): Use editable pip install instead of support_loader
Now that the espp python package is pip-installable, switch the example scripts to a plain
import esppand remove the support_loader.py sys.path shim. The documented developer workflow ispip install -e .from the repo root (orpip install esppfor releases). Set a persistent CMake build-dir in pyproject.toml so re-running the editable install after C++ changes rebuilds incrementally (~45s vs ~4-5min from scratch).fix(lib): Release the GIL in blocking python binding methods
litgen is now imported lazily so the postprocess passes can be reapplied without a litgen environment.
Motivation and Context
Calling Task.stop() (or any binding method that joins the task thread or blocks on I/O) from python deadlocked: the caller held the GIL while joining, and the task thread was waiting for the GIL to invoke its python callback. Add a _add_gil_release_guards postprocess pass to autogenerate_bindings.py that appends py::call_guardpy::gil_scoped_release() to the .def(...) of every blocking method (Task/Timer start/stop/cancel, FtpServer start/stop, Tcp/UdpSocket connect/accept/send/receive, RtspClient request methods, RtspServer start/stop/send_frame, RtspSession packet sends - 42 defs), matching the guard the hand-written rtps bindings already use. The pass is string-literal-aware, idempotent, and applied to the checked-in pybind_espp.cpp.
How has this been tested?
wheel build + import/API smoke test, sdist -> wheel build (the no-prebuilt-wheel pip path, 4.4MB sdist with only the needed submodules), and the standalone lib/build.sh flow.
fresh-venv
pip install -e ., task.py and timer.py run cleanly,and an incremental reinstall after touching task.cpp.
wheel rebuild + test stopping a running Task and Timer from python (previously a hard deadlock, confirmed via stack sample).
Screenshots (if appropriate, e.g. schematic, board, console logs, lab pictures):
Types of changes
Checklist:
Software
.github/workflows/build.ymlfile to add my new test to the automated cloud build github action.