System setup
- Hardware: x86_64 workstation, NVIDIA GeForce RTX 3090
- Driver 580.142, CUDA 12.9 runtime
- Debian trixie (glibc 2.41), Python 3.12, inside a container
- Wheel:
cuvslam-17.0.0+cu12-cp312-abi3-manylinux_2_39_x86_64.whl (v17.0.0)
Description
TL;DR — the released wheel links cuBLAS/cuSOLVER/cuSPARSE but neither
bundles nor declares them, so import cuvslam fails on a clean install.
I've root-caused it and have a fix ready to go — say the word and I'll fork
and open the PR today. Details below.
Installing the released v17.0.0 wheel succeeds, but importing it fails:
$ pip install "https://github.com/nvidia-isaac/cuVSLAM/releases/download/v17.0.0/cuvslam-17.0.0%2Bcu12-cp312-abi3-manylinux_2_39_x86_64.whl"
Successfully installed cuvslam-17.0.0+cu12
$ python3 -c "import cuvslam"
ImportError: libcusolver.so.11: cannot open shared object file: No such file or directory
The root cause looks like an inconsistency in how the wheel is vendored: it
bundles one of the CUDA libraries it links against, but not the other three, and
does not declare them as dependencies either.
libcuvslam.so needs four CUDA libraries:
$ readelf -d cuvslam/libcuvslam.so | grep NEEDED | grep -iE "cu|nv"
NEEDED libcudart-09529672.so.12.6.77
NEEDED libcusolver.so.11
NEEDED libcusparse.so.12
NEEDED libcublas.so.12
Only the first is vendored — cuvslam.libs/ contains exactly one file, and the
RPATH points there:
$ ls cuvslam.libs/
libcudart-09529672.so.12.6.77
$ readelf -d cuvslam/libcuvslam.so | grep RPATH
RPATH [$ORIGIN/../cuvslam.libs]
So the auditwheel vendoring path is working; cuBLAS/cuSOLVER/cuSPARSE appear to
be deliberately excluded, which is very reasonable given their size — the wheel
is already 112 MB. The issue is that nothing was declared to compensate. The
wheel's complete dependency list is:
$ unzip -p cuvslam-17.0.0+cu12-*.whl "*.dist-info/METADATA" | grep -i "^Requires"
Requires-Python: >=3.9
Requires-Dist: pyyaml>=5.3.1
So on any environment that does not already provide the CUDA math libraries, the
dependency is invisible to pip and the import fails.
Why this may not be showing up in CI
scripts/test_pycuvslam_in_docker.sh (invoked by pr-verify.yml) installs from
the source tree rather than from the built wheel, and runs inside cuvslam:local,
which is a CUDA devel image where cuBLAS/cuSOLVER/cuSPARSE are present anyway:
docker run --runtime=nvidia --gpus all --rm ... cuvslam:local bash -c '
pip install /cuvslam/python/
python3 -m unittest discover -v -s /cuvslam/python/test ...'
Both conditions needed to surface this — installing the released wheel, and doing
so where the CUDA toolkit is absent — are outside the current test path.
Workaround (confirmed working)
pip install nvidia-cusolver-cu12 nvidia-cusparse-cu12 nvidia-cublas-cu12
SP=$(python3 -c "import sysconfig; print(sysconfig.get_paths()['purelib'])")/nvidia
export LD_LIBRARY_PATH=$SP/cusolver/lib:$SP/cusparse/lib:$SP/cublas/lib:$LD_LIBRARY_PATH
python3 -c "import cuvslam; print('IMPORT OK')"
Possible fixes
- Declare the libraries in
python/pyproject.toml (currently
dependencies = ["pyyaml>=5.3.1"]), the approach PyTorch uses. The wrinkle is
that the cu12 and cu13 variants need different packages
(nvidia-cublas-cu12 vs nvidia-cublas-cu13), so the list would need to be
driven by the build variant rather than hard-coded — I don't know how you'd
prefer to wire that into the scikit-build-core setup, which is really why I'm
filing this rather than sending a patch.
- Alternatively, set the RPATH to also search the pip layout
($ORIGIN/../nvidia/*/lib) so the wheel-provided libraries resolve without
LD_LIBRARY_PATH.
- At minimum, documenting the requirement would save people the debugging round
trip.
It might also be worth a small CI step that pip installs the built wheel in
a clean image (a plain python:3.12, or a CUDA runtime rather than devel
image) and does nothing but import cuvslam. That would catch this class of
packaging regression permanently, and it's cheap.
I'm offering to do all of this for you.
I've already done the investigation, so the implementation is the easy part and
I'm glad to take it off your plate. Say the word and I'll fork the repo and open
a PR today containing:
- the dependency declarations in
python/pyproject.toml, wired to the build
variant so cu12 and cu13 each get the right nvidia-* packages — whichever
mechanism you'd prefer, just tell me and I'll follow it;
- the CI smoke test above: install the built wheel in a clean image,
import cuvslam, done;
- a short note in the install docs, if you want it.
Equally happy to do only a subset, or to hand the analysis over and let someone
on your side implement it — whatever is least work for you.
One thing worth asking directly, so I don't waste your review time: I noticed
essentially all merged PRs here are from NVIDIA-authored accounts, and a few
community PRs have been open for some months. If external contributions aren't
something the project takes — completely understandable for a repo that mirrors
an internal tree — just say so and I'll leave the report here for your team to
pick up, no hard feelings. But if you do accept PRs from forks, I'd genuinely
like to send this one, and I'll turn it around quickly.
Thanks for publishing the library — the rest of the integration went smoothly.
System setup
cuvslam-17.0.0+cu12-cp312-abi3-manylinux_2_39_x86_64.whl(v17.0.0)Description
Installing the released v17.0.0 wheel succeeds, but importing it fails:
The root cause looks like an inconsistency in how the wheel is vendored: it
bundles one of the CUDA libraries it links against, but not the other three, and
does not declare them as dependencies either.
libcuvslam.soneeds four CUDA libraries:Only the first is vendored —
cuvslam.libs/contains exactly one file, and theRPATH points there:
So the auditwheel vendoring path is working; cuBLAS/cuSOLVER/cuSPARSE appear to
be deliberately excluded, which is very reasonable given their size — the wheel
is already 112 MB. The issue is that nothing was declared to compensate. The
wheel's complete dependency list is:
So on any environment that does not already provide the CUDA math libraries, the
dependency is invisible to pip and the import fails.
Why this may not be showing up in CI
scripts/test_pycuvslam_in_docker.sh(invoked bypr-verify.yml) installs fromthe source tree rather than from the built wheel, and runs inside
cuvslam:local,which is a CUDA devel image where cuBLAS/cuSOLVER/cuSPARSE are present anyway:
Both conditions needed to surface this — installing the released wheel, and doing
so where the CUDA toolkit is absent — are outside the current test path.
Workaround (confirmed working)
Possible fixes
python/pyproject.toml(currentlydependencies = ["pyyaml>=5.3.1"]), the approach PyTorch uses. The wrinkle isthat the
cu12andcu13variants need different packages(
nvidia-cublas-cu12vsnvidia-cublas-cu13), so the list would need to bedriven by the build variant rather than hard-coded — I don't know how you'd
prefer to wire that into the scikit-build-core setup, which is really why I'm
filing this rather than sending a patch.
(
$ORIGIN/../nvidia/*/lib) so the wheel-provided libraries resolve withoutLD_LIBRARY_PATH.trip.
It might also be worth a small CI step that
pip installs the built wheel ina clean image (a plain
python:3.12, or a CUDAruntimerather thandevelimage) and does nothing but
import cuvslam. That would catch this class ofpackaging regression permanently, and it's cheap.
I'm offering to do all of this for you.
I've already done the investigation, so the implementation is the easy part and
I'm glad to take it off your plate. Say the word and I'll fork the repo and open
a PR today containing:
python/pyproject.toml, wired to the buildvariant so
cu12andcu13each get the rightnvidia-*packages — whichevermechanism you'd prefer, just tell me and I'll follow it;
import cuvslam, done;Equally happy to do only a subset, or to hand the analysis over and let someone
on your side implement it — whatever is least work for you.
One thing worth asking directly, so I don't waste your review time: I noticed
essentially all merged PRs here are from NVIDIA-authored accounts, and a few
community PRs have been open for some months. If external contributions aren't
something the project takes — completely understandable for a repo that mirrors
an internal tree — just say so and I'll leave the report here for your team to
pick up, no hard feelings. But if you do accept PRs from forks, I'd genuinely
like to send this one, and I'll turn it around quickly.
Thanks for publishing the library — the rest of the integration went smoothly.