Skip to content

Python wheel links cuBLAS/cuSOLVER/cuSPARSE but neither bundles nor declares them — import cuvslam fails on a clean install #70

Description

@behnamasadi

System setup

  • Hardware: x86_64 workstation, NVIDIA GeForce RTX 3090
  • Driver 580.142, CUDA 12.9 runtime
  • Debian trixie (glibc 2.41), Python 3.12, inside a container
  • Wheel: cuvslam-17.0.0+cu12-cp312-abi3-manylinux_2_39_x86_64.whl (v17.0.0)

Description

TL;DR — the released wheel links cuBLAS/cuSOLVER/cuSPARSE but neither
bundles nor declares them, so import cuvslam fails on a clean install.
I've root-caused it and have a fix ready to go — say the word and I'll fork
and open the PR today.
Details below.

Installing the released v17.0.0 wheel succeeds, but importing it fails:

$ pip install "https://github.com/nvidia-isaac/cuVSLAM/releases/download/v17.0.0/cuvslam-17.0.0%2Bcu12-cp312-abi3-manylinux_2_39_x86_64.whl"
Successfully installed cuvslam-17.0.0+cu12

$ python3 -c "import cuvslam"
ImportError: libcusolver.so.11: cannot open shared object file: No such file or directory

The root cause looks like an inconsistency in how the wheel is vendored: it
bundles one of the CUDA libraries it links against, but not the other three, and
does not declare them as dependencies either.

libcuvslam.so needs four CUDA libraries:

$ readelf -d cuvslam/libcuvslam.so | grep NEEDED | grep -iE "cu|nv"
 NEEDED  libcudart-09529672.so.12.6.77
 NEEDED  libcusolver.so.11
 NEEDED  libcusparse.so.12
 NEEDED  libcublas.so.12

Only the first is vendored — cuvslam.libs/ contains exactly one file, and the
RPATH points there:

$ ls cuvslam.libs/
libcudart-09529672.so.12.6.77

$ readelf -d cuvslam/libcuvslam.so | grep RPATH
 RPATH  [$ORIGIN/../cuvslam.libs]

So the auditwheel vendoring path is working; cuBLAS/cuSOLVER/cuSPARSE appear to
be deliberately excluded, which is very reasonable given their size — the wheel
is already 112 MB. The issue is that nothing was declared to compensate. The
wheel's complete dependency list is:

$ unzip -p cuvslam-17.0.0+cu12-*.whl "*.dist-info/METADATA" | grep -i "^Requires"
Requires-Python: >=3.9
Requires-Dist: pyyaml>=5.3.1

So on any environment that does not already provide the CUDA math libraries, the
dependency is invisible to pip and the import fails.

Why this may not be showing up in CI

scripts/test_pycuvslam_in_docker.sh (invoked by pr-verify.yml) installs from
the source tree rather than from the built wheel, and runs inside cuvslam:local,
which is a CUDA devel image where cuBLAS/cuSOLVER/cuSPARSE are present anyway:

docker run --runtime=nvidia --gpus all --rm ... cuvslam:local bash -c '
    pip install /cuvslam/python/
    python3 -m unittest discover -v -s /cuvslam/python/test ...'

Both conditions needed to surface this — installing the released wheel, and doing
so where the CUDA toolkit is absent — are outside the current test path.

Workaround (confirmed working)

pip install nvidia-cusolver-cu12 nvidia-cusparse-cu12 nvidia-cublas-cu12
SP=$(python3 -c "import sysconfig; print(sysconfig.get_paths()['purelib'])")/nvidia
export LD_LIBRARY_PATH=$SP/cusolver/lib:$SP/cusparse/lib:$SP/cublas/lib:$LD_LIBRARY_PATH
python3 -c "import cuvslam; print('IMPORT OK')"
IMPORT OK

Possible fixes

  1. Declare the libraries in python/pyproject.toml (currently
    dependencies = ["pyyaml>=5.3.1"]), the approach PyTorch uses. The wrinkle is
    that the cu12 and cu13 variants need different packages
    (nvidia-cublas-cu12 vs nvidia-cublas-cu13), so the list would need to be
    driven by the build variant rather than hard-coded — I don't know how you'd
    prefer to wire that into the scikit-build-core setup, which is really why I'm
    filing this rather than sending a patch.
  2. Alternatively, set the RPATH to also search the pip layout
    ($ORIGIN/../nvidia/*/lib) so the wheel-provided libraries resolve without
    LD_LIBRARY_PATH.
  3. At minimum, documenting the requirement would save people the debugging round
    trip.

It might also be worth a small CI step that pip installs the built wheel in
a clean image (a plain python:3.12, or a CUDA runtime rather than devel
image) and does nothing but import cuvslam. That would catch this class of
packaging regression permanently, and it's cheap.

I'm offering to do all of this for you.

I've already done the investigation, so the implementation is the easy part and
I'm glad to take it off your plate. Say the word and I'll fork the repo and open
a PR today containing:

  • the dependency declarations in python/pyproject.toml, wired to the build
    variant so cu12 and cu13 each get the right nvidia-* packages — whichever
    mechanism you'd prefer, just tell me and I'll follow it;
  • the CI smoke test above: install the built wheel in a clean image, import cuvslam, done;
  • a short note in the install docs, if you want it.

Equally happy to do only a subset, or to hand the analysis over and let someone
on your side implement it — whatever is least work for you.

One thing worth asking directly, so I don't waste your review time: I noticed
essentially all merged PRs here are from NVIDIA-authored accounts, and a few
community PRs have been open for some months. If external contributions aren't
something the project takes — completely understandable for a repo that mirrors
an internal tree — just say so and I'll leave the report here for your team to
pick up, no hard feelings. But if you do accept PRs from forks, I'd genuinely
like to send this one, and I'll turn it around quickly.

Thanks for publishing the library — the rest of the integration went smoothly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions