fix: dlsym hook self-lookup degrades to RTLD_DEFAULT, crashing cuDNN 9 apps (silent SIGSEGV at first convolution)#215
Conversation
…9 apps When libvgpu.so is preloaded from a path other than the hardcoded /usr/local/vgpu/libvgpu.so (and CUDA_REDIRECT is unset), the dlopen for the self-handle fails and vgpulib stays NULL. The dlsym hook then calls real_dlsym(NULL, symbol) for every "cu"-prefixed symbol, which glibc treats as RTLD_DEFAULT: a global-scope search. That prefix also matches CUDA *library* symbols, not just driver API. cuDNN 9 is a dispatcher architecture: libcudnn.so.9's cudnnCreate resolves the real engine implementation via dlsym at first use. The global-scope search hands the dispatcher its OWN cudnnCreate export back, and the resulting self-call recurses until stack overflow -- a silent SIGSEGV with no HAMI-core diagnostics. Any LibTorch >= 2.5 cu12x application dies at its first convolution (TorchScript or eager); with cuDNN 8 stacks the same mis-resolution can surface as a livelock instead. Related report: Project-HAMi#169 (vllm 0.18 = torch 2.7 + cu128). Fix, three parts: 1. Recover the self-handle via dladdr on a known own symbol and dlopen(dli_fname, RTLD_LAZY | RTLD_NOLOAD) when the hardcoded path misses, so the lookup is always against libvgpu itself. 2. If the self-handle is still NULL, treat the lookup as a miss and fall through to real_dlsym(handle, symbol) with the caller's own handle -- never RTLD_DEFAULT. 3. Fire preInit only after a confirmed hook hit, so non-driver "cu"-prefixed symbols (cudnnCreate, cublas*, ...) no longer trigger CUDA pre-initialization inside the caller's dynamic-loader call. Verified: a LibTorch 2.7.1+cu128 stereo-depth application that previously segfaulted at its first convolution under every memory cap (including with LD_PRELOAD alone and no cap set) now runs to completion with caps enforced from 12 GB down to 1 GB; a LibTorch 2.4.1+cu118 build of the same application that previously livelocked now completes. No regression on previously-working workloads: PyTorch cu118 and C++ LibTorch cu118 pipelines both pass with unchanged accuracy under the patched library. Deployments using the standard /usr/local/vgpu install path are unaffected by construction (the new code path never executes).
|
Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits. 📝 Please follow instructions in the contributing guide to update your commits with the DCO Full details of the Developer Certificate of Origin can be found at developercertificate.org. The list of commits missing DCO signoff:
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mohhef The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @mohhef! It looks like this is your first PR to Project-HAMi/HAMi-core 🎉 |
📝 WalkthroughWalkthroughThe Changeslibvgpu dlsym Recovery and preInit Reordering
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant dlsym
participant dladdr
participant vgpulib
participant preInit
Caller->>dlsym: dlsym(handle, symbol)
alt vgpulib is NULL
dlsym->>dladdr: dladdr(init_dlsym)
dladdr-->>dlsym: loaded library path
dlsym->>vgpulib: dlopen(path, RTLD_NOLOAD)
end
alt symbol starts with "cu"
dlsym->>vgpulib: real_dlsym(vgpulib, symbol)
alt symbol found and not cuGetExportTable
dlsym->>preInit: pthread_once(preInit)
end
dlsym-->>Caller: return resolved or fallback symbol
else
dlsym->>dlsym: real_dlsym(handle, symbol)
dlsym-->>Caller: return symbol
end
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
1 participant
Symptom
Any LibTorch >= 2.5 cu12x application preloaded with
libvgpu.sofrom a path other than/usr/local/vgpu/libvgpu.sodies with a silent SIGSEGV at its first convolution (no HAMI-core error banner), at everyCUDA_DEVICE_MEMORY_LIMITincluding very loose ones, and even withLD_PRELOADset but no limit configured at all. On cuDNN 8 stacks (torch cu118) the same defect can surface as a livelock instead of a crash. This looks related to the reports in #169 (vllm 0.18 = torch 2.7 + cu128).gdb backtrace at the crash (20 identical frames — infinite self-recursion until stack overflow):
Root cause
In
src/libvgpu.c, the dlsym hook resolves every"cu"-prefixed symbol against a libvgpu self-handle:When libvgpu is preloaded from any other location (bind-mounted container paths,
/opt/hami/..., etc.) andCUDA_REDIRECTis unset,vgpulibis NULL — and on glibc,dlsym(NULL, ...)means RTLD_DEFAULT: a global-scope search.The
"cu"prefix also matches CUDA library symbols, not just driver API. cuDNN 9 is a dispatcher architecture:libcudnn.so.9'scudnnCreatelazily resolves the real engine implementation viadlsymat first use. The interposed global-scope search hands the dispatcher its owncudnnCreateexport back, and the self-call recurses until the stack overflows. Additionally, the unconditionalpthread_once(preInit)for anycu*symbol runs full CUDA pre-initialization inside the caller's dynamic-loader call (the caller is mid-dlopenof cuDNN at that moment).Fix (three small parts, one function)
dladdron a known own symbol,dlopen(dli_fname, RTLD_LAZY | RTLD_NOLOAD), when the hardcoded path misses — the lookup is then always genuinely against libvgpu.real_dlsym(handle, symbol)with the caller's own handle.preInitonly on a confirmed hook hit, so non-drivercu*symbols (cudnnCreate,cublas*,cufft*, ...) no longer trigger CUDA pre-initialization inside the loader. (Hooks self-initialize viaensure_post_init, so deferral is safe.)Deployments using the standard
/usr/local/vgpuinstall path are unaffected by construction: the new code paths never execute there.Verification
/opt/hami/nvidia-smibinding sanity, 2048 MiB capPossible follow-up (not in this PR)
The underlying discrimination mechanism ("prefix-match
cu*, resolve against own exports") could instead gate on the explicit hook table (__dlsym_hook_section), which would also make the behavior independent of ELF export visibility. That requires reworking the table's pre-init early-return, so it is left as a follow-up.Summary by CodeRabbit