GPU backend: failure-path hardening + tests (sticky-error fix, cached-tensor contract, fault-injection hook)#338
Open
noobdev-ph wants to merge 1 commit into
Open
Conversation
Three vendor-neutral fixes to backend_cuda.cu, each with test coverage: 1. Upload check-order: a cached device tensor is now usable when the caller's host pointers are stale or NULL. CUDA_RELEASE_HOST slots null their host pointers after upload; the current engine reaches those tensors through direct handles (coli_cuda_expert_mlp etc.), but any caller going through coli_cuda_matmul/tensor_upload with a cached tensor — as matmul_qt does for QT tensors — hits the !weights check before the cached-tensor branch and fails spuriously. This hardens the API contract rather than fixing a measured regression; the contract is pinned by a 64x sustained-reuse test. 2. Sticky runtime error (real bug, test-caught): a failed allocation left the last-error state set, so the NEXT healthy launch's cudaGetLastError() check reported 'out of memory' and disabled a perfectly good tensor. cuda_ok() now consumes the error on the failure path; regression-covered. 3. COLI_GPU_FAIL_AFTER=N test hook: every GPU compute entry point (19 total: matmul, expert mlp/group, shared mlp, attention ops, pipe ops) reports failure after N successful calls, so the engine's CPU fallbacks and expert_host_ensure rematerialization can be exercised end-to-end without real hardware faults. Unset = zero effect; uploads/queries are never gated. Validated on GLM-5.2: total failure (N=0) completes coherently with every released expert rematerialized. Tests (run via make cuda-test on any CUDA GPU; vendor-neutral source): 64x sustained matmul reuse after host pointers are freed; upload from a scribbled-and-freed temporary; five graceful upload-failure cases with stats-integrity assertions; healthy-launch-after-failed-alloc (the sticky-error regression); fault-hook on/off restore. Verified on AMD RX 9070 XT via the companion HIP PR's compat header (same test source); a make cuda-test run on NVIDIA hardware would complete the matrix.
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The vendor-neutral half of #112, rebuilt against
devas requested — the smallest version that does one thing: failure-path hardening forbackend_cuda.cu, with tests your CI compiles and your hardware can execute.Three changes (2 files, +87/−4):
cudaGetLastError()check reportsout of memoryand permanently disables a good tensor. One failed upload could cascade into disabling valid GPU tensors.cuda_ok()now consumes the error on its failure path; regression-covered.CUDA_RELEASE_HOSTslot lifecycle). API-contract hardening, pinned by a 64× sustained-reuse test — stated plainly: the current engine reaches released tensors via direct handles, so this is defensive, not a measured regression fix.COLI_GPU_FAIL_AFTER=Nfault hook: all 19 compute entry points report failure after N successes (unset = zero effect, uploads/queries never gated). This is what lets the repair machinery be tested: withN=0on GLM-5.2, every GPU call fails, every released expert rematerializes viaexpert_host_ensure, and generation completes coherently. It also surfaced two behaviors worth knowing: MTP speculation disengages entirely under total GPU failure (graceful but silent — maybe worth a log line), and outputs are deterministic given a frozen.coli_usage(pin membership feeds grouped-path selection, so runs diverge as the learning cache evolves — expected, but subtle when comparing runs).Validation
make -C c check(CPU untouched)engine-cuda-syntaxCI job compiles both filesmake cuda-teston NVIDIA hardware — this is the run only you can do, and it completes the verification matrixCompatibility
glm.c) changes at allCompanion PR (independent, either merges first): the HIP single-source port.