You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
add an opt-in COLI_CUDA_WARP_DECODE=1 path for low-row INT4 expert decode on NVIDIA
map one output neuron to one 32-lane warp and eight adjacent outputs to each block
preserve the existing 256-thread logical accumulation and reduction order bit-for-bit
keep HIP on the portable block kernel through the backend capability shim
Validation
make -C c test-c
make -C c cuda-test CUDA=1 COLI_ANS=1 on RTX 5090: pass
CUDA unit test compares WarpDecode and block outputs with memcmp
GLM-5.2 six-GPU A/B used the same 10,628-expert ANS sidecar, prompt, greedy sampling, routing, and 13,757 VRAM-served calls
generated text and routing were identical
path
decode
expert matmul
block control
4.11 tok/s
4.871 s
WarpDecode
4.49 tok/s
4.340 s
change
+9.2%
-10.9%
Dependency
This is intentionally a separate stacked PR on top of #651. It should be reviewed/merged after #651; once that lands, GitHub will reduce this PR to the WarpDecode commit only.
Closing this draft after the broader balanced validation.
The CUDA correctness path remains bit-exact against the block kernel and CI is
green, but the original end-to-end 4.11 -> 4.49 tok/s result did not survive
controlled repeated testing. Later balanced runs were dominated by host
memory-bandwidth variation; the medians did not show a stable WarpDecode gain
and in one balanced series favored the existing block path (8.41 vs 7.56 tok/s).
The branch is also stacked on the pre-rebase commits of #651 and is now 19
commits behind current dev. Rebasing it only to preserve an optional kernel
without a demonstrated end-to-end benefit would add maintenance cost without
a supported performance claim.
If a future workload shows a repeatable low-row benefit, the two WarpDecode
commits can be rebuilt cleanly on current dev with a fresh ABBA benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
COLI_CUDA_WARP_DECODE=1path for low-row INT4 expert decode on NVIDIAValidation
make -C c test-cmake -C c cuda-test CUDA=1 COLI_ANS=1on RTX 5090: passmemcmpDependency
This is intentionally a separate stacked PR on top of #651. It should be reviewed/merged after #651; once that lands, GitHub will reduce this PR to the WarpDecode commit only.