Skip to content

cuda: add bit-exact warp decode path for low-row INT4 - #652

Closed
ZacharyZcR wants to merge 7 commits into
JustVugg:devfrom
ZacharyZcR:exp/warpdecode
Closed

cuda: add bit-exact warp decode path for low-row INT4#652
ZacharyZcR wants to merge 7 commits into
JustVugg:devfrom
ZacharyZcR:exp/warpdecode

Conversation

@ZacharyZcR

Copy link
Copy Markdown
Contributor

Summary

  • add an opt-in COLI_CUDA_WARP_DECODE=1 path for low-row INT4 expert decode on NVIDIA
  • map one output neuron to one 32-lane warp and eight adjacent outputs to each block
  • preserve the existing 256-thread logical accumulation and reduction order bit-for-bit
  • keep HIP on the portable block kernel through the backend capability shim

Validation

  • make -C c test-c
  • make -C c cuda-test CUDA=1 COLI_ANS=1 on RTX 5090: pass
  • CUDA unit test compares WarpDecode and block outputs with memcmp
  • GLM-5.2 six-GPU A/B used the same 10,628-expert ANS sidecar, prompt, greedy sampling, routing, and 13,757 VRAM-served calls
  • generated text and routing were identical
path decode expert matmul
block control 4.11 tok/s 4.871 s
WarpDecode 4.49 tok/s 4.340 s
change +9.2% -10.9%

Dependency

This is intentionally a separate stacked PR on top of #651. It should be reviewed/merged after #651; once that lands, GitHub will reduce this PR to the WarpDecode commit only.

@ZacharyZcR

Copy link
Copy Markdown
Contributor Author

Closing this draft after the broader balanced validation.

The CUDA correctness path remains bit-exact against the block kernel and CI is
green, but the original end-to-end 4.11 -> 4.49 tok/s result did not survive
controlled repeated testing. Later balanced runs were dominated by host
memory-bandwidth variation; the medians did not show a stable WarpDecode gain
and in one balanced series favored the existing block path (8.41 vs 7.56 tok/s).

The branch is also stacked on the pre-rebase commits of #651 and is now 19
commits behind current dev. Rebasing it only to preserve an optional kernel
without a demonstrated end-to-end benefit would add maintenance cost without
a supported performance claim.

If a future workload shows a repeatable low-row benefit, the two WarpDecode
commits can be rebuilt cleanly on current dev with a fresh ABBA benchmark.

@ZacharyZcR ZacharyZcR closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant