CUDA/HIP: support argsort/top-k ncols>1024 on ROCm via hipCUB#14
Closed
DeanoC wants to merge 1 commit into
Closed
Conversation
The GPU argsort/top-k path uses a single-block bitonic sort (block_dims =
ncols_pad), so it caps at 1024 threads/block. On CUDA, ncols > 1024 is handled
by the CUB device-sort path; on HIP that path was compiled out (GGML_CUDA_USE_CUB
is CUDA-only), so ggml_backend_cuda_supports_op reported TOP_K/ARGSORT as
unsupported for ncols > 1024 and they fell back to the CPU reference — 275 of the
test-backend-ops cases (full-vocab sampling, large sorts) on every ROCm build.
Enable the existing device-sort path on HIP via hipCUB (rocPRIM), which provides
the same DeviceRadixSort / DeviceSegmentedSort / DeviceSegmentedRadixSort API:
* common.cuh: define GGML_CUDA_USE_HIPCUB on HIP.
* argsort.cu/.cuh: include <hipcub/hipcub.hpp> and compile argsort_f32_i32_cuda_cub
on HIP too; hipCUB has no CCCL strided iterator, so the init_offsets segment
path is used (already the CUB fallback).
* top-k.cu: on HIP, route ncols > 1024 through the device sort + slice-first-k,
mirroring the CUB branch; the fast partial-bitonic top-k still handles the
common ncols <= 1024 decode/verify case unchanged.
* ggml-cuda.cu: supports_op returns true for TOP_K/ARGSORT when either CUB or
hipCUB is available.
test-backend-ops -o TOP_K and -o ARGSORT: 0 not-supported / 0 fail on gfx1201
(R9700) and gfx1151 (Strix Halo) — the 275 previously-unsupported ncols>1024
cases (up to ne=[262144]) now run on-GPU and are bit-for-bit vs the CPU
reference. Requires the rocm hipcub + rocprim headers (ship with ROCm).
This was referenced Jul 9, 2026
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Internal review PR. Stacked on #13 (base
feat/topk-partial-sort-gfx1201), so the diff below is only the ncols>1024 enablement. Follows up the finding from #13 that N>1024 TOP_K/ARGSORT are unsupported on every ROCm build.Problem
The GPU argsort/top-k uses a single-block bitonic sort (
block_dims = ncols_pad), capped at 1024 threads/block. On CUDA, ncols>1024 is handled by the CUB device-sort path; on HIP that path is compiled out (GGML_CUDA_USE_CUBis CUDA-only), sosupports_opreturnsne[0] <= 1024and larger sorts (full-vocab sampling, etc.) fall back to the CPU reference — 275 test-backend-ops cases on every ROCm build.Fix
Enable the existing device-sort path on HIP via hipCUB (rocPRIM), which provides the same
DeviceRadixSort/DeviceSegmentedSort/DeviceSegmentedRadixSortAPI:common.cuh: defineGGML_CUDA_USE_HIPCUBon HIP.argsort.cu/.cuh:#include <hipcub/hipcub.hpp>and compileargsort_f32_i32_cuda_cubon HIP (no CCCL strided iterator → uses the existinginit_offsetssegment path).top-k.cu: on HIP, route ncols>1024 through the device sort + slice-first-k (mirrors the CUB branch); the fast partial-bitonic top-k still handles the common ncols ≤ 1024 decode/verify case unchanged.ggml-cuda.cu:supports_opreturnstruefor TOP_K/ARGSORT when CUB or hipCUB is present.Validation (
test-backend-ops test)The previously-unsupported ncols>1024 cases (up to ne=[262144]) now run on-GPU and are bit-for-bit vs the CPU reference on both archs. The ncols≤1024 partial-bitonic path (#13) is untouched.
Requires the ROCm
hipcub+rocprimheaders (ship with ROCm; present on our gfx1201 box and lucebox4 gfx1151/opt/rocm).