Skip to content

CUDA/HIP: support argsort/top-k ncols>1024 on ROCm via hipCUB#14

Closed
DeanoC wants to merge 1 commit into
feat/topk-partial-sort-gfx1201from
feat/topk-hip-large-n
Closed

CUDA/HIP: support argsort/top-k ncols>1024 on ROCm via hipCUB#14
DeanoC wants to merge 1 commit into
feat/topk-partial-sort-gfx1201from
feat/topk-hip-large-n

Conversation

@DeanoC

@DeanoC DeanoC commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Internal review PR. Stacked on #13 (base feat/topk-partial-sort-gfx1201), so the diff below is only the ncols>1024 enablement. Follows up the finding from #13 that N>1024 TOP_K/ARGSORT are unsupported on every ROCm build.

Problem

The GPU argsort/top-k uses a single-block bitonic sort (block_dims = ncols_pad), capped at 1024 threads/block. On CUDA, ncols>1024 is handled by the CUB device-sort path; on HIP that path is compiled out (GGML_CUDA_USE_CUB is CUDA-only), so supports_op returns ne[0] <= 1024 and larger sorts (full-vocab sampling, etc.) fall back to the CPU reference — 275 test-backend-ops cases on every ROCm build.

Fix

Enable the existing device-sort path on HIP via hipCUB (rocPRIM), which provides the same DeviceRadixSort / DeviceSegmentedSort / DeviceSegmentedRadixSort API:

  • common.cuh: define GGML_CUDA_USE_HIPCUB on HIP.
  • argsort.cu/.cuh: #include <hipcub/hipcub.hpp> and compile argsort_f32_i32_cuda_cub on HIP (no CCCL strided iterator → uses the existing init_offsets segment path).
  • top-k.cu: on HIP, route ncols>1024 through the device sort + slice-first-k (mirrors the CUB branch); the fast partial-bitonic top-k still handles the common ncols ≤ 1024 decode/verify case unchanged.
  • ggml-cuda.cu: supports_op returns true for TOP_K/ARGSORT when CUB or hipCUB is present.

Validation (test-backend-ops test)

op arch not-supported fail
TOP_K gfx1201 (R9700) 0 (was 275) 0
ARGSORT gfx1201 0 (was 275) 0
TOP_K gfx1151 (Strix Halo) 0 (was 275) 0
ARGSORT gfx1151 0 (was 275) 0

The previously-unsupported ncols>1024 cases (up to ne=[262144]) now run on-GPU and are bit-for-bit vs the CPU reference on both archs. The ncols≤1024 partial-bitonic path (#13) is untouched.

Requires the ROCm hipcub + rocprim headers (ship with ROCm; present on our gfx1201 box and lucebox4 gfx1151 /opt/rocm).

The GPU argsort/top-k path uses a single-block bitonic sort (block_dims =
ncols_pad), so it caps at 1024 threads/block. On CUDA, ncols > 1024 is handled
by the CUB device-sort path; on HIP that path was compiled out (GGML_CUDA_USE_CUB
is CUDA-only), so ggml_backend_cuda_supports_op reported TOP_K/ARGSORT as
unsupported for ncols > 1024 and they fell back to the CPU reference — 275 of the
test-backend-ops cases (full-vocab sampling, large sorts) on every ROCm build.

Enable the existing device-sort path on HIP via hipCUB (rocPRIM), which provides
the same DeviceRadixSort / DeviceSegmentedSort / DeviceSegmentedRadixSort API:

  * common.cuh: define GGML_CUDA_USE_HIPCUB on HIP.
  * argsort.cu/.cuh: include <hipcub/hipcub.hpp> and compile argsort_f32_i32_cuda_cub
    on HIP too; hipCUB has no CCCL strided iterator, so the init_offsets segment
    path is used (already the CUB fallback).
  * top-k.cu: on HIP, route ncols > 1024 through the device sort + slice-first-k,
    mirroring the CUB branch; the fast partial-bitonic top-k still handles the
    common ncols <= 1024 decode/verify case unchanged.
  * ggml-cuda.cu: supports_op returns true for TOP_K/ARGSORT when either CUB or
    hipCUB is available.

test-backend-ops -o TOP_K and -o ARGSORT: 0 not-supported / 0 fail on gfx1201
(R9700) and gfx1151 (Strix Halo) — the 275 previously-unsupported ncols>1024
cases (up to ne=[262144]) now run on-GPU and are bit-for-bit vs the CPU
reference. Requires the rocm hipcub + rocprim headers (ship with ROCm).
@DeanoC

DeanoC commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #15 — collapsed the partial-top-k win (#13) and the hipCUB ncols>1024 support (#14) into a single PR for review; they're complementary changes to the same kernels.

@DeanoC DeanoC closed this Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant