RFC: chunked CUDA prefill kernel for ggml_gated_delta_net

[leonardHONG]

The CUDA implementation of ggml_gated_delta_net (ggml/src/ggml-cuda/gated_delta_net.cu) still has the TODO (from #19504) for a chunked prefill kernel in launch_gated_delta_net().

My reading is that there is still a CUDA-side prefill gap here: larger prompt processing cases still go through the existing token-sequential kernel, while the chunked algorithmic reference lives in build_delta_net_chunking() (src/models/delta-net-base.cpp).

I'd like to work on a chunked CUDA prefill kernel for this op, likely starting with the non-KDA path first.

Tentative plan:

• mirror the existing DeltaNet chunking logic as a fused CUDA path;
• keep small-token cases on the current kernel and dispatch chunked only for larger n_tokens;
• validate against the existing reference path;
• benchmark prefill on Qwen3-Next / Qwen3.5-style GDN workloads;
• KDA path may land as a follow-up if the initial PR grows too large.

Before I start prototyping: any preference for the chunked dispatch threshold here, and is there any existing work or direction I should align with?

[am17an]

The main issue is the maintainability of this kernel. From experience it's going to be a lot of code to get good performance out of this, but you can try if you want. I have a previous attempt at https://github.com/am17an/llama.cpp/tree/gated_delta_net_chunk which uses TF32 tensor cores and is not optimal in any way but beats the AR kernel at higher batch sizes

[leonardHONG]

Thanks, this is really helpful context. I’ll take a look at your branch first and see how far I can get with a small non-KDA prototype.

I agree the maintainability side is probably the hard part here. I can't promise that I’ll end up with something worth merging, but I’m happy to experiment a bit and share numbers / findings if it looks reasonable.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.