metal: fused_gdn_ch (chunked prompt processing) needed for Qwen3.5 on older Apple GPUs

[appwa9]

Request

PR #20244 adds the fused Metal kernel for GGML_OP_GATED_DELTA_NET (autoregressive path). The chunked/prompt processing path (fused_gdn_ch) is still // TODO: implement in llama-context.cpp:154.

Without fused_gdn_ch, prompt processing falls back to decomposed Metal ops (23 kernels per DeltaNet layer), which produce NaN on Apple GPU Family 7 (A14 Bionic). Implementing fused_gdn_ch would likely bypass these decomposed ops and is the most probable path to fixing this — though we haven't been able to verify this since the chunked path doesn't exist yet.

Context

Qwen3.5-2B (6 attention + 18 Gated DeltaNet layers) works correctly on A17 Pro (GPU Family 9) but produces NaN logits during prompt eval on A14 (GPU Family 7). llama_decode() returns 0, but all logits are NaN.

We tested PR #20244 built as a custom xcframework on A14. The fused AR kernel is recognized, but prompt processing still uses the decomposed path and produces NaN — confirming the issue is specifically in the chunked/prompt eval code path.

Other findings:

• CPU-only inference works on A14 — Metal kernel issue, not a model issue
• Qwen3-1.7B (no DeltaNet) works on A14 Metal — A14 Metal is fine for standard transformers
• Disabling flash attention and reducing batch size don't help

The decomposed kernels involved include kernel_solve_tri_f32, kernel_cumsum_blk_f32, kernel_ssm_conv_f32_f32_batched_4, kernel_tri_f32_2/3, kernel_l2_norm_f32_f32_4.

Environment

• Failing: iPad Gen 10 (A14, GPU Family 7), iPadOS 26.3.1, b8253
• Working: iPad Mini 7 (A17 Pro, GPU Family 9), same build
• Model: Qwen3.5-2B Q4_K_M

Related

• metal : add Metal backend for GGML_OP_GATED_DELTA_NET #20244 — Metal fused GDN kernel (AR path only)
• ggml: add GATED_DELTA_NET op #19504 — Fused GDN op (CPU + CUDA)

[ggerganov]

You will need to rebase and adapt your work over #20340

[appwa9]

Thanks for the pointer! To clarify — we're end users reporting the A14 NaN issue, not the author of #20244. We tested #20244 as a custom xcframework to see if the fused kernel would help, but since fused_gdn_ch wasn't implemented yet, prompt eval still hit the decomposed path.

Great to see #20340 enables the chunked fused GDN path — that's exactly what we were hoping for. We'll retest on our A14 device once #20340 and a Metal backend implementation (like #20244) are both merged.

[ggerganov]

Please try #20361

[appwa9]

Tested PR #20361 on iPad 10th generation (A14 Bionic, iPadOS 26.3.1) with Qwen3.5-2B-Q4_K_M — the fused GDN Metal kernel works correctly. No NaN issues.

The fused_gdn_ch implementation bypasses the decomposed path that was producing NaN on A14, which is exactly what we needed. Great work — thank you!

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.