Eval bug: Running Qwen3.5-35B-A3B on CUDA backend is 35% slower than Qwen3-30B-A3B.

[lingyezhixing]

Name and Version

ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 4060 Laptop GPU, compute capability 8.9, VMM: yes
Device 1: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes
load_backend: loaded CUDA backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-cuda.dll
load_backend: loaded RPC backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-rpc.dll
load_backend: loaded CPU backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-cpu-zen4.dll
version: 8149 (a96a112)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CUDA

Hardware

AMD R9 7940H
RTX4060 Laptop
V100-32G-SXM2 <- by setting CUDA_VISIBLE_DEVICES=1 to run the model here

Models

Qwen3-30B-A3B-Thinking-2507-UD-Q5_K_XL.gguf
Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf

Problem description & steps to reproduce

Shouldn't the linear attention of Qwen3.5 run faster? Why is it slower now? Moreover, it has been observed during use that generation involves at least two CPU cores approaching full load, which indicates that llama.cpp's support for this architecture is still insufficient, requiring a fallback to the CPU path?

First Bad Commit

No response

Relevant log output

Logs

E:\models\LLM\GGUF>"D:\LLM\LLM-Manager\backend\llama.cpp\llama-bench.exe" -m Qwen3-30B-A3B-Thinking-2507-UD-Q5_K_XL.gguf
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes
load_backend: loaded CUDA backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-cuda.dll
load_backend: loaded RPC backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-rpc.dll
load_backend: loaded CPU backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-cpu-zen4.dll
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q5_K - Medium |  20.27 GiB |    30.53 B | CUDA       |  99 |           pp512 |       735.81 ± 13.51 |
| qwen3moe 30B.A3B Q5_K - Medium |  20.27 GiB |    30.53 B | CUDA       |  99 |           tg128 |         59.02 ± 0.36 |

build: a96a1120b (8149)

E:\models\LLM\GGUF>"D:\LLM\LLM-Manager\backend\llama.cpp\llama-bench.exe" -m Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes
load_backend: loaded CUDA backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-cuda.dll
load_backend: loaded RPC backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-rpc.dll
load_backend: loaded CPU backend from D:\LLM\LLM-Manager\backend\llama.cpp\ggml-cpu-zen4.dll
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35moe ?B Q5_K - Medium     |  23.02 GiB |    34.66 B | CUDA       |  99 |           pp512 |        570.31 ± 5.38 |
| qwen35moe ?B Q5_K - Medium     |  23.02 GiB |    34.66 B | CUDA       |  99 |           tg128 |         38.36 ± 0.13 |

build: a96a1120b (8149)

[l29ah]

Much slower on Vulkan as well, but no excessive CPU load.

[thomas-0816]

same here ~ 40% slower in token generation, no excessive cpu load, details:

Version:

 ./llama-server --version
load_backend: loaded RPC backend from /home/tb/code/llama-b8148/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (RADV PHOENIX) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /home/tb/code/llama-b8148/libggml-vulkan.so
load_backend: loaded CPU backend from /home/tb/code/llama-b8148/libggml-cpu-zen4.so
version: 8148 (244641955)
built with GNU 11.4.0 for Linux x86_64

System:

Ryzen 7 PRO 7840U w/ Radeon 780M
GTT 24 GB (amd_iommu=off amdgpu.gttsize=24576 ttm.pages_limit=6291456), total ram 32 GB
Kernel: 6.17

Qwen 3.5

./llama-server -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
--jinja --fit on --threads -1 --parallel 1 --ctx-size 8192 \
--temp 0.7 --min-p 0.0 --top-p 0.8 --top-k 20 --presence-penalty 1.5  \
--n_predict 4096 --chat-template-kwargs "{\"enable_thinking\": false}" --direct-io

prompt eval time =     405.56 ms /    22 tokens (   18.43 ms per token,    54.25 tokens per second)
       eval time =   11438.67 ms /   222 tokens (   51.53 ms per token,    19.41 tokens per second)
      total time =   11844.23 ms /   244 tokens

Qwen 3

./llama-server -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL \
--jinja --fit on --threads -1 --parallel 1 --ctx-size 16384 \
--temp 0.7 --min-p 0.0 --top-p 0.8 --top-k 20 --presence-penalty 1.0 \
--n_predict 4096 --direct-io

prompt eval time =     416.28 ms /    18 tokens (   23.13 ms per token,    43.24 tokens per second)
       eval time =   16847.56 ms /   576 tokens (   29.25 ms per token,    34.19 tokens per second)
      total time =   17263.84 ms /   594 tokens

llama-server logs:

Qwen3.5-35B-A3B-GGUF.txt

Qwen3-30B-A3B-GGUF.txt

[engrtipusultan]

Maybe once #18792 and #19504 are done things will further improve in time.

[ckinateder]

Bumping this, i am having the same issue.