Eval bug: Qwen3.5-27B batched performance much lower than expected

[remyoudompheng]

Name and Version

$ ./llama-cli --version
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (RADV PHOENIX) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
version: 8182 (05728db)
built with GNU 15.2.1 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

Radeon 780M iGPU with DDR5 memory

Models

https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/blob/main/Qwen3.5-27B-UD-Q4_K_XL.gguf
https://huggingface.co/unsloth/gemma-3-27b-it-qat-GGUF/blob/main/gemma-3-27b-it-qat-UD-Q4_K_XL.gguf

Problem description & steps to reproduce

When running Qwen3.5-27B the batched performance does not scale in the same way as other dense models with the same size (see performance numbers below).

First Bad Commit

No response

Relevant log output

$ llama-batched-bench -m Qwen3.5-27B-UD-Q4_K_XL.gguf -c 16384 -npp 256 -ntg 64 -npl 1,8,16,32,48

ggml_vulkan: 0 = AMD Radeon 780M Graphics (RADV PHOENIX) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 8182 (05728db18) with GNU 15.2.1 for Linux x86_64
...
sched_reserve: graph nodes  = 7721 (with bs=512), 4745 (with bs=1)

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
256	64	1	320	4.604	55.61	16.691	3.83	21.295	15.03
256	64	8	2560	33.896	60.42	74.504	6.87	108.400	23.62
256	64	16	5120	71.501	57.29	114.469	8.95	185.970	27.53
256	64	32	10240	146.497	55.92	194.407	10.53	340.904	30.04
256	64	48	15360	223.670	54.94	289.033	10.63	512.702	29.96

Comparison with Gemma 3 27B

$ llama-batched-bench -m gemma-3-27b-it-qat-UD-Q4_K_XL.gguf -c 16384 -npp 256 -ntg 64 -npl 1,8,16,32

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 780M Graphics (RADV PHOENIX) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 8182 (05728db18) with GNU 15.2.1 for Linux x86_64
...
sched_reserve: graph nodes  = 2613

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
256	64	1	320	4.035	63.44	14.962	4.28	18.997	16.84
256	64	8	2560	29.486	69.46	58.729	8.72	88.215	29.02
256	64	16	5120	61.090	67.05	61.877	16.55	122.967	41.64
256	64	32	10240	123.017	66.59	67.983	30.13	191.000	53.61

Here is the Vulkan perf dump for a single generation pass:

Qwen 3.5 27B (3.16s with at least 1.8s in non-matmul ops)

Vulkan Timings:
ADD: 224 x 882.614 us = 197706 us
CONCAT: 48 x 162.588 us = 7804.24 us
CONT: 112 x 1225.27 us = 137230 us
CPY: 97 x 1740.66 us = 168844 us
EXP: 48 x 3.904 us = 187.41 us
FLASH_ATTN_EXT dst(256,24,1,32),  q(256,1,24,32),  k(256,512,4,32),  v(256,512,4,32),  m(512,1,1,32): 16 x 612.365 us = 9797.85 us (657.537 GFLOPS/s)
GET_ROWS: 98 x 1249.98 us = 122498 us
GLU: 64 x 104.801 us = 6707.27 us
L2_NORM: 96 x 30.124 us = 2891.93 us
MUL: 352 x 2423.93 us = 853223 us
MUL_MAT q4_K m=1024 n=32 k=5120: 22 x 0.47 us = 10.361 us (712408 GFLOPS/s)
MUL_MAT q4_K m=12288 n=32 k=5120: 16 x 1798.08 us = 28769.3 us (2239.13 GFLOPS/s)
MUL_MAT q4_K m=17408 n=32 k=5120: 128 x 2131.24 us = 272799 us (2676.23 GFLOPS/s)
MUL_MAT q4_K m=48 n=32 k=5120: 96 x 48.829 us = 4687.62 us (322.083 GFLOPS/s)
MUL_MAT q4_K m=5120 n=32 k=17408: 32 x 5852.47 us = 187279 us (974.647 GFLOPS/s)
MUL_MAT q4_K m=5120 n=32 k=6144: 16 x 805.071 us = 12881.1 us (2500.53 GFLOPS/s)
MUL_MAT q5_K m=10240 n=32 k=5120: 48 x 1678.32 us = 80559.5 us (1999.09 GFLOPS/s)
MUL_MAT q5_K m=6144 n=32 k=5120: 48 x 850.027 us = 40801.3 us (2368.24 GFLOPS/s)
MUL_MAT q6_K m=1024 n=32 k=5120: 10 x 0.48 us = 4.8 us (698982 GFLOPS/s)
MUL_MAT q6_K m=248320 n=32 k=5120: 1 x 40075 us = 40075 us (2030.23 GFLOPS/s)
MUL_MAT q6_K m=5120 n=32 k=17408: 32 x 7145.06 us = 228642 us (798.327 GFLOPS/s)
MUL_MAT_VEC q4_K m=5120 n=1 k=6144 batch=32: 48 x 7805.86 us = 374681 us (257.896 GFLOPS/s)
REPEAT: 144 x 1613.54 us = 232350 us
RMS_NORM_MUL RMS_NORM(128,48,1,32): 48 x 96.705 us = 4641.88 us
RMS_NORM_MUL RMS_NORM(256,24,32,1): 16 x 57.325 us = 917.214 us
RMS_NORM_MUL RMS_NORM(256,4,32,1): 16 x 19.874 us = 317.996 us
RMS_NORM_MUL RMS_NORM(5120,32,1,1): 129 x 25.856 us = 3335.47 us
ROPE: 32 x 13.526 us = 432.846 us
SCALE: 48 x 13.642 us = 654.861 us
SET_ROWS: 32 x 7.394 us = 236.632 us
SIGMOID: 64 x 6.323 us = 404.733 us
SILU: 96 x 31.888 us = 3061.29 us
SOFTPLUS: 48 x 3.87 us = 185.768 us
SSM_CONV: 48 x 90.451 us = 4341.67 us
SUB: 48 x 35.706 us = 1713.91 us
SUM_ROWS: 96 x 1350.2 us = 129619 us
Total time: 3.16029e+06 us.

Gemma 3 27B (1.35s)

Vulkan Timings:
ADD: 124 x 39.782 us = 4933.07 us
CPY: 2 x 2.183 us = 4.367 us
FLASH_ATTN_EXT dst(128,32,1,32),  q(128,1,32,32),  k(128,512,16,32),  v(128,512,16,32),  m(512,1,1,32): 62 x 1213.18 us = 75217.5 us (221.265 GFLOPS/s)
GET_ROWS: 2 x 17.553 us = 35.106 us
GLU: 62 x 233.759 us = 14493.1 us
MUL: 62 x 24.827 us = 1539.33 us
MUL_MAT iq4_xs m=2048 n=32 k=5376: 5 x 378.43 us = 1892.15 us (1861.84 GFLOPS/s)
MUL_MAT iq4_xs m=21504 n=32 k=5376: 10 x 4339.82 us = 43398.2 us (1704.69 GFLOPS/s)
MUL_MAT iq4_xs m=4096 n=32 k=5376: 5 x 1231.92 us = 6159.58 us (1143.87 GFLOPS/s)
MUL_MAT q4_K m=2048 n=32 k=5376: 52 x 413.115 us = 21482 us (1705.52 GFLOPS/s)
MUL_MAT q4_K m=21504 n=32 k=5376: 104 x 3293.55 us = 342530 us (2246.22 GFLOPS/s)
MUL_MAT q4_K m=4096 n=32 k=5376: 52 x 1120.59 us = 58270.7 us (1257.51 GFLOPS/s)
MUL_MAT q4_K m=5376 n=32 k=21504: 27 x 8751.36 us = 236287 us (845.421 GFLOPS/s)
MUL_MAT q4_K m=5376 n=32 k=4096: 62 x 677.188 us = 41985.7 us (2080.83 GFLOPS/s)
MUL_MAT q5_K m=2048 n=32 k=5376: 10 x 106.962 us = 1069.62 us (6587.15 GFLOPS/s)
MUL_MAT q5_K m=21504 n=32 k=5376: 10 x 3055.7 us = 30557 us (2421.07 GFLOPS/s)
MUL_MAT q5_K m=4096 n=32 k=5376: 5 x 1009.73 us = 5048.67 us (1395.57 GFLOPS/s)
MUL_MAT q5_K m=5376 n=32 k=21504: 5 x 9979.65 us = 49898.2 us (741.367 GFLOPS/s)
MUL_MAT q6_K m=2048 n=32 k=5376: 57 x 1.333 us = 75.988 us (528517 GFLOPS/s)
MUL_MAT q6_K m=262208 n=32 k=5376: 1 x 48139.9 us = 48139.9 us (1873.87 GFLOPS/s)
MUL_MAT q6_K m=5376 n=32 k=21504: 30 x 11262 us = 337861 us (656.95 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(5376,32,1,1): 249 x 37.906 us = 9438.81 us
RMS_NORM_MUL_ROPE RMS_NORM(128,32,32,1): 62 x 197.689 us = 12256.7 us
RMS_NORM_MUL_ROPE_VIEW_SET_ROWS RMS_NORM(128,16,32,1): 62 x 33.281 us = 2063.45 us
ROPE_VIEW_SET_ROWS ROPE: 62 x 16.62 us = 1030.49 us
SCALE: 63 x 13.547 us = 853.473 us
SET_ROWS: 124 x 19.583 us = 2428.31 us
Total time: 1.34895e+06 us.

[remyoudompheng]

note that rewriting like this seems to help

--- a/src/models/delta-net-base.cpp
+++ b/src/models/delta-net-base.cpp
@@ -342,15 +342,15 @@ std::pair<ggml_tensor *, ggml_tensor *> llm_build_delta_net_base::build_delta_ne
     s = ggml_mul(ctx0, s, g);
 
     ggml_tensor * s_t = ggml_cont(ctx0, ggml_transpose(ctx0, s));
+    ggml_tensor * k_t = ggml_cont(ctx0, ggml_transpose(ctx0, k));
 
     // [1, S_v, H_v, n_seqs]
     ggml_tensor * sk;
-    sk = ggml_mul     (ctx0, s_t, k);
-    sk = ggml_sum_rows(ctx0, sk);
+    sk = ggml_mul_mat(ctx0, s_t, k);
 
     // [S_v, 1, H_v, n_seqs]
     ggml_tensor * d;
-    d = ggml_sub(ctx0, v, ggml_transpose(ctx0, sk));
+    d = ggml_sub(ctx0, v, sk);
     d = ggml_mul(ctx0, d, b);
 
     // [1, S_v, H_v, n_seqs]
@@ -359,17 +359,15 @@ std::pair<ggml_tensor *, ggml_tensor *> llm_build_delta_net_base::build_delta_ne
 
     // [S_v, S_v, H_v, n_seqs]
     ggml_tensor * kd;
-    k  = ggml_repeat(ctx0, k, s);
-    kd = ggml_mul   (ctx0, k, d_t);
+    kd = ggml_mul_mat(ctx0, k_t, d_t);
 
     s_t = ggml_add(ctx0, s_t, kd);
 
     cb(s_t, "dnet_add_ar_state", il);
 
-    ggml_tensor * s_q = ggml_mul     (ctx0, s_t, q);
-    ggml_tensor * o   = ggml_sum_rows(ctx0, s_q);
+    ggml_tensor * o   = ggml_mul_mat(ctx0, s_t, q);
 
-    o = ggml_permute  (ctx0, o, 2, 0, 1, 3); // [S_v, H_v, n_tokens, n_seqs]
+    o = ggml_permute  (ctx0, o, 0, 2, 1, 3); // [S_v, H_v, n_tokens, n_seqs]
     s = ggml_transpose(ctx0, s_t);           // [S_v, S_v, H_v, n_seqs]
 
     return {o, s};

[remyoudompheng]

Also maybe recurrent memory should be allowed to use f16

[remyoudompheng]

FYI it's much better after #20334 but the cost of GET_ROWS/CPY on cache_s_l is still very high

ADD: 176 x 26.994 us = 4751.07 us
CONCAT: 48 x 146.787 us = 7045.8 us
CONT: 64 x 9.708 us = 621.344 us
CPY: 97 x 1437.11 us = 139399 us
FLASH_ATTN_EXT dst(256,24,1,32),  q(256,1,24,32),  k(256,512,4,32),  v(256,512,4,32),  m(512,1,1,32): 16 x 619.839 us = 9917.44 us (649.609 GFLOPS/s)
GATED_DELTA_NET: 48 x 2869.18 us = 137721 us
GET_ROWS: 98 x 1255.6 us = 123048 us
GLU: 64 x 104.874 us = 6711.96 us
L2_NORM: 96 x 17.02 us = 1633.98 us
MUL: 112 x 19.088 us = 2137.89 us
MUL_MAT q4_K m=1024 n=32 k=5120: 22 x 0.35 us = 7.72 us (956121 GFLOPS/s)
MUL_MAT q4_K m=12288 n=32 k=5120: 16 x 1390.18 us = 22242.9 us (2896.12 GFLOPS/s)
MUL_MAT q4_K m=17408 n=32 k=5120: 128 x 1649.72 us = 211164 us (3457.37 GFLOPS/s)
MUL_MAT q4_K m=48 n=32 k=5120: 96 x 43.409 us = 4167.31 us (362.296 GFLOPS/s)
MUL_MAT q4_K m=5120 n=32 k=17408: 32 x 4904.82 us = 156954 us (1162.96 GFLOPS/s)
MUL_MAT q4_K m=5120 n=32 k=6144: 16 x 635.198 us = 10163.2 us (3169.25 GFLOPS/s)
MUL_MAT q5_K m=10240 n=32 k=5120: 48 x 1416.97 us = 68014.4 us (2367.81 GFLOPS/s)
MUL_MAT q5_K m=6144 n=32 k=5120: 48 x 677.275 us = 32509.2 us (2972.3 GFLOPS/s)
MUL_MAT q6_K m=1024 n=32 k=5120: 10 x 0.356 us = 3.56 us (942448 GFLOPS/s)
MUL_MAT q6_K m=248320 n=32 k=5120: 1 x 34936.6 us = 34936.6 us (2328.83 GFLOPS/s)
MUL_MAT q6_K m=5120 n=32 k=17408: 32 x 6155.58 us = 196979 us (926.653 GFLOPS/s)
MUL_MAT_VEC q4_K m=5120 n=1 k=6144 batch=32: 48 x 7788.23 us = 373835 us (258.48 GFLOPS/s)
RMS_NORM_MUL RMS_NORM(128,48,1,32): 48 x 96.129 us = 4614.22 us
RMS_NORM_MUL RMS_NORM(256,24,32,1): 16 x 55.054 us = 880.868 us
RMS_NORM_MUL RMS_NORM(256,4,32,1): 16 x 17.715 us = 283.449 us
RMS_NORM_MUL RMS_NORM(5120,32,1,1): 129 x 24.999 us = 3224.95 us
ROPE: 32 x 12.227 us = 391.284 us
SET_ROWS: 32 x 5.646 us = 180.686 us
SIGMOID: 64 x 7.841 us = 501.851 us
SILU: 96 x 29.619 us = 2843.44 us
SOFTPLUS: 48 x 3.6 us = 172.827 us
SSM_CONV: 48 x 101.029 us = 4849.43 us
Total time: 1.56191e+06 us.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.