Vulkan: MUL_MAT_ID is the dominant prefill bottleneck for large MoE models on gfx1151

[0xSero]

Vulkan: MUL_MAT_ID is the dominant prefill bottleneck for large MoE models on gfx1151

Summary

On a Strix Halo system (gfx1151, 120 GiB GTT), Vulkan MUL_MAT_ID accounts for 42–66% of total prefill time when running Qwen3.5-122B-A10B Q6_K at practical context lengths (≤32k). Even at 128k context, where FLASH_ATTN_EXT takes over, MUL_MAT_ID still contributes 20%.

Patched ROCm (with #21344 MMQ VGPR tuning) achieves 19–35% faster prefill than Vulkan on the same hardware and model. This gap appears to originate primarily from the MUL_MAT_ID path — the routed-expert matrix multiplication used by MoE architectures.

Hardware

Component	Spec
CPU	AMD Ryzen AI MAX+ 395 (Zen 5, 16C/32T)
GPU	Radeon 8060S (gfx1151, RDNA 3.5, 40 CU)
RAM	128 GB LPDDR5X-8000 (~215 GB/s)
GTT	120 GiB
OS	Fedora 43, kernel 6.17.1, mesa-vulkan-drivers 25.3.6

Model

• Qwen3.5-122B-A10B REAP-20 Q6_K (75.73 GiB)
• 205 total experts, 8 active per token
• KV cache: q8_0/q8_0, flash attention enabled

Profiling method

Built llama.cpp b8779 with Vulkan backend. Used GGML_VK_PERF_LOGGER=1 environment variable to capture per-kernel timings during llama-bench runs with -pg (prefill+generate) at multiple context depths.

Results: kernel-level breakdown per layer step (batch=512)

At KV=512 (short context — total 1.70s/step)

Kernel	Time (ms)	% of step
MUL_MAT_ID q6_K m=1024 n=8 k=3072 n_expert=205 batch=512	772.6	45.4%
MUL_MAT_ID q6_K m=3072 n=8 k=1024 n_expert=205 batch=512	352.1	20.7%
MUL_MAT q6_K m=3072 n=512 k=8192	96.1	5.7%
FLASH_ATTN_EXT	9.4	0.6%
MUL_MAT_ID total	1124.7	66.2%

At KV=8192 (practical chat — total 1.84s/step)

Kernel	Time (ms)	% of step
MUL_MAT_ID (both shapes combined)	1059.3	57.6%
FLASH_ATTN_EXT	226.1	12.3%
MUL_MAT q6_K (various)	255.6	13.9%

At KV=32768 (long document — total 2.56s/step)

Kernel	Time (ms)	% of step
FLASH_ATTN_EXT	930.3	36.4%
MUL_MAT_ID (both shapes combined)	1072.6	41.9%
MUL_MAT q6_K (various)	280.8	11.0%

At KV=131072 (max context — total 5.35s/step)

Kernel	Time (ms)	% of step
FLASH_ATTN_EXT	3740.0	69.9%
MUL_MAT_ID (both shapes combined)	1052.9	19.7%
MUL_MAT q6_K (various)	281.6	5.3%

Key observation

MUL_MAT_ID time is essentially constant (~1050 ms) regardless of context length. It does not scale with KV cache size. The expert routing computation is the same cost whether processing 512 or 131072 tokens of context. It is purely a function of batch size and expert count.

End-to-end throughput: Vulkan vs patched ROCm

Test	Vulkan (t/s)	Patched ROCm (t/s)	ROCm advantage
pp512	268.89	354.57	+32%
pp8192+tg128	249.65	302.44	+21%
pp32768+tg128	234.94	291.03	+24%
pp131072+tg128	145.03	171.66	+18%
tg128	23.52	21.00	−11% (Vulkan wins decode)

Relevant prior work

• vulkan: apply MUL_MAT_ID subgroup optimization to non-coopmat devices #15524 (merged): Vulkan MUL_MAT_ID subgroup optimization for non-coopmat devices. Gave 100–660% improvements on smaller MoE models (Qwen3-30B-A3B). This optimization is already active in the Vulkan build used here — the remaining bottleneck is on top of it.

• metal : optimize MoE for large batches #13388 (merged): Metal MoE optimization using map → batched matmul → unmap. Achieved 1.8–4.1× speedups on Qwen3-30B-A3B F16/Q4_0/Q8_0. The key insight was converting per-expert scatter operations into a contiguous batched matmul by remapping inputs per-expert.

• gfx1151 nwarps, tile sizing to curb VGPR pressure #21344 (open): ROCm MMQ VGPR tuning for gfx1151. Achieves the 19–35% prefill gains shown above. This operates on the dense matmul path (MMQ), not MUL_MAT_ID directly, but the ROCm backend's MUL_MAT_ID implementation may handle the expert routing differently.

• ggml-zendnn : add MUL_MAT_ID op support for MoE models #21315 (merged): ZenDNN added MUL_MAT_ID support specifically because MoE throughput justified dedicated backend work.

Analysis

The Vulkan MUL_MAT_ID kernel currently processes n=8 (active experts) × batch=512 tokens against 205 experts with a Q6_K weight matrix. The ~1050 ms constant cost suggests the kernel is not efficiently utilizing the GPU for this workload shape — it is a many-small-matmul pattern that could potentially benefit from the same remap/batch approach used in #13388.

The fact that patched ROCm is 18–32% faster on prefill while Vulkan wins decode by 11% further suggests this is not a memory bandwidth issue but a compute dispatch / kernel efficiency issue specific to the expert routing path.

Suggested direction

The Metal #13388 approach (map inputs → contiguous batched matmul → unmap results) seems like the most promising direction for Vulkan. The current per-expert dispatch pattern appears to leave significant GPU utilization on the table, especially on gfx1151 where the 40 CUs and 215 GB/s bandwidth should support higher throughput.

Alternatively, gfx1151-specific warptile tuning for the MUL_MAT_ID shader (similar in spirit to #21344's MMQ tuning) could help if the existing subgroup optimization from #15524 is hitting register pressure or occupancy limits on RDNA 3.5.

Build info

build: 75f3bc94e (8779)
Vulkan0: Radeon 8060S Graphics (RADV STRIX_HALO), uma: 1, fp16: 1, bf16: 0, warp size: 64, shared memory: 65536, int dot: 1, matrix cores: KHR_coopmat