Eval bug: Vulkan Performance Regression, slower than CPU

[Dnonmi]

Name and Version

$llama-cli --version                         
ggml_vulkan: Found 1 Vulkan devices:          
ggml_vulkan: 0 = Adreno (TM) 740 (Qualcomm Technologies Inc. Adreno Vulkan Driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: none
version: 8345 (cf6d540b6)                     
built with Clang 18.0.2 for Android aarch64

Operating systems

Other? (Please let us know in description)

GGML backends

Vulkan

Hardware

Snapdragon 8Gen2, Adreno 740 GPU

Models

Qwen3.5 2B Q4_K, Q2_K

Problem description & steps to reproduce

Situation

I am working with an "unusual" setup (Android 15). I'm looking to either fix performance regressions or fix output corruption in Qwen3.5.

• Background: I previously used chraac's llama.cpp fork for Hexagon backend support.

• Steps to reproduce:

  • Hexagon Backend:
  • wanted Qwen 3.5 support.
  • rebased the fork onto the full qwen35 support PR #19468.
  • Qwen 3.5 2B (Q2_K_S) successfully processed a "hey" input once, but subsequently I was unable to reproduce this behavior and it was outputting gibberish at ~1-3 tk/s. Other models function correctly, though Hexagon remains just a slow backend.
  • Decided to try vulkan instead.
  • Vulkan (Rebase):
  • also produced gibberish (not chinese) but at a much higher rate of 20–30 tk/s, which is also true for other models.
  • llama-cli with -ngl 0 has correct output, and other models (LFM2.5, llama3.2, qwen2.5) run correctly.
  • git clone latest master (March 13th, 2026).
  • Vulkan (Master - March 13, 2026):
  • clean build from the latest master validated that Qwen 3.5 output is correct. However, generation speed has regressed to 6–10 tk/s—slower than CPU performance and significantly worse than my custom build. Reading is also a lot worse.
  • Back to Rebase branch:
  • traced and merged dependency commits in qwen35.cpp up to add GATED_DELTA_NET op (#19504), including the Vulkan Gated Delta Net commit.
  • still has output corruption.

Debugging

* Verified that `llama-cli` with `-ngl 0` (CPU only) produces correct output for Qwen3.5 in all builds after rebase.
* Confirmed that other models across various quantizations run correctly on the custom build.
* Attempted common fixes from similar Mac/Multi-GPU issues (e.g., `ggml f16` flags, disabling `flash_attn`), but with no improvement to the output.

Summary
While the Qwen3.5 corrupted output issue with Vulkan backend appears resolved in latest master (at least on Adreno GPUs), there's a significant performance regression compared to my custom rebase. Does anyone know which specific commits—beyond the ones I’ve cited—actually fixed the Qwen 3.5 corrupted generation on the Vulkan backend?
The performance regression in master isn't worth at all to run it.

I am hesitant to open a new issue, if this is a known duplicate do point me at the right one.
I found similar issues opened by multi-gpu and Mac users, but nothing this specific for me to polute in those issues.

First Bad Commit

No response

Relevant log output

Logs

* **Model loading:**

Output from llama server page.

How many letter `o` counts in the word "orangotango"?

Reasoning:
elerde5!%/$=A4'#!:8-3<F4E*F,6.G6.8(F23G"2F&/-F

qwen3.5.gguf
Reading
25 tokens
0.8s
31.03 tokens/s
Generation
41 tokens
2.6s
16.03 t/s

[IMbackK]

Comparing the speed of an implementation that generates gibberish is pointless.

[Dnonmi]

Comparing the speed of an implementation that generates gibberish is pointless.

Do you know what a pattern is?
Only Qwen3.5 generates gibberish, other models have the same consistent speed up compared to Hexagon/CPU.
As well as consistent performance regression in master.

Unnecessary comment.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.