Eval bug: quality regression when loading experts on the GPU compared to CPU offloading

[grumd]

Name and Version

latest master branch - b8683-d0a6dfeb2

Operating systems

Linux

GGML backends

CUDA, Vulkan

Hardware

RTX 5080 16 GB

96 GB DDR5 RAM, 9800X3D

Models

unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL

Problem description & steps to reproduce

Depending on the usage of --cpu-moe or -ot, the same model and settings produce very different quality of results.
I'm using this benchmark: https://github.com/nlothian/llm-sql-benchmark

Base command:

llama-server -hf unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL \
  --no-mmproj --no-mmap --jinja -b 4096 -ub 2048 --parallel 1 \
  --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.01 \
  -c 32000 \
  --cache-ram 0 --seed 3407

Testing only the Hard level tests

Adding -ot '.*_exps\.weight=CPU' to the base command (CUDA):
12/13 pass, 1 fail

Adding --cpu-moe to the base command (CUDA):
12/13 pass, 1 fail

Adding -ot '.*_exps\.weight=CPU' to the base command (Vulkan):
10/13 pass, 2 fail, 1 error

Adding --cpu-moe to the base command (Vulkan):
10/13 pass, 2 fail, 1 error

-fitc 32000 instead of -c 32000 (CUDA)
9/13 pass, 3 fail, 1 error

-fitc 32000 instead of -c 32000 (Vulkan)
9/13 pass, 2 fail, 2 error

I've rerun the tests multiple times even changing temp and seeds, very consistently getting the same results -- the same tests Q6, Q9, Q21, Q22 are getting fails or errors in all cases except for full --cpu-moe with CUDA.

First Bad Commit

No response

Relevant log output

Not sure what logs to add here

[grumd]

Not a duplicate of #21255 -- issue still happens with CUDA 13.0, even though incoherent outputs with IQ3 quants are fixed with 13.0

[grumd]

Binary searched the commit history.

This is the culprit: #20340

Before this merge Qwen3.5 122B does 9/13 tasks correctly, after this merge it does 12/13 tasks correctly.

This is nice, but this improvement only applies if I use --cpu-moe and load all experts on the CPU. If I simply rely on --fit and some of the experts move to CUDA0, the improvement doesn't apply and we're back to 9/13 correct tasks.

[nlothian]

@grumd love that you are finding this benchmark useful. LMK if you have any feedback for ways to improve it.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.