Eval bug: qwen35moe produces static degenerate output (all "/" tokens) with CUDA, works on CPU-only

[mythikal03]

Name and Version

$ ./llama-cli --version
version: 8076 (d61290111)
built with GNU 13.3.0 for Linux x86_64

Build flags: -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="120" -DGGML_CUDA_FORCE_CUBLAS=ON -DGGML_CUDA_FA_ALL_QUANTS=ON

Operating systems

Linux (Ubuntu 24.04, kernel 6.8.0-100-generic)

GGML backends

CUDA

Hardware

AMD Threadripper PRO 9975WX (64c/128t), 256GB DDR5 RAM, NVIDIA RTX PRO 6000 Blackwell 96GB (SM120, compute 12.0). CUDA 12.9, driver 580.126.16.

Models

• unsloth/Qwen3.5-397B-A17B-GGUF — UD-Q3_K_XL (176GB, 5 shards)
• unsloth/Qwen3.5-397B-A17B-GGUF — MXFP4_MOE (216GB, 6 shards)
• ubergarm/Qwen3.5-397B-A17B-GGUF — Q3_K (193GB, llama.cpp-native convert_hf_to_gguf.py)

Problem description & steps to reproduce

Every generated token is / (token ID 14) when any GPU layers are active. The logit distribution is static — identical at every position regardless of input, temperature, or sampling parameters. The model loads successfully, processes prompts at expected speed, but the output is entirely degenerate.

CPU-only works perfectly. With -ngl 0, the model produces correct, coherent output with proper reasoning.

Minimal reproduction (broken — GPU):

# Server
./llama-server \
  -m Qwen3.5-397B-A17B-MXFP4_MOE-00001-of-00006.gguf \
  --mmproj mmproj-BF16.gguf \
  -ngl 999 -cmoe -c 131072 -np 1 -t 64 \
  --host 0.0.0.0 --port 8103

# Request
curl http://localhost:8103/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.5","messages":[{"role":"user","content":"What is 2+2?"}]}'

# Result: every token is "/" in both content and reasoning_content

Minimal reproduction (working — CPU-only):

./llama-cli \
  -m Qwen3.5-397B-A17B-MXFP4_MOE-00001-of-00006.gguf \
  -ngl 0 -t 64 -c 4096 -n 256 \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
  -p "What is (2+2)x3?"

# Result: correct coherent output, proper PEMDAS reasoning, answer = 12

What was tested and ruled out:

Variable	Tested	Result
Quant (unsloth UD-Q3_K_XL)	-ngl 999 -cmoe	Slashes
Quant (unsloth MXFP4_MOE)	-ngl 999 -cmoe	Slashes
Quant (ubergarm Q3_K, llama.cpp-native convert)	-ngl 999 -cmoe	Slashes
Auto-fit (no -ngl, no -cmoe)	llama-server	Slashes (extremely slow, 151 graph splits)
CPU-only (-ngl 0)	llama-cli	Works perfectly
With/without --mmproj	-ngl 999 -cmoe	Slashes both ways
With/without -fa on	-ngl 999 -cmoe	Slashes both ways
With/without -ctk q8_0 -ctv q8_0	-ngl 999 -cmoe	Slashes both ways
With/without --jinja	-ngl 999 -cmoe	Slashes both ways
With/without --reasoning-format deepseek	-ngl 999 -cmoe	Slashes both ways
NVIDIA driver 580.105.08 → 580.126.16	Rebuilt llama.cpp after update	Slashes on both
Raw /completion endpoint (no chat template)	-ngl 999 -cmoe	Slashes
Various temperatures (0.0–1.5)	-ngl 999 -cmoe	Slashes

Conclusion: The qwen35moe CUDA compute graph produces a static logit distribution when GPU layers are used. Three independent quants from two different conversion pipelines all produce identical failure. CPU-only inference is correct. The bug appears to be in the GPU forward pass for this architecture, not in model weights or quantization.

First Bad Commit

Cannot bisect — qwen35moe support was introduced in PR #19468 (merged Feb 10), and the bug has been present since the first build that includes it. Tested on b8076 (d612901).

Note: this is distinct from #19676 (long-prompt segfault with --op-offload on multi-GPU). This issue affects single-GPU inference at any prompt length.

Relevant log output

Logs

$ curl -s http://localhost:8103/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"qwen3.5","messages":[{"role":"user","content":"What is 2+2?"}]}'

{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","content":"",
"reasoning_content":"///////////////////////////////////////////////////////////////////////////////..."}}],
"usage":{"completion_tokens":1024,"prompt_tokens":22,"total_tokens":1046},
"timings":{"predicted_per_token_ms":34.39,"predicted_per_second":29.07}}

[JohannesGaessler]

Does it work correctly without -DGGML_CUDA_FORCE_CUBLAS? As described in the build documentation, that option can lead to numerical issues.

[mythikal03]

That was it. Rebuilt without -DGGML_CUDA_FORCE_CUBLAS=ON and the model produces correct, coherent output immediately.

$ curl -s http://localhost:8103/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"qwen3.5","messages":[{"role":"user","content":"What is (2+2)x3?"}],"chat_template_kwargs":{"enable_thinking":false}}'

{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant",
"content":"To solve the expression $(2+2) \\times 3$...The final answer is **12**."}}],
"timings":{"predicted_per_second":24.5}}

24.5 tok/s on MXFP4_MOE with -cmoe, correct reasoning, proper stop token. The FP16 cuBLAS numerical overflow on Blackwell (SM120) was the root cause — not the compute graph.

Build flags before (broken): -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="120" -DGGML_CUDA_FORCE_CUBLAS=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
Build flags after (working): -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="120" -DGGML_CUDA_FA_ALL_QUANTS=ON

Thanks for the quick catch — the build docs warning about numerical overflows (except CDNA/RDNA4) is exactly right. Blackwell is not exempt.