Qwen3-Coder-Next (Qwen3-Next-80B) CPU inference ~5x slower than expected — consumer hardware benchmarks

[ProgenyAlpha]

Summary

Qwen3-Coder-Next (80B MoE, 3B active) achieves ~7.7 tok/s on consumer hardware when benchmarks of comparable MoE models suggest it should be closer to 35-60 tok/s. This was previously reported in #17936 (closed as stale), but we now have fresh controlled benchmarks comparing it against a dense model on the same hardware, showing the anomaly clearly.

Environment

• CPU: AMD Ryzen AI 9 HX PRO 370 (Zen 5, 12c/24t, 4.37 GHz)
• RAM: 96GB DDR5-5600 dual-channel (2x48GB)
• OS: TrueNAS SCALE (Linux 6.12.15)
• Runtime: Ollama 0.15.6 (Docker, CPU-only)
• llama.cpp version: Build 7973 (39bf692)
• CPU features: AVX, AVX2, AVX-512, AVX-512_VBMI, AVX-512_VNNI, AVX-512_BF16, FMA, F16C

Benchmark Results

Same hardware, same conditions, same prompt

Model	Quant	Size	Active Params	Gen tok/s	Architecture
qwen2.5-coder:32b	Q4_K_M	19GB	32.8B (100%)	3.54	Dense
qwen3-coder-next	Q4_K_M	51GB	3B (3.75%)	7.74	MoE

The anomaly

Q3CN activates only 3B of 80B params per token — 3.75% of the model. A dense 32B model reads 100% of its weights per token. By bandwidth math alone:

• Dense 32B reads ~19GB per token → 3.54 tok/s (matches DDR5-5600 bandwidth ceiling of ~65 GB/s)
• MoE 80B reading 3B active should read ~2-3GB per token → expected 20-30+ tok/s
• Actual: 7.74 tok/s — roughly 3-4x slower than the bandwidth math predicts

Cross-reference with #17936

The original reporter tested on AMD EPYC 9454P (48c/96t):

Model	Active Params	tok/s
Qwen3-MoE-30B	~3B	63.10
Qwen3-Next-80B	~3B	11.76
GPT-OSS-120B	~5B	35.75

Same pattern: Qwen3-Next is 5.4x slower than Qwen3-MoE despite identical active parameter counts.

Thread scaling (rules out threading issues)

Threads	Gen tok/s
4	2.72
6	3.37
8	3.49
12 (physical cores)	3.54
24 (with SMT)	2.84

12 threads (physical cores) is optimal. SMT hurts. CPU governor (performance vs powersave) makes zero difference — confirming this is bandwidth-bound, not compute-bound.

Why this matters

Qwen3-Coder-Next is one of the most capable open-weight coding models available. The MoE architecture (3B active / 80B total) was specifically designed to run on consumer hardware. But the CPU inference path in llama.cpp isn't taking advantage of the sparse activation pattern — it appears to be reading far more data than the 3B active params require.

The fork ik_llama.cpp reports ~1.9x faster MoE inference through fused MoE operations, better AVX-512 utilization, and MoE-specific optimizations. This suggests the performance gap is addressable in the llama.cpp backend.

Expected behavior

MoE models with 3B active params should achieve inference speeds comparable to dense 3B models on CPU (adjusted for routing overhead). On DDR5-5600 (~65 GB/s real-world bandwidth), a 3B active model should achieve 20-30+ tok/s generation.

Related

• Feature Request: Qwen3-Next: CPU performance optimization #17936 (original report, closed as stale)
• PR Optimization: Qwen3 next autoregressive pass #17996 (referenced as potential fix, status unclear)
• ik_llama.cpp MoE optimizations (demonstrates fix is possible)

[iz0eyj]

On the same hardware it runs very well using the GPU (#19396)

[pwilkin]

I know @am17an has been quietly working on CPU optimizations (eg. #19422), so maybe he knows how much can be squeezed here.

[ProgenyAlpha]

Thanks for the replies, both.

@iz0eyj — We ended up going the Vulkan route and got all 49 layers on the iGPU (Radeon 890M, 16 RDNA 3.5 CUs). With flash attention enabled: 17.6 tok/s TG, 53.8 tok/s PP, 50% KV cache reduction. The key discovery was dropping --no-mmap — on UMA architecture it forces double allocation from the shared DDR5 pool, which was causing OOM even with 96GB of RAM.

@pwilkin — Good call on #19422. I'm watching the GEMM microkernel PR and seeing how that unfolds. The context-scaling speedups look promising, especially at 16K-32K. Pairing that with DeltaNet support (which Q3CN was actually designed for) could be pretty exciting for CPU inference.

[pwilkin]

@ProgenyAlpha Yeah, now that Delta.net support has stabilized (Kimi Linear uses a different variant, Qwen3.5, which is a big and important model family, is going to use it), we're probably going to introduce a dedicated delta.net OP / kernel. Aman is already working on that, see #19504

[iz0eyj]

@pwilkin - @ProgenyAlpha
I just published an article with in-depth tests on various models of different sizes running on Ryzen AI 9—I think you might find it interesting. It's not in English yet, but a single mouse click is all it takes to translate it.
You can read it here

[moshemalawach]

On a Zen4 with 12 channels of DDR5 ram I also get 7 tok/s where similar models land me more than 10x that speed

[leberschnitzel]

there seems to be a workaround for this:
I'm using qwen/qwen3-coder-next Q4_K_M on a gaming PC with 64GB DDR5 RAM and an AMD Radeon 9070 XT with 16GB VRAM.
Before I offloaded 13 Layers to the GPU and had ~10 tokens / S.
Now I've changed it to offload everything to the GPU but enabled the experimental setting in LM Studio "Number of Layers to force the experts into CPU." and put it on 39 Layers.
Now I have 25 tokens / second.
My guess would be that somehow now the GPU handles the routing and it works as intended.

[pwilkin]

@leberschnitzel that option is also known in mainline llama.cpp as --n-cpu-moe 😆

But honestly, just running llama.cpp directly with your intended context without any settings and relying on -fit will probably yield the best results.

[iz0eyj]

I never use 4-bit models.
If sufficient memory is available and the 8-bit model runs fast enough, I don't see much sense in accepting the quality loss caused by excessive compression.
That said, I admit I've never conducted qualitative tests on 80B models, so I'll try to determine whether dropping below 8 bits might start to make sense in that specific case.
It certainly doesn't make sense with the 30B models I used before switching to Qwen Next, as in that case the model proved unsuitable for complex unsupervised tasks.

[leberschnitzel]

I never use 4-bit models. If sufficient memory is available and the 8-bit model runs fast enough, [...]

Yeah that's the problem :D
On my system the 8 bit model fits juuust into RAM, but it's so close that it crashes sometimes.
The 4 bit one quality wise already blows everything out of the water that I've tested so far on my setup, so I'm really happy with it.

But this is really not something related to this issue ;)

[jdchmiel]

I never use 4-bit models. If sufficient memory is available and the 8-bit model runs fast enough, I don't see much sense in accepting the quality loss caused by excessive compression. That said, I admit I've never conducted qualitative tests on 80B models, so I'll try to determine whether dropping below 8 bits might start to make sense in that specific case. It certainly doesn't make sense with the 30B models I used before switching to Qwen Next, as in that case the model proved unsuitable for complex unsupervised tasks.

I thought that the general consensus was that a 4bit quant was within 1-3 % of full 16bit, and 8 was a fraction of a percent. Did you measure the differences in 4 vs 8 bit for some specific use cases that showed it was better to run 8 bit vs a larger model at 4 bit?
I am trying to determine a good way to test the best accuracy I can achieve within 64g VRAM and had been thinking to investigate some 2 bit quants of larger models against the 4 bit versions of smaller models.

[ProgenyAlpha]

@iz0eyj Interesting stance on 4-bit — we actually just ran controlled PPL benchmarks on Qwen3-Coder-Next across quant levels to see if the quality gap is real on 80B MoE. Results on Radeon 890M Vulkan, wikitext-2-raw corpus:

Quant	Size	PPL	Delta from Q4_K_M
Q4_K_M	45 GB	8.3817	baseline
Q5_K_M	53 GB	8.2484	-0.133
Q6_K	62 GB	8.2277	-0.154
Q8_0	85 GB	8.2298	-0.152

Only 0.15 PPL separates Q4_K_M from Q8_0 across 40 GB more RAM. Q6_K lands right in line — essentially identical to Q8_0 despite being 23 GB smaller. MoE models are unusually quant-tolerant because only ~3B of the 80B params fire per token — quantizing dormant experts barely affects output quality.

For dense models like the 30B you mentioned, the gap would be much wider and your instinct to avoid 4-bit makes sense. But for MoE at this scale, Q4_K_M is essentially free quality-wise and leaves 40+ GB of headroom for context, KV cache, or just running other services alongside.

[jdvpro]

I thought that the general consensus was that a 4bit quant was within 1-3 % of full 16bit, and 8 was a fraction of a percent. Did you measure the differences in 4 vs 8 bit for some specific use cases that showed it was better to run 8 bit vs a larger model at 4 bit? I am trying to determine a good way to test the best accuracy I can achieve within 64g VRAM and had been thinking to investigate some 2 bit quants of larger models against the 4 bit versions of smaller models.

The quality depends heavily on the task. I've noticed that overall the difference can indeed be marginal. But there are specific tasks that statistically deviate quite significantly from the general picture. The difference between Q4, Q6, and Q8 is noticeable just by looking at the output. Long story short, for production I always go with Q8 minimum. This is specifically about the Qwen/Qwen3-30B-A3B-Instruct-2507 model. As for other models — I can't say. Only benchmarks will tell.

[loscrossos]

FYI with DDR4 RAM at 3200Mhz in CPU mode i get also 7t/s. It uses 91GB RAM (have 128).

i would expect DDR5 to be faster. using Q4_K_M