[M3 Ultra] Metal: 13% throughput regression in token generation (tg128) vs March 2026 builds

[jecruz]

Bug Description

M3 Ultra (Apple Silicon, 96GB unified memory) shows significant token generation throughput regression in recent Metal backend builds compared to March 2026 builds.

Hardware

• Apple M3 Ultra
• 96GB unified memory
• macOS (Metal backend)

Model

Qwen3.5-35B-A3B-Q8_0 (34.36 GiB GGUF)

Expected Behavior

Token generation (tg128) should achieve ~85 t/s as seen in March 28, 2026 builds.

Actual Behavior

Token generation (tg128) is limited to ~73-74 t/s — a 13% regression.

Benchmark Data

Test	March 28 (build 178)	Current (build 8860)	Regression
f16 tg128	85.48 t/s	73.43 t/s	-13%
f16 pp512	~1807 t/s	1808 t/s	-1%
f16 pp128+tg128	~137 t/s	137.61 t/s	+1%
f16 pp8192	~2106 t/s	1183 t/s	-44%
turbo3 tg128	78.47 t/s	63.23 t/s	-19%

Note: pp8192 shows a severe 44% regression, suggesting the issue may be related to larger batch/prefill sizes.

Root Cause Investigation

• Same model, same hardware confirmed
• Pre-merge vs post-merge shows same regression (not from any fork-specific changes)
• Upstream base (build 8401, ggml 0.9.7) also shows regression
• Reverting N_FORWARD from 64→8 makes it worse (69.92 t/s), so concurrency changes are not the cause

Suspected Commits (not bisected)

These Metal kernel commits landed between build 178 and current:

1. b54124110 — metal: fix q5_k mul_mv register spill (metal : fix q5_k mul_mv register spill #20399)
2. e22cd0aa1 — metal: extend mul_mv_ext to BF16, Q2_K, Q3_K (metal : extend mul_mv_ext to BF16, Q2_K, Q3_K #20250)
3. 1725e316c — models: optimize qwen3next graph (models : optimizing qwen3next graph #19375)

The pp8192 regression (-44%) suggests the issue may be in graph construction or large batch handling.

Test Command

./build/bin/llama-bench \
  -m model.gguf \
  -ctk f16 -ctv f16 \
  -pg 128,128 -pg 8192,128 \
  -t 1 -r 3 -o md

Tags

• performance
• hardware
• macos

[MarkErik]

@jecruz I'd like to test this out on my M3U as well, but I'm not familiar with the build number you a referencing for March 2026. I'm only familiar with the thousands e.g. 8860 numbered builds.

Is there a specific one from March you'd like me to test?

[pjsgsy]

fwiw - after update, I am seeing MASSIVE t/s regression on this same model, but on Win/CUDA. I offload all to CPU, but t/s dropped from ~21 to as little as 5!

I've been battling with it for a few hours. Quant type does not seem to matter. Pretty useless for me now. I'm not sure exactly when this happened, but it seems to have occurred in the last week since I pulled working 21 t/s binaries.

Happy to try any fixes, as I can build from source (and have my own fork too, that has the same issue AFTER pulling from the latest ggml side!)

[pjsgsy]

I reverted to March 27, tested, and the issue seemed to go away. Ended up removing ec16a07 New mul_mat_vec_q_moe kernel for BS > 1 and it seems I am OK again. That was kind of a massive regression! I am back in spitting distance of my old t/s now. And, just to confirm, this was on Qwen3.5-35B-A3b

[pjsgsy]

I'm assuming this was related to my issue, though I am not Apple. If not, and you want me to open another ticket, let me know

[ggerganov]

@pjsgsy Open a new issue and provide hardware info, commands and logs. Tag @gaugarg-nv

[gaugarg-nv]

@pjsgsy Thanks for reporting the issue. As Georgi mentioned, please file a new issue with more details - HW, CTK version, quants used, commands.
In my and @JohannesGaessler's testing, we had not seen such regressions with PR #20905.

[jecruz]

Hi @MarkErik, the "build 178" / "build 8860" are internal CI build numbers from my own build pipeline, not public identifiers. The regression was introduced on March 29, 2026, by commit ec16a07 (Optimize MOE GEMV kernel for BS > 1) - which was a CUDA MOE kernel optimization, but touches shared MoE code that Metal also uses. Another user (@pjsgsy) independently confirmed that reverting this commit fixed their performance. The last known good commit is f5d1c41. You should see ~85 t/s at that point vs ~73 t/s on the current master (a33faba8e) using Qwen3.5-35B-A3b.

[gaugarg-nv]

but touches shared MoE code that Metal also uses.

I don't think this is correct. This commit doesn't make changes to any shared code with Metal. This is all CUDA-specific.
If there is a regression in the CUDA path, we will need more details for repro.