Feature Request: Support for GLM 5.2

[artraze]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add support for GLM 5.2, which makes use of sparse attention acceleration by skipping top-k index calculation for some layers

Motivation

GLM 5.2 is currently the top ranked open weights model.

Possible Implementation

The main change since 5.1 seems to be the sparse sparse attention, specifically the index_skip_topk_offset: 3 and index_topk_freq: 4 which signal that the sparse attention is only recalculated every 4 layers after the first 3, while in GLM 5.1 it was calculated every layer. A indexer_types: [...] is provided, but seems to just codify the offset+freq.

(I would figure this is being worked on, but I couldn't find any discussion, PR, or issue so I figured I would make one. If no one is actually looking into it, I could give it a shot at some point today.)

[JMPSequeira]

@artraze please do take a shot at it 🥇

[SharkWipf]

FWIW: The model works (seemingly, at a glance, in my quick testing) fine without changes, out of the box, as fine as GLM 5.1 does.
While I'm all for implementing the new architectural changes, they don't seem necessary to run GLM 5.2 in llama.cpp. And keep in mind, even GLM 5.1 isn't fully implemented in llama.cpp yet. There are still several architectural features partially or entirely missing.

[JMPSequeira]

FWIW: The model works (seemingly, at a glance, in my quick testing) fine without changes, out of the box, as fine as GLM 5.1 does. While I'm all for implementing the new architectural changes, they don't seem necessary to run GLM 5.2 in llama.cpp. And keep in mind, even GLM 5.1 isn't fully implemented in llama.cpp yet. There are still several architectural features partially or entirely missing.

What quant and rig you running? and what speeds?

[SharkWipf]

What quant you running?

Unsloth ud_q4-k-xl

and what speeds?

That's very hardware-dependent, heh. But identical to GLM 5.1 on the same hardware and same settings/context lengths, anecdotally, which is around 6-8tok/s, which makes sense since it's running the exact same way.
With the GLM 5.2 architectural enhancements it should be faster than GLM 5.1 at long contexts.
I have not tested actual long context runs though. Even launching at 1m context means I have to shift my offloading to the point where even small generations run at 0.3t/s on my hardware.

Edit:

and rig [edit]

Zen3 threadripper, 512gb ram, rtx 6000 blackwell (cuda), 3090 (cuda), Intel b60 (vulkan), with moe trunk on the rtx 6000 and custom layer-by-layer (not expert by expert) offloading.

Have fun decoding:

./build/bin/llama-server -m /mnt/sda1/models/GLM-5.2-GGUF-Q4_K_XL/GLM-5.2-UD-Q4_K_XL-00001-of-00011.gguf --keep -1 --offline --no-context-shift --flash-attn on --jinja --temp 1 --backend-sampling --parallel 1 --fit off --threads 64 -dev CUDA0,CUDA1,Vulkan1 --tensor-split 1,0,0 --n-gpu-layers 9999 -ot (?:^|\.)blk\.(?:(?:9|1[0-1]))\..*ffn_(gate|up|down)_exps\.=CUDA1,(?:^|\.)blk\.(?:1[2-4])\..*ffn_(gate|up|down)_exps\.=Vulkan1,(?:^|\.)blk\.(?:(?:1[5-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]|6[0-9]|7[0-8]))\..*ffn_(gate|up|down)_exps\.=CPU --batch-size 16384 --ubatch-size 16384 --port 8124 -c 294912

(FWIW: this layer-wise uploading seems to offer roughly twice the performance of naive expert-wise offloading on this hardware/model class)

[artraze]

FWIW: The model works (seemingly, at a glance, in my quick testing) fine without changes, out of the box, as fine as GLM 5.1 does.

Unsloth ud_q4-k-xl

That is a hack unsloth put together... Rather than honoring the index caching, they replicated the index tensors for the cached layers (I'm assuming- their gguf has indexers every layer and that's the only explanation I can think of). Even if we accept that as functional, which it probably is, it still adds size, memory, and a lot of compute overhead.

Worse, though, from this ticket's perspective, the hack hasn't landed in llama.cpp :). So even if this is just a ticket against llama-quantize or convert_hf_to_gguf.py, GLM-5.2 isn't supported.

[SharkWipf]

Okay that's fair. I assumed Unsloth just used the standard conversion script, especially since I've seen a few other pre-Unsloth GLM 5.2 ggufs on HF already as well. Though as mentioned, I see no resource usage increase over GLM 5.1 run in the same way. So whatever they did, it seems to be working as well as (also Unsloth) GLM 5.1 at the very least. In fact I'm anecdotally able to push it slightly further than I could GLM 5.1 Unsloth, apples-to-apples.

[artraze]

even GLM 5.1 isn't fully implemented in llama.cpp yet

Ah, after looking at it a bit, I hadn't realized that llama.cpp still isn't using DSA for GLM-5 (is that correct?). That's probably not good. I guess those hacked tensors could be 0 too, then...

This might end up being a larger project than I thought, and fixing llama-quantize and/or convert_hf_to_gguf.py (i.e. adding support for cached sparse attention) might be the simplest option in the near term.

[SharkWipf]

(is that correct?)

Yeah exactly. Also GLM MTP isn't implemented yet either AFAIK (though that one should be relatively trivial to add now that Qwen/Gemma MTP has been implemented). And I thought there was something else big missing too. Lightning indexer? //CC @fairydreaming
(Though interestingly, even with the reduced output quality from lacking full DSA, GLM-5.1 has been my favorite (non-agentic) model by a long shot, outperforming every other model at many tasks, including the SOTA proprietary ones)

[burnsjordan559-max]

I agree that the lack of full DSA and MTP support is likely impacting the performance and quality of GLM-5.1 and 5.2 in llama.cpp. If MTP implementation is now more straightforward following the Qwen and Gemma updates, that could be a good starting point. I'll look into the feasibility of adding support for the cached sparse attention in the conversion scripts as a near-term fix, but it sounds like there is a fair amount of architectural debt to catch up on for this model family.

…

On Thu, Jun 18, 2026, 10:45 AM artraze ***@***.***> wrote: *artraze* left a comment (ggml-org/llama.cpp#24730) <#24730 (comment)> even GLM 5.1 isn't fully implemented in llama.cpp yet Ah, after looking at it a bit, I hadn't realized that llama.cpp still isn't using DSA for GLM-5 (is that correct?). That's probably not good <https://www.reddit.com/r/LocalLLaMA/comments/1rq8otd/running_deepseek_v32_with_dense_attention_like_in/>. I guess those hacked tensors could be 0 too, then... This might end up being a larger project than I thought, and fixing llama-quantize and/or convert_hf_to_gguf.py (i.e. adding support for cached sparse attention) might be the simplest option in the near term. — Reply to this email directly, view it on GitHub <#24730?email_source=notifications&email_token=CA7KD7EP54D23X2QDD54YOL5AQFBDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZUGM3DQNRRG4ZKM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4743686172>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CA7KD7FCDBSNHQ5XGXT2BML5AQFBDAVCNFSNUABFKJSXA33TNF2G64TZHM3DCMRTGU2DOOBUHNEXG43VMU5TINRYGUYTEMRYGM32C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CA7KD7FQY3MXZWQ2YOXSAS35AQFBDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZUGM3DQNRRG4ZKM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/CA7KD7A5KMETV3L2UBRRAH35AQFBDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZUGM3DQNRRG4ZKM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[fairydreaming]

@artraze @SharkWipf Right, GLM 5/5.1 in llama.cpp still uses dense attention without lightning indexer. But adding a full DSA implementation to the current LLM_ARCH_GLM_DSA architecture doesn't make much sense until either #24231 is merged (so there's a new lightning indexer OP) and is supported by most backends (CUDA etc) OR some magic GGML OP fusion mechanism is implemented that would make it unnecessary. Otherwise people suddenly would find that their favourite model is slightly smarter, but works much slower than expected and uses exorbitant amounts of compute buffer memory (like DeepSeek V3.2 currently).

[artraze]

@fairydreaming Thanks for the insight, and the implementation efforts :). Yeah, I had gone into this assuming that it was just Z.ai's IndexCache that needed to be implemented and hadn't realized that DSA itself was still a WIP. Agreed that it doesn't make much sense to implement it for GLM before it's solved for DS3.2.

[juj]

I have been running llama.cpp with CPU inferencing on 2x Epyc 7763 128-core/256-thread CPU with 512GB of DDR4 RAM on unsloth/GLM-5.2-UD-Q4_K_XL to generate overnight "Read code in this project and analyze it for bugs" reports over a small codebase.

Compared to unsloth/MiniMax M2.7 BF16, the quality that GLM 5.2 produces is a night-and-day difference. MiniMax M2.7 does find some bugs, but also makes a large amount of trivial mistakes. After scanning the project several times with MiniMax M2.7 and no longer finding interesting actionable points, I switched to scanning with GLM 5.2, and wow, the improvement in the quality of reports is outstanding.

However, the difficult part is speed: MiniMax M2.7 took a bit less than three hours to complete a scan through the project. Prompt processing starts off at around 90 t/sec, and token generation at 5.5 t/sec, dropping into pp 50 t/sec, tg ~3t/sec towards the end of the scan.

But GLM 5.2 starts off at only pp ~5 t/sec, tg 2.2 t/sec, and drops into pp 0.8 t/sec, tg 0.3 t/sec towards the end of the scan. And in total, GLM 5.2 takes about 60 hours to run the same prompt as MiniMax M2.7 did in 3h.

So I would like to raise the question as to whether there would exist ways to speed up GLM 5.2 inferencing from the current state in llama.cpp? I know that GLM 5.2 is a quite a bit bigger model than MiniMax M2.7, though is the speed difference explained by the difference in number of neurons alone?

As far as code analysis quality goes, GLM 5.2 has been impressive for me when other large local models have been a bit disappointing. If the performance of GLM 5.2 could be improved to run closer to the current MiniMax M2.7 on CPU inferencing, that would absolutely enable overnight code analysis activities with the model.

[SharkWipf]

Apples to apples, model performance linearly scales with parameter count, and GLM 5.2 is 3x the size of MiniMax M2.7 and thus expected to be at least 3x as slow, under the same conditions.
That said, model architecture also factors in. Q4_K_XL should be faster than bf16 on a memory bandwidth constrained machine, too.
I expect speculative decoding/MTP/Dflash won't help much with pure CPU processing. I'm not sure how much, if any, effect implementing the lightning indexer would have here. And a proper DSA implementation I think (positively) affects output quality, not (positively) output speed.
From what I can tell GLM 5.x also just is a fairly heavy model architecturally and even Zhipu themselves seem to struggle with that.

Edit: Activated parameter count*, not just parameter count.