MLA model support (GLM-5.1 / DeepSeek-style): timeline + missing zhongzhu/glm-mla branch

[gavingavin99]

Hi, thanks for the great work on OSCAR — the INT2 KV-cache results are impressive.

I'm interested in applying OSCAR to MLA-type models (GLM-5.1 / DeepSeek-style, where the KV cache is a compressed latent rather than full per-head K/V). I have two questions:

1. The referenced branch doesn't seem to be public.
   The Model support table lists:

▎ | GLM-5.1 | SGLang zhongzhu/glm-mla | 🧪 experimental (MLA latent) |

but zhongzhu/glm-mla doesn't appear in the branch list and returns 404. The other experimental branches (zhongzhu/hybrid-model, zhongzhu/VL, etc.) are all pushed — is glm-mla just not pushed yet, or
did it get renamed/merged elsewhere?

2. Roadmap for MLA.
   The README notes MLA models are "not on main yet," and the Latest News mentions GLM 5.2 / MiniMax 3 being tested for long-horizon agentic tasks. Could you share:

• a rough timeline for MLA latent KV support (calibration pipeline + serving path), and
• whether OSCAR's covariance-aware rotation is applied to the compressed latent directly, or reconstructed K/V — i.e. how the per-layer rotation/clipping is adapted when there's a single shared latent
  instead of per-head K/V.

Happy to help test on GLM-5.1 if there's a WIP branch I can pull. Thanks!

[zzz0906]

Thanks for the kind words — and good catch on the dead link. On zhongzhu/glm-mla (the 404). You're right: that branch isn't public yet. It's an internal WIP branch and just hasn't been pushed. I plan to push a pullable WIP MLA branch before next weeks once it's cleaned up, and we'd genuinely welcome your help testing on GLM-5.1 / 5.2 / Minimax M3 — I'll follow up on this thread when it's live.

How the rotation adapts to the latent (your Q2b). OSCAR operates on the compressed latent directly, not on reconstructed per-head K/V: - For full-attention models, OSCAR fits two covariance-aware rotations per layer — a query-covariance rotation for K and a score-weighted V-covariance rotation — and these act per-(KV-)head. - For MLA there's no per-head K/V in the cache; what's stored is the single shared latent c_kv (dim kv_lora_rank). So OSCAR fits one rotation per layer on the latent: estimate the latent covariance Σ = c_kvᵀ c_kv over the calibration tokens, eigendecompose (eigh), and order eigenvectors by ascending eigenvalue so low-variance directions pack into the same INT2 groups (which clips cleanly). The result is a single [kv_lora_rank × kv_lora_rank] orthogonal matrix per layer, and the clip thresholds are fit on the rotated latent. - Per-head structure only re-emerges after the kv_b_proj up-projection at attention time — the cache holds the rotated+quantized latent, and the up-projection runs on the dequantized latent, so the rotation never has to be per-head.

The open refinement (and exactly the tradeoff you point at) is making the latent rotation aware of the kv_b_proj up-projection — i.e. shaping the INT2 error in the latent directions that matter most after reconstruction, rather than treating the latent in isolation. That's part of what the WIP branch is for. Happy to ping you here the moment the MLA branch is pushable. Thanks again for the careful read @gavingavin99. Also, very welcome to give a hands-on doing PR for this.

[pqzm1155]

@zzz0906 Hello, thank you for your excellent work. Could you please share any updates regarding the GLM5.2 model? And When will the repo support it?
Many thanks.

[zzz0906]

Hi @pqzm1155, the local code has been finished and verified; I am benchmarking two different designs regarding ROPE+NOPE; I will push out our final design once it's down. It shall be done this week.

[jiashuwu666]

Hi @pqzm1155, the local code has been finished and verified; I am benchmarking two different designs regarding ROPE+NOPE; I will push out our final design once it's down. It shall be done this week.

@zzz0906 Greeting, thanks for your excellent work, just wondering will zhongzhu/glm-mla branch be available soon? Looking forward to it. Also, just wondering is there any plan to support GLM 5.1 & 5.2 for vllm?