Feature Request: Support Kimi K2.5

[AesSedai]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Kimi K2.5 is moonshot's newest Kimi model, and now supports multimedia (image and video) input. I haven't tried to convert it yet, but I suspect due to the new vision component that a conversion would fail.

Similar to Kimi K2 Thinking, this model has the experts in native INT4 quantization.

Motivation

This is the newest iteration / release of Kimi and provides new vision input features, making it uniquely positioned as one of the largest open source LLMs that supports multimodality.

Possible Implementation

No response

[cpumaxx]

python3 /usr/src/llama.cpp/convert_hf_to_gguf.py --outfile /media/models/kimi-k2.5/kimi-k2.5-q8.gguf --outtype q8_0 --verbose /media/models/kimi-k2.5/

NFO:hf-to-gguf:Loading model: kimi-k2.5
WARNING:hf-to-gguf:Failed to load model config from /media/models/kimi-k2.5: The repository /media/models/kimi-k2.5 contains custom code which must be executed to correctly load the model. You can inspect the repository content at /media/models/kimi-k2.5 .
 You can inspect the repository content at https://hf.co//media/models/kimi-k2.5.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:Model architecture: DeepseekV3ForCausalLM
WARNING:hf-to-gguf:Failed to load model config from /media/models/kimi-k2.5: The repository /media/models/kimi-k2.5 contains custom code which must be executed to correctly load the model. You can inspect the repository content at /media/models/kimi-k2.5 .
 You can inspect the repository content at https://hf.co//media/models/kimi-k2.5.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
...snip...
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:blk.0.attn_norm.weight,       torch.bfloat16 --> F32, shape = {7168}
INFO:hf-to-gguf:blk.0.ffn_down.weight,        torch.bfloat16 --> Q8_0, shape = {18432, 7168}
INFO:hf-to-gguf:blk.0.ffn_gate.weight,        torch.bfloat16 --> Q8_0, shape = {7168, 18432}
INFO:hf-to-gguf:blk.0.ffn_up.weight,          torch.bfloat16 --> Q8_0, shape = {7168, 18432}
INFO:hf-to-gguf:blk.0.ffn_norm.weight,        torch.bfloat16 --> F32, shape = {7168}
INFO:hf-to-gguf:blk.0.attn_kv_a_norm.weight,  torch.bfloat16 --> F32, shape = {512}
INFO:hf-to-gguf:blk.0.attn_kv_a_mqa.weight,   torch.bfloat16 --> Q8_0, shape = {7168, 576}
INFO:hf-to-gguf:blk.0.attn_k_b.weight,        torch.bfloat16 --> Q8_0, shape = {128, 512, 64}
INFO:hf-to-gguf:blk.0.attn_v_b.weight,        torch.bfloat16 --> Q8_0, shape = {512, 128, 64}
INFO:hf-to-gguf:blk.0.attn_output.weight,     torch.bfloat16 --> Q8_0, shape = {8192, 7168}
INFO:hf-to-gguf:blk.0.attn_q_a_norm.weight,   torch.bfloat16 --> F32, shape = {1536}
INFO:hf-to-gguf:blk.0.attn_q_a.weight,        torch.bfloat16 --> Q8_0, shape = {7168, 1536}
INFO:hf-to-gguf:blk.0.attn_q_b.weight,        torch.bfloat16 --> Q8_0, shape = {1536, 12288}
INFO:hf-to-gguf:blk.1.attn_norm.weight,       torch.bfloat16 --> F32, shape = {7168}
Traceback (most recent call last):
  File "/usr/src/llama.cpp/convert_hf_to_gguf.py", line 11276, in <module>
    main()
    ~~~~^^
  File "/usr/src/llama.cpp/convert_hf_to_gguf.py", line 11270, in main
    model_instance.write()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/src/llama.cpp/convert_hf_to_gguf.py", line 683, in write
    self.prepare_tensors()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/src/llama.cpp/convert_hf_to_gguf.py", line 7556, in prepare_tensors
    super().prepare_tensors()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/src/llama.cpp/convert_hf_to_gguf.py", line 554, in prepare_tensors
    for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                                 ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/llama.cpp/convert_hf_to_gguf.py", line 7522, in modify_tensors
    datas.append(self._experts[bid][ename])
                 ~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'model.layers.1.mlp.experts.0.down_proj.weight'

I have not tried patching to allow custom code execution.

[Lissanro]

I am still downloading the new model, but since it is INT4 like K2 Thinking, information about creating Q4_X quant to preserve the original quality may still mostly apply (since higher quant like Q8 would not provide any quality increase): https://huggingface.co/ubergarm/Kimi-K2-Thinking-GGUF - of course, this does not help with image/video support, but I thought worth sharing since it is relevant to the text generation part of this model too.

[AesSedai]

Agreed, I've been following @jukofyork's work from this comment locally: #17064 (comment)

Along with the updated quantize_row_q4_K_ref from this comment later on: #17064 (comment)

So far I've had to modify the config.json to look more like the regular Kimi-K2 Thinking config.json (so, removing the text_config key in the config and de-indenting and deduplicating the inner values for that so they sit at the top level of the JSON now, updating the arch to DS and the model_type to just "kimi_k2") and also removing the mm and vision_tower entries from the model.safetensor.index.json.

I've just finished successfully making the decompressed BF16 gguf and am working on the Q4_X quant now. The caveat here being that there's no vision-related elements to the quant, it's text-to-text only, but that's still going to be fun to play with :)

[AesSedai]

I've successfully produced a text-only Q4_X quant following @jukofyork's Q8_0 / Q4_0 schema and I'm running the wiki.test.raw corpus through it for perplexity now and should have results in a couple of hours.

I did a few inference runs on the model and it looks sane, and the PPL chunks so far are in the mid-1.5 / 2.1-ish range so things are looking good.

I've also begun to upload the 556663.41 MiB Q4_X quant and it will be available in a few hours at: https://huggingface.co/AesSedai/Kimi-K2.5-GGUF

Edit: The Q4_X text-only GGUF quant is uploaded now.

[Amerisium]

Is it possible to add mmproj by referring to mistralai/Mistral-Large-3-675B-Instruct-2512 of "Inage-Text to Text" which uses the same DeepSeek architecture?

[AesSedai]

@Amerisium it's not quite that easy, but I'm taking a crack at it. It uses the same MoonViT as the already-supported KimiVLForConditionalGeneration model, but the modeling code is a bit different with how the configs are laid out and a couple other changes.

If I get it working, I'll share the mmproj.

[cybrah]

Reasoning content separation not working for Kimi K2.5

I've been testing Kimi K2.5 with llama-server and found that the reasoning_content field is not being populated in API responses, even with --reasoning-format deepseek enabled.

Model: unsloth/Kimi-K2.5-UD-Q3_K_XL
Flags: --jinja --special --reasoning-format deepseek

The Kimi K2.5 Jinja chat template injects at the end of the prompt to force the model into thinking mode:

"prompt":"...<|im_assistant|>assistant<|im_middle|>"

This means the opening tag is part of the prompt, not the model's output. The model then generates thinking content and outputs only the closing tag followed by the actual response.

The reasoning format parser appears to expect both and in the model's generated output, so it fails to separate the content. The result is that everything gets dumped into message.content with embedded in the middle, and reasoning_content is never populated.

Example response:

json

{
  "message": {
    "role": "assistant",
    "content": "The user is asking... I'll go with a straightforward answer.\n\n2+2 equals 4."
  }
}

The </think> tag is visible in __verbose.content but stripped from the main content without being properly parsed:

"__verbose": {
  "content": "The user is asking... straightforward answer.</think>2+2 equals 4."
}

Expected behavior:
json

{
  "message": {
    "role": "assistant", 
    "content": "2+2 equals 4.",
    "reasoning_content": "The user is asking... I'll go with a straightforward answer."
  }
}

Tested reasoning formats:

    --reasoning-format deepseek - no reasoning_content field
    --reasoning-format deepseek-legacy - no reasoning_content field
    --reasoning-format none - </think> present in content but no opening tag

Not sure if I should raise a separate issue or if this is in scope with OPs issue (it is about supporting k2.5 so put it here anyway)!

[jukofyork]

You have to edit the jinja template:

https://huggingface.co/unsloth/Kimi-K2.5-GGUF/discussions/1#697b46fdf48287bb9c2e92dc

I've figured out what is causing this:

https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/chat_template.jinja

{%- if add_generation_prompt -%}
  <|im_assistant|>assistant<|im_middle|>
  {%- if thinking is defined and thinking is false -%}
  <think></think>
  {%- else -%}
  <think>        # <--- This forced <think> tag doesn't set returned as part of the chat.
  {%- endif -%}
{%- endif -%}

To fix this you need to make a copy of chat_template.jinja and delete the {%- else -%} <think> part at the bottom like this:

{%- if add_generation_prompt -%}
  <|im_assistant|>assistant<|im_middle|>
  {%- if thinking is defined and thinking is false -%}
  <think></think>
  {%- endif -%}
{%- endif -%}

Then run llama.cpp using --jinja --chat-template-file chat_template.jinja options.

(Also: there doesn't seem to be any need/requirement to use --special option now)

[cybrah]

You're a hero @jukofyork

That works perfectly thank you!

[AesSedai]

I've updated my PR that adds image input support to Kimi-K2.5. It'll definitely need review by ngxson, but from my testing it appears to work. I would not be surprised if there were a couple of bugs, but I feel that it's not broken as far as I can tell.

I've also uploaded mmproj files to my repo. You'll need to pull and compile the code from my PR to use the vision component.

[chrislawso]

disregard

[AesSedai]

Merged in #19170