Feature Request: Kimi Linear model (Kimi Delta Attention)

[fizzAI]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Support Kimi Linear architecture models such as moonshotai/Kimi-Linear-48B-A3B-Instruct

Motivation

It a gud model, what can I say :)
It also preemptively adds support for an architecture and attention method that Moonshot devs have hinted at using in their next big model; see i.e. https://x.com/bigeagle_xd/status/1983911519541981247

Possible Implementation

Likely blocked for now by the work going on in #16095, as the token mixing mechanism used (Kimi Delta Attention) is a variant of the Gated Deltanet used in Qwen 3 Next. See also the technical report for more details on it

[mattepiu]

This is what gpt-5 reported of the python code inspection of kda commit into flash-linear-attention repo (maybe useful) and what kernel it produced when asked (likely bugged):
https://chatgpt.com/share/69088b9d-7260-800f-abe6-e0efc26baf4d

[BradKML]

Seconding this, please make it happen for both CUDA and MLX/MPS

[rankaiyx]

There's a patch.
Fix FLA import errors
https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/commit/d64a5299ded33ab2609617e05f6bd2cf9b6eef35

[cacaview]

我完成了部分适配工作，以下是完成的工作

1. GGUF Python库修改

文件: gguf-py/gguf/constants.py

• 添加了 MODEL_ARCH.KIMI 架构定义
• 添加了11个KDA专用tensor类型：
  • KDA_Q_CONV, KDA_K_CONV, KDA_V_CONV (1D卷积)
  • KDA_F_A, KDA_F_B (门控投影)
  • KDA_G_A, KDA_G_B (额外门控)
  • KDA_B (beta投影)
  • KDA_A_LOG (A矩阵对数)
  • KDA_O_NORM (输出归一化)
  • KDA_DT_BIAS (delta time偏置)
• 添加了MLA支持：ATTN_Q_A, ATTN_Q_B, ATTN_KV_A_MQA, ATTN_KV_B, ATTN_Q_A_NORM, ATTN_KV_A_NORM
• 添加了MoE支持：expert weights, shared experts, router bias

文件: gguf-py/gguf/tensor_mapping.py

• 映射了所有Kimi特定的HuggingFace tensor名称到GGUF tensor类型
• 支持MoE: block_sparse_moe.experts.*, shared_experts.*, gate.e_score_correction_bias

2. 模型转换脚本

文件: convert_hf_to_gguf.py

• 实现了 KimiLinearModel 类
• MoE tensor合并: 自动合并256个专家的权重 (w1, w2, w3) → FFN_*_EXP
• Tokenizer支持: 覆盖 get_vocab_base()使用 trust_remote_code=True 加载Tiktoken
• 参数提取: 使用 find_hparam() 灵活处理不同的配置键

还有一些C++代码尚未提交，因为它们还未经测试，明天我再发布这些代码
目前模型可以被转换为gguf格式的文件

项目仓库地址在这里：https://github.com/cacaview/llama.cpp

English version uses AI translation

I've completed part of the adaptation work; here's what's done

1. GGUF Python Library Modifications

File: gguf-py/gguf/constants.py

• Added MODEL_ARCH.KIMI architecture definition
• Added 11 KDA-specific tensor types:
  • KDA_Q_CONV, KDA_K_CONV, KDA_V_CONV (1D convolution)
  • KDA_F_A, KDA_F_B (gating projection)
  • KDA_G_A, KDA_G_B (additional gating)
  • KDA_B (beta projection)
  • KDA_A_LOG (A matrix logarithm)
  • KDA_O_NORM (output normalization)
  • KDA_DT_BIAS (delta time bias)
• Added MLA support: ATTN_Q_A, ATTN_Q_B, ATTN_KV_A_MQA, ATTN_KV_B, ATTN_Q_A_NORM, ATTN_KV_A_NORM
• Added MoE support: expert weights, shared experts, router bias

File: gguf-py/gguf/tensor_mapping.py

• Mapped all Kimi-specific HuggingFace tensor names to GGUF tensor types
• MoE support: block_sparse_moe.experts.*, shared_experts.*, gate.e_score_correction_bias

2. Model Conversion Script

File: convert_hf_to_gguf.py

• Implemented the KimiLinearModel class
• MoE tensor merging: Automatically merged weights of 256 experts (w1, w2, w3) → FFN_*_EXP
• Tokenizer support: Overrode get_vocab_base() to load Tiktoken with trust_remote_code=True
• Parameter extraction: Used find_hparam() to flexibly handle different configuration keys

There are still some C++ codes not submitted as they haven't been tested yet; I'll release them tomorrow
Currently, the model can be converted to files in GGUF format

Project repository is here：https://github.com/cacaview/llama.cpp

[jacekpoplawski]

@cacaview could you create a pull request?

[cacaview]

Completed Work

1. Architecture Registration (llama-arch.cpp/h)

• Register the LLM_ARCH_KIMI architecture
• Define tensor name mappings for all KDA/MLA-related components

2. Model Loading (llama-model.cpp/h)

• load_hparams: Load Kimi model hyperparameters (MLA/KDA dimensions, MoE configuration)
• load_tensors: Load 603 tensors (KDA layers, MLA layers, MoE FFN)
• build_graph: Call llm_build_kimi

3. Inference Graph Construction (src/models/kimi.cpp)

• build_kda_attn: KDA linear attention layer
• build_mla_attn: MLA low-rank attention layer
• build_moe_ffn_layer: MoE feed-forward network

4. Model Conversion (convert_hf_to_gguf.py)

• KimiLinearModel class: Handle TikToken tokenizer
• Correctly map tensor names and dimensions

5. Running Status

PS H:\llama\llama.cpp> cmd /c ".\build-cuda\bin\Release\llama-cli.exe -m kimi-Q4_K_M.gguf -p ""1+1="" -n 20 --no-warmup -no-cnv -ngl 10 2>&1" | Select-Object -Last 30
llama_context: graph nodes  = 1387
llama_context: graph splits = 241 (with bs=512), 3 (with bs=1)        
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
main: llama threadpool init, n_threads = 10

system_info: n_threads = 10 (n_threads_batch = 10) / 16 | CUDA : ARCHS = 860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

sampler seed: 3726446019
sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000       
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist     
generate: n_ctx = 4096, n_batch = 2048, n_predict = 20, n_keep = 0    

1+1=ongongongongongongongongongongongongongongongongongongongong      

common_perf_print:    sampling time =       6.49 ms
common_perf_print:    samplers time =       2.41 ms /    24 tokens    
common_perf_print:        load time =     544.43 ms
common_perf_print: prompt eval time =     541.39 ms /     4 tokens (  135.35 ms per token,     7.39 tokens per second)
common_perf_print:        eval time =     777.07 ms /    19 runs   (   40.90 ms per token,    24.45 tokens per second)
common_perf_print:       total time =    1328.62 ms /    23 tokens    
common_perf_print: unaccounted time =       3.67 ms /   0.3 %      (total - sampling - prompt eval - eval) / (total)
common_perf_print:    graphs reused =         18
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free     self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - CUDA0 (RTX 4070)   | 12281 =    0 + (12142 = 10876 +     640 +     626) +         138 |
llama_memory_breakdown_print: |   - Host               |              
   18527 = 17435 +    1088 +       4                |

I've completed some work – the model is generating outputs, but it's not functioning properly.（output like 1+1=ongongong...） I suspect the issue lies with the attention layers.
I'll continue to fix the KDA and MLA layers.

[cacaview]

@cacaview could you create a pull request?

Sorry for getting back to you so late. Since I have classes most of the time and don’t have much free time, I can’t reply to messages promptly. I plan to submit the PR after debugging the code until the model runs correctly. If you’re interested, you can check out my forked repository first.

[engrtipusultan]

@pwilkin when you are free from Qwen3 next. Will you be creating PR for this?

[pwilkin]

@engrtipusultan Yup, should be up in a few days.

[engrtipusultan]

@engrtipusultan Yup, should be up in a few days.

A big thank you to you and all the contributors and maintainers

[cacaview]

I think the model is working properly now

This is the log file：log.md

[pwilkin]

@cacaview Nice job! I've looked at your code and it looks really promising. A couple of remarks:

-> the ggml_kda_scan has to be rewritten to GGML-style tensor code similar to what has been done in Qwen3Next, I suspect you might want to use SOLVE_TRI for that part (I know the KDA kernel uses it)
-> the convention with tensor names is we try to use the existing tensor mappings as much as possible instead of adding new ones, so you should reuse the existing tensors (SSM_CONV1D, SSM_DT, SSM_OUT, SSM_A - since it's not SSM_A_LOG as it's pre-converted) where possible and add new ones in the general SSM_ prefix instead of the specific KDA one. Also, you added them in the wrong section (vision projector).
-> the invalid branches shouldn't have fallbacks, but GGML_ABORT() with an error message instead