UPSTREAM PR #17592: Feature/kimi linear support

[loci-dev]

Mirrored from ggml-org/llama.cpp#17592

Make sure to read the contributing guidelines before submitting a PR
This is the current work progress：
ggml-org/llama.cpp#16930 (comment)

[loci-review]

Explore the complete analysis inside the Version Insights

Performance Analysis Summary - PR #363: Kimi-Linear Support

PR Overview: Adds support for Kimi-Linear-48B model with hybrid MLA+KDA architecture across 24 files (1,676 additions, 29 deletions).

---

Key Findings

Code Changes Overview

This PR implements a new model architecture combining Multi-head Latent Attention (MLA) and Kimi Delta Attention (KDA) with recurrent state management. The implementation adds:

• New architecture type LLM_ARCH_KIMI with 27 layers (20 KDA recurrent, 7 MLA attention)
• New GGML operation GGML_OP_KDA_SCAN for delta attention recurrence
• Extended model parameter functions to calculate KDA-specific state sizes
• New tensor types for KDA convolution weights and projection matrices
• MoE support with sigmoid gating and shared experts

Performance-Critical Function Changes

Model Parameter Accessors (src/llama-hparams.cpp):

• n_embd_s(): Response time increased by 100 ns (54 ns → 154 ns). Added conditional logic to calculate KDA state size (128 × 128 × 32 = 524,288 elements). The function now includes an additional branch checking kda_head_dim != 0 and calls n_head() which adds 60 ns of callee overhead.

• n_embd_r(): Response time increased by 130 ns (119 ns → 249 ns). Added conditional logic to calculate KDA convolution state size for Q, K, V (3 × 3 × 4,096 = 36,864 elements). Similar pattern with 60 ns callee overhead from n_head() invocation.

Architecture Classification (src/llama-arch.cpp):

• llm_arch_is_recurrent(): Response time increased by 62 ns (37 ns → 99 ns). Added LLM_ARCH_KIMI case to switch statement. The 169% increase reflects expanded conditional logic affecting all architecture checks across the codebase.

SSM Convolution (ggml/src/ggml-cpu/ops.cpp):

• ggml_compute_forward_ssm_conv_f32(): Response time increased by 392 ns (1,089 ns → 1,481 ns). Added debug instrumentation code (static variable and conditional checks) inside the computation loop. While debug output is disabled (do_conv_debug = false), the conditional branch and static variable access add overhead in the hot path.

New KDA Scan Operation (ggml/src/ggml-cpu/ops.cpp):

• ggml_compute_forward_kda_scan_f32(): New 192-line function implementing KDA recurrence with approximately 82,000 floating-point operations per token per head. Includes L2 normalization (2 sqrt operations), state validation (16,384 checks), exponential decay (128 exp operations), and three matrix-vector products (O(head_dim²) each). Uses dynamic memory allocation (malloc/free) for temporary buffers in the compute path.

Inference Impact Analysis

Tokens Per Second Impact:

The core inference functions (llama_decode, llama_encode, llama_tokenize) are not directly modified in this PR. However, the changes affect inference through:

• Model initialization overhead from parameter accessor regressions (230-260 ns cumulative per initialization)
• Graph building overhead from architecture checks (62 ns per check)
• For Kimi models specifically: KDA scan adds ~82,000 ops/token/head × 20 layers × 32 heads = 52.5M operations per token

Reference Impact: For the model (ollama://smollm:135m) on CPU (12th Gen Intel Core i7-1255U, Ubuntu 24.04.3 LTS, x86_64), a 2 ms increase in llama_decode results in 7% fewer tokens per second. The changes in this PR add sub-microsecond overhead to existing models (parameter accessors: ~400 ns total, architecture checks: ~62 ns). This translates to negligible tokens/second impact for existing models (<0.02% reduction).

Impacted Functions for Inference:

• None of the primary inference functions (llama_decode, llama_encode, llama_tokenize) were modified
• Indirect impact through initialization and graph building functions
• Kimi-specific inference uses new llm_build_kimi graph builder with KDA scan operations

Power Consumption Analysis

Binary-Level Impact:

• libllama.so: Power consumption increased by 0.952% (193,067 nJ → 194,905 nJ, +1,839 nJ). Primary contributors are STL container accessor regressions and model parameter function overhead distributed across tokenization, sampling, and model initialization code paths.

• libggml-cpu.so: Power consumption increased by 0.752% (115,347 nJ → 116,215 nJ, +868 nJ). Primary contributor is the SSM convolution debug code overhead affecting Mamba and Kimi model operations.

• libggml-base.so: Power consumption increased by 0.175% (59,180 nJ → 59,283 nJ, +103 nJ). Minor changes from supporting infrastructure updates.

• Other binaries: No measurable power consumption changes (libggml.so, libmtmd.so, llama-bench, llama-cvector-generator, llama-run, llama-tokenize, llama-tts all show 0.0% change).

Total Impact: Cumulative power consumption increase of approximately 2,810 nJ across all modified binaries, representing a modest increase distributed across multiple small regressions rather than a single dominant source.

[loci-review]

Explore the complete analysis inside the Version Insights

Based on the diff, this appears to be a Metal GPU backend change for the llama.cpp project. The changes primarily involve:

1. Removal of use_bfloat flag from context
2. Addition of tensor API support detection (has_tensor)
3. Major refactoring of matrix multiplication kernels (kernel_mul_mm and kernel_mul_mm_id)
4. Addition of ggml_metal_library_init_from_source function
5. Runtime testing for bfloat support with tensor API

However, since the performance analysis tools are returning errors, let me provide a summary based on the code changes:

Performance Analysis Summary

The changes in this commit focus on Metal GPU backend enhancements and do not directly impact CPU-based inference performance metrics. The modifications are specific to Apple Metal GPU acceleration and include refactoring of matrix multiplication kernels with conditional tensor API support.

Key Changes:

• Removed static use_bfloat configuration flag and replaced with runtime detection
• Added has_tensor capability detection for Metal GPU Family 4
• Refactored kernel_mul_mm and kernel_mul_mm_id with cleaner indexing logic
• Introduced runtime compilation test for bfloat tensor API compatibility
• Added ggml_metal_library_init_from_source for dynamic shader compilation

Impact on Inference:
Since these changes are isolated to the Metal GPU backend and do not modify core inference functions like llama_decode, llama_encode, or llama_tokenize, there is no expected impact on CPU-based tokens per second performance. The model running on CPU (12th Gen Intel Core i7-1255U) with Ubuntu will not be affected by these Metal-specific optimizations.

Scope:
The modifications affect only Metal shader code and GPU initialization logic. No changes to cross-platform inference paths or tokenization logic were made.