DeepSeek-V4-Flash 284B on 4x RTX 3090 — 1,900 t/s prefill, 131K context (ds4-longctx)

This branch runs the full DeepSeek-V4-Flash 284B MoE (2-bit expert GGUF, 85-91 GB) on four RTX 3090s: 96 GB of VRAM total, sm_86, no FP8, PCIe only. It builds on cchuter's V4 CUDA port. Every change sits behind a DSV4_* environment variable; with the flags unset the code paths are stock.

Two checkpoints are supported on this 96 GB rig: the full 284B model at 2-bit (Unsloth UD-IQ2_M) with 131K context, or the REAP-pruned 180B at ~4-bit (K160 GGUF) with 262K context — same VRAM budget, quality parity between them (see the 2026-08-05 section below).

2026-08-06: decode night — 39.3 → 44.5 t/s (+13%), prefill untouched

One night of decode optimization on the UD-IQ2_M config (131K ctx, temp 0, 256-token bursts through the prod server):

step	decode, short ctx	decode @130K
start (220 W power limit)	39.34 t/s	37.61 t/s
+ clocks locked at 1995 MHz, 270 W (scripts/ds4-gpu-clocks.sh)	42.8 t/s	39.35 t/s
+ sinkhorn / fp8-KV quantize kernel rewrites	44.50 t/s	40.81 t/s

Prefill is not hurt: 1,774 t/s @32K on the final build (1,675-1,761 reference). The wins came from a register-based fast path for the head-compression split sinkhorn (shuffle reductions over the accumulator) and a two-pass, bit-exact fp8 KV quantize kernel. Decode is a ~21 ms serial GPU chain per token; syncs were profiled and are not the problem — the next lever is MoE expert GEMV bandwidth (i-quant MMVQ port, parked). A DSpark speculative-decoding port is built and measured but also parked: acceptance 27.5% with a Q4_K draft, yet e2e is slower than plain decode because the draft runs uncaptured and costs VRAM. Per-hypothesis write-ups live in docs/ds4/ (DS4_H*_2026-08-0*.md).

2026-08-05: REAP K160 GGUF — 256K context on 96 GB, quality parity with the 2-bit full model

anonymousmaharaj/DeepSeek-V4-Flash-0731-REAP-K160-GGUF is a GGUF of 0xSero's REAP K160 checkpoint (160 of 256 routed experts kept per MoE scope, 180.4B params after pruning), converted in-house on this branch: routed experts Q3_K/Q4_K (down-proj is the sensitive one and gets Q4_K), attention/shared experts/indexer/output Q8_0, compressor APE tensors pinned to F32, chat template embedded. 89.9 GB, 3.99 bpw, four shards.

Same 90 GB budget, two ways to spend it:

checkpoint	parameters	weight bits	max context on 96 GB
Unsloth UD-IQ2_M (unpruned)	284B	2.56 bpw	131,072
REAP K160 Q3_K/Q4_K (this repo)	180B	3.99 bpw	262,144

Quality A/B at temperature 0 (benchlocal-cli, 150 scenarios):

pack	K160 Q3_K/Q4_K	UD-IQ2_M
medium (ToolCall / InstructFollow / StructOutput / DataExtract / ReasonMath, 75)	67	67
cli-40	23	22
hermesagent-20	14	14
bugfind-15	14	14
total	118/150	117/150

One scenario out of 150 is noise — read it as a tie. The error profiles differ: K160 wins investigation and multi-step scenarios, the 2-bit build wins exact-format text tasks. Known limitation: non-English output (tested with Russian) is broken on the K160 build — that comes from the expert pruning, not the quantization, so use UD-IQ2_M if you need anything but English.

Speed on the rig (262,144 ctx, ~92/96 GB filled, 350 W): prefill 1,675 t/s @32K, 1,766-1,771 t/s @64-130K, marginal ~1,830-1,850 t/s; decode 34-36 t/s (up to 38.9 t/s on single requests). Launch it exactly like the UD checkpoint, with a bigger context:

MODEL=/path/to/DeepSeek-V4-Flash-0731-REAP-K160-Q3_K_Q4_K-00001-of-00004.gguf \
CTX=262144 bash scripts/ds4-prod-serve.sh

Full conversion recipe, conversion traps (transformers pin, APE/F32, chat template injection) and quality details: DS4HANDOFF.md section 0f.

2026-08-04: capture-always CUDA graphs — prefill 433 → 1,900 t/s in one session

Prefill was host-bound: one thread pushing ~1,700 kernel launches per ubatch into contended command channels (151 us each) kept four GPUs 27-30% busy. DSV4_PREFILL_GRAPHS=1 skips the CUDA-graph stability gate for prefill ubatches and captures every one — launches are recorded host-side, then a single cudaGraphLaunch submits the whole ~2,200-node split. The GPUs now sit at 93-96% and the power limit becomes the binding constraint.

Measured on DeepSeek-V4-Flash-0731 (Unsloth UD-IQ2_M, 91 GB; this quant ships no MTP head), nvidia-smi -pl 350, warm server:

prompt depth	prefill	decode
32K	1,862 t/s	38-40 t/s
98K	1,906 t/s	37.0 t/s
130K — fills `--ctx-size 131072`	1,807-1,823 t/s	36.9 t/s

Same-day controls without the graphs: 513 / 436 / ~444 t/s; decode is untouched (36.97 without vs 37.02 with, same build). Power curve at warm 130K: 220 W → 1,381, 280 W → 1,679, 300 W → 1,742, 350 W → 1,823 t/s. The first request at a new depth pays a one-time allocator climb (~550 t/s) — start with scripts/ds4-prod-serve.sh, which warms it away at startup. Validated end to end: greedy-checkable answers through the capture path, an 11-turn agent loop at 115-131K depth, and prefix-cache reuse (~1.4K reprocessed tokens per agent turn with default context checkpoints).

Earlier checkpoint: IQ2_XXS + MTP — 164K context at 41 t/s decode

Measured on this box (IQ2_XXS/Q2_K experts, Q8_0 attention, imatrix; 220 W per GPU). Speed against prompt depth, through llama-server with the production flag set:

prompt depth	prefill	decode, MTP on	decode, no MTP
~1K	—	50.1 t/s	38.1 t/s
105K	414 t/s	41.6 t/s	34.6 t/s
127K — fills `--ctx-size 131072`	396 t/s	42.6 t/s	—
156K — fills `--ctx-size 163840`	367 t/s	41.4 t/s	—

Speculative decoding (MTP) is worth +31% at short context and +20% at depth. Decode degrades gently with depth: 42.6 t/s at 127K, 41.4 t/s at 156K.

Prefill at shallower depths, via llama-batched-bench:

prompt	prefill
512	582 t/s
8K	556 t/s
32K	487 t/s
97K	495 t/s

Maximum context with MTP is --ctx-size 163840, verified filled. 262144 does not fit with MTP loaded — it fails at ~205K depth on a depth-scaled compute buffer; the projected ceiling is ~205-220K. Without MTP the branch has completed a 253K prompt (ctx 262144) and passes needle retrieval from 200K depth.

The 105K row uses the older -ts 1,1,1,0.85 split; the two deeper rows use -ts 1,1,0.90,0.95, which is now the recommended one — see below.

What is in the branch

Capture-always CUDA graphs for prefill (DSV4_PREFILL_GRAPHS): prefill ubatch graphs can never satisfy the replay stability gate (rotating pipeline input slots, ARANGE op_params, growing indexer shapes), so the backend captures every ubatch and submits it as one graph launch — the per-ubatch submission storm disappears. Exactly one graph + instance alive per device (bounded memory, learned from an agent-loop OOM), instantiate/update/launch failures fall back to direct execution, and decode keeps its own keyed replay path untouched. +2.9-3.4x prefill.
Prefill scheduler fixes (DSV4_STABLE_TOPO, GGML_GALLOC_STICKY): constant prefill graph topology plus grow-only per-family galloc plans — kills a per-ubatch realloc storm that synchronized all backends once per ubatch, and is also what makes graph capture possible at all.
GGML_OP_REPEAT on I32/I16 runs on CUDA instead of silently falling back to the CPU: removes 172 backend boundaries per graph coming from the 4-way hyper-connections (decode graph splits 364 → 20).
Sparse top-k FlashAttention for the CSA prompt chunks (DSV4_SPARSE_FA): the FA kernel gathers only the 512 KV positions the lightning indexer selected, with cp.async loads on full tiles and per-Q-tile union lists (DSV4_FA_UNION).
MoE MMQ tile fix (DSV4_MOE_TILE): the ids path sized tiles for the worst-case column bound, which wasted 87% of the MACs at ubatch 512. Sizing from the actual per-expert token count gave +19% end to end. Related finding: IQ2_XXS runs 1.74x faster than Q2_K in MMQ on Ampere.
Lightning indexer variants: causal skip (DSV4_IDX_SKIP), a q-tiled WMMA kernel for 200K+ depths (DSV4_IDX_QTILE).
Constant-shape decode graphs (DSV4_CONSTANT_SHAPE): depth-bucketed shapes so CUDA graphs replay instead of rebuilding every token.
MTP speculative decoding, K=1, end to end: the MTP head is side-loaded from a separate GGUF (DSV4_MTP_GGUF), drafts are computed inside the main graph, and llama-server picks them up via --spec-type dsv4-mtp. Accept rate is 85-100% on greedy decoding. Note (2026-08): this legacy MTP path has been superseded in src/models/deepseek4.cpp by a DSpark port (layer taps + markov bias, built via scripts/ds4-convert-mtp.py) that is measured but currently parked — slower e2e than plain decode; see docs/ds4/DS4_H1_DSPARK_SPEC_2026-08-05.md.
Tool-call fixes for agent clients: the DSML parser now accepts tool parameters in any order. Before this, a well-formed call failed to parse whenever the model ordered parameters differently from the JSON schema, so clients such as opencode never executed the tool.
MMVQ small_k boundary fix (DSV4_MMVQ_SMALLK): both routed expert matmuls land exactly on the small_k trigger that a strict < excludes — IQ2_XXS up/gate at 4096/256 = 16 blocks against a threshold of 16, Q2_K down at 2048/256 = 8 against 8. Relaxing it to <= gives +5% decode with prefill unchanged.
Batched radix-select Top-K for prefill (DSV4_PREFILL_RADIX_TOPK): the prefill indexer top-k went through ggml_argsort_top_k, which fully sorts the compressed width and allocates a [n_comp, n_tokens] i32 result plus CUB temp storage — both scaling with context, and both fatal past ~90K tokens. Replaced with a one-block-per-row radix-select behind GGML_OP_TOP_K that needs no context-scaled scratch. Same prefill speed, no ceiling.

See DS4_OPTIMIZATION_2026-07-27.md for the mechanism behind the last two, the full measurements, and the hypotheses that were rejected. DS4HANDOFF.md is the consolidated handoff (rig, flags, launch commands); older dated research notes live in docs/ds4/.

Production launch

Current checkpoint (0731 UD-IQ2_M) — one command, auto-warmed

This is the recommended interactive/agent server configuration for the 4x RTX 3090 host. The 0731 UD-IQ2_M checkpoint has no compatible MTP head, so do not add the older DSV4_MTP_* flags to this launch.

Build after changing the source:

cmake --build build-v4-cuda --target llama-server -j 8

The fast prefill measurements use a 350 W power limit. It is not persistent across reboots:

sudo nvidia-smi -pl 350

Start the server:

MODEL=/path/to/DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf \
bash scripts/ds4-prod-serve.sh

The script starts llama-server with the full ship flag set on 0.0.0.0:18080, waits for readiness, then runs one synthetic full-depth prefill so the sticky allocator plans and CUDA pools reach their high-water marks — after that every request, including the first real one, runs at warm speed instead of paying the ~3x allocator climb. Wait for this line before recording a benchmark or opening the server to real traffic:

=== production server warm and ready on 0.0.0.0:18080 ===

The WebUI is at http://SERVER_IP:18080/; the OpenAI-compatible endpoint is http://SERVER_IP:18080/v1/chat/completions.

Verified defaults and what they affect

setting	production default	effect / guidance
`CTX`	`131072`	maximum context allocated at startup; larger values increase compressed-KV and graph memory
`BATCH`	`8192`	outer prompt batch; keep this for throughput
`UBATCH`	`384`	GPU microbatch; divisible by the 128-token DS4 compressor ratio and verified without CUDA graph failures
`TS`	`1,1,0.95,1.05`	layer/weight distribution across GPUs; keeps the tight GPU2 safer than an equal split
`CTX_CHECKPOINTS`	`1`	one correctness-safe host checkpoint for agent prefix reuse
`CHECKPOINT_EVERY_NT`	`-1`	disables periodic checkpoints; the aligned tail checkpoint is sufficient and avoids repeated large state copies
`WARM`	`1`	performs the required one-time full-depth allocator/CUDA-pool warm pass; use `0` only for diagnostics
`LOG`	`/tmp/ds4-prod-server.log`	server stdout/stderr; override it for durable benchmark logs
`PORT`	`18080`	WebUI and API port
`NGL`	`999`	offload all possible layers; do not lower for normal production
`FIT_TARGET`	unset	optional auto-fit reserve target; diagnostic only, not part of the verified production layout

Examples:

# Use the rig's standard model path and keep a durable log.
LOG=$HOME/ds4-sweep/results/prod.log bash scripts/ds4-prod-serve.sh

# Start without the expensive warm pass for a quick diagnostic only.
WARM=0 PORT=18081 bash scripts/ds4-prod-serve.sh

# Override context deliberately; the warm pass follows CTX.
CTX=65536 LOG=/tmp/ds4-64k.log bash scripts/ds4-prod-serve.sh

Ship flags, briefly

group	flags	what they do
agent rollback	`DSV4_AGENT_CKPT_TAIL=1`	keeps one checkpoint on a 128-token boundary; without the alignment, incremental prefill falls into the slow per-token compressor path
prefill submission	`DSV4_PREFILL_GRAPHS=1`, `DSV4_STABLE_TOPO=1`, `GGML_GALLOC_STICKY=1`	capture each large prefill graph and prevent per-ubatch allocator replanning/all-GPU drains
prefill attention	`DSV4_SPARSE_FA=1`, `DSV4_FA_UNION=1`, `DSV4_IDX_SKIP=1`, `DSV4_PREFILL_RADIX_TOPK=1`	sparse compressed attention and bounded Top-K memory; radix Top-K is required beyond about 90K
MoE prefill	`DSV4_MOE_TILE=1`, `DSV4_MOE_RESIDENT=1`, `DSV4_GLU_FUSE=1`, `DSV4_MOE_FUSE=1`	remove dead MMQ work and host synchronization, then fuse routed-expert work
decode	`DSV4_CONSTANT_SHAPE=1`, `DSV4_DECODE_FUSED_IDX=1`, `DSV4_DECODE_RADIX_TOPK=1`, `DSV4_MMVQ_SMALLK=1`	stable decode graph plus faster Lightning Indexer and exact Top-512 selection
multi-GPU	`GGML_CUDA_P2P=1`	enables peer copies between the four cards

All flags remain individually overridable. Useful kill switches are DSV4_PREFILL_GRAPHS=0 for prefill capture and GGML_CUDA_DISABLE_GRAPHS=1 for all CUDA graphs. They are diagnostic controls, not recommended production settings.

Agent behavior and expected speed

Do not pass --ctx-checkpoints 0: this SWA/recurrent model then has to re-prefill the whole transcript after a generated tail diverges from the next request. The production launcher instead keeps one full host checkpoint and rounds its replay position down to a 128-token boundary. This preserves the batched compressor and, unlike the rejected recurrent-only device snapshot, matches a fresh full recompute byte-for-byte.

With a 31K seed and roughly 10K-token additions, the verified six-turn run measured 1,475 / 1,543 / 1,483 / 1,292 / 1,354 / 1,242 prompt tokens/s through approximately 95K context, with 38.6-39.7 decode tokens/s. The first addition was 296 tokens/s before checkpoint alignment. See DS4HANDOFF.md for the full table and correctness hash.

Logs and basic operations

# Health and current listener.
curl -s http://127.0.0.1:18080/health
lsof -nP -iTCP:18080 -sTCP:LISTEN

# Follow the default log.
tail -f /tmp/ds4-prod-server.log

# Stop only the exact PID shown by lsof; never use pkill -f on the rig.
kill SERVER_PID

If prefill is unexpectedly slow, first verify that the startup warm pass finished, the power limit is still 350 W, no other process is using the GPUs, and the launch log shows ubatch=384, one context checkpoint, and the ship flags. A cold first request or an unaligned agent checkpoint can both look like a kernel regression while the kernels themselves are fine.

Two operational rules learned the hard way:

do not disable the launcher's aligned context checkpoint when serving agents;
the power limit is worth raising (sudo nvidia-smi -pl 350, resets at boot): warm prefill is power-bound, and 220 → 350 W is +31% prefill with the cards peaking at 83 C in bursts.

Earlier checkpoint (IQ2_XXS + MTP)

Historical numbers, measured before the 2026-08 DSpark rewrite of the MTP path — treat the launch below as a record of that checkpoint, not a current production recipe. Today's production configs are UD-IQ2_M (131K) and REAP K160 (262K), both launched via scripts/ds4-prod-serve.sh.

CUDA_VISIBLE_DEVICES=0,1,2,3 GGML_CUDA_P2P=1 \
DSV4_CONSTANT_SHAPE=1 DSV4_SPARSE_FA=1 DSV4_FA_UNION=1 DSV4_IDX_SKIP=1 \
DSV4_MOE_TILE=1 DSV4_MOE_RESIDENT=1 DSV4_GLU_FUSE=1 DSV4_MOE_FUSE=1 \
DSV4_DECODE_FUSED_IDX=1 DSV4_DECODE_RADIX_TOPK=1 \
DSV4_MMVQ_SMALLK=1 DSV4_PREFILL_RADIX_TOPK=1 \
DSV4_MTP_SPEC=1 DSV4_MTP_GGUF=/path/to/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf \
./llama-server -m DeepSeek-V4-Flash-IQ2XXS-...-imatrix.gguf \
  -ngl 999 --split-mode layer --flash-attn on --no-repack \
  --ctx-size 163840 --batch-size 4096 --ubatch-size 512 \
  -ts 1,1,0.90,0.95 --spec-type dsv4-mtp --parallel 1 \
  --jinja --reasoning on --reasoning-format deepseek --reasoning-budget 2048

Notes: tested only on Ampere (CUDA 12.6).

DSV4_PREFILL_RADIX_TOPK=1 is not optional at long context — without it this command dies at 90,112 prompt tokens in the prefill top-k.

-ts 1,1,0.90,0.95 replaces the older 1,1,1,0.85. Both place the same 81,686 MiB of weights, but 1,1,1,0.85 leaves CUDA2 with 10 MiB free, and the MTP decode graph cannot be instantiated there past ~110K tokens. Shifting one layer off CUDA2 onto CUDA3 raises the minimum free VRAM across devices to 684 MiB and lets the full context fill.

Agent clients should send temperature: 0; sampling at 0.7 on 2-bit weights measurably degrades tool selection, and greedy decoding also keeps the MTP draft gate open.

I post benchmarks from this rig on X: @superalesha.

Original llama.cpp README below.

llama.cpp

Manifesto / ggml / ops

LLM inference in C/C++

Recent API changes

Hot topics

Hugging Face cache migration: models downloaded with -hf are now stored in the standard Hugging Face cache directory, enabling sharing with other HF tools.
guide : using the new WebUI of llama.cpp
guide : running gpt-oss with llama.cpp
[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗
Support for the gpt-oss model with native MXFP4 format has been added | PR | Collaboration with NVIDIA | Comment
Multimodal support arrived in llama-server: #12898 | documentation
VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode
Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim
Hugging Face Inference Endpoints now support GGUF out of the box! ggml-org#9669
Hugging Face GGUF editor: discussion | tool

Quick start

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

Install llama.cpp using brew, nix or winget
Run with Docker - see our Docker documentation
Download pre-built binaries from the releases page
Build from source by cloning this repository - check out our build guide

Once installed, you'll need a model to work with. Head to the Obtaining and quantizing models section to learn more.

Example command:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

Multimodal

Bindings

Python: ddh0/easy-llama
Python: abetlen/llama-cpp-python
Go: go-skynet/go-llama.cpp
Node.js: withcatai/node-llama-cpp
JS/TS (llama.cpp server client): lgrammel/modelfusion
JS/TS (Programmable Prompt Engine CLI): offline-ai/cli
JavaScript/Wasm (works in browser): tangledgroup/llama-cpp-wasm
Typescript/Wasm (nicer API, available on npm): ngxson/wllama
Ruby: yoshoku/llama_cpp.rb
Rust (more features): edgenai/llama_cpp-rs
Rust (nicer API): mdrokz/rust-llama.cpp
Rust (more direct bindings): utilityai/llama-cpp-rs
Rust (automated build from crates.io): ShelbyJenkins/llm_client
C#/.NET: SciSharp/LLamaSharp
C#/VB.NET (more features - community license): LM-Kit.NET
Scala 3: donderom/llm4s
Clojure: phronmophobic/llama.clj
React Native: mybigday/llama.rn
Java: kherud/java-llama.cpp
Java: QuasarByte/llama-cpp-jna
Zig: deins/llama.cpp.zig
Flutter/Dart: netdur/llama_cpp_dart
Flutter: xuegao-tzx/Fllama
PHP (API bindings and features built on top of llama.cpp): distantmagic/resonance (more info)
Guile Scheme: guile_llama_cpp
Swift srgtuszy/llama-cpp-swift
Swift ShenghaiWang/SwiftLlama
Delphi Embarcadero/llama-cpp-delphi
Go (no CGo needed): hybridgroup/yzma
Android: llama.android

UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

AI Sublime Text plugin (MIT)
BonzAI App (proprietary)
cztomsik/ava (MIT)
Dot (GPL)
eva (MIT)
iohub/collama (Apache-2.0)
janhq/jan (AGPL)
johnbean393/Sidekick (MIT)
KanTV (Apache-2.0)
KodiBot (GPL)
llama.vim (MIT)
LARS (AGPL)
Llama Assistant (GPL)
LlamaLib (Apache-2.0)
LLMFarm (MIT)
LLMUnity (MIT)
LMStudio (proprietary)
LocalAI (MIT)
LostRuins/koboldcpp (AGPL)
MindMac (proprietary)
MindWorkAI/AI-Studio (FSL-1.1-MIT)
Mobile-Artificial-Intelligence/maid (MIT)
Mozilla-Ocho/llamafile (Apache-2.0)
nat/openplayground (MIT)
nomic-ai/gpt4all (MIT)
ollama/ollama (MIT)
oobabooga/text-generation-webui (AGPL)
PocketPal AI (MIT)
psugihara/FreeChat (MIT)
ptsochantaris/emeltal (MIT)
pythops/tenere (AGPL)
ramalama (MIT)
semperai/amica (MIT)
withcatai/catai (MIT)
Autopen (GPL)

Tools

akx/ggify – download PyTorch models from Hugging Face Hub and convert them to GGML
akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp
crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage
Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)
unslothai/unsloth – 🦥 exports/saves fine-tuned and trained models to GGUF (Apache-2.0)

Infrastructure

Paddler - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure
GPUStack - Manage GPU clusters for running LLMs
llama_cpp_canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
llama-swap - transparent proxy that adds automatic model switching with llama-server
Kalavai - Crowdsource end to end LLM deployment at any scale
llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes.
LLMKube - Kubernetes operator for llama.cpp with multi-GPU and Apple Silicon Metal support"

Games

Lucy's Labyrinth - A simple maze game where agents controlled by an AI model will try to trick you.

Supported backends

Backend	Target devices
Metal	Apple Silicon
BLAS	All
BLIS	All
SYCL	Intel and Nvidia GPU
OpenVINO [In Progress]	Intel CPUs, GPUs, and NPUs
MUSA	Moore Threads GPU
CUDA	Nvidia GPU
HIP	AMD GPU
ZenDNN	AMD CPU
Vulkan	GPU
CANN	Ascend NPU
OpenCL	Adreno GPU
IBM zDNN	IBM Z & LinuxONE
WebGPU [In Progress]	All
RPC	All
Hexagon [In Progress]	Snapdragon
VirtGPU	VirtGPU APIR

Obtaining and quantizing models

The Hugging Face platform hosts a number of LLMs compatible with llama.cpp:

You can either manually download the GGUF file or directly use any llama.cpp-compatible models from Hugging Face or other model hosting sites, by using this CLI argument: -hf <user>/<model>[:quant]. For example:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable MODEL_ENDPOINT. The MODEL_ENDPOINT must point to a Hugging Face compatible API endpoint.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

Use the GGUF-my-repo space to convert to GGUF format and quantize model weights to smaller sizes
Use the GGUF-my-LoRA space to convert LoRA adapters to GGUF format (more info: ggml-org#10123)
Use the GGUF-editor space to edit GGUF meta data in the browser (more info: ggml-org#9268)
Use the Inference Endpoints to directly host llama.cpp in the cloud (more info: ggml-org#9669)

To learn more about model quantization, read this documentation

`llama-cli`

A CLI tool for accessing and experimenting with most of `llama.cpp`'s functionality.

Run in conversation mode

Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding -cnv and specifying a suitable chat template with --chat-template NAME

llama-cli -m model.gguf

# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!

Run in conversation mode with custom chat template

# use the "chatml" template (use -h to see the list of supported templates)
llama-cli -m model.gguf -cnv --chat-template chatml

# use a custom template
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'

Constrain the output with a custom grammar
```
llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'

# {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}
```
The grammars/ folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.

For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/

`llama-server`

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

Start a local HTTP server with default configuration on port 8080

llama-server -m model.gguf --port 8080

# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions

Support multiple-users and parallel decoding

# up to 4 concurrent requests, each with 4096 max context
llama-server -m model.gguf -c 16384 -np 4

Enable speculative decoding

# the draft.gguf model should be a small variant of the target model.gguf
llama-server -m model.gguf -md draft.gguf

Serve an embedding model

# use the /embedding endpoint
llama-server -m model.gguf --embedding --pooling cls -ub 8192

Serve a reranking model

# use the /reranking endpoint
llama-server -m model.gguf --reranking

Constrain all outputs with a grammar

# custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf

`llama-perplexity`

A tool for measuring the perplexity ¹ (and other quality metrics) of a model over a given text.

Measure the perplexity over a text file

llama-perplexity -m model.gguf -f file.txt

# [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
# Final estimate: PPL = 5.4007 +/- 0.67339

Measure KL divergence
```
# TODO
```

`llama-bench`

Benchmark the performance of the inference for various parameters.

Run default benchmark

llama-bench -m model.gguf

# Output:
# | model               |       size |     params | backend    | threads |          test |                  t/s |
# | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |
#
# build: 3e0ba0e60 (4229)

`llama-simple`

A minimal example for implementing apps with `llama.cpp`. Useful for developers.

Basic text completion

llama-simple -m model.gguf

# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of

Contributing

Contributors can open PRs
Collaborators will be invited based on contributions
Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
Any help with managing issues, PRs and projects is very appreciated!
See good first issues for tasks suitable for first contributions
Read the CONTRIBUTING.md for more information
Make sure to read this: Inference at the edge
A bit of backstory for those who are interested: Changelog podcast

XCFramework

The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS, and macOS. It can be used in Swift projects without the need to compile the library from source. For example:

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(
            name: "MyLlamaPackage",
            dependencies: [
                "LlamaFramework"
            ]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
            checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
        )
    ]
)

The above example is using an intermediate build b5046 of the library. This can be modified to use a different version by changing the URL and checksum.

Completions

Command-line completion is available for some environments.

Bash Completion

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

Dependencies

yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
subprocess.h - Single-header process launching solution for C and C++ - Public domain

https://huggingface.co/docs/transformers/perplexity ↩

Name		Name	Last commit message	Last commit date
Latest commit History 9,413 Commits
.devops		.devops
.gemini		.gemini
.github		.github
.pi/gg		.pi/gg
benches		benches
ci		ci
cmake		cmake
common		common
docs		docs
examples		examples
ggml		ggml
gguf-py		gguf-py
grammars		grammars
include		include
licenses		licenses
media		media
models		models
pocs		pocs
requirements		requirements
scripts		scripts
src		src
tests		tests
tools		tools
vendor		vendor
.clang-format		.clang-format
.clang-tidy		.clang-tidy
.dockerignore		.dockerignore
.ecrc		.ecrc
.editorconfig		.editorconfig
.flake8		.flake8
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitmodules		.gitmodules
.pre-commit-config.yaml		.pre-commit-config.yaml
AGENTS.md		AGENTS.md
AUTHORS		AUTHORS
CLAUDE.md		CLAUDE.md
CMakeLists.txt		CMakeLists.txt
CMakePresets.json		CMakePresets.json
CODEOWNERS		CODEOWNERS
CONTRIBUTING.md		CONTRIBUTING.md
DS4HANDOFF.md		DS4HANDOFF.md
DS4_OPTIMIZATION_2026-07-27.md		DS4_OPTIMIZATION_2026-07-27.md
LICENSE		LICENSE
Makefile		Makefile
P2P_RECOVERY_PLAN_610.md		P2P_RECOVERY_PLAN_610.md
README.md		README.md
SECURITY.md		SECURITY.md
build-xcframework.sh		build-xcframework.sh
convert_hf_to_gguf.py		convert_hf_to_gguf.py
convert_hf_to_gguf_update.py		convert_hf_to_gguf_update.py
convert_llama_ggml_to_gguf.py		convert_llama_ggml_to_gguf.py
convert_lora_to_gguf.py		convert_lora_to_gguf.py
dev-team.json		dev-team.json
flake.nix		flake.nix
mypy.ini		mypy.ini
pyproject.toml		pyproject.toml
pyrightconfig.json		pyrightconfig.json
requirements.txt		requirements.txt
ty.toml		ty.toml

Folders and files

Latest commit

History

Repository files navigation

DeepSeek-V4-Flash 284B on 4x RTX 3090 — 1,900 t/s prefill, 131K context (ds4-longctx)

2026-08-06: decode night — 39.3 → 44.5 t/s (+13%), prefill untouched

2026-08-05: REAP K160 GGUF — 256K context on 96 GB, quality parity with the 2-bit full model

2026-08-04: capture-always CUDA graphs — prefill 433 → 1,900 t/s in one session

Earlier checkpoint: IQ2_XXS + MTP — 164K context at 41 t/s decode

What is in the branch

Production launch

Current checkpoint (0731 UD-IQ2_M) — one command, auto-warmed

Verified defaults and what they affect

Ship flags, briefly

Agent behavior and expected speed

Logs and basic operations

Earlier checkpoint (IQ2_XXS + MTP)

llama.cpp

Recent API changes

Hot topics

Quick start

Description

Text-only

Multimodal

Supported backends

Obtaining and quantizing models

llama-cli

A CLI tool for accessing and experimenting with most of llama.cpp's functionality.

llama-server

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

llama-perplexity

A tool for measuring the perplexity 1 (and other quality metrics) of a model over a given text.

llama-bench

Benchmark the performance of the inference for various parameters.

llama-simple

A minimal example for implementing apps with llama.cpp. Useful for developers.

Contributing

Other documentation

Development documentation

Seminal papers and background on the models

XCFramework

Completions

Bash Completion

Dependencies

Footnotes

About

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`llama-cli`

A CLI tool for accessing and experimenting with most of `llama.cpp`'s functionality.

`llama-server`

`llama-perplexity`

A tool for measuring the perplexity ¹ (and other quality metrics) of a model over a given text.

`llama-bench`

`llama-simple`

A minimal example for implementing apps with `llama.cpp`. Useful for developers.