Skip to content

Skylark-Software/EagleBranch

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8,178 Commits
.devops
.devops
 
 
.gemini
.gemini
 
 
.github
.github
 
 
benches
benches
 
 
ci
ci
 
 
cmake
cmake
 
 
common
common
 
 
docs
docs
 
 
examples
examples
 
 
ggml
ggml
 
 
gguf-py
gguf-py
 
 
grammars
grammars
 
 
include
include
 
 
licenses
licenses
 
 
media
media
 
 
models
models
 
 
pocs
pocs
 
 
requirements
requirements
 
 
scripts
scripts
 
 
src
src
 
 
tests
tests
 
 
tools
tools
 
 
vendor
vendor
 
 
.clang-format
.clang-format
 
 
.clang-tidy
.clang-tidy
 
 
.dockerignore
.dockerignore
 
 
.ecrc
.ecrc
 
 
.editorconfig
.editorconfig
 
 
.flake8
.flake8
 
 
.gitignore
.gitignore
 
 
.gitmodules
.gitmodules
 
 
.pre-commit-config.yaml
.pre-commit-config.yaml
 
 
AGENTS.md
AGENTS.md
 
 
AUTHORS
AUTHORS
 
 
CLAUDE.md
CLAUDE.md
 
 
CMakeLists.txt
CMakeLists.txt
 
 
CMakePresets.json
CMakePresets.json
 
 
CODEOWNERS
CODEOWNERS
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
SECURITY.md
SECURITY.md
 
 
SETUP.md
SETUP.md
 
 
UPSTREAM_PR_NOTES.md
UPSTREAM_PR_NOTES.md
 
 
USAGE.md
USAGE.md
 
 
build-xcframework.sh
build-xcframework.sh
 
 
convert_hf_to_gguf.py
convert_hf_to_gguf.py
 
 
convert_hf_to_gguf_update.py
convert_hf_to_gguf_update.py
 
 
convert_llama_ggml_to_gguf.py
convert_llama_ggml_to_gguf.py
 
 
convert_lora_to_gguf.py
convert_lora_to_gguf.py
 
 
flake.lock
flake.lock
 
 
flake.nix
flake.nix
 
 
mypy.ini
mypy.ini
 
 
poetry.lock
poetry.lock
 
 
pyproject.toml
pyproject.toml
 
 
pyrightconfig.json
pyrightconfig.json
 
 
requirements.txt
requirements.txt
 
 

Repository files navigation

Skylark Software

EagleBranch

EagleBranch is a llama.cpp fork with TurboQuant 3-bit KV cache compression and speculative decoding (EAGLE v1/v2/v3, NextN/MTP) for current, legacy, and CPU inference. Includes multiple bug fixes that enable legacy NVIDIA hardware and a fused rotated-domain matvec kernel ("Lane B") that closes most of the throughput gap to IQ4_NL on MLA models.

Download

📦 Download v1.0 (Linux x86_64 + CUDA, 183 MB)

All releases · v1.0 release notes · Step-by-step setup guide (SETUP.md)

# Quick install — verify, extract, smoke test
curl -LO https://github.com/Skylark-Software/EagleBranch/releases/download/v1.0/skylark-llama-server-v1.0-linux-x86_64-cuda.tar.gz
echo "fd99d81e0d46047c890ea13b188b046b28a2f7411c9a13e41d771321cf14addb  skylark-llama-server-v1.0-linux-x86_64-cuda.tar.gz" | sha256sum -c
tar -xzf skylark-llama-server-v1.0-linux-x86_64-cuda.tar.gz
cd skylark-llama-server-v1.0-linux-x86_64-cuda
./llama-server --version

See SETUP.md for full step-by-step instructions including hardware checks, GPU compatibility verification, model downloads, and first-inference examples.

Usage

Once the server is running (default http://localhost:8080), it speaks the OpenAI-compatible Chat Completions API — any OpenAI client library works by pointing base_url at it:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

response = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=60,
)
print(response.choices[0].message.content)

Or via curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"Hello!"}],"max_tokens":60}'

See USAGE.md for streaming, native llama.cpp endpoints, generation parameters, speculative decoding (-md draft.gguf), multimodal/vision (--mmproj), multi-instance setup, and other practical patterns.

Hardware targets

  • Pre-SM80 NVIDIA GPUs — Pascal P40, P100, GTX 1080/1080 Ti — where upstream's Flash Attention can't reach MLA's head_dim=576 and quantized K-cache configurations crash without additional graph fixes. This is the primary development target.
  • CPU-only inference — the 3-bit KV cache makes large KV footprints fit in a fraction of memory, and the fused CPU FlashAttention path outperforms GPU KV on MHA models.
  • Modern NVIDIA (Ampere / Ada / Hopper) — also supported; builds cleanly SM61 through SM90.
  • Apple Silicon (Mac Studio and similar) — untested. Design concept fits (unified memory + KV compression is ideal for big MoE), but the binary is Linux x86_64 + CUDA — it won't run on macOS as-is. Building from source on macOS would compile, and the EAGLE/MTP speculative decoding runtime is architecture-agnostic, but the TurboQuant kernels are CUDA only; a Metal port would be real work. The CPU path is plain C with no NEON acceleration, so it would be correct but not competitively fast on ARM.

Large MoE models fit well on modest hardware as a side effect of the compression: Kimi K2.5 (1T params / 32B active) runs at 4.72 tok/s on four P40s; Mistral Large 3 (675B) at 3.69 tok/s.

Featured capability — 3-bit KV cache compression (binary)

Mainline llama.cpp's smallest KV cache quantization is IQ4_NL at 4.5 bits per weight. This fork's binary distribution includes two novel 3-bit KV cache types built on Google's TurboQuant algorithm (ICLR 2026), Pascal-compatible end-to-end:

Type bits/weight vs f16 vs IQ4_NL PPL penalty
TBQ3_1 3.125 5.12× smaller 30% smaller +2.27%
TBQ3_2 3.5 4.57× smaller 22% smaller +2.44%
  • Quality-preserving: on Mistral Small 24B (wiki.test.raw, 30 chunks, ctx=512), perplexity rises only 2.27% from 5.3663 to 5.4881 at 5.12× compression. Error bars overlap f16 on many prompts.
  • Pascal (SM61) compatible: no Flash Attention requirement, works on P40 / P100 / GTX 1080 class hardware.
  • CPU path often faster than GPU: on Mistral Small 24B @ 32K ctx the TBQ3_1 CPU KV path delivers 15.9 tok/s — actually beating IQ4_NL GPU KV at 15.4 tok/s — because fused CPU FlashAttention amortizes the dequant cost better than the separate CUDA MUL_MAT path can.
  • MLA-ready: TBQ3_2 (block=32) works on DeepSeek and Mistral Large 3's MLA head_dim=576. A fused rotated-domain matvec kernel ("Lane B") closes most of the throughput gap to IQ4_NL.

This is the only 3-bit KV cache implementation we know of for pre-Ampere NVIDIA hardware.

Featured capability — speculative decoding for massive MoE models (binary)

Native support for the two speculative decoding methods that ship with modern MLA-architecture draft heads — neither available in upstream llama.cpp for these specific model families.

Method Target model Draft head source Measured acceptance
MTP / NextN DeepSeek R1 (671B) built-in nextn_layer_parameters 47% at 3.90 tok/s (+4.6% vs baseline)
EAGLE v1 Mistral Large 3 (675B) Mistral Eagle head on HF 64.5% (validated correct; net-zero on 1085-split MoE target)
EAGLE v2 eagle3_ds arch any compatible supported
  • Custom eagle3_ds architecture — handles MLA (kv_lora_rank=512, head_dim=576) + MoE target models in a single decoder graph.
  • Auto-detection of EAGLE method from GGUF metadata (eagle_method = "eagle" → v1; no metadata → Eagle-3).
  • GGUF converter for Mistral Eagle and DeepSeek R1 NextN draft heads, including FP8 dequantization and block-wise weight scale handling.
  • Server-mode fixes that mainline's llama-server lacked: target model auto-setup for draft feature extraction, eagle3_n_past reset between requests (fixed a second-request crash), and correct encoder FC sizing for v1 vs MTP modes.

What else this enables

  • Trillion-parameter MoE inference on 4× P40 (~$2K of hardware) — Kimi K2.5 (1T parameters, 32B active) runs at 4.72 tok/s with vision.
  • Multimodal vision on 1T-parameter Kimi K2.5 and 675B Mistral Large 3 — both use pixtral-family mmproj encoders on top of MLA text decoders. Kimi K2.5 requires the kimik25 projector type (upstream PR #19170); Mistral Large 3 uses a 4.9 GB F16 pixtral encoder. Both work end-to-end on Pascal alongside the MIT quantized-K fixes on this branch.
  • MLA quantized K cache actually works on Pascal — upstream crashes at load time (unsupported type combination (iq4_nl to iq4_nl)); the MIT fixes on this branch make it run.

Measured performance (Tesla P40, SM61)

Large MoE models

Model Parameters / Active Hardware tok/s Notes
Kimi K2.5 (vision) 1T / 32B 4× P40 4.72 K=q8_0, V=f16, 131K ctx
Mistral Large 3 675B / 41B 4× P40 3.69 K=q8_0, V=f16, 16K ctx
Mistral Large 3 675B / 41B 4× P40 3.32 K=iq4_nl, V=f16 — needs fixes in this branch
DeepSeek R1 671B / 37B 4× P40 4.00 K=iq4_nl, V=f16, 6K ctx
DeepSeek V2 Lite 16B / 2.4B 1× P40 72 K=iq4_nl, V=f16

KV cache compression — Mistral Small 24B @ 32K ctx

KV config tok/s KV size Compression PPL vs f16
f16 baseline 17.5 5120 MB 1× 0
IQ4_NL 15.4 1440 MB 3.56× +0.52%
TBQ3_1 CPU path (binary) 15.9 1000 MB 5.12× +2.27%

CPU-path TurboQuant actually beats GPU KV on MHA models because fused CPU FlashAttention amortizes the dequant cost better than the separate CUDA MUL_MAT path.

What's included on this branch (MIT)

General-purpose fixes for anyone running MLA + quantized KV on pre-SM80 hardware. These are the bug fixes upstream lacks; they work on any llama.cpp deployment regardless of which models you target.

  • MLA quantized-K graph fix — unblocks --cache-type-k iq4_nl on all MLA models (Mistral Large 3, DeepSeek R1/V2/V3, Kimi K2/K2.5)
  • CUDA CPY dispatches for IQ4_NL → F32 / F16 — fused dequant + cast
  • Non-contiguous same-type quant copy kernel — fixes a load-time crash when the server's speculative probe encounters reshape/flatten copies with quantized data

What's distributed as a binary

Proprietary extensions are built on top of this public branch and shipped as a stripped release bundle. Not source-available.

  • TBQ3_1 / TBQ3_2 KV cache compression (3-bit and 3.5-bit) — CPU and CUDA implementations, Pascal-compatible, including a fused rotated- domain matvec kernel for MLA ("Lane B") that closes most of the gap to IQ4_NL's throughput
  • EAGLE v1/v2 speculative decoding — custom eagle3_ds architecture with MLA + MoE, auto-detects EAGLE method from GGUF metadata
  • MTP / NextN speculative decoding — converter (convert_hf_to_gguf.py) and server runtime for DeepSeek R1/V3/V2

See a release bundle's NOTICE for full copyright / license terms. Contact info@skylarksoftware.me for binary access and licensing inquiries.

The MIT fixes in detail

MLA V-cast graph fix

Mainline bug: at MLA's non-FA attention node, V is a view of the quantized K cache. ggml_cuda_cpy hits unsupported type combination (iq4_nl to iq4_nl) when the graph inserts ggml_cont(ggml_transpose(v)) — no backend has a transposed quant-to-quant CPY kernel.

Fix: insert ggml_cast(v, F32) before the transpose+cont in llama-graph.cpp's build_attn_mha (non-FA branch). Supported via to_fp32_cuda (see also the new CPY dispatches below). Four commits progressively refine this — F16 intermediate was tried but reverted because Pascal's fp16 compute tax outweighs the halved bandwidth.

Affects: DeepSeek R1/V2/V3, Mistral Large 3, Kimi K2/K2.5 with --cache-type-k iq4_nl on P40-class hardware.

CUDA CPY dispatches for IQ4_NL → F32 / F16

Mainline only had quant-to-quant CPY. With the V-cast fix above we need IQ4_NL → F32 (and F16). Adds ggml_cuda_op_cpy_iq4_nl_f32 and the F16 variant — dequant + cast in a single kernel. Keeps the dequant on GPU instead of round-tripping through CPU.

Non-contiguous same-type quant copy

Mainline bug: server's common_speculative_is_compat load-time probe schedules iq4_nl → iq4_nl with non-contiguous strides. The existing same-type path uses cudaMemcpyAsync (contiguous only) and falls through to the unsupported-combination abort.

Fix: cpy_blck_quant_same — a generic kernel that walks the linear block index through each tensor's own block layout (mirrors ggml_compute_forward_dup_bytes on CPU). Block identity is ordinal (src block N → dst block N), so it works across reshape/flatten copies.

Building

Same as upstream — this branch is a superset of a specific upstream commit, not a refactor. Standard CMake:

cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

Verified on CUDA 12.x with SM61 (Pascal P40) through SM90 (Hopper).

Relationship to upstream

Based on upstream llama.cpp. This branch is shared under MIT so anyone needing these fixes can use it directly or lift individual commits into their own work.

llama.cpp (upstream — below this point)

llama

License: MIT Release Server

Manifesto / ggml / ops

LLM inference in C/C++

Recent API changes

Hot topics

Quick start

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

Once installed, you'll need a model to work with. Head to the Obtaining and quantizing models section to learn more.

Example command:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

Multimodal

Bindings
UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

Tools
  • akx/ggify – download PyTorch models from HuggingFace Hub and convert them to GGML
  • akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp
  • crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
  • gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage
  • Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)
  • unslothai/unsloth – 🦥 exports/saves fine-tuned and trained models to GGUF (Apache-2.0)
Infrastructure
  • Paddler - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure
  • GPUStack - Manage GPU clusters for running LLMs
  • llama_cpp_canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
  • llama-swap - transparent proxy that adds automatic model switching with llama-server
  • Kalavai - Crowdsource end to end LLM deployment at any scale
  • llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes.
Games
  • Lucy's Labyrinth - A simple maze game where agents controlled by an AI model will try to trick you.

Supported backends

Backend Target devices
Metal Apple Silicon
BLAS All
BLIS All
SYCL Intel and Nvidia GPU
MUSA Moore Threads GPU
CUDA Nvidia GPU
HIP AMD GPU
ZenDNN AMD CPU
Vulkan GPU
CANN Ascend NPU
OpenCL Adreno GPU
IBM zDNN IBM Z & LinuxONE
WebGPU [In Progress] All
RPC All
Hexagon [In Progress] Snapdragon
VirtGPU VirtGPU APIR

Obtaining and quantizing models

The Hugging Face platform hosts a number of LLMs compatible with llama.cpp:

You can either manually download the GGUF file or directly use any llama.cpp-compatible models from Hugging Face or other model hosting sites, such as ModelScope, by using this CLI argument: -hf <user>/<model>[:quant]. For example:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable MODEL_ENDPOINT. For example, you may opt to downloading model checkpoints from ModelScope or other model sharing communities by setting the environment variable, e.g. MODEL_ENDPOINT=https://www.modelscope.cn/.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

To learn more about model quantization, read this documentation

llama-cli

A CLI tool for accessing and experimenting with most of llama.cpp's functionality.

  • Run in conversation mode

    Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding -cnv and specifying a suitable chat template with --chat-template NAME

    llama-cli -m model.gguf

# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!
  • Run in conversation mode with custom chat template 
    # use the "chatml" template (use -h to see the list of supported templates)
llama-cli -m model.gguf -cnv --chat-template chatml

# use a custom template
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'
  • Constrain the output with a custom grammar 
    llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'

# {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}

    The grammars/ folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.

    For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/

llama-server

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

  • Start a local HTTP server with default configuration on port 8080 
    llama-server -m model.gguf --port 8080

# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions
  • Support multiple-users and parallel decoding 
    # up to 4 concurrent requests, each with 4096 max context
llama-server -m model.gguf -c 16384 -np 4
  • Enable speculative decoding 
    # the draft.gguf model should be a small variant of the target model.gguf
llama-server -m model.gguf -md draft.gguf
  • Serve an embedding model 
    # use the /embedding endpoint
llama-server -m model.gguf --embedding --pooling cls -ub 8192
  • Serve a reranking model 
    # use the /reranking endpoint
llama-server -m model.gguf --reranking
  • Constrain all outputs with a grammar 
    # custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf

llama-perplexity

A tool for measuring the perplexity 1 (and other quality metrics) of a model over a given text.

  • Measure the perplexity over a text file 
    llama-perplexity -m model.gguf -f file.txt

# [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
# Final estimate: PPL = 5.4007 +/- 0.67339
  • Measure KL divergence 
    # TODO

llama-bench

Benchmark the performance of the inference for various parameters.

  • Run default benchmark 
    llama-bench -m model.gguf

# Output:
# | model               |       size |     params | backend    | threads |          test |                  t/s |
# | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |
#
# build: 3e0ba0e60 (4229)

llama-simple

A minimal example for implementing apps with llama.cpp. Useful for developers.

  • Basic text completion 
    llama-simple -m model.gguf

# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of

Contributing

  • Contributors can open PRs
  • Collaborators will be invited based on contributions
  • Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Any help with managing issues, PRs and projects is very appreciated!
  • See good first issues for tasks suitable for first contributions
  • Read the CONTRIBUTING.md for more information
  • Make sure to read this: Inference at the edge
  • A bit of backstory for those who are interested: Changelog podcast

Other documentation

Development documentation

Seminal papers and background on the models

If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:

XCFramework

The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS, and macOS. It can be used in Swift projects without the need to compile the library from source. For example:

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(
            name: "MyLlamaPackage",
            dependencies: [
                "LlamaFramework"
            ]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
            checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
        )
    ]
)

The above example is using an intermediate build b5046 of the library. This can be modified to use a different version by changing the URL and checksum.

Completions

Command-line completion is available for some environments.

Bash Completion

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

Dependencies

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • subprocess.h - Single-header process launching solution for C and C++ - Public domain

Footnotes

  1. https://huggingface.co/docs/transformers/perplexity ↩

About

EagleBranch is a llama.cpp fork with TurboQuant 3-bit KV cache compression and speculative decoding (EAGLE v1/v2/v3, NextN/MTP) for current, legacy, and CPU inference. Includes multiple bug fixes that enable legacy NVIDIA hardware and a fused rotated-domain matvec kernel ("Lane B") that closes most of the throughput gap to IQ4_NL on MLA models.

skylarksoftware.me

Resources

Readme

License

MIT license

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

7 tags

Packages

 
 
 

Contributors

Languages