💡 Ultra-fast local LLM and embedding inference directly inside your JVM process — Zero-copy, Zero HTTP overhead, C++ native speed.
FastAIModel is a retained-memory local inference engine for Java that wraps llama.cpp (for GGUF) and ONNX Runtime (for ONNX) using direct JNI bindings. It is the engine that drives offline execution in the FastJava Ecosystem, giving Java developers native LLM and embedding capabilities without keeping heavy external apps (like LM Studio or Ollama) open.
- Why FastAIModel?
- Quick Start
- Demos
- Technical Highlights
- Installation
- API Reference
- Performance
- Project Structure
- Roadmap
- License
import fastaimodel.FastAIModel;
public class Demo {
public static void main(String[] args) {
// Load local GGUF model directly into memory
try (FastAIModel model = new FastAIModel("models/qwen2.5-coder-1.5b.gguf", 2048, 0.7f)) {
model.generate("Write a quicksort in Java:", token -> {
System.out.print(token);
System.out.flush();
});
}
}
}Running LLMs locally in Java typically requires invoking external subprocesses or running local HTTP servers. FastAIModel eliminates this bloat by running the model directly inside your Java process:
- True In-Process Execution — Runs the model in the same process space, bypassing system context-switches and network sockets.
- Zero HTTP/JSON Overhead — Text and tokens flow directly between Java and C++ memory.
- Low Memory Overhead — Eliminates the footprint of keeping GUI-based desktop inference servers running in the background.
- 🚀 Native llama.cpp Performance — Direct integration with CPU AVX2/AVX512 instruction sets and GPU computation (Vulkan/CUDA).
- 🌊 Direct Token Streaming — Direct native callbacks stream tokens back to your Java consumer in real-time.
- 📦 GGUF & ONNX Support — Native compatibility with GGUF quantized models (Llama, Qwen, Mistral, Gemma) and local TTS ONNX models (Piper) in one unified engine.
- 🧠 Zero-Copy Memory — Shared token handling minimizing garbage collection strain on the JVM.
To run the local demos, you first need to download the model files. You can find them and thousands of other open-source models on the major generic AI model portals:
- 🌐 Hugging Face Hub — The largest community repository for LLMs, Audio, and Vision models (GGUF, ONNX, PyTorch, etc.).
- 🌐 ModelScope Hub — A comprehensive open-source model registry and repository for diverse AI domains.
- 🌐 Civitai — A platform specializing in community-shared diffusion and neural network models.
Specifically for our demos, you can download:
- 🔊 ONNX TTS Model: en_US-danny-low.onnx (and its config en_US-danny-low.onnx.json). Explore more voices in the Piper Voices Database.
- 🧠 GGUF LLM Model: Llama-3.2-1B-Instruct-Q8_0.gguf. Find other models in the Hugging Face GGUF Database.
You can run the local demos using the provided batch files. Specify your model path as an argument to test:
Piper's en_US-danny-low.onnx model runs through ONNX Runtime via direct JNI bindings. Generated audio is streamed straight into Java Sound—no temporary files, no external players.
run-onnx-demo.bat "..\FastBot\examples\Demo\en_US-danny-low.onnx"The Llama 3.2 model runs in "silent mode," producing blazing-fast token streams without printing engine diagnostics—perfect for background agents and FastBot integrations.
run-silent-demo.bat "..\FastAI\examples\Demo\models\Llama-3.2-1B-Instruct-Q8_0.gguf"Meta’s Llama 3.2 1B model is executed with full diagnostics enabled, showing layer loading, quantization details, token timings, and memory stats.
run-gguf-demo.bat "..\FastAI\examples\Demo\models\Llama-3.2-1B-Instruct-Q8_0.gguf"- ⚡ In-Process Execution: FastAIModel loads ONNX and GGUF models directly inside the JVM, using native bindings to llama.cpp and ONNX Runtime. This eliminates socket overhead, HTTP servers, client processes, and inter-process copies. Everything runs in the same memory space.
- 🧊 Zero-Copy Memory: Token buffers are shared between native and Java without duplication, minimizing GC pressure and keeping latency stable even during long context windows.
- 📴 Offline-First Architecture: Built for real local inference: offline agents, private LLM workflows, and on-device AI.
- 🔊 Audio Integration: TTS output is streamed directly into Java Sound APIs—no WAV files, no external players, just pure real-time audio synthesis inside the JVM.
Add the JitPack repository and the dependencies to your pom.xml:
<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
</repository>
</repositories>
<dependencies>
<!-- FastAIModel Engine -->
<dependency>
<groupId>com.github.andrestubbe</groupId>
<artifactId>FastAIModel</artifactId>
<version>0.1.1</version>
</dependency>
<!-- FastCore (Mandatory Native DLL Loader) -->
<dependency>
<groupId>com.github.andrestubbe</groupId>
<artifactId>FastCore</artifactId>
<version>0.1.1</version>
</dependency>
</dependencies>repositories {
maven { url 'https://jitpack.io' }
}
dependencies {
implementation 'com.github.andrestubbe:FastAIModel:0.1.1'
implementation 'com.github.andrestubbe:FastCore:0.1.1'
}- ROADMAP.md: Planned milestone features and performance extensions.
- REFERENCE.md: JNI contracts and configuration options.
- PHILOSOPHY.md: In-process design decisions.
- CHANGELOG.md: Releases history.
| Platform | Status |
|---|---|
| Windows 10/11 (x64) | ✅ Fully Supported |
| Linux | 🚧 Planned |
| macOS | 🚧 Planned |
- FastAI - Unified AI client interface for Java
- FastAIMemory - Unified conversation history and prompt formatters
- FastCore - Unified JNI loader and platform abstraction
MIT License — See LICENSE file for details.
Part of the FastJava Ecosystem — Making the JVM faster. Small package. Maximum speed. Zero bloat. 🚀📋