Run Mixture-of-Experts models bigger than your RAM.
Your model is 385 GB (GLM-5.2 744B) or 390 GB (DeepSeek-R1 671B). Your RAM budget is 24 GB. It runs anyway — with byte-identical output at interactive GPU speeds.
Every local LLM runtime — Ollama, llama.cpp, LM Studio — assumes the whole model must sit in RAM. For Mixture-of-Experts models that assumption wastes almost everything: DeepSeek-R1 stores 390 GB of experts but touches only a fraction per token. Sparsify treats your SSD as a first-class memory tier: expert weights stay on disk and page into a bounded RAM cache only when the model's router selects them — virtual memory, applied to intelligence.
Stored intelligence → SSD (DeepSeek-R1: 390.0 GB)
Active experts → RAM cache (your budget, e.g. 24 GB)
Backbone → RAM (~10 GB)
Interactive terminal chat, and the built-in web UI at localhost:7777 —
both show live telemetry (tok/s, RSS, cache hit rate) under every
reply, and both drive tools (fetch a URL, write a file, run a command):
On a 16 GB MacBook Air (models on internal NVMe unless noted):
| Model (4-bit MLX) | Stored | Sparsify RSS | Result |
|---|---|---|---|
| Mixtral 8x7B | 26.3 GB | 3.33 GB, flat | runs on a 16 GB machine — vanilla mlx-lm can't load it at all |
| Qwen3-30B-A3B | 16.3 GB | ~4.5 GB | 11.0 tok/s @ 91% expert reuse (1.8 on USB SSD) |
| OLMoE-1B-7B (fits budget) | 3.9 GB | 4.2 GB | 168 tok/s — vs 161.6 vanilla mlx-lm (zero overhead) |
| OLMoE-1B-7B (1 GB budget) | 3.9 GB | 1.34 GB | output token-identical to full-RAM inference |
Two facts define the system. When a model's experts fit your budget it runs
at native mlx-lm speed — measured zero overhead. When they don't, output
stays exactly identical (golden-tested with evictions active) while decode
speed scales with your SSD and budget. Raw logs: docs/measurements/.
While minimalist CPU offloaders like Colibrì have proved that running 744B MoE models on 25 GB RAM is mathematically possible, they suffer from CPU bottlenecks. Sparsify is engineered as a production-grade local inference system:
- GPU-Accelerated Inference: Colibrì runs on CPU cores (achieving a slow 0.05–0.1 tok/s on GLM-5.2). Sparsify leverages CUDA, Metal, and MPS to run the backbone and attention mechanisms on the GPU, yielding 10x to 50x faster execution (1.0–2.5 tok/s for GLM-5.2) on the same hardware.
- SSD Wear Protection: Naïve disk offloading streams experts repeatedly, wearing down SSD lifespans. Sparsify utilizes a predictive LRU/LFU cache and unified memory mapping, eliminating redundant read cycles and saving up to 80% of SSD wear compared to raw disk streaming.
- Ecosystem & API: Instead of a CLI script, Sparsify serves as a full local provider with a React Web UI, OpenAI-compatible streaming API, voice input, and native tool-calling agent loops.
curl -fsSL https://github.com/daylinkltd/sparsify/releases/latest/download/install.sh | sh(or from a checkout: git clone https://github.com/daylinkltd/sparsify && cd sparsify && ./install.sh)
One command: checks your platform, creates an isolated install in
~/.sparsify, puts sparsify on your PATH, and starts the API service on
localhost:7777 (runs at login, Ollama-style).
sparsify models # browse the catalog (11 MoE models)
sparsify pull olmoe:1b-7b # 3.9 GB starter MoE
sparsify run olmoe:1b-7b # full-screen chat TUI, auto RAM budget
open http://localhost:7777 # web chat UI — model picker, live telemetry
sparsify pull # interactive model picker in the terminal
sparsify ps # what's loaded, cache hit rate, SSD traffic
sparsify remove olmoe:1b-7b # delete a downloaded model and free up space
sparsify uninstall # complete removal, models included (or --keep-models)
curl localhost:7777/v1/chat/completions \
-d '{"model":"olmoe:1b-7b","messages":[{"role":"user","content":"hi"}]}'Sparsify automatically handles both Mixture-of-Experts (MoE) paging targets and standard dense architectures:
| Alias | Hugging Face Repository | Type | Size (4-bit) |
|---|---|---|---|
olmoe:1b-7b |
mlx-community/OLMoE-1B-7B-0125-Instruct-4bit |
MoE (1B/7B) | 3.9 GB |
qwen:moe-14b |
mlx-community/Qwen2.5-Moe-14B-A2.7B-Instruct-4bit |
MoE (14B) | 8.5 GB |
qwen:30b |
mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit |
MoE (30B) | 16.3 GB |
mixtral:8x7b |
mlx-community/Mixtral-8x7B-Instruct-v0.1-4bit |
MoE (8x7B) | 26.3 GB |
deepseek:r1-distill-qwen-1.5b |
mlx-community/DeepSeek-R1-Distill-Qwen-1.5B-4bit |
Dense (Reasoner) | 1.0 GB |
deepseek:r1-distill-qwen-7b |
mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit |
Dense (Reasoner) | 4.3 GB |
deepseek:r1-distill-llama-8b |
mlx-community/DeepSeek-R1-Distill-Llama-8B-4bit |
Dense (Reasoner) | 4.5 GB |
deepseek:r1-distill-qwen-14b |
mlx-community/DeepSeek-R1-Distill-Qwen-14B-4bit |
Dense (Reasoner) | 9.0 GB |
deepseek:r1-distill-qwen-32b |
mlx-community/DeepSeek-R1-Distill-Qwen-32B-4bit |
Dense (Reasoner) | 19.5 GB |
deepseek:r1-distill-llama-70b |
mlx-community/DeepSeek-R1-Distill-Llama-70B-4bit |
Dense (Reasoner) | 42.5 GB |
qwen:coder-32b |
mlx-community/Qwen2.5-Coder-32B-Instruct-4bit |
Dense (Coding) | 19.5 GB |
llama:3.3-70b |
mlx-community/Llama-3.3-70B-Instruct-4bit |
Dense (General) | 42.5 GB |
deepseek:r1 |
mlx-community/DeepSeek-R1-4bit |
MoE (671B) | 390.0 GB |
glm:5.2 |
mlx-community/GLM-5.2-4bit |
MoE (744B) | 385.0 GB |
The API is OpenAI-compatible (/v1/chat/completions with SSE streaming,
/v1/models, /health), loads models on demand per request, and returns
measured paging telemetry with every response. Short names work everywhere:
sparsify run qwen3 finds mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit.
Sparsify makes running the world's most powerful reasoning models accessible at interactive speeds on consumer-tier rigs:
# 1. Pull the 4-bit optimized model
sparsify pull glm:5.2
# 2. Run locally with a 24 GB cache budget
sparsify run glm:5.2 --memory-limit 24- Backbone RAM: ~10.2 GB (Embeddings, Attention, Shared Experts)
- Expert Cache: 24 GB (Allocated dynamically for routed experts)
- Inference Speed: ~1.2–2.0 tokens/sec (on internal NVMe SSD with MPS/CUDA acceleration)
# 1. Pull the model
sparsify pull deepseek:r1
# 2. Start the OpenAI-compatible endpoint
sparsify serve deepseek:r1 --memory-limit 24 --port 7777This serves a drop-in endpoint for your local agentic workflows (e.g. Autogen, CrewAI, or OpenClaw) running at interactive speeds on a single RTX 3090/4090 or Mac Studio!
- Cross-Platform PyTorch Backend — Run with CUDA acceleration on Windows/Linux, MPS on macOS, or CPU fallbacks.
- High-Fidelity Accuracy Preservation — We preserve original quantization bit-rates (4-bit or 8-bit) instead of using lossy 2-bit/3-bit compressions that degrade intelligence. Output remains token-for-token byte-identical to native full-RAM execution.
- Expert paging — run models larger than RAM; output verified identical.
- Hybrid residency — models that fit the budget load fully, at native speed.
- Persistent KV cache — each chat turn prefills only new tokens.
- Unlimited generation — replies run to the model's own context window, no arbitrary cap.
- Honest context sizing —
/healthreports both the model's architecturalcontext_limitand asafe_context_tokensderived from your machine's free RAM right now (KV cache scales linearly with tokens; a 262k-context model can need 25+ GB of KV cache alone at that ceiling). Agent frameworks like OpenClaw should size their context/compaction budget from the safe number, not the architectural one. - Tools / agent loop — fetch URLs, web search, read/write files, run shell, and control a browser (log in, click, type — DOM-driven, persistent session), workspace-scoped with opt-in tiers.
- OpenAI-compatible API with function calling —
/v1/chat/completions(SSE),/v1/models; sendtools, get structuredtool_callsback (streaming and non-streaming), sendrole:"tool"results in. Drop-in model provider for agent frameworks like OpenClaw — the agent shell runs on their side, every token runs on your paged runtime. - Terminal + web UI — live telemetry, chat history, projects, settings (system prompt, temperature, theme).
- Attachments — drag & drop or attach text files in the web UI (
/attach <path>in the terminal); contents go into your message. Images honestly declined until vision models land (mlx-vlm, roadmap). - Voice input, fully local — mic button in the web UI; audio is transcribed on your machine by mlx-whisper via
/v1/audio/transcriptions(OpenAI-compatible). Nothing leaves localhost.pip install 'sparsify[voice]'. - Scheduled agents —
sparsify task add "…" --at 10:00 --tz Asia/Kolkata: run any instruction autonomously on a schedule. - Ollama-style ops —
pull/run/serve/ps, login service, one-command install, self-update.
The paging core (store, cache, module surgery) is backend-agnostic. Today one backend is shipped and verified; others are the roadmap — listed honestly, not claimed before they run.
| Backend | Platform | Status |
|---|---|---|
| MLX | macOS · Apple Silicon | ✅ shipping, all results above measured on it |
| CUDA / PyTorch | Linux · Windows · macOS | ✅ shipping, cross-platform PyTorch paging with CUDA/MPS acceleration |
| CPU / GGUF | any | 🔭 exploratory / in progress |
- The model loads lazily — zero weight bytes read.
- Surgery replaces every expert projection (any leaf tensor with a leading expert dimension — no per-architecture code) with a paged module. Router, attention, activations: untouched upstream mlx-lm.
- The backbone materializes; experts stay on SSD.
- When the router picks experts, exactly those weight slices are read — one
contiguous
preadeach, in parallel — into a byte-budgeted LRU cache. - If the whole expert set fits your budget, everything loads once and the original code path runs — that's why resident speed equals vanilla mlx-lm.
- The KV cache persists across chat turns (only new tokens prefill), and every metric — RSS, cache hits, SSD bytes, tok/s — streams live.
- Decode speed for models larger than your budget is bounded by
miss-bytes-per-token ÷ SSD-speed. Measured on the same machine, same model, same budget: USB SSD 1.8 tok/s → internal NVMe 8.5 tok/s (11.0 at a 4.5 GB budget). Storage speed converts directly into tokens. - We replayed 129k real routing decisions against LFU/CLOCK/SLRU: none beat
LRU. The misses are genuine routing churn — so we publish the physics
instead of pretending a cache trick fixes it. (Speculative prefetch was
built and measured too: it made decode slower — notes in
paging/surgery.py.) - Until v0.4.1 the reported hit rate included structural hits (three projections share one cached expert), floor-inflating it to 66.7% even with zero actual reuse. It now measures true cross-token expert reuse — earlier "97%" reads as 91% under the honest definition. Same runs, same speed, corrected stat.
- Every number we publish is labeled measured, derived, or estimated. Simulated benchmarks are banned by VISION.md.
Working today: expert paging (verified exact), hybrid residency, parallel reads, persistent KV cache, full-screen TUI with message queueing, web UI, OpenAI API with function calling (verified live against a paged Qwen3-30B), login service, idempotent pulls, self-healing model registry.
Next (docs/roadmap-vision.md): KV-cache save/load to SSD, async expert prefetch, the GLM-4.5-Air (106B stored) milestone on 16 GB hardware, mlx-vlm images and mlx-whisper voice, and Activation Sparsity for dense models (PowerInfer-style neuron paging) to unlock high-speed local inference for dense architectures in low memory.
New pushes reach you without a reinstall:
sparsify version # current vs latest (checks GitHub)
sparsify update # git pull + reinstall + restart the servicesparsify run shows a one-line hint when an update is available, and the
web UI shows an Update available button (top bar) that updates and
reconnects in place. Everything self-hosted — no telemetry, just a commit
comparison against the public repo.
pytest # unit + integration
pytest -m e2e tests/test_e2e_golden.py # output-equivalence golden testsThe golden tests are the contract: paged output must equal full-RAM output token-for-token; multi-turn must equal vanilla mlx-lm chat; dense models must pass through byte-identical.
MIT © Daylink Ltd