A distributed AI inference platform built on three consumer-grade AMD machines. Routes inference requests across heterogeneous compute (CPU, GPU, NPU) based on task complexity. Targets $0 cloud spend for privacy-sensitive workloads, with optional cloud burst when local capacity is exhausted.
Built and maintained by John Pritchard. Live fleet benchmarks at rawmark.io.
Frontier AI APIs are powerful but they are also a privacy boundary, a recurring cost line, and a single point of failure. BIFROST is the answer to: what if a small consulting practice or regional firm wanted Claude-class capability for compliance-sensitive workloads, on hardware they already own, with cloud burst as an option rather than a default?
The platform routes each request to the smallest tier that can handle it — a 1B-parameter NPU model for trivial classification, a 70B GGUF on llama-server for hard reasoning, and several stops in between. Cascade fallbacks handle overload. Failed-tier escalation is observable, not silent.
This repository is the open-source core. Commercial deployments add data-localization, custom skill packages, and on-site or remote operator support.
flowchart TB
Client[Client request<br/>OpenAI / Anthropic API shape]
Client --> Router
subgraph Router["BIFROST Router (Hearth k3d, port 8089)"]
Classify[Classifier<br/>complexity band:<br/>TRIVIAL / MODERATE / COMPLEX]
Cascade[Cascade selector<br/>per-band tier order]
Dispatch[Tier dispatch<br/>Ollama / llama-server / FLM]
Classify --> Cascade --> Dispatch
end
Dispatch --> Bifrost
Dispatch --> Hearth
Dispatch --> Forge
subgraph Bifrost["Bifrost · Ryzen 9 3950X · RX 9070 XT 16GB RDNA4"]
BR1[Ollama :11434<br/>T1B coder + instruct]
end
subgraph Hearth["Hearth · Ryzen 7 5700G · RX 5700 XT + Vega8"]
HR1[Ollama :11434<br/>T1A-hearth fallback]
HR2[Ollama :11436<br/>Vega8 embeddings]
end
subgraph Forge["Forge · Ryzen AI Max+ 395 · 96GB unified · XDNA2 NPU"]
FR1[Ollama :11434<br/>T2 26B + T1 phi4-mini]
FR2[llama-server :11438<br/>T2.5 Llama 3.3 70B]
FR3[FLM :8003<br/>NPU 1B-class]
end
Forge -.bench data.-> Rawmark[rawmark.io<br/>live fleet metrics]
Three nodes, six tiers, one routing layer. Each tier is a real running model on real silicon. Cascade tables map complexity bands to tier-order — TRIVIAL hits NPU first, COMPLEX hits the 70B first, with documented fallbacks when a tier is unhealthy. The Router exposes both /v1/chat/completions (OpenAI shape) and /v1/messages (Anthropic shape) so clients written for either provider work without modification.
Hardware diversity is intentional. RDNA4 (Bifrost), RDNA1 (Hearth), and XDNA2 NPU + iGPU (Forge) cover the realistic spread of consumer-grade AMD silicon a privacy-sensitive client would already own.
The full fleet requires three machines. The single-node demo below runs the Router + a local Ollama instance on one box and demonstrates routing with one tier.
# 1. Clone and set up
git clone https://github.com/jhpritch-dev/bifrost-platform.git
cd bifrost-platform
# 2. Start a local Ollama instance with one model (8GB+ RAM)
ollama serve &
ollama pull qwen2.5-coder:7b
# 3. Start the Router pointed at local Ollama
docker run --rm -p 8089:8080 \
-e BIFROST_TIER_OVERRIDE=1a-coder=http://host.docker.internal:11434 \
bifrost-router:latest
# 4. Send a request
curl http://localhost:8089/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"messages": [{"role":"user","content":"Write a Python one-liner that reverses a string."}]
}'For the full three-node deployment (k3d on Hearth + Ollama on Bifrost + Ollama / llama-server / FLM on Forge), see docs/DEPLOYMENT.md.
bifrost-platform/
├── router/ # Routing service (FastAPI + dispatch logic)
├── classifier/ # Complexity classification
├── docs/ # Architecture, deployment, ops runbooks
├── benchmarks/ # Fleet benchmark harness (rawmark.io source)
└── README.md # You are here
Sibling repos:
jhpritch-dev/bifrost-skills— Anthropic SKILL.md registry served over HTTP for n8n + OpenCode + cross-fleet consumers- Live benchmark data: rawmark.io
Active development. Routing layer, cascade fallback, classifier, Anthropic API compatibility shim, and observability gates are all live. Latest router release: v92.
Recent work:
- Anthropic Messages API shim so Claude Code clients can route through BIFROST without modification
- Reasoning-coalesce to handle Ollama thinking-models that emit reasoning side-channel and leave
contentempty - Pre-cycle tier health gates that catch silent backend failures before they degrade accept rates
- Mainline kernel 6.19 migration on Forge to work around an upstream IOMMU-SVA regression in 6.17/6.18.y that was breaking NPU SVA initialization
A more detailed log lives in docs/SESSION-LOG.md.
Issues and PRs welcome. The codebase is set up for iterative agent-driven development — there is a strong convention of writing intent briefs before changes, and the decisions.md log captures every accepted/rejected change for both human review and future training.
If you are interested in the commercial side (privacy-sensitive deployments for regional firms, on-site model hosting, RFP work), see rawmark.io for contact.
Apache 2.0. See LICENSE.
John Pritchard — building distributed AI infrastructure on consumer hardware. Open to AI infrastructure / ML engineering / MLOps roles.
LinkedIn · GitHub · rawmark.io