UGR — AI Runtime Operating System

┌──────────────────────────────────────────────────────────────┐
│                        Application                           │
├──────────────────────────────────────────────────────────────┤
│                    OpenAI-Compatible API                      │
├──────────────────────────────────────────────────────────────┤
│                            Planner                            │
├──────────────────────────────────────────────────────────────┤
│                Runtime Intelligence Engine                    │
├──────────────────────┬───────────────────────────────────────┤
│      Scheduler       │              Simulation               │
├──────────────────────┴───────────────────────────────────────┤
│                Virtual Resource Manager (VRM)                 │
├──────────────────────────────────────────────────────────────┤
│               Virtual Memory Fabric (VMF) *                   │
├──────────────────────────────────────────────────────────────┤
│                   Driver Abstraction Layer                    │
├──────────────┬──────────────┬────────────────────────────────┤
│  llama.cpp   │  TensorRT *  │           Vulkan *             │
│   (CUDA/CPU) │  (planned)   │          (planned)             │
└──────────────┴──────────────┴────────────────────────────────┘

formerly Adaptive Memory Fabric (AMF). Virtual Memory Fabric better communicates that this is a virtual memory system, not a hardware fabric.

TensorRT and Vulkan backends are planned but not yet implemented.

Why UGR Exists

Modern inference engines optimize execution within a single memory domain (GPU VRAM or CPU RAM). UGR extends the execution model across multiple storage tiers and heterogeneous hardware, treating model components as virtualized resources instead of static files. This enables adaptive scheduling, predictive resource management, and execution strategies that are impractical in monolithic inference runtimes.

When a model exceeds VRAM, most runtimes OOM. UGR pages overflowing tensors to RAM, then to NVMe, then to remote storage — transparently, with heat-based eviction and prefetch. When you want to switch models, most runtimes reload from disk. UGR keeps a registry of cold/warm/hot models and switches in milliseconds. When you ask "will this model run on my hardware?", UGR simulates before loading a single byte.

How It Works: Execution Lifecycle

GGUF File
    │
    ▼
┌──────────┐
│  Planner │  Parse → Extract Metadata → Build Resource Graph
└────┬─────┘
     │
     ▼
┌──────────────┐
│  Simulation  │  Predict TPS, VRAM, bottlenecks (no tensor loading)
└──────┬───────┘
       │
       ▼
┌──────────────────┐
│  Memory Plan     │  Assign resources to tiers (VRAM / RAM / NVMe)
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  Resource Graph  │  Decomposed model: layers → experts → projections
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│  Timeline        │  Sorted: prefetch → compute → migrate → evict
└──────┬───────────┘
       │
       ▼
┌──────┐
│  VMF  │  Move tensors between tiers as execution progresses
└──┬───┘
   │
   ▼
┌──────────┐
│  Backend │  llama.cpp / CPU / (future) TensorRT
└────┬─────┘
     │
     ▼
┌───────────┐
│ Inference │  Streaming tokens back to the user
└───────────┘

Design Principles

• Resource virtualization over monolithic model loading — every tensor, KV cache slot, and expert is a first-class resource with its own lifecycle.
• Backend independence through a stable driver API — swap llama.cpp for TensorRT without changing application code.
• Adaptive execution based on hardware capabilities — detect, profile, and persist hardware characteristics; tune parameters automatically.
• Predictive scheduling instead of reactive paging — simulate before you load, prefetch before you compute, evict before you OOM.
• Measure before optimizing — every inference updates the capability database; the runtime improves over time.
• Favor portability over hardware-specific tuning — the same binary runs on an A100 workstation, a laptop with integrated graphics, and a headless CPU server.

vs. Other Approaches

llama.cpp

llama.cpp	UGR
Single model, manual reload	Multi-model registry with instant /switch
OOM on VRAM overflow	Virtual Memory Fabric pages to RAM → NVMe
All MoE experts loaded	Per-expert NVMe slices; only active 8-10 loaded
No hardware awareness	Auto-detects VRAM, PCIe, NUMA, CPU features
No pre-execution prediction	ugr simulate detects OOM before loading
Hardcoded build paths	backends.yaml — swap backends from a config file

Ollama

Ollama	UGR
Orchestration CLI + llama.cpp	Full resource-virtualized microkernel
Single-tier memory	5-tier VMF with heat-based eviction, prefetch, compression
No MoE paging	Per-expert NVMe slice paging
No scheduling	5 policies (latency, capacity, balanced, adaptive)
No runtime learning	RIE adapts policies from telemetry history
No simulation	ugr simulate with capability profile + bottleneck analysis

Simulation: The Flagship Feature

Before downloading or loading a model, UGR can tell you whether it will run, how fast, and where the bottleneck is:

$ ugr simulate gemma-4-31B-it-Q4_K_M --prompt 512 --output 128

═══════════════════════════════════════
  UGR Simulation Report
═══════════════════════════════════════

Hardware
  Quadro M1200         4 GB VRAM
  32 GB RAM            PCIe Gen3

Model
  Gemma-4-31B          18.3 GB
  62 layers            1536 embedding

── Feasibility ────────────────────────
  ✓ Runnable                       Peak VRAM:  3.8 GB
                                   Peak RAM:    28 GB
                                   Peak NVMe:   6 GB

── Performance ───────────────────────
  Estimated TPS:     1.9 tok/s     TTFT:       3.2 sec
  RAM-only TPS:      0.8 tok/s     Mixed TPS:  1.9 tok/s

── Memory Traffic ────────────────────
  Total Migrations:  34            Cache Hit:  83%
  Migration BW:      142 MB/s      Per-Token:  1.2 MB

── Analysis ──────────────────────────
  Bottleneck:   PCIe Bandwidth
  GPU Util:     84%
  PCIe Saturation:  YES

  Recommendation: Increase prefetch window, switch to Capacity scheduler

This works against any registered model. It uses your hardware's capability database (~/.ugr/hardware.json) to get accurate bandwidth figures. No tensor data is loaded — only the GGUF metadata is parsed.

Benchmarks

UGR's strengths aren't about peak tok/s on a single model — they're about what you can run and manage on constrained hardware.

Metric	llama.cpp alone	With UGR
Largest runnable model (4 GB VRAM, 32 GB RAM)	~3.5 GB (fits in VRAM)	~31 GB (pages through RAM + NVMe)
Model switch time	5-30s (full reload)	<100ms (session switch, no reload)
MoE expert management	All 512 experts loaded	Only 10 active experts per forward pass
Pre-load OOM detection	None (load until crash)	ugr simulate before downloading
Multi-model serving	Manual port management	Single daemon, instant /switch
Hardware adaptation	Manual --gpu-layers	Auto-detection + persisted profile + RIE tuning

Measured Throughput

Hardware: Quadro M1200 (4 GB VRAM, Pascal), 32 GB RAM

Model	Size	Layers	TPS (CPU)	TPS (GPU 4 layers)
Qwen3.5-2B-Instruct (Q4_K_M)	1.3 GB	25	10.60	15.2
Gemma-4-31B-it (Q4_K_M)	18.3 GB	62	0.96	2.1

GPU offload tested with --gpu-layers 4 and GGML_CUDA_ENABLE_UNIFIED_MEMORY=0.

MoE Per-Expert NVMe Paging

For Mixture-of-Experts (MoE) models (e.g. Mixtral 8x7B, Qwen-MoE, DeepSeek-MoE), UGR provides dynamic per-expert slice paging. Traditional inference engines load all $N$ experts into RAM simultaneously (causing OOM on large models). UGR's pure-Go inference path (--pure-go) routes tokens through the gating network and reads only the top-$K$ active experts per token directly from the mmap'd GGUF file into an LRU memory cache.

# Run a 31 GB MoE model on 4 GB VRAM + 32 GB RAM using MoE expert paging
./ugr run mixtral-8x7B-Q4_K_M --pure-go -i

Outputs live telemetry per token:

• Active routed experts per layer
• RAM cache hit rate & LRU eviction
• NVMe/Disk slice read bandwidth

Quick Start

# Build (CPU-only)
cd ugr && go build -o ugr ./cmd/ugr

# Pull a model
./ugr pull Qwen/Qwen3.5-2B-Instruct:Q4_K_M

# Simulate before running
./ugr simulate Qwen3.5-2B-Q4_K_M

# Interactive inference (standard engine)
./ugr run Qwen3.5-2B-Q4_K_M -i

# Interactive inference (MoE expert paging mode)
./ugr run mixtral-8x7B-Q4_K_M --pure-go -i

# System dashboard
./ugr stats

Prerequisites

• Go 1.22+, Linux
• CUDA toolkit and llama.cpp build (optional, for GPU offload)

Full installation guide: docs/installation_guide.md

CLI Reference

Command	What it does
pull <model>	Download from HuggingFace or register local GGUF
run <model> [--pure-go]	Interactive (-i) or one-shot inference (--pure-go for MoE paging)
simulate <model>	Predict TPS, peak memory, bottleneck before loading
list	Registered models with state (cold/warm/hot)
info <model>	Full architecture breakdown
rm <model>	Remove model and all files
ps	Active sessions
stats	System dashboard (GPU, RAM, NVMe, model states)

Daemon (ugrd)

OpenAI-compatible REST API:

ugrd --model Qwen/Qwen3.5-2B-Instruct:Q4_K_M --addr :8080

curl http://localhost:8080/v1/chat/completions \
  -d '{"model":"qwen","messages":[{"role":"user","content":"Hello"}]}'

Architecture

Full documentation: docs/architecture.md

Subsystem design (RFC-style): docs/design/

Project Status

Feature	Status
Multi-backend inference (llama.cpp, CPU)	Stable
GGUF parser (pure Go, MoE-aware)	Stable
Model registry + CLI	Stable
Daemon (OpenAI-compatible API)	Stable
Virtual Memory Fabric (page files, eviction, migration)	Stable
Resource pool (VRAM/RAM budget, LRU eviction)	Stable
Multi-modal input (image/audio/video)	Stable
MoE per-expert NVMe paging (--pure-go)	Stable
Runtime Intelligence Engine	Experimental
Simulation engine	Experimental
Capability database	Experimental
TensorRT backend	Planned
Vulkan backend	Planned