Hybrid GPU/CPU offloading for MoE models

[Defilan]

Summary

Add first-class support for hybrid GPU/CPU inference, enabling large Mixture-of-Experts (MoE) models to run on VRAM-constrained hardware by offloading expert weights and/or KV cache to system RAM.

Motivation

MoE models like Qwen3-30B-A3B (30B total params, ~3B active per token) and DeepSeek V3 are increasingly popular but their total parameter count far exceeds what consumer GPUs can hold. However, because only a small fraction of experts activate per token, keeping the dense layers (attention, embeddings, shared experts) on GPU while placing routed expert weights in CPU RAM achieves near-GPU generation speed at a fraction of the VRAM cost.

Real-world example: Dual RTX 5060 Ti 16GB running Qwen3-30B-A3B with 90K context by keeping active experts on GPU and offloading inactive experts + KV cache to CPU RAM.

This also applies to extended-context dense model inference — --no-kv-offload lets the KV cache live in system RAM, freeing VRAM for model weights when context windows exceed what VRAM can hold.

Current State

LLMKube already has the building blocks but lacks MoE-specific and hybrid-specific controls:

Feature	Status
--n-gpu-layers (partial layer offload)	Implemented (spec.hardware.gpu.layers)
KV cache quantization (--cache-type-k/v)	Implemented (cacheTypeK/cacheTypeV)
Multi-GPU layer sharding	Implemented
--cpu-moe (MoE expert offload to CPU)	Not implemented
--n-cpu-moe (partial MoE expert offload)	Not implemented
--no-kv-offload (KV cache in system RAM)	Not implemented
--override-tensor (fine-grained tensor placement)	Not implemented
Host memory resource requests for hybrid pods	Not implemented

Why MoE Models Specifically Benefit

Dense models pay a steep penalty for CPU offload because every layer must transit PCIe on every token. MoE models are fundamentally different:

• Only 2-8 experts activate per token out of potentially 64-256 total
• Activation vectors crossing PCIe are small (kilobytes, not full weight tensors)
• CPU executes expert matrix multiply locally using DDR bandwidth
• The non-selected experts incur zero cost regardless of placement
• ~15-20% of experts receive ~80% of routing traffic, creating a natural hot/cold tier

Result: MoE models lose far less performance from expert-on-CPU than dense models lose from layer-on-CPU.

Performance Characteristics

Configuration	Generation	Prompt Processing	Context Capacity
Full GPU (fits in VRAM)	20-60 t/s	500-2000 t/s	Limited by VRAM
Hybrid GPU+CPU (MoE experts on CPU)	3-12 t/s	30-150 t/s	VRAM + RAM
Full CPU	1-3 t/s	5-30 t/s	RAM only

Proposed Implementation

Phase 1: Core hybrid offloading flags

New fields on InferenceServiceSpec:

• moeCPUOffload (bool) — maps to --cpu-moe. Keeps all routed expert FFN weights in system RAM while attention and dense layers remain on GPU.
• moeCPULayers (int32) — maps to --n-cpu-moe N. Partial expert offload for first N layers only.
• noKVOffload (bool) — maps to --no-kv-offload. Forces KV cache to system RAM instead of VRAM. Critical for extended context windows.

New field on InferenceResourceRequirements:

• hostMemory (string, e.g. "64Gi") — explicit system RAM request for hybrid pods. Translated to resources.requests.memory on the pod spec. Without this, K8s has no idea the pod will consume 64GB of RAM alongside its GPU, causing OOM kills after model load.

Phase 2: Fine-grained control

• tensorOverrides ([]string) — maps to --override-tensor. Power-user escape hatch for regex-based tensor placement (e.g. ["exps=CPU", "blk\\.(0|1)\\.ffn_.*=CUDA0"]).
• batchSize / uBatchSize (int32) — maps to --batch-size / --ubatch-size. Critical for hybrid performance; larger batches amortize PCIe overhead.

Phase 3: Operator intelligence

• Validation: emit Warning event when moeCPUOffload=true but hostMemory is not set
• Auto-detection: if GGUF metadata indicates MoE architecture, emit suggestion event recommending hybrid offload for VRAM-constrained nodes
• Status field: HybridOffloadActive on InferenceServiceStatus for dashboards

Kubernetes Scheduling Considerations

Standard GPU pods request minimal memory. Hybrid pods need accurate RAM requests:

# Without hybrid awareness (current) - pod gets OOM killed
resources:
  limits:
    nvidia.com/gpu: "2"
  requests:
    memory: "2Gi"

# With hybrid awareness (proposed) - scheduler places correctly
resources:
  limits:
    nvidia.com/gpu: "2"
  requests:
    memory: "72Gi"  # expert weights (50GB) + KV cache (20GB) + overhead
    cpu: "8"         # CPU cores matter for expert throughput

NUMA topology also matters — CPU sockets on the same NUMA domain as the GPU get better PCIe bandwidth. Document recommending topology-manager-policy=best-effort for hybrid nodes.

Use Cases This Unlocks

1. Qwen3-30B-A3B on dual consumer GPUs — 32GB combined VRAM holds active path, experts in 64GB+ RAM, 90K+ context
2. DeepSeek V3/R1 671B on 4x RTX 4090 — 96GB VRAM for dense layers, expert weights in 256GB DDR5, enabling a 671B model on consumer hardware at 2-4 t/s
3. Extended context dense models — Llama 70B on A100-80GB with 128K context by pushing KV cache to RAM
4. Enterprise batch agentic inference — on-prem nodes with 2x A100 + 512GB RAM serve larger models overnight for non-real-time workloads

Competitive Position

Only llama.cpp and its derivatives provide genuine MoE-expert-aware CPU offloading. SGLang and vLLM are architected around tensor parallelism on homogeneous GPU clusters. This is a competitive moat for LLMKube — no other K8s operator exposes these controls.

References

• llama.cpp MoE offload guide (HuggingFace blog)
• llama.cpp --n-cpu-moe PR #15077
• llama.cpp --override-tensor PR #11397
• Two-tier expert cache feature request - Issue #20757
• ik_llama.cpp fork — enhanced CPU performance for MoE workloads

[Defilan]

Shipped in two phases: Phase 1 (#281) added moeCPUOffload, moeCPULayers, noKvOffload, hostMemory. Phase 2 (#283) added tensorOverrides, batchSize, uBatchSize. Both included in the 0.7.0 release.