CPU AI Runtime

CPU AI Runtime

Purpose

This page describes the CPU-only model runtime assumptions for XAI OS.

Contents

• CPU-Only Model Serving
• Shared Weights and Private State
• Model Loader
• Tokenizer Boundary
• Acceleration Boundary

CPU-Only Model Serving

XAI OS targets CPU-bound AI models. The runtime should support quantized local models and predictable inference loops without CUDA, Metal, GPU, or NPU dependency.

The first runtime should be minimal and measurable. It does not need to support every model format before the OS proves boot, memory, core isolation, networking, and telemetry.

Shared Weights and Private State

Shared read-only model weights are central to XAI OS. A host running many app agents should avoid loading duplicate copies of the same model.

Each agent gets:

• a shared model-weight mapping;
• private KV/cache arenas;
• private tokenizer/request state;
• private source-code context;
• explicit permissions.

Model Loader

The loader should:

• validate model metadata;
• map weights read-only;
• prefer hugepage/large-page mappings where available;
• prefault hot regions before READY;
• expose model identity for sharing;
• report memory placement and faults.

Initial model format support is an open implementation choice. GGUF is a practical first candidate for quantized CPU-only local models. SafeTensors may be useful for tensor metadata and validation later, but no format should be treated as final until recorded in Open Decisions.

For the QEMU full-OS path, the current MVP uses a small deterministic model file at /models/cpu-ai-mvp.xaiosmodel inside the VirtIO-backed read-only filesystem. This is a real file-loader boundary for QEMU correctness, not a production model format decision.

The QEMU model file records:

• magic and version;
• CPU-only flags;
• 8-bit quantization metadata;
• shared weight range;
• byte-table tokenizer range;
• deterministic CPU runtime id;
• required private KV/cache bytes;
• FNV-1a payload checksum.

Invalid manifests, checksum failures, undersized tokenizer tables, undersized KV/cache bindings, missing model files, and GPU-required manifests fail closed and increment telemetry.

Model Arena Layout

model identity metadata
  -> read-only weight mapping
  -> optional hugepage alignment padding
  -> per-platform CPU dispatch metadata

per-agent private state
  -> tokenizer/request state
  -> KV/cache arena
  -> source-context references

The model arena is shared read-only. The KV/cache arena is private per embedded app agent. This avoids duplicate model copies while preserving per-agent inference state.

Tokenizer Boundary

Tokenization can run outside the hottest inference loop if that improves isolation. Tokenizer memory and source-context memory should be accounted separately from model weights and KV/cache.

The QEMU MVP keeps tokenization and deterministic decode as separate runtime stages. The tokenizer binds a 256-entry byte table from the loaded model file. The CPU dispatch boundary calls the deterministic CPU kernel through the runtime id and records dispatch telemetry.

This is not a production model implementation; it is a correctness gate that proves AI Cells bind shared read-only weights, private KV/cache arenas, and tokenizer/runtime state through explicit contracts.

QEMU Telemetry

Milestone 37 requires QEMU telemetry for:

• cpu_ai_model_file_loads;
• cpu_ai_model_file_rejects;
• cpu_ai_model_bytes_loaded;
• cpu_ai_manifest_validations;
• cpu_ai_tokenizer_binds;
• cpu_ai_kernel_dispatches;
• cpu_ai_admission_rejects;
• cpu_ai_checksum_failures;
• the earlier model-load, tokenizer-call, runtime-call, KV-write, shared-weight, and GPU-reject counters.

Acceleration Boundary

Architecture-specific CPU acceleration is allowed through generic fallbacks:

• scalar fallback first;
• x86-64 vector paths where available;
• AArch64 vector paths where available;
• AMX or similar matrix extensions only behind feature detection.

GPU acceleration is not a requirement and must not be necessary for correctness.

Implementation Requirements

• Model identity must include format, quantization, checksum, and architecture compatibility.
• Shared model weights must be mapped read-only in every AI Cell.
• Private KV/cache arenas must never alias another agent's private state.
• CPU feature dispatch must fall back safely when AMX, AVX-family features, or AArch64 vector features are unavailable.
• Tokenizer errors must fail the request without corrupting model arena state.

Related Pages

• Memory System
• Performance Targets
• Intel Xeon Port
• Benchmark Result Format