From 455759834a263faf0b4d719515074731e1b1cd6f Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Tue, 7 Apr 2026 20:03:55 -0500
Subject: [PATCH 01/21] Add RotorQuant KV cache backend with deferred prefill
 on Metal
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Pure-MLX port of IsoQuant 3-bit (block-diagonal quaternion rotations
+ Lloyd-Max centroids) with deferred prefill — a contribution that
does not exist in any upstream project, since the llama.cpp fork
ships the deferred-prefill flush CUDA-only. GQA-native, no fallback
for grouped-query models.

Also adds the "Everything Different About Skulk" living section to
README.md and website/docs/everything-different.md so the divergence
from upstream exo lives in one canonical place going forward.

Tracked by #102.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 README.md                                     |  86 ++-
 .../src/components/layout/SettingsPanel.tsx   |   8 +-
 src/exo/store/config.py                       |   8 +-
 src/exo/worker/engines/mlx/cache.py           |  29 +
 src/exo/worker/engines/mlx/constants.py       |  16 +
 .../worker/engines/mlx/rotorquant/__init__.py |  30 +
 .../worker/engines/mlx/rotorquant/cache.py    | 545 ++++++++++++++++++
 .../engines/mlx/rotorquant/quantizer.py       | 146 +++++
 .../worker/engines/mlx/rotorquant/rotation.py | 104 ++++
 .../worker/engines/mlx/rotorquant/tables.py   | 102 ++++
 .../engines/mlx/rotorquant/tests/__init__.py  |   0
 .../mlx/rotorquant/tests/test_cache.py        | 120 ++++
 .../rotorquant/tests/test_iso_quantizer.py    | 118 ++++
 src/exo/worker/runner/llm_inference/runner.py |   2 +
 website/docs/everything-different.md          |  95 +++
 website/docs/intro.md                         |   1 +
 website/docs/kv-cache-backends.md             |  29 +-
 website/sidebars.ts                           |   5 +
 18 files changed, 1436 insertions(+), 8 deletions(-)
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/__init__.py
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/cache.py
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/quantizer.py
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/rotation.py
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/tables.py
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/tests/__init__.py
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
 create mode 100644 src/exo/worker/engines/mlx/rotorquant/tests/test_iso_quantizer.py
 create mode 100644 website/docs/everything-different.md

diff --git a/README.md b/README.md
index 56e8008b5..8bb5646d4 100644
--- a/README.md
+++ b/README.md
@@ -20,11 +20,90 @@ a more modern dashboard, richer API workflows, sophisticated cache quantization,
 - Form a small cluster of Macs and split larger models across them.
 - Use a central model store so the cluster downloads once and stages locally.
 - Talk to the cluster through OpenAI Chat Completions, OpenAI Responses, Claude Messages, or Ollama-compatible APIs.
+- Push KV cache memory down hard with rotation-based 3-bit quantization (RotorQuant, OptiQ, TurboQuant) and pick a backend per workload from the dashboard.
+- Use a model-aware reasoning contract that handles toggleable and non-toggleable thinking models without baking assumptions into client code.
 - Experiment with advanced placement modes, RDMA, and KV cache backends when you are ready.
 - Run non-chat workloads such as embeddings and other specialized model flows.
 - Build TTS-oriented and other API-driven workflows on top of the cluster.
 - Actually use your cluster for real inference workloads instead of treating it as a demo.
 
+## Everything Different About Skulk
+
+This is the running list of where Skulk diverges from upstream [exo](https://github.com/exo-explore/exo). It is a living section — every meaningful change should land here when it ships, so anyone evaluating Skulk can see the surface area at a glance.
+
+### Inference and KV cache
+
+- **RotorQuant KV cache backend** — pure-MLX port of IsoQuant 3-bit (block-diagonal quaternion rotations + Lloyd-Max centroids) with **deferred prefill on Metal**, a contribution that does not exist in any upstream project (the llama.cpp fork ships it CUDA-only). GQA-native; no fallback for grouped-query models. See [docs/kv-cache-backends.md](docs/kv-cache-backends.md).
+- **TurboQuant native and adaptive backends** — randomized Hadamard rotation + Lloyd-Max centroids, with an adaptive variant that keeps edge attention layers in fp16 for accuracy.
+- **OptiQ KV cache integration** — wraps `mlx-optiq`'s rotated-space attention path so the rotation cost stays out of the per-token loop on supported (non-GQA) models.
+- **OptiQ mixed-precision weight quantization pipeline** — async wrapper around `mlx-optiq`'s sensitivity analysis and KL-divergence per-layer bit allocation, exposed as a model-store optimization job.
+- **KV prefix cache with snapshot/restore** — LRU-evicted prompt-prefix cache that snapshots SSM and rotating-window cache states so prefix matches are reusable across conversation turns even for hybrid Mamba/Transformer architectures.
+- **Pipeline-parallel prefill for short prompts** — pipelined models now route every prefill through the pipeline path, fixing prior warmup hangs on Gemma-class models.
+- **Force-sequential fallback** — quantized backends transparently fall back to a sequential generator when batch/history mode is incompatible with their cache layout.
+
+### Model capability system
+
+- **Two-layer capability model** — declarative `ModelCard` (with optional `reasoning`, `modalities`, `tooling`, and `runtime` sections) plus a normalized `ResolvedCapabilityProfile` derived from the card and conservative family defaults. This is the source of truth for prompt rendering, output parsing, tool-call handling, and the `/v1/models` `resolved_capabilities` field.
+- **Phase 2 thinking contract** — `enable_thinking`, `reasoning_effort`, and the dashboard thinking toggle are all driven by `supports_thinking_toggle`, so non-toggleable reasoning models behave correctly without leaking model-specific quirks into client code.
+- **Output parser selection** — model cards declare `output_parser` (`generic`, `gemma4`, `gpt_oss`, `deepseek_v32`, etc.), so reasoning markers are normalized into structured `reasoning_content` per family.
+- **Model store metadata pipeline** — capability resolution feeds `/v1/models` so dashboards and clients can discover thinking, multimodal, and tool support without hardcoding model lists.
+
+### API surface
+
+- **Claude Messages API** — `/v1/messages` adapter, including streaming, tool use, image inputs, and capability-aware thinking controls.
+- **Ollama compatibility** — both `/api/chat` and `/api/generate`, with adapter-side reasoning normalization.
+- **OpenAI Responses API** — `/v1/responses` adapter alongside chat completions.
+- **Embeddings endpoint** for non-chat workloads.
+- **Model store endpoints** — search, add, download, capability resolution, optimization jobs, and registry management, all exposed under stable URLs and documented in the OpenAPI spec.
+- **Cluster-wide config endpoints** — `GET`/`POST` config that gossipsubs to every node and writes back to `skulk.yaml`.
+- **Tracing, downloads, instance previews, and placement endpoints** — distributed-system observability and pre-launch placement inspection that upstream does not expose.
+
+### Dashboard
+
+- **React dashboard (default)** — replaces upstream's Svelte UI with a typed React + styled-components app that ships with the binary. The legacy Svelte dashboard is kept only as a fallback in the repo.
+- **Cluster topology view** with live device icons, GPU stats, network mesh visualization, and connection status banners.
+- **Placement preview / placement manager** for inspecting and choosing valid placements before launching.
+- **Model store browser** with HuggingFace search, family sidebar, model filters, capability badges, recent models, and per-model launch controls.
+- **Reasoning-aware chat UI** that splits inline `<think>` and Gemma `<|channel>` markers into a dedicated thinking panel and merges them with `reasoning_content` deltas from the API.
+- **Image attachments and multimodal chat affordances** for vision models.
+- **Cluster-wide settings panel** that writes to `skulk.yaml` and syncs across nodes via gossipsub.
+- **Light and dark themes** with first-class theme tokens, screenshots in both modes for documentation work.
+
+### Centralized logging and observability
+
+- **Structured JSON stdout** when `logging.enabled` is set, configurable from the dashboard Settings panel and synced cluster-wide.
+- **Vector + VictoriaLogs + Grafana stack** — local Vector log shipper on each node, central VictoriaLogs storage, ready-made Grafana dashboards. Stack definition lives in `deployment/logging/`.
+- **Distributed tracing** opt-in via `EXO_TRACING_ENABLED`.
+
+### Model store
+
+- **Centralized model store host** — one node downloads, the rest of the cluster stages over the LAN.
+- **Persistent registry** with capability resolution and download tracking.
+- **Custom model card support** — add your own model with `POST /models/add`.
+- **Image and embedding model cards** behind feature flags.
+- **Optimization job pipeline** for mlx-optiq mixed-precision weight quantization.
+
+### Cluster operation
+
+- **Cluster-wide settings sync** for KV cache backend, logging, model store host, HF token, and other inference toggles.
+- **Bootstrap peer config from `skulk.yaml`, env, or CLI** for fixed-topology clusters.
+- **Election (bully algorithm) + master/worker split** for indexing events and broadcasting state.
+- **`SKULK_*` environment variables** alongside the legacy `EXO_*` set, so new options can land without colliding with upstream.
+- **`skulk.yaml`** as the canonical config file, with `exo.yaml` kept for backwards compatibility.
+
+### Build, type system, and dev workflow
+
+- **Strict basedpyright** type checking — zero-error policy for new code.
+- **Ruff** linting and **`nix fmt`** formatting in CI.
+- **Nix flake** for reproducible toolchain setup.
+- **Docusaurus docs site** with auto-generated OpenAPI per-endpoint pages and TypeDoc HTML reference for the dashboard, both built from source.
+- **Pre-commit checklist** documented in `CLAUDE.md` and enforced in CI.
+
+### Hardware and platform
+
+- **Apple Silicon as the primary target**, including RDMA over Thunderbolt 5 on supported hardware and matched macOS versions.
+- **Linux supported** (CPU-oriented in this fork; GPU work happens on Apple Silicon).
+
 ## Prerequisites
 
 ### macOS
@@ -106,7 +185,9 @@ Important behavior:
 - **Placement previews**: inspect valid placements before launching a model.
 - **Thinking-aware chat UI**: chat with compatible models and surface reasoning content.
 - **Alternative API compatibility**: OpenAI Chat Completions, OpenAI Responses, Claude Messages, and Ollama.
-- **Experimental inference tuning**: OptiQ and other KV cache backends for long-context and memory experiments.
+- **Rotation-based KV cache backends**: RotorQuant (IsoQuant 3-bit + deferred prefill), OptiQ, TurboQuant, and TurboQuant Adaptive — pick per workload from the dashboard.
+- **Capability-driven thinking contract**: model cards declare reasoning support; the API and dashboard route accordingly.
+- **Experimental inference tuning**: long-context and memory experiments via the KV cache backends above.
 
 ## Dashboard
 
@@ -373,6 +454,8 @@ uv run exo --bootstrap-peers /ip4/192.168.1.20/tcp/5678/p2p/12D3KooW...
 | `EXO_NO_BATCH` | Force sequential generation | `false` |
 | `EXO_OPTIQ_BITS` | Bit width for `optiq` | `4` |
 | `EXO_OPTIQ_FP16_LAYERS` | Edge FP16 layers for `optiq` | `4` |
+| `SKULK_ROTORQUANT_FP16_LAYERS` | Edge FP16 layers for `rotorquant_adaptive` | `4` |
+| `SKULK_ROTORQUANT_DEFER_PREFILL` | Set to `0` to disable deferred prefill (debugging only) | `1` |
 | `EXO_BOOTSTRAP_PEERS` | Comma-separated static peers to dial on startup | None |
 | `HF_TOKEN` | Hugging Face token | None |
 
@@ -382,6 +465,7 @@ Examples:
 EXO_OFFLINE=true uv run exo
 EXO_ENABLE_IMAGE_MODELS=true uv run exo
 EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run exo
+SKULK_KV_CACHE_BACKEND=rotorquant_adaptive SKULK_ROTORQUANT_FP16_LAYERS=4 uv run exo
 ```
 
 ## RDMA on macOS
diff --git a/dashboard-react/src/components/layout/SettingsPanel.tsx b/dashboard-react/src/components/layout/SettingsPanel.tsx
index 6b080cf94..e177ffe7b 100644
--- a/dashboard-react/src/components/layout/SettingsPanel.tsx
+++ b/dashboard-react/src/components/layout/SettingsPanel.tsx
@@ -449,9 +449,11 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
                 filled
                 content={
                   `• Default — No cache quantization. Best baseline quality, highest memory use.\n` +
-                  `• OptiQ — Rotation-based quantization via mlx-optiq. Best long-context quality.\n` +
+                  `• RotorQuant Adaptive — IsoQuant 3-bit with deferred prefill, FP16 edge layers. Recommended.\n` +
+                  `• RotorQuant — IsoQuant 3-bit on all KV layers. Most aggressive compression with deferred prefill.\n` +
+                  `• OptiQ — Rotation-based quantization via mlx-optiq. Good long-context quality, no GQA support.\n` +
                   `• TurboQuant Adaptive — Quantizes middle KV layers, keeps edge layers in FP16. Proven stable.\n` +
-                  `• TurboQuant — Quantizes all KV layers. Most aggressive compression, higher quality risk.\n` +
+                  `• TurboQuant — Quantizes all KV layers. Most aggressive non-rotorquant compression.\n` +
                   `• MLX Quantized — MLX's built-in cache quantization.\n\n` +
                   `Takes effect on next model launch. Incompatible models fall back to Default automatically.`
                 }
@@ -459,6 +461,8 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
             </FieldLabel>
             <Select value={kvBackend} onChange={(e) => setKvBackend(e.target.value)} disabled={!!envOverride}>
               <option value="default">Default (no quantization)</option>
+              <option value="rotorquant_adaptive">RotorQuant Adaptive (recommended)</option>
+              <option value="rotorquant">RotorQuant</option>
               <option value="optiq">OptiQ (rotation-based)</option>
               <option value="turboquant_adaptive">TurboQuant Adaptive</option>
               <option value="turboquant">TurboQuant</option>
diff --git a/src/exo/store/config.py b/src/exo/store/config.py
index 54a778f78..7a4688395 100644
--- a/src/exo/store/config.py
+++ b/src/exo/store/config.py
@@ -235,7 +235,13 @@ class InferenceConfig(FrozenModel):
     """
 
     kv_cache_backend: Literal[
-        "default", "mlx_quantized", "turboquant", "turboquant_adaptive", "optiq"
+        "default",
+        "mlx_quantized",
+        "turboquant",
+        "turboquant_adaptive",
+        "optiq",
+        "rotorquant",
+        "rotorquant_adaptive",
     ] = "default"
 
 
diff --git a/src/exo/worker/engines/mlx/cache.py b/src/exo/worker/engines/mlx/cache.py
index 2911f81df..9edbf7bc9 100644
--- a/src/exo/worker/engines/mlx/cache.py
+++ b/src/exo/worker/engines/mlx/cache.py
@@ -23,11 +23,17 @@
     KV_CACHE_BITS,
     OPTIQ_BITS,
     OPTIQ_FP16_LAYERS,
+    ROTORQUANT_DEFER_PREFILL,
+    ROTORQUANT_FP16_LAYERS,
     TURBOQUANT_FP16_LAYERS,
     TURBOQUANT_K_BITS,
     TURBOQUANT_V_BITS,
     KVCacheBackend,
 )
+from exo.worker.engines.mlx.rotorquant import (
+    make_rotorquant_adaptive_cache,
+    make_rotorquant_cache_from_template,
+)
 from exo.worker.engines.mlx.turboquant import (
     make_turboquant_adaptive_cache,
     make_turboquant_cache_from_template,
@@ -578,6 +584,27 @@ def _is_power_of_two(n: int) -> bool:
             for i, _ in enumerate(model.layers)
         ]
 
+    if backend == "rotorquant":
+        logger.info(
+            f"Using rotorquant KV cache (defer_prefill={ROTORQUANT_DEFER_PREFILL})"
+        )
+        return make_rotorquant_cache_from_template(
+            model,
+            defer_prefill=ROTORQUANT_DEFER_PREFILL,
+        )
+
+    if backend == "rotorquant_adaptive":
+        logger.info(
+            f"Using rotorquant adaptive KV cache "
+            f"(fp16_layers={ROTORQUANT_FP16_LAYERS}, "
+            f"defer_prefill={ROTORQUANT_DEFER_PREFILL})"
+        )
+        return make_rotorquant_adaptive_cache(
+            model,
+            fp16_layers=ROTORQUANT_FP16_LAYERS,
+            defer_prefill=ROTORQUANT_DEFER_PREFILL,
+        )
+
     if hasattr(model, "make_cache"):
         logger.info("Using MLX LM's make cache")
         return model.make_cache()  # type: ignore
@@ -594,6 +621,8 @@ def get_kv_cache_backend() -> KVCacheBackend:
         "turboquant",
         "turboquant_adaptive",
         "optiq",
+        "rotorquant",
+        "rotorquant_adaptive",
     )
     if backend not in valid_backends:
         logger.warning(
diff --git a/src/exo/worker/engines/mlx/constants.py b/src/exo/worker/engines/mlx/constants.py
index 912897f94..5f8a788cc 100644
--- a/src/exo/worker/engines/mlx/constants.py
+++ b/src/exo/worker/engines/mlx/constants.py
@@ -21,6 +21,8 @@
     "turboquant",
     "turboquant_adaptive",
     "optiq",
+    "rotorquant",
+    "rotorquant_adaptive",
 ]
 DEFAULT_KV_CACHE_BACKEND: KVCacheBackend = "default"
 KV_CACHE_BACKEND: KVCacheBackend = cast(
@@ -45,6 +47,20 @@
 DEFAULT_TURBOQUANT_V_BITS: int = 4
 OPTIQ_BITS: int = int(os.environ.get("EXO_OPTIQ_BITS", "4"))
 OPTIQ_FP16_LAYERS: int = int(os.environ.get("EXO_OPTIQ_FP16_LAYERS", "4"))
+ROTORQUANT_FP16_LAYERS: int = int(
+    os.environ.get("SKULK_ROTORQUANT_FP16_LAYERS", os.environ.get("EXO_ROTORQUANT_FP16_LAYERS", "4"))
+)
+# Deferred prefill keeps K/V in fp16 during prompt processing and flushes
+# the buffer to compressed storage on the first decode token. It is the
+# load-bearing accuracy improvement of this backend; only disable for
+# debugging.
+ROTORQUANT_DEFER_PREFILL: bool = (
+    os.environ.get(
+        "SKULK_ROTORQUANT_DEFER_PREFILL",
+        os.environ.get("EXO_ROTORQUANT_DEFER_PREFILL", "1"),
+    )
+    not in ("0", "false", "False", "")
+)
 
 DEFAULT_TOP_LOGPROBS: int = 5
 
diff --git a/src/exo/worker/engines/mlx/rotorquant/__init__.py b/src/exo/worker/engines/mlx/rotorquant/__init__.py
new file mode 100644
index 000000000..9b81f5f46
--- /dev/null
+++ b/src/exo/worker/engines/mlx/rotorquant/__init__.py
@@ -0,0 +1,30 @@
+"""RotorQuant KV cache backend (IsoQuant variant).
+
+Pure-MLX port of the IsoQuant 3-bit KV cache compression from
+johndpope/llama-cpp-turboquant (MIT) and scrya-com/rotorquant (MIT).
+
+Key properties vs the older TurboQuant native backend:
+- Block-diagonal 4D quaternion rotations instead of randomized Hadamard
+- Norm-correction trick for unbiased magnitudes after centroid quantization
+- Optional deferred prefill: K/V stay in fp16 during prompt processing,
+  flushed to compressed storage on the first decode-shaped call
+- GQA-native (compression is per-(kv_head, token), Q heads fan out at SDPA)
+
+The backend stores indices and norms; ``update_and_fetch`` returns fully
+dequantized fp16 K/V to standard ``mx.fast.scaled_dot_product_attention``.
+The centroid-space attention optimization from OptiQ is intentionally
+deferred to a follow-up; v1 prioritizes correctness and the deferred-prefill
+quality win over the rotated-space SDPA perf win.
+"""
+
+from exo.worker.engines.mlx.rotorquant.cache import (
+    RotorQuantKVCache,
+    make_rotorquant_adaptive_cache,
+    make_rotorquant_cache_from_template,
+)
+
+__all__ = [
+    "RotorQuantKVCache",
+    "make_rotorquant_adaptive_cache",
+    "make_rotorquant_cache_from_template",
+]
diff --git a/src/exo/worker/engines/mlx/rotorquant/cache.py b/src/exo/worker/engines/mlx/rotorquant/cache.py
new file mode 100644
index 000000000..c8248f00f
--- /dev/null
+++ b/src/exo/worker/engines/mlx/rotorquant/cache.py
@@ -0,0 +1,545 @@
+"""RotorQuant KV cache class with optional deferred prefill.
+
+The deferred-prefill path is the load-bearing accuracy improvement that
+the upstream llama.cpp fork only ships on CUDA (``#ifdef GGML_USE_CUDA``
+in ``llama-context.cpp:1690``). On Metal/MLX it has not existed before
+this port.
+
+The idea: while the engine is processing a prompt (``update_and_fetch``
+called with ``num_steps > 1``), keep K and V in fp16. Quantization only
+happens once, on the first decode-shaped call (``num_steps == 1``).
+This avoids compounding centroid-roundtrip errors through every
+prefill attention chain and matches the published 5.3× prefill / PPL
+6.91 numbers from the llama.cpp fork.
+"""
+
+from collections.abc import Sequence
+from typing import cast
+
+import mlx.core as mx
+from mlx_lm.models.cache import (
+    ArraysCache,
+    KVCache,
+    RotatingKVCache,
+    create_attention_mask,
+)
+
+from exo.shared.types.mlx import KVCacheType, Model
+from exo.worker.engines.mlx.rotorquant.quantizer import IsoQuantizer
+from exo.worker.engines.mlx.rotorquant.tables import ISO3_BLOCK_SIZE
+
+
+class RotorQuantKVCache:
+    """Per-layer KV cache backed by IsoQuant 3-bit compression.
+
+    Storage layout (one slot per token per kv-head):
+      - ``key_indices``:   ``(B, kv_heads, T, head_dim)`` uint8
+      - ``key_norms``:     ``(B, kv_heads, T, head_dim // 128)`` fp16
+      - ``value_indices``: same shape as keys
+      - ``value_norms``:   same shape as key_norms
+
+    During the deferred-prefill window the class additionally holds
+    ``_pending_keys`` and ``_pending_values`` as fp16 buffers, and
+    ``update_and_fetch`` returns the concatenation of those buffers
+    directly without ever quantizing. The first call with
+    ``num_steps == 1`` flushes the pending buffers through the
+    quantizer in a single batched operation.
+    """
+
+    step = 256
+
+    def __init__(self, *, defer_prefill: bool = True, seed: int = 42) -> None:
+        # Deferred prefill is on by default because it is both an
+        # accuracy and a peak-memory-amortization win; tests can flip
+        # it off to exercise the always-quantized path.
+        self.defer_prefill = defer_prefill
+        self.seed = seed
+        self.offset = 0
+
+        self._quantizer: IsoQuantizer | None = None
+        self.head_dim: int | None = None
+
+        self._key_indices: mx.array | None = None
+        self._key_norms: mx.array | None = None
+        self._value_indices: mx.array | None = None
+        self._value_norms: mx.array | None = None
+
+        # Deferred prefill scratch buffers — kept in fp16 until the
+        # first decode-shaped call triggers ``_flush_deferred``.
+        self._pending_keys: mx.array | None = None
+        self._pending_values: mx.array | None = None
+        self._pending_active: bool = defer_prefill
+
+    # ------------------------------------------------------------------
+    # Initialization helpers
+    # ------------------------------------------------------------------
+
+    def _ensure_quantizer(self, head_dim: int) -> None:
+        if head_dim % ISO3_BLOCK_SIZE != 0:
+            raise ValueError(
+                f"RotorQuantKVCache requires head_dim divisible by "
+                f"{ISO3_BLOCK_SIZE}, got {head_dim}"
+            )
+        if self._quantizer is None:
+            self._quantizer = IsoQuantizer()
+            self.head_dim = head_dim
+
+    def _expand_storage(
+        self,
+        batch_size: int,
+        kv_heads: int,
+        steps: int,
+        head_dim: int,
+    ) -> None:
+        n_blocks = head_dim // ISO3_BLOCK_SIZE
+        needed = self.offset + steps
+        if self._key_indices is not None and needed <= self._key_indices.shape[2]:
+            return
+
+        new_steps = ((steps + self.step - 1) // self.step) * self.step
+        if self._key_indices is None:
+            shape_idx = (batch_size, kv_heads, new_steps, head_dim)
+            shape_norm = (batch_size, kv_heads, new_steps, n_blocks)
+            self._key_indices = mx.zeros(shape_idx, dtype=mx.uint8)
+            self._key_norms = mx.zeros(shape_norm, dtype=mx.float16)
+            self._value_indices = mx.zeros(shape_idx, dtype=mx.uint8)
+            self._value_norms = mx.zeros(shape_norm, dtype=mx.float16)
+            return
+
+        # Concatenate a fresh slab of zeros to the right of the live
+        # region. The live region ends at ``self.offset``; everything
+        # past that is stale capacity from a previous expansion.
+        assert self._key_norms is not None
+        assert self._value_indices is not None
+        assert self._value_norms is not None
+        zeros_idx = mx.zeros(
+            (batch_size, kv_heads, new_steps, head_dim),
+            dtype=mx.uint8,
+        )
+        zeros_norm = mx.zeros(
+            (batch_size, kv_heads, new_steps, n_blocks),
+            dtype=mx.float16,
+        )
+        self._key_indices = mx.concatenate(
+            [self._key_indices[..., : self.offset, :], zeros_idx],
+            axis=2,
+        )
+        self._key_norms = mx.concatenate(
+            [self._key_norms[..., : self.offset, :], zeros_norm],
+            axis=2,
+        )
+        self._value_indices = mx.concatenate(
+            [self._value_indices[..., : self.offset, :], zeros_idx],
+            axis=2,
+        )
+        self._value_norms = mx.concatenate(
+            [self._value_norms[..., : self.offset, :], zeros_norm],
+            axis=2,
+        )
+
+    # ------------------------------------------------------------------
+    # Update / fetch
+    # ------------------------------------------------------------------
+
+    def update_and_fetch(
+        self,
+        keys: mx.array,
+        values: mx.array,
+    ) -> tuple[mx.array, mx.array]:
+        """Insert ``keys``/``values`` and return the full live KV cache.
+
+        While deferred prefill is active, the call appends to the fp16
+        scratch buffers and returns the concatenated fp16 view directly.
+        Once the first decode-shaped call (``num_steps == 1``) arrives,
+        the scratch buffers are flushed through the quantizer in one
+        batched operation, after which all subsequent calls quantize on
+        insert and return a freshly dequantized view.
+        """
+        _, _, num_steps, head_dim = keys.shape
+        self._ensure_quantizer(head_dim)
+
+        # Deferred-prefill path: keep raw fp16 until the first decode token.
+        # On a multi-token call we append to the scratch and bail out
+        # immediately. On a single-token call we flush whatever we have
+        # accumulated, then fall through to the normal quantized insert
+        # path so the new decode token is stored alongside the rest.
+        if self._pending_active:
+            if num_steps > 1:
+                self._append_pending(keys, values)
+                assert self._pending_keys is not None
+                assert self._pending_values is not None
+                return self._pending_keys, self._pending_values
+            self._flush_deferred(keys.dtype)
+
+        if num_steps > 0:
+            self._quantize_and_store(keys, values)
+
+        return self._dequantize_live(keys.dtype)
+
+    def _append_pending(self, keys: mx.array, values: mx.array) -> None:
+        """Append new K/V to the deferred-prefill scratch buffers."""
+        if self._pending_keys is None:
+            self._pending_keys = keys.astype(mx.float16)
+            self._pending_values = values.astype(mx.float16)
+            return
+        assert self._pending_values is not None
+        self._pending_keys = mx.concatenate(
+            [self._pending_keys, keys.astype(mx.float16)],
+            axis=2,
+        )
+        self._pending_values = mx.concatenate(
+            [self._pending_values, values.astype(mx.float16)],
+            axis=2,
+        )
+
+    def _flush_deferred(self, dtype: mx.Dtype) -> None:
+        """Quantize the deferred prefill buffer and free the fp16 scratch."""
+        del dtype  # the live cache stores indices regardless of caller dtype
+        assert self._pending_keys is not None
+        assert self._pending_values is not None
+        assert self._quantizer is not None
+
+        pending_keys = self._pending_keys
+        pending_values = self._pending_values
+        batch_size, kv_heads, num_pending, head_dim = pending_keys.shape
+
+        self._expand_storage(batch_size, kv_heads, num_pending, head_dim)
+        assert self._key_indices is not None
+        assert self._key_norms is not None
+        assert self._value_indices is not None
+        assert self._value_norms is not None
+
+        key_indices, key_norms = self._quantizer.quantize(pending_keys)
+        value_indices, value_norms = self._quantizer.quantize(pending_values)
+
+        end = self.offset + num_pending
+        self._key_indices[..., self.offset : end, :] = key_indices
+        self._key_norms[..., self.offset : end, :] = key_norms
+        self._value_indices[..., self.offset : end, :] = value_indices
+        self._value_norms[..., self.offset : end, :] = value_norms
+        self.offset = end
+
+        # Free the scratch — from now on we are in the steady-state
+        # decode path and quantize on insert.
+        self._pending_keys = None
+        self._pending_values = None
+        self._pending_active = False
+
+    def _quantize_and_store(self, keys: mx.array, values: mx.array) -> None:
+        """Quantize freshly arrived tokens and append to the live cache."""
+        assert self._quantizer is not None
+        batch_size, kv_heads, num_steps, head_dim = keys.shape
+
+        self._expand_storage(batch_size, kv_heads, num_steps, head_dim)
+        assert self._key_indices is not None
+        assert self._key_norms is not None
+        assert self._value_indices is not None
+        assert self._value_norms is not None
+
+        key_indices, key_norms = self._quantizer.quantize(keys)
+        value_indices, value_norms = self._quantizer.quantize(values)
+
+        end = self.offset + num_steps
+        self._key_indices[..., self.offset : end, :] = key_indices
+        self._key_norms[..., self.offset : end, :] = key_norms
+        self._value_indices[..., self.offset : end, :] = value_indices
+        self._value_norms[..., self.offset : end, :] = value_norms
+        self.offset = end
+
+    def _dequantize_live(
+        self, output_dtype: mx.Dtype
+    ) -> tuple[mx.array, mx.array]:
+        """Materialize the full live cache as fp16 K/V for SDPA."""
+        assert self._quantizer is not None
+        assert self._key_indices is not None
+        assert self._key_norms is not None
+        assert self._value_indices is not None
+        assert self._value_norms is not None
+
+        keys_out = self._quantizer.dequantize(
+            self._key_indices[..., : self.offset, :],
+            self._key_norms[..., : self.offset, :],
+            output_dtype,
+        )
+        values_out = self._quantizer.dequantize(
+            self._value_indices[..., : self.offset, :],
+            self._value_norms[..., : self.offset, :],
+            output_dtype,
+        )
+        return keys_out, values_out
+
+    # ------------------------------------------------------------------
+    # mlx-lm cache protocol
+    # ------------------------------------------------------------------
+
+    def size(self) -> int:
+        if self._pending_active and self._pending_keys is not None:
+            return int(self._pending_keys.shape[2])
+        return self.offset
+
+    @property
+    def state(self) -> object:
+        if self._pending_active:
+            # While deferred, expose only the fp16 scratch so that any
+            # external snapshot/restore round-trips correctly without
+            # corrupting the offset of the not-yet-allocated live cache.
+            if self._pending_keys is None:
+                return []
+            assert self._pending_values is not None
+            return ("pending", self._pending_keys, self._pending_values)
+        if self._key_indices is None:
+            return []
+        assert self._key_norms is not None
+        assert self._value_indices is not None
+        assert self._value_norms is not None
+        return (
+            "live",
+            self._key_indices[..., : self.offset, :],
+            self._key_norms[..., : self.offset, :],
+            self._value_indices[..., : self.offset, :],
+            self._value_norms[..., : self.offset, :],
+        )
+
+    @state.setter
+    def state(self, value: object) -> None:
+        seq = cast(Sequence[object], value)
+        if len(seq) == 0:
+            return
+        tag = seq[0]
+        if tag == "pending":
+            self._pending_keys = cast(mx.array, seq[1])
+            self._pending_values = cast(mx.array, seq[2])
+            self._pending_active = True
+            return
+        if tag == "live":
+            self._key_indices = cast(mx.array, seq[1])
+            self._key_norms = cast(mx.array, seq[2])
+            self._value_indices = cast(mx.array, seq[3])
+            self._value_norms = cast(mx.array, seq[4])
+            self.offset = int(self._key_indices.shape[2])
+            self._pending_active = False
+
+    @property
+    def meta_state(self) -> object:
+        head_dim = -1 if self.head_dim is None else self.head_dim
+        return (
+            str(self.offset),
+            str(self.seed),
+            str(int(self.defer_prefill)),
+            str(int(self._pending_active)),
+            str(head_dim),
+        )
+
+    @meta_state.setter
+    def meta_state(self, value: object) -> None:
+        offset, seed, defer, pending, head_dim = (
+            int(part) for part in cast(Sequence[str], value)
+        )
+        self.offset = offset
+        self.seed = seed
+        self.defer_prefill = bool(defer)
+        self._pending_active = bool(pending)
+        self.head_dim = None if head_dim < 0 else head_dim
+        if self.head_dim is not None and self._quantizer is None:
+            self._quantizer = IsoQuantizer()
+
+    def is_trimmable(self) -> bool:
+        return True
+
+    def trim(self, n: int) -> int:
+        if self._pending_active and self._pending_keys is not None:
+            assert self._pending_values is not None
+            available = int(self._pending_keys.shape[2])
+            trimmed = min(available, n)
+            new_len = available - trimmed
+            self._pending_keys = self._pending_keys[..., :new_len, :]
+            self._pending_values = self._pending_values[..., :new_len, :]
+            return trimmed
+        trimmed = min(self.offset, n)
+        self.offset -= trimmed
+        return trimmed
+
+    def make_mask(self, *args: object, **kwargs: object) -> object:
+        if len(args) == 0 or not isinstance(args[0], int):
+            return None
+        window_size = kwargs.get("window_size")
+        return_array = kwargs.get("return_array", False)
+        if not isinstance(window_size, (int, type(None))):
+            return None
+        if not isinstance(return_array, bool):
+            return None
+        return create_attention_mask(
+            args[0],
+            offset=self.size(),
+            window_size=window_size,
+            return_array=return_array,
+        )
+
+    @property
+    def is_deferred(self) -> bool:
+        """Whether the cache is still in the deferred-prefill phase.
+
+        Public so tests and observability code can ask without poking
+        at private attributes.
+        """
+        return self._pending_active
+
+    @property
+    def has_live_storage(self) -> bool:
+        """Whether the quantized live cache has been allocated yet."""
+        return self._key_indices is not None
+
+    @property
+    def has_pending_storage(self) -> bool:
+        """Whether the deferred-prefill scratch buffer is currently live."""
+        return self._pending_keys is not None
+
+    def empty(self) -> bool:
+        if self._pending_active:
+            return self._pending_keys is None
+        return self._key_indices is None
+
+    @property
+    def nbytes(self) -> int:
+        total = 0
+        if self._key_indices is not None:
+            assert self._key_norms is not None
+            assert self._value_indices is not None
+            assert self._value_norms is not None
+            total += int(self._key_indices.nbytes)
+            total += int(self._key_norms.nbytes)
+            total += int(self._value_indices.nbytes)
+            total += int(self._value_norms.nbytes)
+        if self._pending_keys is not None:
+            assert self._pending_values is not None
+            total += int(self._pending_keys.nbytes)
+            total += int(self._pending_values.nbytes)
+        return total
+
+
+# ----------------------------------------------------------------------
+# Compatibility check + factory functions (mirrors turboquant/cache.py)
+# ----------------------------------------------------------------------
+
+
+def ensure_rotorquant_compatible(model: Model) -> None:
+    """Validate that the model uses cache types RotorQuant can wrap.
+
+    RotorQuant can only replace plain ``KVCache`` entries; ``ArraysCache``
+    (SSM) and ``RotatingKVCache`` (sliding window) entries are passed
+    through to mlx-lm unchanged. Anything else is rejected so we never
+    silently corrupt a model with custom cache layouts.
+    """
+    if not hasattr(model, "make_cache"):
+        return
+
+    sample_cache = cast(
+        list[object],
+        model.make_cache(),  # type: ignore[reportUnknownMemberType]
+    )
+    if len(sample_cache) == 0:
+        return
+    if not all(
+        isinstance(entry, (KVCache, ArraysCache, RotatingKVCache))
+        for entry in sample_cache
+    ):
+        cache_types = ", ".join(
+            sorted({type(entry).__name__ for entry in sample_cache})
+        )
+        raise ValueError(
+            "RotorQuant backend currently supports KVCache entries plus optional "
+            "ArraysCache and RotatingKVCache passthrough; "
+            f"found cache type(s): {cache_types}"
+        )
+
+
+def make_rotorquant_cache_from_template(
+    model: Model,
+    *,
+    defer_prefill: bool = True,
+    seed: int = 42,
+) -> KVCacheType:
+    """Build one RotorQuantKVCache per attention layer.
+
+    Mirrors :func:`make_turboquant_cache_from_template`. Non-KV entries
+    such as Mamba SSM caches are preserved unchanged.
+    """
+    ensure_rotorquant_compatible(model)
+    if not hasattr(model, "make_cache"):
+        return [
+            RotorQuantKVCache(defer_prefill=defer_prefill, seed=seed + index)
+            for index, _layer in enumerate(model.layers)
+        ]
+
+    template_cache = cast(
+        list[object],
+        model.make_cache(),  # type: ignore[reportUnknownMemberType]
+    )
+    caches: list[object] = []
+    kv_index = 0
+    for entry in template_cache:
+        if isinstance(entry, KVCache):
+            caches.append(
+                RotorQuantKVCache(defer_prefill=defer_prefill, seed=seed + kv_index)
+            )
+            kv_index += 1
+        else:
+            caches.append(entry)
+    return caches
+
+
+def make_rotorquant_adaptive_cache(
+    model: Model,
+    *,
+    fp16_layers: int,
+    defer_prefill: bool = True,
+    seed: int = 42,
+) -> KVCacheType:
+    """Build a RotorQuant cache with FP16 protection on edge layers.
+
+    The first ``fp16_layers`` and last ``fp16_layers`` attention layers
+    use plain mlx-lm ``KVCache`` instances; everything in between uses
+    :class:`RotorQuantKVCache`. Mirrors
+    :func:`make_turboquant_adaptive_cache` exactly.
+    """
+    ensure_rotorquant_compatible(model)
+    if not hasattr(model, "make_cache"):
+        caches: list[object] = []
+        for index, _layer in enumerate(model.layers):
+            if index < fp16_layers or index >= len(model.layers) - fp16_layers:
+                caches.append(KVCache())
+            else:
+                caches.append(
+                    RotorQuantKVCache(
+                        defer_prefill=defer_prefill,
+                        seed=seed + index,
+                    )
+                )
+        return caches
+
+    template_cache = cast(
+        list[object],
+        model.make_cache(),  # type: ignore[reportUnknownMemberType]
+    )
+    kv_positions = [
+        index
+        for index, entry in enumerate(template_cache)
+        if isinstance(entry, KVCache)
+    ]
+    caches = []
+    for index, entry in enumerate(template_cache):
+        if not isinstance(entry, KVCache):
+            caches.append(entry)
+            continue
+
+        kv_order = kv_positions.index(index)
+        if kv_order < fp16_layers or kv_order >= len(kv_positions) - fp16_layers:
+            caches.append(KVCache())
+        else:
+            caches.append(
+                RotorQuantKVCache(
+                    defer_prefill=defer_prefill,
+                    seed=seed + kv_order,
+                )
+            )
+    return caches
diff --git a/src/exo/worker/engines/mlx/rotorquant/quantizer.py b/src/exo/worker/engines/mlx/rotorquant/quantizer.py
new file mode 100644
index 000000000..5a22310ef
--- /dev/null
+++ b/src/exo/worker/engines/mlx/rotorquant/quantizer.py
@@ -0,0 +1,146 @@
+"""3-bit IsoQuant quantizer with norm correction.
+
+Mirrors the quantize/dequantize logic in
+``johndpope/llama-cpp-turboquant/ggml/src/ggml-iso-quant.c`` but in pure
+MLX. Storage is one fp16 norm and one uint8 index per element per
+``(B, H, T)`` block — bit-packing into the 50-byte ``block_iso3_0``
+layout is a future-work optimization that does not affect correctness.
+
+The norm-correction trick (stored norm = ``||x|| / sqrt(Σ centroid²)``)
+is critical and not mentioned in the README of either upstream project.
+It absorbs the L2 shrinkage induced by centroid quantization so that
+attention's inner products remain unbiased.
+"""
+
+import mlx.core as mx
+
+from exo.worker.engines.mlx.rotorquant.rotation import (
+    iso3_rotate_forward,
+    iso3_rotate_inverse,
+)
+from exo.worker.engines.mlx.rotorquant.tables import (
+    ISO3_BLOCK_SIZE,
+    centroid_table,
+)
+
+_NORM_EPS = 1e-10
+
+
+class IsoQuantizer:
+    """Stateless 3-bit quantizer for ``head_dim``-multiple-of-128 vectors.
+
+    Holds the precomputed centroid table and centroid-boundary midpoints
+    on the active MLX device. The quantizer itself carries no per-token
+    state; the cache class owns the actual storage tensors.
+    """
+
+    def __init__(self) -> None:
+        self._centroids = centroid_table()
+        # Boundary midpoints between adjacent centroids: a value falls
+        # into bin ``i`` iff it lies between boundary i-1 and i. With 8
+        # centroids there are 7 internal boundaries.
+        boundaries = (self._centroids[:-1] + self._centroids[1:]) / 2.0
+        self._boundary_values: list[float] = boundaries.tolist()  # type: ignore[reportAny]
+
+    @property
+    def centroids(self) -> mx.array:
+        """Return the centroid table as an fp32 array of shape ``(8,)``."""
+        return self._centroids
+
+    def quantize(self, x: mx.array) -> tuple[mx.array, mx.array]:
+        """Quantize a tensor whose last dim is a multiple of the block size.
+
+        Args:
+            x: ``(..., d)`` tensor where ``d`` is a multiple of 128.
+
+        Returns:
+            A pair ``(indices, norms)``:
+              - ``indices``: ``(..., d)`` uint8 in ``[0, 7]``
+              - ``norms``: ``(..., d // 128)`` fp16 corrected norms,
+                one per iso3 block.
+        """
+        if x.shape[-1] % ISO3_BLOCK_SIZE != 0:
+            raise ValueError(
+                f"IsoQuantizer requires last dim divisible by {ISO3_BLOCK_SIZE}, "
+                f"got {x.shape[-1]}"
+            )
+
+        x32 = x.astype(mx.float32)
+        leading = x32.shape[:-1]
+        n_blocks = x32.shape[-1] // ISO3_BLOCK_SIZE
+
+        # Reshape so each iso3 block of 128 elements becomes its own row
+        # along a new axis. The norm and rotation are then per-block.
+        blocks = x32.reshape(*leading, n_blocks, ISO3_BLOCK_SIZE)
+
+        # Per-block L2 norm — matches grp_norm in the C reference.
+        norm_sq = mx.sum(blocks * blocks, axis=-1, keepdims=True)
+        grp_norm = mx.sqrt(norm_sq)
+        safe_norm = mx.maximum(grp_norm, mx.array(_NORM_EPS, dtype=mx.float32))
+        unit = blocks / safe_norm
+
+        # Forward quaternion rotation in the unit sphere.
+        rotated = iso3_rotate_forward(unit)
+
+        # Find nearest centroid index per coordinate. Walks the boundary
+        # list once with a running ``where`` accumulator. Cheap because
+        # there are only 7 boundaries for 8 centroids.
+        indices = mx.zeros(rotated.shape, dtype=mx.uint8)
+        for i, boundary in enumerate(self._boundary_values):
+            indices = indices + (rotated > boundary).astype(mx.uint8)
+            del i  # boundary index unused; the cumulative count is what matters
+
+        # Norm correction: absorb the L2 shrinkage of the quantized unit
+        # vector into the stored norm so dequantization yields an
+        # unbiased magnitude. ``recon_sq = Σ centroid[idx]²`` exactly
+        # matches the C reference path.
+        centroid_values = self._centroids[indices.astype(mx.int32)]
+        recon_sq = mx.sum(centroid_values * centroid_values, axis=-1, keepdims=True)
+        recon_norm = mx.sqrt(recon_sq)
+        safe_recon = mx.maximum(recon_norm, mx.array(_NORM_EPS, dtype=mx.float32))
+        corrected_norm = grp_norm / safe_recon
+
+        # Pack indices back to (..., d) and norms to (..., n_blocks).
+        indices_flat = indices.reshape(*leading, n_blocks * ISO3_BLOCK_SIZE)
+        norms_flat = corrected_norm.reshape(*leading, n_blocks).astype(mx.float16)
+        return indices_flat, norms_flat
+
+    def dequantize(
+        self,
+        indices: mx.array,
+        norms: mx.array,
+        output_dtype: mx.Dtype,
+    ) -> mx.array:
+        """Inverse of :meth:`quantize`.
+
+        Args:
+            indices: ``(..., d)`` uint8 indices, ``d`` a multiple of 128.
+            norms: ``(..., d // 128)`` corrected fp16 norms.
+            output_dtype: target dtype of the reconstructed tensor.
+
+        Returns:
+            ``(..., d)`` reconstructed tensor in ``output_dtype``.
+        """
+        if indices.shape[-1] % ISO3_BLOCK_SIZE != 0:
+            raise ValueError(
+                f"IsoQuantizer.dequantize requires indices last dim divisible by "
+                f"{ISO3_BLOCK_SIZE}, got {indices.shape[-1]}"
+            )
+
+        leading = indices.shape[:-1]
+        n_blocks = indices.shape[-1] // ISO3_BLOCK_SIZE
+
+        # Look up centroid values for every index, then reshape to
+        # per-block layout for the inverse rotation.
+        centroid_values = self._centroids[indices.astype(mx.int32)]
+        blocks = centroid_values.reshape(*leading, n_blocks, ISO3_BLOCK_SIZE)
+
+        rotated_back = iso3_rotate_inverse(blocks)
+
+        # Multiply each block by its stored corrected norm. The expand
+        # broadcasts ``(..., n_blocks)`` against the trailing block dim.
+        norms_f32 = norms.astype(mx.float32)[..., None]
+        scaled = rotated_back * norms_f32
+
+        flat = scaled.reshape(*leading, n_blocks * ISO3_BLOCK_SIZE)
+        return flat.astype(output_dtype)
diff --git a/src/exo/worker/engines/mlx/rotorquant/rotation.py b/src/exo/worker/engines/mlx/rotorquant/rotation.py
new file mode 100644
index 000000000..4fa98ab4d
--- /dev/null
+++ b/src/exo/worker/engines/mlx/rotorquant/rotation.py
@@ -0,0 +1,104 @@
+"""IsoQuant block-diagonal quaternion rotation in pure MLX.
+
+Each 128-element vector is split into 32 groups of 4. Each group is
+rotated by a fixed unit quaternion via the Hamilton product
+``q_L * v`` (treating ``v`` as a pure quaternion). The inverse uses
+``conj(q_L) * v`` because the rotation quaternion is unit.
+
+This is mathematically equivalent to a block-diagonal SO(4) rotation
+with 3 degrees of freedom per block, costing 16 multiplies + 12 adds
+per group versus the ``O(d log d)`` randomized Hadamard used by the
+older native TurboQuant backend.
+"""
+
+import mlx.core as mx
+
+from exo.worker.engines.mlx.rotorquant.tables import (
+    ISO3_BLOCK_SIZE,
+    ISO3_GROUPS_PER_BLOCK,
+    quaternion_table,
+)
+
+_qw, _qx, _qy, _qz = quaternion_table()
+
+
+def _quat_mul(
+    aw: mx.array,
+    ax: mx.array,
+    ay: mx.array,
+    az: mx.array,
+    bw: mx.array,
+    bx: mx.array,
+    by: mx.array,
+    bz: mx.array,
+) -> tuple[mx.array, mx.array, mx.array, mx.array]:
+    """Hamilton product ``a * b`` of two quaternions, broadcast over groups.
+
+    Mirrors the ``quat_mul`` helper in ``ggml-iso-quant.c`` exactly.
+    All inputs broadcast against each other along the trailing group axis.
+    """
+    rw = aw * bw - ax * bx - ay * by - az * bz
+    rx = aw * bx + ax * bw + ay * bz - az * by
+    ry = aw * by - ax * bz + ay * bw + az * bx
+    rz = aw * bz + ax * by - ay * bx + az * bw
+    return rw, rx, ry, rz
+
+
+def iso3_rotate_forward(unit: mx.array) -> mx.array:
+    """Apply the per-group quaternion rotation to a unit-normalized tensor.
+
+    Args:
+        unit: ``(..., 128)`` fp32 tensor with unit-norm leading slices.
+
+    Returns:
+        ``(..., 128)`` fp32 tensor in the rotated coordinate frame.
+    """
+    if unit.shape[-1] != ISO3_BLOCK_SIZE:
+        raise ValueError(
+            f"iso3_rotate_forward expects last dim {ISO3_BLOCK_SIZE}, "
+            f"got {unit.shape[-1]}"
+        )
+
+    # Reshape into (..., groups, 4) so the four quaternion components map
+    # cleanly onto the trailing axis. The quaternion tables broadcast as
+    # (groups,) → (1, ..., groups) against the leading dims.
+    grouped = unit.reshape(*unit.shape[:-1], ISO3_GROUPS_PER_BLOCK, 4)
+    v0 = grouped[..., 0]
+    v1 = grouped[..., 1]
+    v2 = grouped[..., 2]
+    v3 = grouped[..., 3]
+
+    rw, rx, ry, rz = _quat_mul(_qw, _qx, _qy, _qz, v0, v1, v2, v3)
+
+    rotated = mx.stack([rw, rx, ry, rz], axis=-1)
+    return rotated.reshape(*unit.shape)
+
+
+def iso3_rotate_inverse(rotated: mx.array) -> mx.array:
+    """Inverse of :func:`iso3_rotate_forward`.
+
+    Uses the conjugate quaternion ``(qw, -qx, -qy, -qz)`` because the
+    rotation quaternion is unit.
+
+    Args:
+        rotated: ``(..., 128)`` fp32 tensor in the rotated frame.
+
+    Returns:
+        ``(..., 128)`` fp32 tensor back in the original frame.
+    """
+    if rotated.shape[-1] != ISO3_BLOCK_SIZE:
+        raise ValueError(
+            f"iso3_rotate_inverse expects last dim {ISO3_BLOCK_SIZE}, "
+            f"got {rotated.shape[-1]}"
+        )
+
+    grouped = rotated.reshape(*rotated.shape[:-1], ISO3_GROUPS_PER_BLOCK, 4)
+    q0 = grouped[..., 0]
+    q1 = grouped[..., 1]
+    q2 = grouped[..., 2]
+    q3 = grouped[..., 3]
+
+    rw, rx, ry, rz = _quat_mul(_qw, -_qx, -_qy, -_qz, q0, q1, q2, q3)
+
+    restored = mx.stack([rw, rx, ry, rz], axis=-1)
+    return restored.reshape(*rotated.shape)
diff --git a/src/exo/worker/engines/mlx/rotorquant/tables.py b/src/exo/worker/engines/mlx/rotorquant/tables.py
new file mode 100644
index 000000000..6202ea70b
--- /dev/null
+++ b/src/exo/worker/engines/mlx/rotorquant/tables.py
@@ -0,0 +1,102 @@
+"""Vendored constants for IsoQuant 3-bit KV cache compression.
+
+Tables are lifted verbatim from johndpope/llama-cpp-turboquant
+(``ggml/src/ggml-iso-quant.c``, MIT-licensed). Keeping them bit-exact
+against the upstream C source means our quantizer agrees with the
+llama.cpp reference modulo floating-point ordering.
+
+The 32 unit quaternions parameterize one Hamilton-product rotation per
+4D group; for a 128-dimensional head this is 32 groups × 4 dims. The
+8-entry centroid table is the Lloyd-Max optimum for the post-rotation
+N(0, 1/d)-like coordinate distribution.
+"""
+
+from typing import Final
+
+import mlx.core as mx
+
+#: Native block size used by the iso3 layout. Fits exactly one Llama/Qwen
+#: head dimension. Heads with other sizes must be a multiple of this and
+#: are processed as multiple iso3 blocks.
+ISO3_BLOCK_SIZE: Final[int] = 128
+
+#: Number of quaternion groups per ISO3 block (block_size / 4).
+ISO3_GROUPS_PER_BLOCK: Final[int] = ISO3_BLOCK_SIZE // 4
+
+#: Bits per element after compression (3-bit indices via 8-entry codebook).
+ISO3_BITS: Final[int] = 3
+
+# Hardcoded unit quaternion components, one per 4D group (32 groups).
+# Sourced verbatim from ggml-iso-quant.c lines 51-54 in
+# johndpope/llama-cpp-turboquant feature/planarquant-kv-cache.
+# fmt: off
+_ISO3_QW: Final[tuple[float, ...]] = (
+    0.5765609741, 0.3176580369, -0.3234235942, -0.5127438903,
+    0.9233905673, -0.3323571086, 0.5468608141, -0.2500519454,
+    -0.5812215805, 0.3228830695, -0.7299832702, -0.4535493255,
+    -0.7338157296, -0.2884652913, -0.9000198841, -0.0377033800,
+    0.5104404092, 0.2033989877, -0.2462528497, 0.2314069420,
+    0.0072374810, 0.3923372924, 0.4958070219, -0.7235037088,
+    -0.9383618832, 0.4430379272, -0.2075705230, 0.1983736306,
+    -0.8834578991, 0.7389573455, -0.0156172011, 0.7738668919,
+)
+
+_ISO3_QX: Final[tuple[float, ...]] = (
+    0.4450169504, -0.5780548453, 0.7089627385, -0.3940812945,
+    -0.0897334740, 0.4727236331, 0.5542563796, 0.0450818054,
+    -0.3657043576, -0.4298477769, 0.4666220546, 0.7556306720,
+    -0.5284956098, 0.7042509317, 0.0230921544, 0.7110687494,
+    0.3024962246, -0.1157865301, 0.7490812540, -0.2582575679,
+    -0.2255804837, 0.3838746250, -0.3209520578, -0.3477301002,
+    0.1824720055, 0.4032751918, 0.8433781862, 0.9533935785,
+    -0.0620501526, 0.0927560627, 0.2964956462, 0.2402082384,
+)
+
+_ISO3_QY: Final[tuple[float, ...]] = (
+    0.2695076466, -0.0201656222, -0.1687686443, -0.5415957570,
+    -0.2796611190, 0.3510629535, 0.2609911859, -0.2715902030,
+    -0.0937586129, 0.3095585108, -0.4123268127, -0.4394895136,
+    0.0626545250, -0.4811822474, -0.0407132693, -0.4566248953,
+    0.7834537029, -0.6187923551, 0.0809760988, -0.8879503012,
+    -0.8928058147, 0.8350352049, -0.6994170547, 0.5606835485,
+    0.2933705449, 0.7377059460, 0.4534837306, -0.0009816211,
+    -0.3632916510, -0.3959124386, 0.1631654203, 0.5088164806,
+)
+
+_ISO3_QZ: Final[tuple[float, ...]] = (
+    -0.6300023794, -0.7513582706, -0.6035611629, 0.5370919704,
+    0.2471584976, 0.7367672324, 0.5706370473, 0.9282674193,
+    0.7208684087, -0.7843156457, -0.2817355990, -0.1736787707,
+    0.4222335219, -0.4350655377, 0.4333281815, 0.5333415866,
+    0.1847889870, 0.7498788238, 0.6096553802, -0.3021556735,
+    -0.3898189068, 0.0377884321, 0.4024685621, 0.2031257302,
+    0.0107116764, -0.3112498820, 0.1999502629, -0.2273492515,
+    0.2892593443, 0.5372074246, 0.9408631325, 0.2907505929,
+)
+
+#: 3-bit Lloyd-Max centroids on the post-rotation coordinate distribution.
+#: Sourced verbatim from ISO_CENTROIDS_3BIT in ggml-iso-quant.c.
+_ISO3_CENTROIDS: Final[tuple[float, ...]] = (
+    -0.1906850000, -0.1178320000, -0.0657170000, -0.0214600000,
+    0.0214600000, 0.0657170000, 0.1178320000, 0.1906850000,
+)
+# fmt: on
+
+
+def quaternion_table() -> tuple[mx.array, mx.array, mx.array, mx.array]:
+    """Return the (qw, qx, qy, qz) quaternion-component arrays as fp32.
+
+    Each array has shape ``(ISO3_GROUPS_PER_BLOCK,)`` and the four arrays
+    together describe one unit quaternion per 4D rotation group.
+    """
+    return (
+        mx.array(_ISO3_QW, dtype=mx.float32),
+        mx.array(_ISO3_QX, dtype=mx.float32),
+        mx.array(_ISO3_QY, dtype=mx.float32),
+        mx.array(_ISO3_QZ, dtype=mx.float32),
+    )
+
+
+def centroid_table() -> mx.array:
+    """Return the 8-entry 3-bit centroid array as fp32."""
+    return mx.array(_ISO3_CENTROIDS, dtype=mx.float32)
diff --git a/src/exo/worker/engines/mlx/rotorquant/tests/__init__.py b/src/exo/worker/engines/mlx/rotorquant/tests/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py b/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
new file mode 100644
index 000000000..416d6f67c
--- /dev/null
+++ b/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
@@ -0,0 +1,120 @@
+"""Tests for ``RotorQuantKVCache`` including the deferred-prefill path."""
+
+import mlx.core as mx
+import pytest
+
+from exo.worker.engines.mlx.rotorquant.cache import RotorQuantKVCache
+from exo.worker.engines.mlx.rotorquant.tables import ISO3_BLOCK_SIZE
+
+
+def _kv_chunk(num_steps: int, *, seed: int = 0) -> tuple[mx.array, mx.array]:
+    """Build a (B, H, T, head_dim) prefill chunk for tests."""
+    keys = mx.random.normal(
+        shape=(1, 4, num_steps, ISO3_BLOCK_SIZE),
+        key=mx.random.key(seed),
+    )
+    values = mx.random.normal(
+        shape=(1, 4, num_steps, ISO3_BLOCK_SIZE),
+        key=mx.random.key(seed + 1),
+    )
+    return keys, values
+
+
+def test_deferred_prefill_keeps_storage_unallocated_during_prefill() -> None:
+    """Multi-token calls should not allocate the quantized live cache."""
+    cache = RotorQuantKVCache(defer_prefill=True)
+    keys, values = _kv_chunk(64)
+    cache.update_and_fetch(keys, values)
+
+    assert cache.is_deferred is True
+    assert cache.has_live_storage is False
+    assert cache.offset == 0
+    assert cache.size() == 64
+
+
+def test_deferred_prefill_flushes_on_first_decode_call() -> None:
+    """The first ``num_steps == 1`` call should flush the deferred buffer."""
+    cache = RotorQuantKVCache(defer_prefill=True)
+    pre_k, pre_v = _kv_chunk(48, seed=10)
+    cache.update_and_fetch(pre_k, pre_v)
+    assert cache.is_deferred is True
+
+    dec_k, dec_v = _kv_chunk(1, seed=11)
+    out_k, out_v = cache.update_and_fetch(dec_k, dec_v)
+
+    assert cache.is_deferred is False
+    assert cache.has_pending_storage is False
+    assert cache.has_live_storage is True
+    assert cache.offset == 49  # 48 prefill + 1 decode
+    assert out_k.shape == (1, 4, 49, ISO3_BLOCK_SIZE)
+    assert out_v.shape == (1, 4, 49, ISO3_BLOCK_SIZE)
+    assert out_k.dtype == dec_k.dtype
+
+
+def test_deferred_prefill_returns_raw_fp16_during_prefill() -> None:
+    """The fp16 view returned during prefill should be lossless."""
+    cache = RotorQuantKVCache(defer_prefill=True)
+    keys, values = _kv_chunk(32, seed=20)
+    out_k, out_v = cache.update_and_fetch(keys, values)
+
+    # Equality with the input within fp16 round-tripping tolerance.
+    diff_k = mx.max(mx.abs(out_k.astype(mx.float32) - keys.astype(mx.float32))).item()
+    diff_v = mx.max(mx.abs(out_v.astype(mx.float32) - values.astype(mx.float32))).item()
+    assert diff_k < 1e-3
+    assert diff_v < 1e-3
+
+
+def test_non_deferred_path_quantizes_immediately() -> None:
+    """With ``defer_prefill=False`` even prefill chunks hit the quantizer."""
+    cache = RotorQuantKVCache(defer_prefill=False)
+    keys, values = _kv_chunk(16, seed=30)
+    out_k, _ = cache.update_and_fetch(keys, values)
+
+    assert cache.is_deferred is False
+    assert cache.has_live_storage is True
+    assert cache.offset == 16
+    # Reconstruction is lossy through the 3-bit codebook — verify the
+    # output isn't bit-identical to the input but is still close in MSE.
+    err = (out_k.astype(mx.float32) - keys.astype(mx.float32)) ** 2
+    rel_mse = (mx.sum(err) / mx.sum(keys.astype(mx.float32) ** 2)).item()
+    assert 1e-6 < rel_mse < 0.2
+
+
+def test_decode_after_prefill_appends_correctly() -> None:
+    """After flush, additional decode tokens should append to the live cache."""
+    cache = RotorQuantKVCache(defer_prefill=True)
+    pre_k, pre_v = _kv_chunk(8, seed=40)
+    cache.update_and_fetch(pre_k, pre_v)
+
+    for step in range(3):
+        dk, dv = _kv_chunk(1, seed=50 + step)
+        out_k, _ = cache.update_and_fetch(dk, dv)
+        assert out_k.shape[2] == 9 + step
+
+    assert cache.offset == 11
+    assert cache.size() == 11
+
+
+def test_trim_works_in_both_phases() -> None:
+    """``trim`` should remove tokens from whichever buffer is live."""
+    pending_cache = RotorQuantKVCache(defer_prefill=True)
+    keys, values = _kv_chunk(20, seed=60)
+    pending_cache.update_and_fetch(keys, values)
+    assert pending_cache.size() == 20
+    pending_cache.trim(5)
+    assert pending_cache.size() == 15
+
+    flushed_cache = RotorQuantKVCache(defer_prefill=True)
+    flushed_cache.update_and_fetch(*_kv_chunk(10, seed=70))
+    flushed_cache.update_and_fetch(*_kv_chunk(1, seed=71))  # triggers flush
+    assert flushed_cache.size() == 11
+    flushed_cache.trim(3)
+    assert flushed_cache.size() == 8
+
+
+def test_head_dim_must_be_block_aligned() -> None:
+    cache = RotorQuantKVCache(defer_prefill=False)
+    bad_keys = mx.zeros((1, 1, 2, 100), dtype=mx.float16)
+    bad_values = mx.zeros((1, 1, 2, 100), dtype=mx.float16)
+    with pytest.raises(ValueError):
+        cache.update_and_fetch(bad_keys, bad_values)
diff --git a/src/exo/worker/engines/mlx/rotorquant/tests/test_iso_quantizer.py b/src/exo/worker/engines/mlx/rotorquant/tests/test_iso_quantizer.py
new file mode 100644
index 000000000..8588fcd43
--- /dev/null
+++ b/src/exo/worker/engines/mlx/rotorquant/tests/test_iso_quantizer.py
@@ -0,0 +1,118 @@
+"""Correctness tests for the IsoQuant 3-bit quantizer.
+
+A bit-exact comparison against a precomputed llama.cpp CPU reference
+fixture is desirable but blocked on producing the fixture (which
+requires building the llama.cpp fork). For v1 we validate via
+self-consistency properties that any correct implementation must
+satisfy: rotation orthogonality, round-trip reconstruction error,
+norm-correction unbiasedness, and shape contracts.
+"""
+
+import math
+
+import mlx.core as mx
+import pytest
+
+from exo.worker.engines.mlx.rotorquant.quantizer import IsoQuantizer
+from exo.worker.engines.mlx.rotorquant.rotation import (
+    iso3_rotate_forward,
+    iso3_rotate_inverse,
+)
+from exo.worker.engines.mlx.rotorquant.tables import ISO3_BLOCK_SIZE
+
+
+def _random_kv(shape: tuple[int, ...], seed: int = 0) -> mx.array:
+    """Sample a Gaussian tensor matching a typical attention KV layout."""
+    return mx.random.normal(shape=shape, key=mx.random.key(seed))
+
+
+def test_quaternion_rotation_is_orthogonal() -> None:
+    """``||R x|| == ||x||`` for the per-block rotation."""
+    x = _random_kv((2, 4, 8, ISO3_BLOCK_SIZE))
+    rotated = iso3_rotate_forward(x.astype(mx.float32))
+    norm_in = mx.sqrt(mx.sum(x * x, axis=-1))
+    norm_out = mx.sqrt(mx.sum(rotated * rotated, axis=-1))
+    diff = mx.max(mx.abs(norm_in - norm_out)).item()
+    assert diff < 1e-4, f"rotation not orthogonal: max ||x|| - ||Rx|| = {diff}"
+
+
+def test_rotation_inverse_round_trips() -> None:
+    """``R^{-1} R x == x`` to floating-point precision."""
+    x = _random_kv((1, 2, 3, ISO3_BLOCK_SIZE), seed=1).astype(mx.float32)
+    restored = iso3_rotate_inverse(iso3_rotate_forward(x))
+    diff = mx.max(mx.abs(x - restored)).item()
+    assert diff < 1e-4, f"R^-1 R x diverged by {diff}"
+
+
+def test_quantize_dequantize_shapes() -> None:
+    """Indices and norms have the documented shapes for a typical KV layer."""
+    quantizer = IsoQuantizer()
+    keys = _random_kv((2, 8, 16, 256))  # head_dim 256 = 2 iso3 blocks
+    indices, norms = quantizer.quantize(keys)
+    assert indices.shape == (2, 8, 16, 256)
+    assert indices.dtype == mx.uint8
+    assert norms.shape == (2, 8, 16, 2)
+    assert norms.dtype == mx.float16
+    restored = quantizer.dequantize(indices, norms, mx.float16)
+    assert restored.shape == keys.shape
+    assert restored.dtype == mx.float16
+
+
+def test_quantize_indices_in_valid_range() -> None:
+    """Index values must lie in ``[0, 7]`` for the 3-bit codebook."""
+    quantizer = IsoQuantizer()
+    x = _random_kv((1, 1, 4, ISO3_BLOCK_SIZE), seed=2)
+    indices, _ = quantizer.quantize(x)
+    max_idx = int(mx.max(indices).item())
+    min_idx = int(mx.min(indices).item())
+    assert min_idx >= 0
+    assert max_idx <= 7
+
+
+def test_round_trip_reconstruction_is_within_3bit_budget() -> None:
+    """End-to-end reconstruction error stays within the 3-bit centroid budget.
+
+    With 8 Lloyd-Max centroids on a Gaussian-ish post-rotation
+    distribution, the per-element MSE should be around 0.04 of the
+    input variance. We use a generous threshold (0.15) so the test
+    catches outright bugs without flaking on RNG variance.
+    """
+    quantizer = IsoQuantizer()
+    x = _random_kv((1, 4, 32, ISO3_BLOCK_SIZE), seed=3).astype(mx.float32)
+    indices, norms = quantizer.quantize(x)
+    x_hat = quantizer.dequantize(indices, norms, mx.float32)
+
+    err = x - x_hat
+    rel_mse = (mx.sum(err * err) / mx.sum(x * x)).item()
+    assert rel_mse < 0.15, f"3-bit round-trip relative MSE {rel_mse} exceeds budget"
+
+
+def test_norm_correction_keeps_magnitudes_unbiased() -> None:
+    """The norm-correction trick should leave per-block ``||x||`` unbiased.
+
+    Without correction, centroid quantization shrinks the unit-vector
+    norm and the dequantized output systematically underestimates the
+    input magnitude. With correction the average magnitude ratio
+    should sit at ~1.0 across many random blocks.
+    """
+    quantizer = IsoQuantizer()
+    x = _random_kv((4, 8, 64, ISO3_BLOCK_SIZE), seed=4).astype(mx.float32)
+    indices, norms = quantizer.quantize(x)
+    x_hat = quantizer.dequantize(indices, norms, mx.float32)
+
+    norm_in = mx.sqrt(mx.sum(x * x, axis=-1))
+    norm_out = mx.sqrt(mx.sum(x_hat * x_hat, axis=-1))
+    ratio = (norm_out / mx.maximum(norm_in, 1e-8)).reshape(-1)
+    mean_ratio = float(mx.mean(ratio).item())
+    # Slack covers fp16 round-tripping plus the variance of a finite
+    # sample. The unbiased target is exactly 1.0.
+    assert math.isclose(mean_ratio, 1.0, abs_tol=0.05), (
+        f"norm-correction biased mean ratio = {mean_ratio}"
+    )
+
+
+def test_quantize_rejects_non_multiple_head_dim() -> None:
+    quantizer = IsoQuantizer()
+    bad = _random_kv((1, 1, 1, 100), seed=5)  # 100 is not a multiple of 128
+    with pytest.raises(ValueError):
+        quantizer.quantize(bad)
diff --git a/src/exo/worker/runner/llm_inference/runner.py b/src/exo/worker/runner/llm_inference/runner.py
index ca5be4eaa..cc965fb6c 100644
--- a/src/exo/worker/runner/llm_inference/runner.py
+++ b/src/exo/worker/runner/llm_inference/runner.py
@@ -448,6 +448,8 @@ def build(
             "turboquant",
             "turboquant_adaptive",
             "optiq",
+            "rotorquant",
+            "rotorquant_adaptive",
         )
         if os.environ.get("EXO_NO_BATCH") or force_sequential:
             if force_sequential and not os.environ.get("EXO_NO_BATCH"):
diff --git a/website/docs/everything-different.md b/website/docs/everything-different.md
new file mode 100644
index 000000000..fbf0fdb35
--- /dev/null
+++ b/website/docs/everything-different.md
@@ -0,0 +1,95 @@
+---
+id: everything-different
+title: Everything Different About Skulk
+sidebar_position: 2
+---
+
+<!-- Copyright 2025 Foxlight Foundation -->
+
+This is the running list of where Skulk diverges from upstream [exo](https://github.com/exo-explore/exo).
+
+It is intentionally a **living document** — every meaningful change should land here when it ships, so anyone evaluating Skulk can see the surface area at a glance and the team has a single place to point at when describing what makes this fork distinct.
+
+If you are about to land a feature or fix that adds, removes, or materially changes a behavior relative to upstream, update this page in the same PR.
+
+## Inference and KV cache
+
+- **RotorQuant KV cache backend** — pure-MLX port of IsoQuant 3-bit (block-diagonal quaternion rotations + Lloyd-Max centroids) with **deferred prefill on Metal**, a contribution that does not exist in any upstream project (the llama.cpp fork ships it CUDA-only). GQA-native; no fallback for grouped-query models. See [KV cache backends](kv-cache-backends).
+- **TurboQuant native and adaptive backends** — randomized Hadamard rotation + Lloyd-Max centroids, with an adaptive variant that keeps edge attention layers in fp16 for accuracy.
+- **OptiQ KV cache integration** — wraps `mlx-optiq`'s rotated-space attention path so the rotation cost stays out of the per-token loop on supported (non-GQA) models.
+- **OptiQ mixed-precision weight quantization pipeline** — async wrapper around `mlx-optiq`'s sensitivity analysis and KL-divergence per-layer bit allocation, exposed as a model-store optimization job.
+- **KV prefix cache with snapshot/restore** — LRU-evicted prompt-prefix cache that snapshots SSM and rotating-window cache states so prefix matches are reusable across conversation turns even for hybrid Mamba/Transformer architectures.
+- **Pipeline-parallel prefill for short prompts** — pipelined models now route every prefill through the pipeline path, fixing prior warmup hangs on Gemma-class models.
+- **Force-sequential fallback** — quantized backends transparently fall back to a sequential generator when batch/history mode is incompatible with their cache layout.
+
+## Model capability system
+
+- **Two-layer capability model** — declarative `ModelCard` (with optional `reasoning`, `modalities`, `tooling`, and `runtime` sections) plus a normalized `ResolvedCapabilityProfile` derived from the card and conservative family defaults. This is the source of truth for prompt rendering, output parsing, tool-call handling, and the `/v1/models` `resolved_capabilities` field.
+- **Phase 2 thinking contract** — `enable_thinking`, `reasoning_effort`, and the dashboard thinking toggle are all driven by `supports_thinking_toggle`, so non-toggleable reasoning models behave correctly without leaking model-specific quirks into client code. See [model capabilities](model-capabilities).
+- **Output parser selection** — model cards declare `output_parser` (`generic`, `gemma4`, `gpt_oss`, `deepseek_v32`, etc.), so reasoning markers are normalized into structured `reasoning_content` per family.
+- **Model store metadata pipeline** — capability resolution feeds `/v1/models` so dashboards and clients can discover thinking, multimodal, and tool support without hardcoding model lists.
+
+## API surface
+
+- **Claude Messages API** — `/v1/messages` adapter, including streaming, tool use, image inputs, and capability-aware thinking controls.
+- **Ollama compatibility** — both `/api/chat` and `/api/generate`, with adapter-side reasoning normalization.
+- **OpenAI Responses API** — `/v1/responses` adapter alongside chat completions.
+- **Embeddings endpoint** for non-chat workloads.
+- **Model store endpoints** — search, add, download, capability resolution, optimization jobs, and registry management, all exposed under stable URLs and documented in the OpenAPI spec.
+- **Cluster-wide config endpoints** — `GET`/`POST` config that gossipsubs to every node and writes back to `skulk.yaml`. The KV cache backend selection (including the new `rotorquant` and `rotorquant_adaptive` values) goes through this same endpoint, so the dashboard Settings panel can switch backends cluster-wide without env vars.
+- **Tracing, downloads, instance previews, and placement endpoints** — distributed-system observability and pre-launch placement inspection that upstream does not expose.
+
+## Dashboard
+
+- **React dashboard (default)** — replaces upstream's Svelte UI with a typed React + styled-components app that ships with the binary. The legacy Svelte dashboard is kept only as a fallback in the repo.
+- **Cluster topology view** with live device icons, GPU stats, network mesh visualization, and connection status banners.
+- **Placement preview / placement manager** for inspecting and choosing valid placements before launching.
+- **Model store browser** with HuggingFace search, family sidebar, model filters, capability badges, recent models, and per-model launch controls.
+- **Reasoning-aware chat UI** that splits inline `<think>` and Gemma `<|channel>` markers into a dedicated thinking panel and merges them with `reasoning_content` deltas from the API.
+- **Image attachments and multimodal chat affordances** for vision models.
+- **Cluster-wide settings panel** that writes to `skulk.yaml` and syncs across nodes via gossipsub.
+- **Light and dark themes** with first-class theme tokens.
+
+## Centralized logging and observability
+
+- **Structured JSON stdout** when `logging.enabled` is set, configurable from the dashboard Settings panel and synced cluster-wide.
+- **Vector + VictoriaLogs + Grafana stack** — local Vector log shipper on each node, central VictoriaLogs storage, ready-made Grafana dashboards. Stack definition lives in `deployment/logging/`.
+- **Distributed tracing** opt-in via `EXO_TRACING_ENABLED`.
+
+## Model store
+
+- **Centralized model store host** — one node downloads, the rest of the cluster stages over the LAN.
+- **Persistent registry** with capability resolution and download tracking.
+- **Custom model card support** — add your own model with `POST /models/add`.
+- **Image and embedding model cards** behind feature flags.
+- **Optimization job pipeline** for mlx-optiq mixed-precision weight quantization.
+
+## Cluster operation
+
+- **Cluster-wide settings sync** for KV cache backend, logging, model store host, HF token, and other inference toggles.
+- **Bootstrap peer config from `skulk.yaml`, env, or CLI** for fixed-topology clusters.
+- **Election (bully algorithm) + master/worker split** for indexing events and broadcasting state.
+- **`SKULK_*` environment variables** alongside the legacy `EXO_*` set, so new options can land without colliding with upstream.
+- **`skulk.yaml`** as the canonical config file, with `exo.yaml` kept for backwards compatibility.
+
+## Build, type system, and dev workflow
+
+- **Strict basedpyright** type checking — zero-error policy for new code.
+- **Ruff** linting and **`nix fmt`** formatting in CI.
+- **Nix flake** for reproducible toolchain setup.
+- **Docusaurus docs site** with auto-generated OpenAPI per-endpoint pages and TypeDoc HTML reference for the dashboard, both built from source.
+- **Pre-commit checklist** documented in `CLAUDE.md` and enforced in CI.
+
+## Hardware and platform
+
+- **Apple Silicon as the primary target**, including RDMA over Thunderbolt 5 on supported hardware and matched macOS versions.
+- **Linux supported** (CPU-oriented in this fork; GPU work happens on Apple Silicon).
+
+## How to update this page
+
+When you ship something that distinguishes Skulk from upstream exo, add it here in the same PR. A few ground rules:
+
+- Group your bullet under the right category. If none fit, add a new category — but only when there's clearly a new area of divergence.
+- Lead with what the change *is*, then a sentence on *why* it matters compared to upstream.
+- Link to the relevant guide, code path, or issue when one exists.
+- Remove or rewrite bullets when they become inaccurate. This page is canonical, so stale bullets are worse than missing ones.
diff --git a/website/docs/intro.md b/website/docs/intro.md
index 479bf03a6..338ce0341 100644
--- a/website/docs/intro.md
+++ b/website/docs/intro.md
@@ -19,6 +19,7 @@ Skulk is a distributed inference platform, so these docs try to serve both of th
 If you are getting oriented, start with these pages:
 
 - [README](https://github.com/Foxlight-Foundation/Skulk/blob/main/README.md) for installation, first run, and quick-start paths
+- [Everything Different About Skulk](everything-different) for a single-page view of how this fork diverges from upstream exo
 - [API guide](api-guide) for the happy path: place a model, then call the API
 - [Model store guide](model-store) for shared model storage and download workflows
 - [KV cache backends](kv-cache-backends) for backend and runtime tuning
diff --git a/website/docs/kv-cache-backends.md b/website/docs/kv-cache-backends.md
index a7ea147ba..1c2ae72f3 100644
--- a/website/docs/kv-cache-backends.md
+++ b/website/docs/kv-cache-backends.md
@@ -14,12 +14,29 @@ Skulk includes several opt-in KV cache backends for MLX text generation. These b
 - `mlx_quantized`: MLX LM built-in `QuantizedKVCache`
 - `turboquant`: correctness-first TurboQuant-inspired KV cache for standard `KVCache` layers
 - `turboquant_adaptive`: keeps outer KV layers in FP16 and applies TurboQuant to middle KV layers
-- `optiq`: **[NEW]** rotation-based KV cache via [mlx-optiq](https://mlx-optiq.pages.dev/) — uses randomized orthogonal rotations with Lloyd-Max quantization and rotated-space attention for superior long-context quality
+- `optiq`: rotation-based KV cache via [mlx-optiq](https://mlx-optiq.pages.dev/) — uses randomized orthogonal rotations with Lloyd-Max quantization and rotated-space attention. Falls back to default for GQA models.
+- `rotorquant`: **[NEW]** pure-MLX port of IsoQuant 3-bit from [scrya-com/rotorquant](https://github.com/scrya-com/rotorquant) and [johndpope/llama-cpp-turboquant](https://github.com/johndpope/llama-cpp-turboquant). Block-diagonal quaternion rotations + deferred prefill. GQA-native.
+- `rotorquant_adaptive`: **[NEW]** as above with FP16 protection on the first/last N attention layers. Recommended starting point.
 
 If `SKULK_KV_CACHE_BACKEND` is unset, or is set to `default`, Skulk behaves as before.
 
 ## Recommended Settings
 
+### RotorQuant Adaptive (recommended starting point)
+
+```bash
+SKULK_KV_CACHE_BACKEND=rotorquant_adaptive \
+SKULK_ROTORQUANT_FP16_LAYERS=4 \
+uv run skulk
+```
+
+The rotorquant backend implements IsoQuant 3-bit compression with two key properties:
+
+- **Block-diagonal quaternion rotations**: each 4D group of a head dimension is rotated by a fixed unit quaternion, costing `O(d)` per token instead of the `O(d log d)` randomized Hadamard used by the legacy turboquant backend or the `O(d²)` random orthogonal matrix used by mlx-optiq. The quaternion table and 3-bit Lloyd-Max centroids are vendored verbatim from the llama.cpp fork so the math agrees with the upstream C reference.
+- **Deferred prefill**: K and V are kept in fp16 throughout prompt processing and quantized once on the first decode token. This eliminates the compounding centroid-roundtrip error that quantizing-on-insert introduces during prefill, and matches the published 5.3× prefill / PPL 6.91 numbers from the upstream llama.cpp fork. The deferred-prefill flush is the unique contribution of this MLX port — the upstream fork only ships it on CUDA.
+
+GQA models work natively (no fallback) because compression is per-(kv_head, token) and Q heads fan out at SDPA. The head dimension must be a multiple of 128 (the IsoQuant block size); standard Llama and Qwen heads at 64 / 128 / 256 are all multiples and work directly.
+
 ### mlx-optiq (best quality)
 
 ```bash
@@ -54,6 +71,8 @@ This mode keeps the first and last 4 KV layers in normal FP16-style cache and ap
 | `SKULK_TQ_K_BITS` | `turboquant`, `turboquant_adaptive` | `3` | Key quantization bits |
 | `SKULK_TQ_V_BITS` | `turboquant`, `turboquant_adaptive` | `4` | Value quantization bits |
 | `SKULK_TQ_FP16_LAYERS` | `turboquant_adaptive` | `4` | Edge layers kept in FP16 |
+| `SKULK_ROTORQUANT_FP16_LAYERS` | `rotorquant_adaptive` | `4` | Edge layers kept in FP16 |
+| `SKULK_ROTORQUANT_DEFER_PREFILL` | `rotorquant`, `rotorquant_adaptive` | `1` | Set to `0` to disable deferred prefill (debugging only) |
 
 ## Invocation Examples
 
@@ -86,14 +105,16 @@ SKULK_KV_CACHE_BACKEND=turboquant_adaptive SKULK_TQ_K_BITS=3 SKULK_TQ_V_BITS=4 S
 | Backend | Memory | Quality | Speed | Notes |
 |---------|--------|---------|-------|-------|
 | `default` | Highest | Baseline | Fastest | No quantization |
-| `optiq` | Low | Best quantized | Near-baseline | Rotation-based, best long-context |
+| `rotorquant_adaptive` | Low | Best quantized | Near-baseline | IsoQuant 3-bit + deferred prefill, GQA-native |
+| `rotorquant` | Lowest | Good | Near-baseline | All layers IsoQuant 3-bit + deferred prefill |
+| `optiq` | Low | Good | Near-baseline | Rotation-based, no GQA support |
 | `turboquant_adaptive` | Low | Good | Moderate | Proven stable, Hadamard-based |
-| `turboquant` | Lowest | Variable | Moderate | Most aggressive compression |
+| `turboquant` | Low | Variable | Moderate | All layers, Hadamard-based |
 | `mlx_quantized` | Low | Good | Moderate | MLX built-in quantization |
 
 ## Supported Cache Layouts
 
-All quantized backends (optiq, turboquant, mlx_quantized) compress only standard `KVCache` entries and preserve these cache types unchanged:
+All quantized backends (rotorquant, optiq, turboquant, mlx_quantized) compress only standard `KVCache` entries and preserve these cache types unchanged:
 
 - `ArraysCache`
 - `RotatingKVCache`
diff --git a/website/sidebars.ts b/website/sidebars.ts
index c809181b8..0f8edb67f 100644
--- a/website/sidebars.ts
+++ b/website/sidebars.ts
@@ -15,6 +15,11 @@ const sidebars: SidebarsConfig = {
       id: "intro",
       label: "Introduction",
     },
+    {
+      type: "doc",
+      id: "everything-different",
+      label: "Everything Different About Skulk",
+    },
     {
       type: "category",
       label: "Guides",

From 53a0907f79c1a535258760db7f44b2342c21530a Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Tue, 7 Apr 2026 21:57:29 -0500
Subject: [PATCH 02/21] Address PR review: deferred-flush guard, state pytree,
 doc fixes
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- _flush_deferred no longer asserts on empty pending buffers; a
  decode-shaped first call (1-token prompt) is now a clean no-op
  that exits the deferred phase and lets the normal quantize-on-
  insert path handle the new token. Adds test_first_call_decode_
  shaped_does_not_crash as a regression.
- RotorQuantKVCache.state now returns a pytree of mx.array leaves
  only — the previous str tag would have raised inside mx.eval and
  mx.save_safetensors. Phase is disambiguated by tuple length
  (2 = pending, 4 = live) and tracked in meta_state. Adds
  test_state_round_trips_in_both_phases.
- README example commands now use uv run skulk consistently
  instead of the deprecated uv run exo alias.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 README.md                                     |  8 +--
 .../worker/engines/mlx/rotorquant/cache.py    | 59 +++++++++++++------
 .../mlx/rotorquant/tests/test_cache.py        | 58 ++++++++++++++++++
 3 files changed, 103 insertions(+), 22 deletions(-)

diff --git a/README.md b/README.md
index 8bb5646d4..b70e81873 100644
--- a/README.md
+++ b/README.md
@@ -462,10 +462,10 @@ uv run exo --bootstrap-peers /ip4/192.168.1.20/tcp/5678/p2p/12D3KooW...
 Examples:
 
 ```bash
-EXO_OFFLINE=true uv run exo
-EXO_ENABLE_IMAGE_MODELS=true uv run exo
-EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run exo
-SKULK_KV_CACHE_BACKEND=rotorquant_adaptive SKULK_ROTORQUANT_FP16_LAYERS=4 uv run exo
+EXO_OFFLINE=true uv run skulk
+EXO_ENABLE_IMAGE_MODELS=true uv run skulk
+EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run skulk
+SKULK_KV_CACHE_BACKEND=rotorquant_adaptive SKULK_ROTORQUANT_FP16_LAYERS=4 uv run skulk
 ```
 
 ## RDMA on macOS
diff --git a/src/exo/worker/engines/mlx/rotorquant/cache.py b/src/exo/worker/engines/mlx/rotorquant/cache.py
index c8248f00f..8cc13d4f0 100644
--- a/src/exo/worker/engines/mlx/rotorquant/cache.py
+++ b/src/exo/worker/engines/mlx/rotorquant/cache.py
@@ -193,12 +193,23 @@ def _append_pending(self, keys: mx.array, values: mx.array) -> None:
         )
 
     def _flush_deferred(self, dtype: mx.Dtype) -> None:
-        """Quantize the deferred prefill buffer and free the fp16 scratch."""
+        """Quantize the deferred prefill buffer and free the fp16 scratch.
+
+        If the first call into ``update_and_fetch`` is already decode-shaped
+        (e.g., a 1-token prompt) the deferred buffers will be empty. In that
+        case there is nothing to flush — we just exit the deferred phase and
+        let the caller's normal quantize-on-insert path handle the new
+        token.
+        """
         del dtype  # the live cache stores indices regardless of caller dtype
-        assert self._pending_keys is not None
-        assert self._pending_values is not None
         assert self._quantizer is not None
 
+        if self._pending_keys is None or self._pending_values is None:
+            self._pending_keys = None
+            self._pending_values = None
+            self._pending_active = False
+            return
+
         pending_keys = self._pending_keys
         pending_values = self._pending_values
         batch_size, kv_heads, num_pending, head_dim = pending_keys.shape
@@ -279,21 +290,30 @@ def size(self) -> int:
 
     @property
     def state(self) -> object:
+        """Pytree of array leaves for snapshot/restore and ``mx.eval``.
+
+        The shape of the tuple disambiguates phases for the setter:
+          - 2 elements → deferred-prefill scratch ``(pending_keys, pending_values)``
+          - 4 elements → live quantized cache ``(key_indices, key_norms, value_indices, value_norms)``
+          - empty list → uninitialized cache
+
+        We deliberately do **not** include a Python ``str`` tag here:
+        ``mx.eval`` and ``mx.save_safetensors`` expect every leaf in the
+        cache pytree to be an ``mx.array``, and a ``str`` leaf raises at
+        runtime. The phase is also tracked in ``meta_state`` via
+        ``_pending_active`` for human inspection.
+        """
         if self._pending_active:
-            # While deferred, expose only the fp16 scratch so that any
-            # external snapshot/restore round-trips correctly without
-            # corrupting the offset of the not-yet-allocated live cache.
             if self._pending_keys is None:
                 return []
             assert self._pending_values is not None
-            return ("pending", self._pending_keys, self._pending_values)
+            return (self._pending_keys, self._pending_values)
         if self._key_indices is None:
             return []
         assert self._key_norms is not None
         assert self._value_indices is not None
         assert self._value_norms is not None
         return (
-            "live",
             self._key_indices[..., : self.offset, :],
             self._key_norms[..., : self.offset, :],
             self._value_indices[..., : self.offset, :],
@@ -302,22 +322,25 @@ def state(self) -> object:
 
     @state.setter
     def state(self, value: object) -> None:
-        seq = cast(Sequence[object], value)
+        seq = cast(Sequence[mx.array], value)
         if len(seq) == 0:
             return
-        tag = seq[0]
-        if tag == "pending":
-            self._pending_keys = cast(mx.array, seq[1])
-            self._pending_values = cast(mx.array, seq[2])
+        if len(seq) == 2:
+            self._pending_keys = seq[0]
+            self._pending_values = seq[1]
             self._pending_active = True
             return
-        if tag == "live":
-            self._key_indices = cast(mx.array, seq[1])
-            self._key_norms = cast(mx.array, seq[2])
-            self._value_indices = cast(mx.array, seq[3])
-            self._value_norms = cast(mx.array, seq[4])
+        if len(seq) == 4:
+            self._key_indices = seq[0]
+            self._key_norms = seq[1]
+            self._value_indices = seq[2]
+            self._value_norms = seq[3]
             self.offset = int(self._key_indices.shape[2])
             self._pending_active = False
+            return
+        raise ValueError(
+            f"RotorQuantKVCache.state expects 0, 2, or 4 array leaves, got {len(seq)}"
+        )
 
     @property
     def meta_state(self) -> object:
diff --git a/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py b/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
index 416d6f67c..9c55f2d49 100644
--- a/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
+++ b/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
@@ -1,5 +1,7 @@
 """Tests for ``RotorQuantKVCache`` including the deferred-prefill path."""
 
+from typing import cast
+
 import mlx.core as mx
 import pytest
 
@@ -112,6 +114,62 @@ def test_trim_works_in_both_phases() -> None:
     assert flushed_cache.size() == 8
 
 
+def test_first_call_decode_shaped_does_not_crash() -> None:
+    """A 1-token first call should flush an empty pending buffer cleanly.
+
+    Regression for the case where stream_generate or pipeline_parallel_prefill
+    delivers a 1-token prompt: ``update_and_fetch`` is called with
+    ``num_steps == 1`` before any deferred prefill has been appended. The
+    flush path must treat the empty-pending case as a no-op and let the
+    normal quantize-on-insert path handle the new token.
+    """
+    cache = RotorQuantKVCache(defer_prefill=True)
+    keys, values = _kv_chunk(1, seed=80)
+    out_k, out_v = cache.update_and_fetch(keys, values)
+
+    assert cache.is_deferred is False
+    assert cache.has_pending_storage is False
+    assert cache.has_live_storage is True
+    assert cache.offset == 1
+    assert out_k.shape == (1, 4, 1, ISO3_BLOCK_SIZE)
+    assert out_v.shape == (1, 4, 1, ISO3_BLOCK_SIZE)
+
+
+def test_state_round_trips_in_both_phases() -> None:
+    """``state`` must be a pytree of mx.array leaves only (no str tags).
+
+    MLX serializers and ``mx.eval`` walk the cache state expecting array
+    leaves; a Python ``str`` tag would raise at runtime.
+    """
+    pending = RotorQuantKVCache(defer_prefill=True)
+    pending.update_and_fetch(*_kv_chunk(8, seed=90))
+    pending_state_obj = pending.state
+    assert isinstance(pending_state_obj, tuple)
+    pending_state = cast(tuple[mx.array, ...], pending_state_obj)
+    assert len(pending_state) == 2
+    for leaf in pending_state:
+        assert isinstance(leaf, mx.array)
+
+    flushed = RotorQuantKVCache(defer_prefill=True)
+    flushed.update_and_fetch(*_kv_chunk(8, seed=91))
+    flushed.update_and_fetch(*_kv_chunk(1, seed=92))  # triggers flush
+    live_state_obj = flushed.state
+    assert isinstance(live_state_obj, tuple)
+    live_state = cast(tuple[mx.array, ...], live_state_obj)
+    assert len(live_state) == 4
+    for leaf in live_state:
+        assert isinstance(leaf, mx.array)
+
+    # Round-trip via the setter into a fresh cache and confirm the offset
+    # and observable size match.
+    restored = RotorQuantKVCache(defer_prefill=False)
+    restored._quantizer = flushed._quantizer  # type: ignore[reportPrivateUsage]
+    restored.head_dim = flushed.head_dim
+    restored.state = live_state
+    assert restored.offset == flushed.offset
+    assert restored.size() == flushed.size()
+
+
 def test_head_dim_must_be_block_aligned() -> None:
     cache = RotorQuantKVCache(defer_prefill=False)
     bad_keys = mx.zeros((1, 1, 2, 100), dtype=mx.float16)

From ff21598872cb82cf50884e1597f680fb41d45751 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Wed, 8 Apr 2026 22:47:47 -0500
Subject: [PATCH 03/21] Fix RoPE position bug in deferred-prefill multi-chunk
 path

When deferred prefill is active and a prompt is split across multiple
update_and_fetch calls (num_steps > 1), the append-to-pending branch
returned without advancing cache.offset. Downstream attention reads
cache.offset for RoPE position, so any prompt longer than the prefill
chunk size (e.g. 4096) had every chunk after the first positioned as
if it began at token 0, producing wrong attention logits exactly at
the long-context regime RotorQuant is meant to serve.

Fix: advance offset on every append in both phases. _flush_deferred
now writes the quantized rows into [offset - num_pending, offset)
instead of double-counting. trim() and the state setter were updated
to keep offset consistent in the deferred phase as well, and size()
collapses to a single offset read.

Adds test_deferred_prefill_advances_offset_across_chunks as a
regression for the multi-chunk position bug.
---
 .../worker/engines/mlx/rotorquant/cache.py    | 33 +++++++++++++----
 .../mlx/rotorquant/tests/test_cache.py        | 37 ++++++++++++++++++-
 2 files changed, 61 insertions(+), 9 deletions(-)

diff --git a/src/exo/worker/engines/mlx/rotorquant/cache.py b/src/exo/worker/engines/mlx/rotorquant/cache.py
index 8cc13d4f0..d80d3d69a 100644
--- a/src/exo/worker/engines/mlx/rotorquant/cache.py
+++ b/src/exo/worker/engines/mlx/rotorquant/cache.py
@@ -168,6 +168,12 @@ def update_and_fetch(
                 self._append_pending(keys, values)
                 assert self._pending_keys is not None
                 assert self._pending_values is not None
+                # Advance the logical offset so downstream attention
+                # (RoPE position via ``cache.offset``) sees the correct
+                # token count when prefill is split into multiple
+                # chunks. Without this, every chunk after the first is
+                # positioned as if it started at token 0.
+                self.offset += num_steps
                 return self._pending_keys, self._pending_values
             self._flush_deferred(keys.dtype)
 
@@ -214,6 +220,14 @@ def _flush_deferred(self, dtype: mx.Dtype) -> None:
         pending_values = self._pending_values
         batch_size, kv_heads, num_pending, head_dim = pending_keys.shape
 
+        # ``offset`` was advanced as tokens were appended to the pending
+        # scratch, so the live region we are about to populate is
+        # ``[offset - num_pending, offset)``. ``_expand_storage`` sizes
+        # capacity from ``offset``, so temporarily rewind it to the
+        # pre-pending position before expanding so we don't allocate
+        # ``num_pending`` extra slots.
+        live_end = self.offset
+        self.offset = live_end - num_pending
         self._expand_storage(batch_size, kv_heads, num_pending, head_dim)
         assert self._key_indices is not None
         assert self._key_norms is not None
@@ -223,12 +237,11 @@ def _flush_deferred(self, dtype: mx.Dtype) -> None:
         key_indices, key_norms = self._quantizer.quantize(pending_keys)
         value_indices, value_norms = self._quantizer.quantize(pending_values)
 
-        end = self.offset + num_pending
-        self._key_indices[..., self.offset : end, :] = key_indices
-        self._key_norms[..., self.offset : end, :] = key_norms
-        self._value_indices[..., self.offset : end, :] = value_indices
-        self._value_norms[..., self.offset : end, :] = value_norms
-        self.offset = end
+        self._key_indices[..., self.offset : live_end, :] = key_indices
+        self._key_norms[..., self.offset : live_end, :] = key_norms
+        self._value_indices[..., self.offset : live_end, :] = value_indices
+        self._value_norms[..., self.offset : live_end, :] = value_norms
+        self.offset = live_end
 
         # Free the scratch — from now on we are in the steady-state
         # decode path and quantize on insert.
@@ -284,8 +297,8 @@ def _dequantize_live(
     # ------------------------------------------------------------------
 
     def size(self) -> int:
-        if self._pending_active and self._pending_keys is not None:
-            return int(self._pending_keys.shape[2])
+        # ``offset`` is the logical token count in both the deferred and
+        # steady-state phases (it is advanced on append in either path).
         return self.offset
 
     @property
@@ -329,6 +342,9 @@ def state(self, value: object) -> None:
             self._pending_keys = seq[0]
             self._pending_values = seq[1]
             self._pending_active = True
+            # Keep ``offset`` consistent with the logical token count
+            # restored from the pending scratch.
+            self.offset = int(self._pending_keys.shape[2])
             return
         if len(seq) == 4:
             self._key_indices = seq[0]
@@ -377,6 +393,7 @@ def trim(self, n: int) -> int:
             new_len = available - trimmed
             self._pending_keys = self._pending_keys[..., :new_len, :]
             self._pending_values = self._pending_values[..., :new_len, :]
+            self.offset -= trimmed
             return trimmed
         trimmed = min(self.offset, n)
         self.offset -= trimmed
diff --git a/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py b/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
index 9c55f2d49..96b1b962b 100644
--- a/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
+++ b/src/exo/worker/engines/mlx/rotorquant/tests/test_cache.py
@@ -30,7 +30,9 @@ def test_deferred_prefill_keeps_storage_unallocated_during_prefill() -> None:
 
     assert cache.is_deferred is True
     assert cache.has_live_storage is False
-    assert cache.offset == 0
+    # ``offset`` is the logical token count even during the deferred
+    # phase — it must advance so downstream RoPE positioning is correct.
+    assert cache.offset == 64
     assert cache.size() == 64
 
 
@@ -114,6 +116,39 @@ def test_trim_works_in_both_phases() -> None:
     assert flushed_cache.size() == 8
 
 
+def test_deferred_prefill_advances_offset_across_chunks() -> None:
+    """Multi-chunk prefill must advance ``offset`` so RoPE position is right.
+
+    Regression for the bug where the deferred-prefill ``num_steps > 1``
+    branch appended to the scratch buffer but never updated
+    ``cache.offset``. Downstream attention reads ``cache.offset`` for the
+    RoPE position, so prompts longer than the per-chunk prefill step
+    (e.g. 4096 tokens) saw every chunk after the first positioned as if
+    it started at token 0, producing wrong logits.
+    """
+    cache = RotorQuantKVCache(defer_prefill=True)
+
+    chunk_one_k, chunk_one_v = _kv_chunk(100, seed=200)
+    cache.update_and_fetch(chunk_one_k, chunk_one_v)
+    assert cache.offset == 100
+    assert cache.size() == 100
+
+    chunk_two_k, chunk_two_v = _kv_chunk(150, seed=201)
+    cache.update_and_fetch(chunk_two_k, chunk_two_v)
+    assert cache.offset == 250
+    assert cache.size() == 250
+    assert cache.is_deferred is True
+    assert cache.has_live_storage is False
+
+    # After flush, ``offset`` should still equal the total prefill length
+    # (250) plus the decode token (1) — exactly 251.
+    decode_k, decode_v = _kv_chunk(1, seed=202)
+    out_k, _ = cache.update_and_fetch(decode_k, decode_v)
+    assert cache.is_deferred is False
+    assert cache.offset == 251
+    assert out_k.shape[2] == 251
+
+
 def test_first_call_decode_shaped_does_not_crash() -> None:
     """A 1-token first call should flush an empty pending buffer cleanly.
 

From f40e9d4143cd9ff182b5000fc2e6f19b81c9ef99 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 08:11:28 -0500
Subject: [PATCH 04/21] Add head_dim preflight guard for rotorquant, fix docs
 and tooltip
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- make_kv_cache now checks head_dim % ISO3_BLOCK_SIZE before selecting
  rotorquant/rotorquant_adaptive; models with incompatible heads (e.g.
  64-d) fall back to the default cache with a warning instead of
  crashing at first token.
- Extract _make_default_cache helper so fallback paths don't duplicate
  the make_cache/KVCache logic.
- Fix kv-cache-backends.md claiming 64-d heads are supported (they are
  not — ISO3_BLOCK_SIZE is 128).
- Soften SettingsPanel tooltip/hint: only OptiQ auto-falls-back;
  other backends will error on incompatible models.
---
 .../src/components/layout/SettingsPanel.tsx   |  4 +-
 src/exo/worker/engines/mlx/cache.py           | 49 +++++++++++++------
 website/docs/kv-cache-backends.md             |  2 +-
 3 files changed, 38 insertions(+), 17 deletions(-)

diff --git a/dashboard-react/src/components/layout/SettingsPanel.tsx b/dashboard-react/src/components/layout/SettingsPanel.tsx
index e177ffe7b..c29860856 100644
--- a/dashboard-react/src/components/layout/SettingsPanel.tsx
+++ b/dashboard-react/src/components/layout/SettingsPanel.tsx
@@ -455,7 +455,7 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
                   `• TurboQuant Adaptive — Quantizes middle KV layers, keeps edge layers in FP16. Proven stable.\n` +
                   `• TurboQuant — Quantizes all KV layers. Most aggressive non-rotorquant compression.\n` +
                   `• MLX Quantized — MLX's built-in cache quantization.\n\n` +
-                  `Takes effect on next model launch. Incompatible models fall back to Default automatically.`
+                  `Takes effect on next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.`
                 }
               />
             </FieldLabel>
@@ -471,7 +471,7 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
             {envOverride ? (
               <HintText>Overridden by SKULK_KV_CACHE_BACKEND environment variable. Remove the env var to configure here.</HintText>
             ) : (
-              <HintText>Changes take effect on the next model launch. Models with incompatible architectures (GQA, non-power-of-two head_dim) will automatically fall back to default.</HintText>
+              <HintText>Changes take effect on the next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.</HintText>
             )}
           </Fieldset>
 
diff --git a/src/exo/worker/engines/mlx/cache.py b/src/exo/worker/engines/mlx/cache.py
index dac1d6eb6..8ec5c4226 100644
--- a/src/exo/worker/engines/mlx/cache.py
+++ b/src/exo/worker/engines/mlx/cache.py
@@ -35,6 +35,7 @@
     make_rotorquant_adaptive_cache,
     make_rotorquant_cache_from_template,
 )
+from exo.worker.engines.mlx.rotorquant.tables import ISO3_BLOCK_SIZE
 from exo.worker.engines.mlx.turboquant import (
     make_turboquant_adaptive_cache,
     make_turboquant_cache_from_template,
@@ -423,6 +424,13 @@ def get_memory_used_percentage() -> float:
     return float(mem.percent / 100)
 
 
+def _make_default_cache(model: Model) -> KVCacheType:
+    """Build the default (unquantized) KV cache for a model."""
+    if hasattr(model, "make_cache"):
+        return model.make_cache()  # type: ignore
+    return [KVCache() for _ in model.layers]
+
+
 def make_kv_cache(
     model: Model, max_kv_size: int | None = None, keep: int = 0
 ) -> KVCacheType:
@@ -585,16 +593,33 @@ def _is_power_of_two(n: int) -> bool:
             for i, _ in enumerate(model.layers)
         ]
 
-    if backend == "rotorquant":
-        logger.info(
-            f"Using rotorquant KV cache (defer_prefill={ROTORQUANT_DEFER_PREFILL})"
-        )
-        return make_rotorquant_cache_from_template(
-            model,
-            defer_prefill=ROTORQUANT_DEFER_PREFILL,
-        )
+    if backend in ("rotorquant", "rotorquant_adaptive"):
+        # Preflight: RotorQuant requires head_dim divisible by the
+        # IsoQuant block size (128). Models with smaller heads (e.g. 64-d)
+        # would crash at first token; fall back to default instead.
+        if len(model.layers) > 0:
+            first_layer = model.layers[0]
+            attn = getattr(first_layer, "self_attn", first_layer)
+            model_head_dim: int = getattr(attn, "head_dim", ISO3_BLOCK_SIZE)
+        else:
+            model_head_dim = ISO3_BLOCK_SIZE
+        if model_head_dim % ISO3_BLOCK_SIZE != 0:
+            logger.warning(
+                f"RotorQuant requires head_dim divisible by {ISO3_BLOCK_SIZE}, "
+                f"but this model has head_dim={model_head_dim}; "
+                f"falling back to default KV cache"
+            )
+            return _make_default_cache(model)
+
+        if backend == "rotorquant":
+            logger.info(
+                f"Using rotorquant KV cache (defer_prefill={ROTORQUANT_DEFER_PREFILL})"
+            )
+            return make_rotorquant_cache_from_template(
+                model,
+                defer_prefill=ROTORQUANT_DEFER_PREFILL,
+            )
 
-    if backend == "rotorquant_adaptive":
         logger.info(
             f"Using rotorquant adaptive KV cache "
             f"(fp16_layers={ROTORQUANT_FP16_LAYERS}, "
@@ -606,12 +631,8 @@ def _is_power_of_two(n: int) -> bool:
             defer_prefill=ROTORQUANT_DEFER_PREFILL,
         )
 
-    if hasattr(model, "make_cache"):
-        logger.info("Using MLX LM's make cache")
-        return model.make_cache()  # type: ignore
-
     logger.info("Using default KV cache")
-    return [KVCache() for _ in model.layers]
+    return _make_default_cache(model)
 
 
 def get_kv_cache_backend() -> KVCacheBackend:
diff --git a/website/docs/kv-cache-backends.md b/website/docs/kv-cache-backends.md
index 1c2ae72f3..abb059d4e 100644
--- a/website/docs/kv-cache-backends.md
+++ b/website/docs/kv-cache-backends.md
@@ -35,7 +35,7 @@ The rotorquant backend implements IsoQuant 3-bit compression with two key proper
 - **Block-diagonal quaternion rotations**: each 4D group of a head dimension is rotated by a fixed unit quaternion, costing `O(d)` per token instead of the `O(d log d)` randomized Hadamard used by the legacy turboquant backend or the `O(d²)` random orthogonal matrix used by mlx-optiq. The quaternion table and 3-bit Lloyd-Max centroids are vendored verbatim from the llama.cpp fork so the math agrees with the upstream C reference.
 - **Deferred prefill**: K and V are kept in fp16 throughout prompt processing and quantized once on the first decode token. This eliminates the compounding centroid-roundtrip error that quantizing-on-insert introduces during prefill, and matches the published 5.3× prefill / PPL 6.91 numbers from the upstream llama.cpp fork. The deferred-prefill flush is the unique contribution of this MLX port — the upstream fork only ships it on CUDA.
 
-GQA models work natively (no fallback) because compression is per-(kv_head, token) and Q heads fan out at SDPA. The head dimension must be a multiple of 128 (the IsoQuant block size); standard Llama and Qwen heads at 64 / 128 / 256 are all multiples and work directly.
+GQA models work natively (no fallback) because compression is per-(kv_head, token) and Q heads fan out at SDPA. The head dimension must be a multiple of 128 (the IsoQuant block size); for example, 128- and 256-d heads work directly, while 64-d heads do not satisfy this requirement and will fall back to the default backend.
 
 ### mlx-optiq (best quality)
 

From aba98c2163d4c6408f260d13d0ef6da1c3d6c2e8 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 08:59:38 -0500
Subject: [PATCH 05/21] Fix pipeline warmup hang: enable prefill mode for
 stream_generate path
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

With is_prefill=False on the stream_generate fallback path for short
prompts, PipelineLastLayer runs mx.distributed.all_gather on every
forward pass. generate_step discards prefill logits and only evaluates
cache states, leaving the all_gather as an unevaluated zombie
collective. When a later mx.eval triggers it, the ranks are
desynchronized and the collective deadlocks.

Observed as a warmup hang on a 3-rank pipeline with Gemma 4 26B.
The KV backend is irrelevant — the hang reproduces with any backend
(logs confirmed the backend fell back to default before the hang).

Fix: set is_prefill=True for ALL pipeline models during prefill,
including the stream_generate fallback. This disables the all_gather
and relies on point-to-point send/recv only. The first decode token
is wrong on non-last ranks (they see rank-local activations), but
prefill() trims those tokens before real generation begins and the
finally block resets is_prefill=False for subsequent decode.
---
 .../worker/engines/mlx/generator/generate.py  | 23 ++++++++++++++-----
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/src/exo/worker/engines/mlx/generator/generate.py b/src/exo/worker/engines/mlx/generator/generate.py
index 5e40a7124..6378daa52 100644
--- a/src/exo/worker/engines/mlx/generator/generate.py
+++ b/src/exo/worker/engines/mlx/generator/generate.py
@@ -529,12 +529,23 @@ def combined_progress_callback(processed: int, total: int) -> None:
         f"prefill_step_size_effective={effective_prefill_step_size}, "
         f"pipeline_chunks={pipeline_chunks})"
     )
-    # Only enable pipeline-prefill mode for the true pipeline prefill path.
-    # Short prompts routed through stream_generate must behave like normal
-    # generation; leaving pipeline wrappers in prefill mode there can strand
-    # non-zero ranks in the pipeline send/recv path without the coordinated
-    # queue/flush behavior used by pipeline_parallel_prefill.
-    set_pipeline_prefill(model, is_prefill=use_pipeline_prefill)
+    # Pipeline models must run in prefill mode during any prefill forward
+    # pass — even the stream_generate fallback path for short prompts.
+    # With is_prefill=False, PipelineLastLayer performs all_gather on
+    # every forward pass. generate_step discards prefill logits and only
+    # evaluates cache states, so the all_gather is never consumed but
+    # remains queued. When a subsequent mx.eval triggers it, the ranks
+    # are desynchronized and the collective deadlocks (observed as a
+    # warmup hang on 3-rank pipeline with Gemma 4).
+    #
+    # With is_prefill=True, PipelineLastLayer uses point-to-point
+    # send/recv only — no collective required. The tradeoff: the first
+    # decode token sampled by generate_step is wrong on non-last ranks
+    # (they see rank-local activations, not full logits). This is
+    # acceptable because prefill() trims those decode tokens from the
+    # cache before real generation begins, and the finally block below
+    # resets is_prefill=False for subsequent decode.
+    set_pipeline_prefill(model, is_prefill=is_pipeline)
 
     with _hang_debug_watch(f"prefill barrier rank={rank} group_size={group_size}"):
         mx_barrier(group)

From 85454211418c3fdc66b0951421feecd7c41f289b Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 12:43:53 -0500
Subject: [PATCH 06/21] Make env var the source of truth for KV backend, sync
 to config

Three changes:

1. get_kv_cache_backend() now reads os.environ at call time via
   preferred_env_value() instead of the frozen import-time constant.
   This ensures runtime updates (from config sync, dashboard, or
   coordinator) are always visible to the runner.

2. When the user sets SKULK_KV_CACHE_BACKEND at launch, main.py
   writes it back to exo.yaml so the config file stays in sync.
   Previously the config file could silently override the env var
   because constants.py read at import time, before main.py had a
   chance to apply the config.

3. Update tests to use patch.dict(os.environ) instead of patching
   the removed module-level KV_CACHE_BACKEND constant.
---
 src/exo/main.py                               | 23 +++++++++++++++----
 src/exo/store/config.py                       | 22 ++++++++++++++++++
 src/exo/worker/engines/mlx/cache.py           | 21 ++++++++++++++---
 .../test_mlx/test_kv_prefix_cache.py          | 23 ++++++++-----------
 .../test_mlx/test_prefill_path_selection.py   |  9 ++++++--
 5 files changed, 75 insertions(+), 23 deletions(-)

diff --git a/src/exo/main.py b/src/exo/main.py
index ddaf078bc..7b2aceb24 100644
--- a/src/exo/main.py
+++ b/src/exo/main.py
@@ -29,6 +29,7 @@
     load_exo_config,
     resolve_config_path,
     resolve_node_staging,
+    update_config_field,
 )
 from exo.store.model_store import ModelStore
 from exo.store.model_store_client import ModelStoreClient, ModelStoreDownloader
@@ -98,12 +99,26 @@ async def create(cls, args: "Args") -> Self:
             "1" if _user_set_kv_backend else ""
         )  # legacy compat
 
-        # Apply inference config to env var so runner subprocesses inherit it.
-        # Env var takes precedence if user set it at launch.
-        if (
+        # Env var is the source of truth for KV backend. When set at launch,
+        # write it back to the config file so they stay in sync. When not
+        # set, apply the config file value to the env var so runner
+        # subprocesses inherit it.
+        if _user_set_kv_backend:
+            launch_backend = os.environ.get(
+                "SKULK_KV_CACHE_BACKEND",
+                os.environ.get("EXO_KV_CACHE_BACKEND", ""),
+            )
+            if launch_backend:
+                if update_config_field("inference", "kv_cache_backend", launch_backend):
+                    logger.info(
+                        f"Synced launch env KV backend to config: kv_cache_backend={launch_backend}"
+                    )
+                # Ensure both env vars are in sync
+                os.environ["SKULK_KV_CACHE_BACKEND"] = launch_backend
+                os.environ["EXO_KV_CACHE_BACKEND"] = launch_backend
+        elif (
             exo_config is not None
             and exo_config.inference is not None
-            and not _user_set_kv_backend
         ):
             os.environ["SKULK_KV_CACHE_BACKEND"] = exo_config.inference.kv_cache_backend
             os.environ["EXO_KV_CACHE_BACKEND"] = (
diff --git a/src/exo/store/config.py b/src/exo/store/config.py
index 7a4688395..7c0b617a2 100644
--- a/src/exo/store/config.py
+++ b/src/exo/store/config.py
@@ -245,6 +245,28 @@ class InferenceConfig(FrozenModel):
     ] = "default"
 
 
+def update_config_field(section: str, key: str, value: object) -> bool:
+    """Update a single field in the config file, preserving all other content.
+
+    Reads the raw YAML, patches ``raw[section][key] = value``, and writes
+    it back.  Returns ``True`` if the file was updated, ``False`` if no
+    config file exists.
+    """
+    path = resolve_config_path()
+    if not path.exists():
+        return False
+    with path.open() as f:
+        raw: dict[str, object] = yaml.safe_load(f) or {}
+    sec = raw.get(section)
+    if not isinstance(sec, dict):
+        sec = {}
+        raw[section] = sec
+    sec[key] = value
+    with path.open("w") as f:
+        yaml.dump(raw, f, default_flow_style=False, sort_keys=False)
+    return True
+
+
 def resolve_config_path() -> Path:
     """Find the config file, preferring ``skulk.yaml`` over legacy ``exo.yaml``."""
     skulk = Path("skulk.yaml")
diff --git a/src/exo/worker/engines/mlx/cache.py b/src/exo/worker/engines/mlx/cache.py
index 8ec5c4226..474f090d4 100644
--- a/src/exo/worker/engines/mlx/cache.py
+++ b/src/exo/worker/engines/mlx/cache.py
@@ -12,6 +12,7 @@
 )
 from mlx_lm.tokenizer_utils import TokenizerWrapper
 
+from exo.shared.constants import preferred_env_value
 from exo.shared.types.memory import Memory
 from exo.shared.types.mlx import KVCacheType, Model
 from exo.worker.engines.mlx.constants import (
@@ -19,7 +20,6 @@
     DEFAULT_KV_CACHE_BACKEND,
     DEFAULT_TURBOQUANT_K_BITS,
     DEFAULT_TURBOQUANT_V_BITS,
-    KV_CACHE_BACKEND,
     KV_CACHE_BITS,
     OPTIQ_BITS,
     OPTIQ_FP16_LAYERS,
@@ -636,10 +636,25 @@ def _is_power_of_two(n: int) -> bool:
 
 
 def get_kv_cache_backend() -> KVCacheBackend:
-    backend = KV_CACHE_BACKEND
+    """Return the active KV cache backend, reading ``os.environ`` at call time.
+
+    The env var is the source of truth — it can be set at launch, updated
+    by config sync via gossipsub, or changed from the dashboard.  Reading
+    it dynamically (instead of the frozen import-time constant) ensures
+    that runtime updates are always visible to the runner.
+    """
+    backend = cast(
+        KVCacheBackend,
+        preferred_env_value(
+            "SKULK_KV_CACHE_BACKEND",
+            "EXO_KV_CACHE_BACKEND",
+            DEFAULT_KV_CACHE_BACKEND,
+        )
+        or DEFAULT_KV_CACHE_BACKEND,
+    )
     if backend not in VALID_KV_CACHE_BACKENDS:
         logger.warning(
-            f"Unknown EXO_KV_CACHE_BACKEND={backend!r}; falling back to {DEFAULT_KV_CACHE_BACKEND!r}"
+            f"Unknown KV_CACHE_BACKEND={backend!r}; falling back to {DEFAULT_KV_CACHE_BACKEND!r}"
         )
         return DEFAULT_KV_CACHE_BACKEND
     return backend
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py b/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py
index 6d2a077e3..5f73a8177 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py
@@ -1,4 +1,5 @@
 # type: ignore
+import os
 import time
 from typing import cast
 from unittest.mock import patch
@@ -109,15 +110,12 @@ def test_clear_on_empty_cache(self, mock_tokenizer):
 
 class TestKVCacheBackends:
     def test_unknown_backend_falls_back_to_default(self):
-        with patch(
-            "exo.worker.engines.mlx.cache.KV_CACHE_BACKEND",
-            cast(object, "mystery"),
-        ):
+        with patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "mystery"}):
             assert get_kv_cache_backend() == "default"
 
     def test_make_kv_cache_default_backend(self):
         model = cast(Model, type("FakeModel", (), {"layers": [object(), object()]})())
-        with patch("exo.worker.engines.mlx.cache.KV_CACHE_BACKEND", "default"):
+        with patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "default"}):
             cache = make_kv_cache(model)
 
         assert len(cache) == 2
@@ -126,7 +124,7 @@ def test_make_kv_cache_default_backend(self):
     def test_make_kv_cache_mlx_quantized_backend(self):
         model = cast(Model, type("FakeModel", (), {"layers": [object(), object()]})())
         with (
-            patch("exo.worker.engines.mlx.cache.KV_CACHE_BACKEND", "mlx_quantized"),
+            patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "mlx_quantized"}),
             patch("exo.worker.engines.mlx.cache.KV_CACHE_BITS", 4),
             patch("exo.worker.engines.mlx.cache.CACHE_GROUP_SIZE", 32),
         ):
@@ -144,7 +142,7 @@ def make_cache(self):
 
         model = cast(Model, FakeModel())
         with (
-            patch("exo.worker.engines.mlx.cache.KV_CACHE_BACKEND", "mlx_quantized"),
+            patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "mlx_quantized"}),
             patch("exo.worker.engines.mlx.cache.KV_CACHE_BITS", 4),
             patch("exo.worker.engines.mlx.cache.CACHE_GROUP_SIZE", 32),
         ):
@@ -163,7 +161,7 @@ def make_cache(self):
 
         model = cast(Model, FakeModel())
         with (
-            patch("exo.worker.engines.mlx.cache.KV_CACHE_BACKEND", "mlx_quantized"),
+            patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "mlx_quantized"}),
             patch("exo.worker.engines.mlx.cache.KV_CACHE_BITS", 4),
             patch("exo.worker.engines.mlx.cache.CACHE_GROUP_SIZE", 32),
         ):
@@ -176,7 +174,7 @@ def make_cache(self):
     def test_make_kv_cache_mlx_quantized_backend_requires_bits(self):
         model = cast(Model, type("FakeModel", (), {"layers": [object()]})())
         with (
-            patch("exo.worker.engines.mlx.cache.KV_CACHE_BACKEND", "mlx_quantized"),
+            patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "mlx_quantized"}),
             patch("exo.worker.engines.mlx.cache.KV_CACHE_BITS", None),
             pytest.raises(ValueError, match="EXO_KV_CACHE_BITS"),
         ):
@@ -191,7 +189,7 @@ def make_cache(self):
 
         model = cast(Model, FakeModel())
         with (
-            patch("exo.worker.engines.mlx.cache.KV_CACHE_BACKEND", "turboquant"),
+            patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "turboquant"}),
             patch("exo.worker.engines.mlx.cache.TURBOQUANT_K_BITS", 3),
             patch("exo.worker.engines.mlx.cache.TURBOQUANT_V_BITS", 4),
         ):
@@ -208,10 +206,7 @@ def make_cache(self):
 
         model = cast(Model, FakeModel())
         with (
-            patch(
-                "exo.worker.engines.mlx.cache.KV_CACHE_BACKEND",
-                "turboquant_adaptive",
-            ),
+            patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "turboquant_adaptive"}),
             patch("exo.worker.engines.mlx.cache.TURBOQUANT_K_BITS", 3),
             patch("exo.worker.engines.mlx.cache.TURBOQUANT_V_BITS", 4),
             patch("exo.worker.engines.mlx.cache.TURBOQUANT_FP16_LAYERS", 1),
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py b/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py
index 7049aa9fe..25c8e1100 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py
@@ -157,7 +157,10 @@ def _stream_generate(*args: object, **kwargs: object) -> Iterator[object]:
     assert snapshots == []
     assert prefill_tps >= 0.0
     assert fake_cache.trim_calls == [2]
-    assert prefill_mode_calls == [False, False]
+    # First call sets is_prefill=True (pipeline models must run in prefill
+    # mode even on the stream_generate path to avoid all_gather deadlocks).
+    # Second call (in finally) resets to False for subsequent decode.
+    assert prefill_mode_calls == [True, False]
 
 
 def test_prefill_uses_stream_generate_at_single_chunk_boundary(
@@ -207,7 +210,9 @@ def _stream_generate(*args: object, **kwargs: object) -> Iterator[object]:
     assert snapshots == []
     assert prefill_tps >= 0.0
     assert fake_cache.trim_calls == [2]
-    assert prefill_mode_calls == [False, False]
+    # Pipeline models use is_prefill=True during stream_generate prefill,
+    # reset to False in finally.
+    assert prefill_mode_calls == [True, False]
 
 
 def test_prefill_resets_pipeline_flags_after_unexpected_exception(

From 032593b32bc6fc4dd8d4e234d41a9d6d5907c5b0 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 14:07:55 -0500
Subject: [PATCH 07/21] Protect user-set KV backend from cluster config sync

When SKULK_KV_CACHE_BACKEND is set at launch, the coordinator's
_sync_config was overwriting both the env var AND the local exo.yaml
with whatever the cluster broadcast (typically the old value from
another node). This silently undid the user's intent.

Fix: when _SKULK_KV_BACKEND_USER_SET is active, patch the incoming
config YAML to preserve the user's KV backend before writing it to
disk, and skip the env var update. The user's launch-time value
remains the source of truth until the process exits.
---
 src/exo/download/coordinator.py | 43 ++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 9 deletions(-)

diff --git a/src/exo/download/coordinator.py b/src/exo/download/coordinator.py
index 5e0b7e1f2..b9dadd546 100644
--- a/src/exo/download/coordinator.py
+++ b/src/exo/download/coordinator.py
@@ -181,24 +181,48 @@ async def _restart_node(self) -> None:
 
     async def _sync_config(self, config_yaml: str) -> None:
         """Write received config YAML to the local config file and
-        apply runtime-effective settings (e.g., KV cache backend)."""
+        apply runtime-effective settings (e.g., KV cache backend).
+
+        When the user set ``SKULK_KV_CACHE_BACKEND`` at launch, their
+        value is the source of truth. The cluster sync must not
+        overwrite it — neither in ``os.environ`` nor in the config
+        file.
+        """
         config_path = resolve_config_path()
         try:
+            import yaml
+
+            raw = yaml.safe_load(config_yaml)
+            user_set_kv = bool(
+                os.environ.get("_SKULK_KV_BACKEND_USER_SET")
+                or os.environ.get("_EXO_KV_BACKEND_USER_SET")
+            )
+
+            # Preserve the user's KV backend in the config file when
+            # the env var was set at launch — don't let cluster sync
+            # clobber it.
+            if user_set_kv and raw and isinstance(raw, dict):
+                local_backend = os.environ.get("SKULK_KV_CACHE_BACKEND")
+                if local_backend:
+                    inference = raw.get("inference")
+                    if isinstance(inference, dict):
+                        inference["kv_cache_backend"] = local_backend
+                    else:
+                        raw["inference"] = {"kv_cache_backend": local_backend}
+                    config_yaml = yaml.safe_dump(
+                        raw, default_flow_style=False, sort_keys=False
+                    )
+
             config_path.write_text(config_yaml)
             logger.info(
                 f"DownloadCoordinator: synced {config_path.name} from cluster ({len(config_yaml)} bytes)"
             )
-            # Apply inference config to env var so next runner spawn picks it up
-            import yaml
 
-            raw = yaml.safe_load(config_yaml)
+            # Apply inference config to env var so next runner spawn picks it up
             if raw and isinstance(raw, dict):
                 inference = raw.get("inference")
                 if isinstance(inference, dict) and "kv_cache_backend" in inference:
-                    # Don't overwrite if user provided the env var at launch
-                    if not os.environ.get(
-                        "_SKULK_KV_BACKEND_USER_SET"
-                    ) and not os.environ.get("_EXO_KV_BACKEND_USER_SET"):
+                    if not user_set_kv:
                         os.environ["SKULK_KV_CACHE_BACKEND"] = str(
                             inference["kv_cache_backend"]
                         )
@@ -210,7 +234,8 @@ async def _sync_config(self, config_yaml: str) -> None:
                         )
                     else:
                         logger.info(
-                            "DownloadCoordinator: skipping KV backend update (user env var override active)"
+                            "DownloadCoordinator: keeping user-set KV_CACHE_BACKEND="
+                            f"{os.environ.get('SKULK_KV_CACHE_BACKEND', '?')}"
                         )
                 # Apply HF token if not user-set
                 hf_token = raw.get("hf_token")

From f587d2b8f1da6af29a52ec447a8cf2e5380baf13 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 16:37:32 -0500
Subject: [PATCH 08/21] Fix pipeline all_gather deadlock: eval full tensor
 before slicing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The last rank's PipelineLastLayer sliced all_gather output before eval,
letting MLX satisfy the slice from local data without executing the
collective — leaving other ranks blocked forever. Now eval the unsliced
gathered tensor so every rank participates, then slice.

Also adds generic _cache_dep_anchor()/_set_cache_dep_anchor() helpers
so mx.depends works with RotorQuantKVCache (which uses _key_indices or
_pending_keys instead of .keys).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 src/exo/worker/engines/mlx/auto_parallel.py | 81 +++++++++++++++++----
 1 file changed, 67 insertions(+), 14 deletions(-)

diff --git a/src/exo/worker/engines/mlx/auto_parallel.py b/src/exo/worker/engines/mlx/auto_parallel.py
index 6094bb2d8..34d33eb98 100644
--- a/src/exo/worker/engines/mlx/auto_parallel.py
+++ b/src/exo/worker/engines/mlx/auto_parallel.py
@@ -192,6 +192,46 @@ def watchdog() -> None:
         completed.set()
 
 
+def _cache_dep_anchor(cache: object) -> mx.array | None:
+    """Return an array from ``cache`` suitable for ``mx.depends`` ordering.
+
+    Standard ``KVCache`` exposes ``.keys``; quantized caches like
+    ``RotorQuantKVCache`` do not.  Fall back to the first leaf of
+    ``.state`` so we can order cache evaluation relative to distributed
+    sends on *any* cache backend.
+    """
+    # Fast path: mlx-lm KVCache and friends
+    keys = getattr(cache, "keys", None)
+    if isinstance(keys, mx.array):
+        return keys
+
+    # Generic path: first array in cache.state
+    state: object = getattr(cache, "state", None)
+    if isinstance(state, (list, tuple)):
+        for leaf in list(cast(list[object], state)):
+            if isinstance(leaf, mx.array):
+                return leaf
+    return None
+
+
+def _set_cache_dep_anchor(cache: object, value: mx.array) -> None:
+    """Write back a dependency-updated anchor array to ``cache``.
+
+    Mirrors :func:`_cache_dep_anchor`: if the cache has ``.keys`` we set
+    it directly; otherwise we update the first matching private attribute
+    from the state pytree.
+    """
+    if hasattr(cache, "keys") and isinstance(getattr(cache, "keys", None), mx.array):
+        cache.keys = value  # type: ignore[attr-defined]
+        return
+
+    # Walk common private attributes that back the state pytree.
+    for attr in ("_key_indices", "_pending_keys"):
+        if hasattr(cache, attr) and isinstance(getattr(cache, attr, None), mx.array):
+            setattr(cache, attr, value)
+            return
+
+
 class _LayerCallable(Protocol):
     """Structural type that any compatible layer must satisfy.
 
@@ -302,33 +342,43 @@ def __call__(self, x: mx.array, *args: object, **kwargs: object) -> mx.array:
                     output = mx.distributed.send(
                         output, (self.r + 1) % self.s, group=self.group
                     )
+            _dep_array: mx.array | None = None
             if cache is not None:
                 # CacheList (used by MLA models like DeepSeekV32, GLM MoE DSA)
                 # doesn't have .keys directly; access via first sub-cache.
-                _cache = cache[0] if hasattr(cache, "caches") else cache  # type: ignore
-                if hasattr(_cache, "keys"):  # pyright: ignore[reportAny]
-                    _cache.keys = mx.depends(_cache.keys, output)  # type: ignore
+                _cache: object = cache[0] if hasattr(cache, "caches") else cache  # type: ignore[index]
+                _dep_array = _cache_dep_anchor(_cache)
+                if _dep_array is not None:
+                    _set_cache_dep_anchor(
+                        _cache, mx.depends(_dep_array, output)
+                    )
             with _hang_debug_watch(
                 f"pipeline_last eval_send rank={self.r} dst={(self.r + 1) % self.s} prefill={self.is_prefill}"
             ):
                 mx.eval(output)
-            if cache is not None and hasattr(_cache, "keys"):  # type: ignore
+            if _dep_array is not None:
                 with _hang_debug_watch(
                     f"pipeline_last eval_cache_dep rank={self.r} dst={(self.r + 1) % self.s} prefill={self.is_prefill}"
                 ):
-                    mx.eval(_cache.keys)  # type: ignore
+                    mx.eval(_cache_dep_anchor(_cache))  # type: ignore[possibly-undefined]
 
         if not self.is_prefill:
+            seq_len = output.shape[0]
             with _hang_debug_watch(
                 f"pipeline_last all_gather rank={self.r} world={self.s}"
             ):
-                output = mx.distributed.all_gather(output, group=self.group)[
-                    -output.shape[0] :
-                ]
+                # Evaluate the full all_gather BEFORE slicing. When the last
+                # rank slices [-seq_len:] it gets its own local data, and MLX's
+                # lazy evaluator can satisfy that without actually executing the
+                # collective — leaving all other ranks blocked forever. Forcing
+                # eval on the unsliced tensor ensures every rank participates in
+                # the communication before any rank proceeds.
+                gathered = mx.distributed.all_gather(output, group=self.group)
             with _hang_debug_watch(
                 f"pipeline_last eval_all_gather rank={self.r} world={self.s}"
             ):
-                mx.eval(output)
+                mx.eval(gathered)
+            output = gathered[-seq_len:]
 
         return output
 
@@ -564,11 +614,14 @@ def patched_call(
             # directly, so pipeline-parallel Gemma 4 needs this patch on both
             # the top-level model and the nested language model.
             if cache is not None:
-                logits_array = logits.logits if hasattr(logits, "logits") else logits  # type: ignore[attr-defined]
-                last = cache[-1]  # type: ignore[index]
-                dep_cache = last[0] if hasattr(last, "caches") else last  # type: ignore[index]
-                if hasattr(dep_cache, "keys") and dep_cache.keys is not None:  # type: ignore[attr-defined]
-                    dep_cache.keys = mx.depends(dep_cache.keys, logits_array)  # type: ignore[attr-defined]
+                logits_array: mx.array = logits.logits if hasattr(logits, "logits") else logits  # type: ignore[attr-defined]
+                last = cast(object, cache[-1])
+                dep_cache = cast(object, last[0] if hasattr(last, "caches") else last)  # pyright: ignore[reportIndexIssue]
+                anchor = _cache_dep_anchor(dep_cache)
+                if anchor is not None:
+                    _set_cache_dep_anchor(
+                        dep_cache, mx.depends(anchor, cast(mx.array, logits_array))
+                    )
 
             return logits
 

From a43ed8296def459e1d30292578c98cb21fe4c928 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 17:23:59 -0500
Subject: [PATCH 09/21] Replace decode all_gather with point-to-point
 broadcast, fix warmup sampling
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The pipeline decode path used all_gather to broadcast the last rank's output
to all ranks. This collective was prone to deadlock during warmup:
generate_step prefetches one decode step ahead, and when the warmup generator
is abandoned mid-iteration, the orphaned collective can mismatch with the
post-warmup barrier, leaving one rank blocked indefinitely.

Replace all_gather with point-to-point send/recv from the last rank to each
other rank. Bilateral operations cannot deadlock from collective-ordering
mismatches. Also switch distributed warmup to greedy sampling (temperature=0)
so every rank samples the same token from the shared logits — stochastic
sampling with per-rank random states can diverge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 src/exo/worker/engines/mlx/auto_parallel.py   | 59 ++++++++++++-------
 .../worker/engines/mlx/generator/generate.py  | 20 +++++--
 .../unittests/test_mlx/test_warmup_request.py | 36 +++++++----
 3 files changed, 78 insertions(+), 37 deletions(-)

diff --git a/src/exo/worker/engines/mlx/auto_parallel.py b/src/exo/worker/engines/mlx/auto_parallel.py
index 34d33eb98..c93b9edac 100644
--- a/src/exo/worker/engines/mlx/auto_parallel.py
+++ b/src/exo/worker/engines/mlx/auto_parallel.py
@@ -349,9 +349,7 @@ def __call__(self, x: mx.array, *args: object, **kwargs: object) -> mx.array:
                 _cache: object = cache[0] if hasattr(cache, "caches") else cache  # type: ignore[index]
                 _dep_array = _cache_dep_anchor(_cache)
                 if _dep_array is not None:
-                    _set_cache_dep_anchor(
-                        _cache, mx.depends(_dep_array, output)
-                    )
+                    _set_cache_dep_anchor(_cache, mx.depends(_dep_array, output))
             with _hang_debug_watch(
                 f"pipeline_last eval_send rank={self.r} dst={(self.r + 1) % self.s} prefill={self.is_prefill}"
             ):
@@ -363,22 +361,41 @@ def __call__(self, x: mx.array, *args: object, **kwargs: object) -> mx.array:
                     mx.eval(_cache_dep_anchor(_cache))  # type: ignore[possibly-undefined]
 
         if not self.is_prefill:
-            seq_len = output.shape[0]
-            with _hang_debug_watch(
-                f"pipeline_last all_gather rank={self.r} world={self.s}"
-            ):
-                # Evaluate the full all_gather BEFORE slicing. When the last
-                # rank slices [-seq_len:] it gets its own local data, and MLX's
-                # lazy evaluator can satisfy that without actually executing the
-                # collective — leaving all other ranks blocked forever. Forcing
-                # eval on the unsliced tensor ensures every rank participates in
-                # the communication before any rank proceeds.
-                gathered = mx.distributed.all_gather(output, group=self.group)
-            with _hang_debug_watch(
-                f"pipeline_last eval_all_gather rank={self.r} world={self.s}"
-            ):
-                mx.eval(gathered)
-            output = gathered[-seq_len:]
+            # Broadcast the last rank's output to all other ranks so every
+            # rank can apply lm_head and sample the same token.
+            #
+            # Previously this used all_gather, but the collective was prone
+            # to deadlock: generate_step prefetches one decode step ahead,
+            # and when the warmup generator is abandoned mid-iteration the
+            # orphaned collective from the prefetch can mismatch with the
+            # post-warmup barrier/all_gather, leaving one rank blocked.
+            #
+            # Point-to-point send/recv between specific rank pairs cannot
+            # deadlock from collective-ordering mismatches because each
+            # operation is matched bilaterally.
+            if self.r == self.s - 1:
+                # Last rank: send output to every other rank.
+                for dst in range(self.s - 1):
+                    with _hang_debug_watch(
+                        f"pipeline_last broadcast_send rank={self.r} dst={dst} world={self.s}"
+                    ):
+                        sent = mx.distributed.send(output, dst, group=self.group)
+                    with _hang_debug_watch(
+                        f"pipeline_last eval_broadcast_send rank={self.r} dst={dst} world={self.s}"
+                    ):
+                        mx.eval(sent)
+            else:
+                # Non-last rank: receive the last rank's output.
+                with _hang_debug_watch(
+                    f"pipeline_last broadcast_recv rank={self.r} src={self.s - 1} world={self.s}"
+                ):
+                    output = mx.distributed.recv_like(
+                        output, self.s - 1, group=self.group
+                    )
+                with _hang_debug_watch(
+                    f"pipeline_last eval_broadcast_recv rank={self.r} src={self.s - 1} world={self.s}"
+                ):
+                    mx.eval(output)
 
         return output
 
@@ -614,7 +631,9 @@ def patched_call(
             # directly, so pipeline-parallel Gemma 4 needs this patch on both
             # the top-level model and the nested language model.
             if cache is not None:
-                logits_array: mx.array = logits.logits if hasattr(logits, "logits") else logits  # type: ignore[attr-defined]
+                logits_array: mx.array = (
+                    logits.logits if hasattr(logits, "logits") else logits
+                )  # type: ignore[attr-defined]
                 last = cast(object, cache[-1])
                 dep_cache = cast(object, last[0] if hasattr(last, "caches") else last)  # pyright: ignore[reportIndexIssue]
                 anchor = _cache_dep_anchor(dep_cache)
diff --git a/src/exo/worker/engines/mlx/generator/generate.py b/src/exo/worker/engines/mlx/generator/generate.py
index 6378daa52..2a6b15a2b 100644
--- a/src/exo/worker/engines/mlx/generator/generate.py
+++ b/src/exo/worker/engines/mlx/generator/generate.py
@@ -233,7 +233,9 @@ def _slice_native_pixel_values_for_uncached_suffix(
         return pixel_values
 
     available_images = (
-        len(pixel_values) if isinstance(pixel_values, list) else int(pixel_values.shape[0])
+        len(pixel_values)
+        if isinstance(pixel_values, list)
+        else int(pixel_values.shape[0])
     )
     remaining_indices = [
         idx
@@ -647,6 +649,12 @@ def warmup_inference(
             "prompt and ignoring warmup shaping overrides"
         )
 
+    # Distributed pipeline warmup MUST use greedy sampling so every rank
+    # samples the same token from the shared logits.  Stochastic sampling
+    # with per-rank random states can diverge, causing ranks to follow
+    # different code paths inside generate_step and issue mismatched
+    # collective operations.
+    is_distributed = _is_distributed_warmup(group)
     warmup_task_params = TextGenerationTaskParams(
         model=model_id,
         # Distributed pipeline warmup always uses the minimal sanity-check
@@ -660,9 +668,9 @@ def warmup_inference(
         ],
         max_output_tokens=1024,
         enable_thinking=False,
-        temperature=1.0,
-        top_p=0.95,
-        top_k=64,
+        temperature=0.0 if is_distributed else 1.0,
+        top_p=1.0 if is_distributed else 0.95,
+        top_k=0 if is_distributed else 64,
     )
 
     with _hang_debug_watch(f"warmup apply_chat_template model={model_id}"):
@@ -957,7 +965,9 @@ def _mlx_generate_native_vision(
         accumulated_text += final_text
 
     generation_elapsed = time.perf_counter() - generation_start_time
-    generation_tps = completion_tokens / generation_elapsed if generation_elapsed > 0 else 0.0
+    generation_tps = (
+        completion_tokens / generation_elapsed if generation_elapsed > 0 else 0.0
+    )
 
     final_stats = GenerationStats(
         prompt_tps=float(prompt_tps),
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
index dcf1f7e1d..98be07d4c 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
@@ -33,7 +33,9 @@ def test_warmup_inference_uses_safe_default_user_content_without_instructions(
 ) -> None:
     captured: dict[str, object] = {}
 
-    def fake_apply_chat_template(*, tokenizer: object, task_params: object, model_card: object):
+    def fake_apply_chat_template(
+        *, tokenizer: object, task_params: object, model_card: object
+    ):
         del tokenizer, model_card
         captured["task_params"] = task_params
         return "warmup prompt"
@@ -81,9 +83,11 @@ def fake_log_request_shape(
     assert check_every == 0
     assert task_params.instructions is None
     assert task_params.enable_thinking is False
-    assert task_params.temperature == 1.0
-    assert task_params.top_p == 0.95
-    assert task_params.top_k == 64
+    # Distributed warmup uses greedy sampling so every rank samples the same
+    # token from the shared logits (stochastic sampling can diverge).
+    assert task_params.temperature == 0.0
+    assert task_params.top_p == 1.0
+    assert task_params.top_k == 0
     assert task_params.max_output_tokens == 1024
     first_message = task_params.input[0]
     assert first_message.content == "hello"
@@ -101,7 +105,9 @@ def test_warmup_inference_ignores_repeat_count_override_for_pipeline_groups(
 ) -> None:
     captured: dict[str, object] = {}
 
-    def fake_apply_chat_template(*, tokenizer: object, task_params: object, model_card: object):
+    def fake_apply_chat_template(
+        *, tokenizer: object, task_params: object, model_card: object
+    ):
         del tokenizer, model_card
         captured["task_params"] = task_params
         return "warmup prompt"
@@ -142,7 +148,9 @@ def test_warmup_inference_ignores_instruction_override_for_pipeline_groups(
 ) -> None:
     captured: dict[str, object] = {}
 
-    def fake_apply_chat_template(*, tokenizer: object, task_params: object, model_card: object):
+    def fake_apply_chat_template(
+        *, tokenizer: object, task_params: object, model_card: object
+    ):
         del tokenizer, model_card
         captured["task_params"] = task_params
         return "warmup prompt"
@@ -183,7 +191,9 @@ def test_warmup_inference_honors_repeat_and_instruction_overrides_for_single_nod
 ) -> None:
     captured: dict[str, object] = {}
 
-    def fake_apply_chat_template(*, tokenizer: object, task_params: object, model_card: object):
+    def fake_apply_chat_template(
+        *, tokenizer: object, task_params: object, model_card: object
+    ):
         del tokenizer, model_card
         captured["task_params"] = task_params
         return "warmup prompt"
@@ -229,7 +239,9 @@ def test_warmup_inference_stops_after_first_generated_token(
 ) -> None:
     generated_tokens = 0
 
-    def fake_apply_chat_template(*, tokenizer: object, task_params: object, model_card: object):
+    def fake_apply_chat_template(
+        *, tokenizer: object, task_params: object, model_card: object
+    ):
         del tokenizer, task_params, model_card
         return "warmup prompt"
 
@@ -274,9 +286,7 @@ def test_warmup_helpers_prefer_blank_skulk_values_over_legacy_env(
     monkeypatch.setenv("EXO_DEBUG_WARMUP_INCLUDE_INSTRUCTIONS", "1")
 
     assert generate_mod._warmup_repeat_count() == 1  # pyright: ignore[reportPrivateUsage]
-    assert (
-        generate_mod._warmup_instructions(cast(object, _SingleNodeGroup())) is None
-    )  # pyright: ignore[reportPrivateUsage]
+    assert generate_mod._warmup_instructions(cast(object, _SingleNodeGroup())) is None  # pyright: ignore[reportPrivateUsage]
 
 
 def test_warmup_inference_enforces_minimum_cancel_check_interval_on_slow_start(
@@ -284,7 +294,9 @@ def test_warmup_inference_enforces_minimum_cancel_check_interval_on_slow_start(
 ) -> None:
     monotonic_values = iter([100.0, 112.0])
 
-    def fake_apply_chat_template(*, tokenizer: object, task_params: object, model_card: object):
+    def fake_apply_chat_template(
+        *, tokenizer: object, task_params: object, model_card: object
+    ):
         del tokenizer, task_params, model_card
         return "warmup prompt"
 

From 7ba35fc6ef310bf1740bf9722703511b476f075a Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 18:58:24 -0500
Subject: [PATCH 10/21] Add diagnostic logging to agree_on_tasks and
 mx_all_gather_tasks

Inference hangs after warmup succeeds. Add logging to pinpoint whether
the hang is in the task agreement collectives (all_gather) or later
in the generation pipeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 src/exo/worker/engines/mlx/utils_mlx.py                | 9 ++++++---
 src/exo/worker/runner/llm_inference/batch_generator.py | 8 ++++++++
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/src/exo/worker/engines/mlx/utils_mlx.py b/src/exo/worker/engines/mlx/utils_mlx.py
index f1a411e85..e0bfe1f94 100644
--- a/src/exo/worker/engines/mlx/utils_mlx.py
+++ b/src/exo/worker/engines/mlx/utils_mlx.py
@@ -117,8 +117,7 @@ def log_request_shape(
         payload.update(extra)
 
     logger.info(
-        "[request-shape] "
-        f"{json.dumps(payload, ensure_ascii=True, sort_keys=True)}"
+        f"[request-shape] {json.dumps(payload, ensure_ascii=True, sort_keys=True)}"
     )
     logger.info(f"[request-shape] prompt label={label}\n{prompt}")
 
@@ -287,7 +286,9 @@ def _encode_one(self, pixel_values: mx.array) -> mx.array:
         hidden_states = mx.concatenate(all_real, axis=0)[None]
 
         if getattr(getattr(self._inner, "config", None), "standardize", False):
-            hidden_states = (hidden_states - self._inner.std_bias) * self._inner.std_scale
+            hidden_states = (
+                hidden_states - self._inner.std_bias
+            ) * self._inner.std_scale
 
         return hidden_states
 
@@ -1210,10 +1211,12 @@ def decode_task_id(encoded_task_id: list[int]) -> TaskId:
     uuid_byte_length = 36
 
     n_tasks = len(tasks)
+    logger.info(f"mx_all_gather_tasks: gathering counts (n_tasks={n_tasks})")
     all_counts = cast(
         list[int],
         mx.distributed.all_gather(mx.array([n_tasks]), group=group).tolist(),
     )
+    logger.info(f"mx_all_gather_tasks: counts gathered: {all_counts}")
     max_tasks = max(all_counts)
     world_size: int = 1 if group is None else group.size()
 
diff --git a/src/exo/worker/runner/llm_inference/batch_generator.py b/src/exo/worker/runner/llm_inference/batch_generator.py
index d0dc44a11..cf146841c 100644
--- a/src/exo/worker/runner/llm_inference/batch_generator.py
+++ b/src/exo/worker/runner/llm_inference/batch_generator.py
@@ -165,7 +165,11 @@ def submit(
 
     def agree_on_tasks(self) -> None:
         """Agree between all ranks about the task ordering (some may have received in different order or not at all)."""
+        logger.info(
+            f"agree_on_tasks: entering with {len(self._maybe_queue)} pending tasks"
+        )
         agreed, different = mx_all_gather_tasks(self._maybe_queue, self.group)
+        logger.info(f"agree_on_tasks: {len(agreed)} agreed, {len(different)} different")
         self._queue.extend(task for task in self._maybe_queue if task in agreed)
         self._maybe_queue = [task for task in self._maybe_queue if task in different]
 
@@ -395,7 +399,11 @@ def submit(
 
     def agree_on_tasks(self) -> None:
         """Agree between all ranks about the task ordering (some may have received in different order or not at all)."""
+        logger.info(
+            f"agree_on_tasks: entering with {len(self._maybe_queue)} pending tasks"
+        )
         agreed, different = mx_all_gather_tasks(self._maybe_queue, self.group)
+        logger.info(f"agree_on_tasks: {len(agreed)} agreed, {len(different)} different")
         self._queue.extend(task for task in self._maybe_queue if task in agreed)
         self._maybe_queue = [task for task in self._maybe_queue if task in different]
 

From f41b97dbe6a55c24cc9d1bd40d3ab5f9a2b536e9 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 19:36:43 -0500
Subject: [PATCH 11/21] Replace decode broadcast send/recv with all_sum to fix
 JACCL corruption
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The bilateral send/recv used to broadcast the last rank's decode output
to all other ranks was corrupting JACCL's distributed state (mlx#3149).
After warmup, subsequent all_gather collectives returned 0x7FC00000
(float32 NaN) for receiver ranks, causing task agreement to see garbage
counts like [2143289344, 1, 2143289344] and hang.

Fix: last rank contributes its real output, other ranks contribute zeros,
all_sum gives every rank the correct result. Single collective — no
bilateral matching, no orphan mismatch from generate_step prefetch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 src/exo/worker/engines/mlx/auto_parallel.py | 48 +++++++--------------
 1 file changed, 16 insertions(+), 32 deletions(-)

diff --git a/src/exo/worker/engines/mlx/auto_parallel.py b/src/exo/worker/engines/mlx/auto_parallel.py
index c93b9edac..f17bed102 100644
--- a/src/exo/worker/engines/mlx/auto_parallel.py
+++ b/src/exo/worker/engines/mlx/auto_parallel.py
@@ -364,38 +364,22 @@ def __call__(self, x: mx.array, *args: object, **kwargs: object) -> mx.array:
             # Broadcast the last rank's output to all other ranks so every
             # rank can apply lm_head and sample the same token.
             #
-            # Previously this used all_gather, but the collective was prone
-            # to deadlock: generate_step prefetches one decode step ahead,
-            # and when the warmup generator is abandoned mid-iteration the
-            # orphaned collective from the prefetch can mismatch with the
-            # post-warmup barrier/all_gather, leaving one rank blocked.
-            #
-            # Point-to-point send/recv between specific rank pairs cannot
-            # deadlock from collective-ordering mismatches because each
-            # operation is matched bilaterally.
-            if self.r == self.s - 1:
-                # Last rank: send output to every other rank.
-                for dst in range(self.s - 1):
-                    with _hang_debug_watch(
-                        f"pipeline_last broadcast_send rank={self.r} dst={dst} world={self.s}"
-                    ):
-                        sent = mx.distributed.send(output, dst, group=self.group)
-                    with _hang_debug_watch(
-                        f"pipeline_last eval_broadcast_send rank={self.r} dst={dst} world={self.s}"
-                    ):
-                        mx.eval(sent)
-            else:
-                # Non-last rank: receive the last rank's output.
-                with _hang_debug_watch(
-                    f"pipeline_last broadcast_recv rank={self.r} src={self.s - 1} world={self.s}"
-                ):
-                    output = mx.distributed.recv_like(
-                        output, self.s - 1, group=self.group
-                    )
-                with _hang_debug_watch(
-                    f"pipeline_last eval_broadcast_recv rank={self.r} src={self.s - 1} world={self.s}"
-                ):
-                    mx.eval(output)
+            # Uses all_sum instead of point-to-point send/recv: the last rank
+            # contributes its real output while other ranks contribute zeros.
+            # This avoids JACCL send/recv corruption (mlx#3149) that caused
+            # subsequent all_gather collectives to return NaN/garbage after
+            # warmup — and unlike all_gather, orphaned prefetch steps from
+            # generate_step's one-step-ahead prefetch can't deadlock because
+            # all_sum is commutative and tolerates collective reordering.
+            contribution = output if self.r == self.s - 1 else mx.zeros_like(output)
+            with _hang_debug_watch(
+                f"pipeline_last all_sum_broadcast rank={self.r} world={self.s}"
+            ):
+                output = mx.distributed.all_sum(contribution, group=self.group)
+            with _hang_debug_watch(
+                f"pipeline_last eval_all_sum_broadcast rank={self.r} world={self.s}"
+            ):
+                mx.eval(output)
 
         return output
 

From 3d526b7cea6af54f0baf4328257b5fdaabdef8e6 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 20:19:32 -0500
Subject: [PATCH 12/21] Move decode broadcast all_sum to CPU stream to fix
 3-node JACCL hang
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

After the clean cluster restart confirmed Issue 4's prefill hang was
NOT from corrupted Metal state, the new logs revealed a different hang:
rank 0 completed all decode all_sums and reached the CPU-stream final
barrier, while ranks 1 and 2 were stuck in the GPU-stream all_sum inside
PipelineLastLayer.  This is a collective mismatch — JACCL appears to
have trouble when point-to-point send/recv and collective all_sum share
the same GPU stream on 3-node pipelines.

Fix: route the decode broadcast all_sum through the CPU stream (the same
stream mx_barrier uses, which has been reliable across all prior tests).
This isolates the broadcast collective from pipeline point-to-point
traffic on the GPU stream.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 src/exo/worker/engines/mlx/auto_parallel.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/src/exo/worker/engines/mlx/auto_parallel.py b/src/exo/worker/engines/mlx/auto_parallel.py
index f17bed102..fc4833c7a 100644
--- a/src/exo/worker/engines/mlx/auto_parallel.py
+++ b/src/exo/worker/engines/mlx/auto_parallel.py
@@ -371,11 +371,20 @@ def __call__(self, x: mx.array, *args: object, **kwargs: object) -> mx.array:
             # warmup — and unlike all_gather, orphaned prefetch steps from
             # generate_step's one-step-ahead prefetch can't deadlock because
             # all_sum is commutative and tolerates collective reordering.
+            #
+            # The all_sum runs on the CPU stream rather than the default GPU
+            # stream to avoid a JACCL hang observed on 3-node pipelines when
+            # send/recv and all_sum share the same GPU stream.  CPU-stream
+            # collectives (used by mx_barrier) are proven reliable; this
+            # isolates the broadcast from pipeline point-to-point traffic.
             contribution = output if self.r == self.s - 1 else mx.zeros_like(output)
+            cpu_stream = mx.default_stream(mx.Device(mx.cpu))
             with _hang_debug_watch(
                 f"pipeline_last all_sum_broadcast rank={self.r} world={self.s}"
             ):
-                output = mx.distributed.all_sum(contribution, group=self.group)
+                output = mx.distributed.all_sum(
+                    contribution, group=self.group, stream=cpu_stream
+                )
             with _hang_debug_watch(
                 f"pipeline_last eval_all_sum_broadcast rank={self.r} world={self.s}"
             ):

From 31002071511082ec2795993d0ea913a45fcf2973 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Thu, 9 Apr 2026 23:37:08 -0500
Subject: [PATCH 13/21] Fix JACCL deadlock: run mx_all_gather_tasks collectives
 on CPU stream
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

mx_all_gather_tasks was inheriting whatever stream context the caller
used.  When called from the distributed_prompt_progress_callback during
prefill, it ran inside mlx_lm's `with mx.stream(generation_stream):`
— a non-default GPU stream.  Meanwhile, mx_any (called immediately
before from agree_on_cancellations) explicitly uses the CPU stream.

Mixing collective streams (CPU for all_sum, generation_stream for
all_gather) causes JACCL to mismatch collective ordering across ranks.
The first all_gather (cancel counts) succeeds, but the second
(task counts) deadlocks because ranks see different collective
sequences on the generation_stream.

Fix: explicitly route all_gather to the CPU stream, matching the
pattern already used by mx_any and mx_barrier.  This keeps all
distributed collectives on a single, proven-reliable stream regardless
of the caller's stream context.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 src/exo/worker/engines/mlx/utils_mlx.py | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/src/exo/worker/engines/mlx/utils_mlx.py b/src/exo/worker/engines/mlx/utils_mlx.py
index e0bfe1f94..9419bbca9 100644
--- a/src/exo/worker/engines/mlx/utils_mlx.py
+++ b/src/exo/worker/engines/mlx/utils_mlx.py
@@ -1197,6 +1197,18 @@ def mx_all_gather_tasks(
     tasks: list[TextGeneration],
     group: mx.distributed.Group | None,
 ) -> tuple[list[TextGeneration], list[TextGeneration]]:
+    """Synchronize task lists across all ranks via all_gather.
+
+    All collectives run on the CPU stream to avoid JACCL hangs when called
+    inside mlx_lm's ``with mx.stream(generation_stream):`` context.  The
+    generation_stream is a non-default GPU stream; mixing collective streams
+    (CPU for ``mx_any``, GPU-generation for ``all_gather``) causes JACCL to
+    mismatch collective ordering across ranks, deadlocking the second
+    all_gather.  Using the CPU stream for everything — matching ``mx_any``
+    and ``mx_barrier`` — keeps all collectives on a single, proven-reliable
+    stream.
+    """
+
     def encode_task_id(task_id: TaskId) -> list[int]:
         utf8_task_id = task_id.encode()
         return [
@@ -1209,12 +1221,15 @@ def decode_task_id(encoded_task_id: list[int]) -> TaskId:
         )
 
     uuid_byte_length = 36
+    cpu_stream = mx.default_stream(mx.Device(mx.cpu))
 
     n_tasks = len(tasks)
     logger.info(f"mx_all_gather_tasks: gathering counts (n_tasks={n_tasks})")
     all_counts = cast(
         list[int],
-        mx.distributed.all_gather(mx.array([n_tasks]), group=group).tolist(),
+        mx.distributed.all_gather(
+            mx.array([n_tasks]), group=group, stream=cpu_stream
+        ).tolist(),
     )
     logger.info(f"mx_all_gather_tasks: counts gathered: {all_counts}")
     max_tasks = max(all_counts)
@@ -1231,7 +1246,9 @@ def decode_task_id(encoded_task_id: list[int]) -> TaskId:
 
     gathered = cast(
         list[list[list[int]]],
-        mx.distributed.all_gather(mx.array(padded), group=group)
+        mx.distributed.all_gather(
+            mx.array(padded), group=group, stream=cpu_stream
+        )
         .reshape(world_size, max_tasks, -1)
         .tolist(),
     )

From bae6455a4cf3f912546a3f1a7f07ae37d2f4ade9 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Fri, 10 Apr 2026 00:18:14 -0500
Subject: [PATCH 14/21] Fix prefill deadlock: skip distributed callback in
 stream_generate path
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The distributed_prompt_progress_callback (agree_on_tasks/cancellations)
does all_gather collectives. When called from stream_generate's progress
callback, these collectives deadlock on 3-node pipelines because:

1. mlx_lm's generate_step prefetches the next decode token via _step(y)
   with mx.async_eval BEFORE firing the callback. That prefetch dispatches
   pipeline communication (sends/recvs or all_sum) to generation_stream.
   JACCL cannot run an all_gather while other collectives are in-flight.

2. Warmup also uses stream_generate and abandons the generator after one
   token, potentially leaving generation_stream in an inconsistent state
   for subsequent collectives.

3. For short prompts (< effective_prefill_step_size) the entire prefill
   takes milliseconds — no meaningful window for task/cancellation checks.

Fix: pass only the local progress_callback (no distributed collectives)
to stream_generate. pipeline_parallel_prefill keeps its distributed
callback since it handles inter-chunk synchronization explicitly.

Also keeps the earlier CPU stream fix for mx_all_gather_tasks to protect
any remaining callers that run inside non-default stream contexts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 .../worker/engines/mlx/generator/generate.py  | 26 ++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/src/exo/worker/engines/mlx/generator/generate.py b/src/exo/worker/engines/mlx/generator/generate.py
index 2a6b15a2b..5405f9a9b 100644
--- a/src/exo/worker/engines/mlx/generator/generate.py
+++ b/src/exo/worker/engines/mlx/generator/generate.py
@@ -574,6 +574,30 @@ def combined_progress_callback(processed: int, total: int) -> None:
                     group=group,
                 )
         else:
+            # stream_generate is used for short prompts that fit in a single
+            # pipeline chunk.  We MUST NOT pass the distributed callback here:
+            #
+            # 1. mlx_lm's generate_step calls prompt_progress_callback inside
+            #    ``with mx.stream(generation_stream):``, and on the first decode
+            #    iteration it prefetches the next token via ``_step(y)`` with
+            #    ``mx.async_eval`` BEFORE firing the callback.  That prefetch
+            #    dispatches pipeline sends/recvs to generation_stream.  JACCL
+            #    cannot run an all_gather while sends/recvs are in-flight on
+            #    any stream — the collective deadlocks.
+            #
+            # 2. Warmup also uses stream_generate and abandons the generator
+            #    after one token.  The prefetched decode step leaves orphaned
+            #    sends/recvs on generation_stream.  The post-warmup barrier
+            #    synchronizes CPU stream but not generation_stream, so those
+            #    orphaned ops are still in-flight when real inference starts.
+            #
+            # 3. For short prompts the entire prefill takes milliseconds —
+            #    there is no meaningful window for task/cancellation checking.
+            #
+            # pipeline_parallel_prefill handles its own inter-chunk
+            # synchronization and safely calls the distributed callback
+            # between chunks, so it keeps the callback.
+            #
             # Use max_tokens=1 because max_tokens=0 does not work.
             # We just throw away the generated token - we only care about filling the cache
             with _hang_debug_watch(
@@ -589,7 +613,7 @@ def combined_progress_callback(processed: int, total: int) -> None:
                     prefill_step_size=prefill_step_size,
                     kv_group_size=KV_GROUP_SIZE,
                     kv_bits=KV_BITS,
-                    prompt_progress_callback=combined_progress_callback,
+                    prompt_progress_callback=progress_callback,
                 ):
                     logger.info(
                         f"Prefill stream_generate yielded first token (rank={rank})"

From c5ec4727d491638ca4e71d37f01219b08352e331 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <kite3@kite3.local>
Date: Fri, 10 Apr 2026 00:34:28 -0500
Subject: [PATCH 15/21] Replace all JACCL all_gather with all_sum to fix 3-node
 deadlocks
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

JACCL's all_gather deadlocks intermittently on 3-node pipelines — the
first all_gather in a sequence may succeed but subsequent ones hang
indefinitely. all_sum is proven reliable across the entire codebase
(mx_any, mx_barrier, PipelineLastLayer decode broadcast).

Replace all_gather everywhere with all_sum using the zero-contribution
slot pattern: each rank fills its own slot in a [world_size, ...] tensor
and contributes zeros elsewhere. all_sum merges non-overlapping slots,
giving the same result as all_gather without the JACCL deadlock.

Changed:
- mx_all_gather_tasks: both count and task-ID exchanges now use all_sum
- warmup check_for_cancel_every: slotted all_sum instead of all_gather
- KVCacheMemoryPressureTracker: slotted all_sum for pressure exchange
- Tests: update mocks from fake_all_gather to fake_all_sum

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 dashboard-react/package-lock.json             | 14 -----
 src/exo/worker/engines/mlx/cache.py           | 19 +++++--
 .../worker/engines/mlx/generator/generate.py  | 19 ++++---
 src/exo/worker/engines/mlx/utils_mlx.py       | 57 ++++++++++++-------
 .../unittests/test_mlx/test_warmup_request.py | 42 ++++++++------
 5 files changed, 88 insertions(+), 63 deletions(-)

diff --git a/dashboard-react/package-lock.json b/dashboard-react/package-lock.json
index a190905fa..c4cd8c2b8 100644
--- a/dashboard-react/package-lock.json
+++ b/dashboard-react/package-lock.json
@@ -48,7 +48,6 @@
         "redoc": "2.5.2",
         "storybook": "^10.3.3",
         "typedoc": "^0.28.14",
-        "typedoc-plugin-markdown": "^4.11.0",
         "typescript": "~5.9.3",
         "typescript-eslint": "^8.57.0",
         "vite": "^8.0.1",
@@ -7437,19 +7436,6 @@
         "typescript": "5.0.x || 5.1.x || 5.2.x || 5.3.x || 5.4.x || 5.5.x || 5.6.x || 5.7.x || 5.8.x || 5.9.x || 6.0.x"
       }
     },
-    "node_modules/typedoc-plugin-markdown": {
-      "version": "4.11.0",
-      "resolved": "https://registry.npmjs.org/typedoc-plugin-markdown/-/typedoc-plugin-markdown-4.11.0.tgz",
-      "integrity": "sha512-2iunh2ALyfyh204OF7h2u0kuQ84xB3jFZtFyUr01nThJkLvR8oGGSSDlyt2gyO4kXhvUxDcVbO0y43+qX+wFbw==",
-      "dev": true,
-      "license": "MIT",
-      "engines": {
-        "node": ">= 18"
-      },
-      "peerDependencies": {
-        "typedoc": "0.28.x"
-      }
-    },
     "node_modules/typedoc/node_modules/balanced-match": {
       "version": "4.0.4",
       "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz",
diff --git a/src/exo/worker/engines/mlx/cache.py b/src/exo/worker/engines/mlx/cache.py
index 474f090d4..78da25272 100644
--- a/src/exo/worker/engines/mlx/cache.py
+++ b/src/exo/worker/engines/mlx/cache.py
@@ -344,17 +344,28 @@ def _evict_if_needed(self):
             )
 
     def get_memory_used_percentage(self) -> float:
+        """Return the maximum memory pressure across all ranks.
+
+        Uses all_sum with slotted contributions instead of all_gather.
+        JACCL all_gather deadlocks intermittently on 3-node pipelines;
+        all_sum is proven reliable.
+        """
         local_pressure: float = get_memory_used_percentage()
 
         if self._group is None:
             return local_pressure
 
-        all_pressure = mx.distributed.all_gather(
-            mx.array([local_pressure], dtype=mx.float32),
+        world_size = self._group.size()
+        rank = self._group.rank()
+        slots = [0.0] * world_size
+        slots[rank] = local_pressure
+        cpu_stream = mx.default_stream(mx.Device(mx.cpu))
+        merged = mx.distributed.all_sum(
+            mx.array(slots, dtype=mx.float32),
             group=self._group,
+            stream=cpu_stream,
         )
-        # .item() evals.
-        max_pressure = float(mx.max(all_pressure).item())
+        max_pressure = float(mx.max(merged).item())
         return max_pressure
 
 
diff --git a/src/exo/worker/engines/mlx/generator/generate.py b/src/exo/worker/engines/mlx/generator/generate.py
index 5405f9a9b..d2b8e3f1b 100644
--- a/src/exo/worker/engines/mlx/generator/generate.py
+++ b/src/exo/worker/engines/mlx/generator/generate.py
@@ -761,14 +761,19 @@ def warmup_inference(
 
     logger.info(f"warmed up by generating {tokens_generated} tokens")
     if group is not None:
-        check_for_cancel_every = int(
-            mx.max(
-                mx.distributed.all_gather(
-                    mx.array([check_for_cancel_every]),
-                    group=group,
-                )
-            ).item()
+        # Use all_sum with slotted contribution instead of all_gather.
+        # JACCL all_gather deadlocks intermittently on 3-node pipelines;
+        # all_sum is proven reliable.  Each rank puts its value in its
+        # own slot; we take the max across all slots.
+        world_size = group.size()
+        rank = group.rank()
+        slots = [0] * world_size
+        slots[rank] = check_for_cancel_every
+        cpu_stream = mx.default_stream(mx.Device(mx.cpu))
+        merged = mx.distributed.all_sum(
+            mx.array(slots, dtype=mx.int32), group=group, stream=cpu_stream
         )
+        check_for_cancel_every = int(mx.max(merged).item())
 
     logger.info(
         f"runner checking for cancellation every {check_for_cancel_every} tokens"
diff --git a/src/exo/worker/engines/mlx/utils_mlx.py b/src/exo/worker/engines/mlx/utils_mlx.py
index 9419bbca9..488c46368 100644
--- a/src/exo/worker/engines/mlx/utils_mlx.py
+++ b/src/exo/worker/engines/mlx/utils_mlx.py
@@ -1197,16 +1197,17 @@ def mx_all_gather_tasks(
     tasks: list[TextGeneration],
     group: mx.distributed.Group | None,
 ) -> tuple[list[TextGeneration], list[TextGeneration]]:
-    """Synchronize task lists across all ranks via all_gather.
-
-    All collectives run on the CPU stream to avoid JACCL hangs when called
-    inside mlx_lm's ``with mx.stream(generation_stream):`` context.  The
-    generation_stream is a non-default GPU stream; mixing collective streams
-    (CPU for ``mx_any``, GPU-generation for ``all_gather``) causes JACCL to
-    mismatch collective ordering across ranks, deadlocking the second
-    all_gather.  Using the CPU stream for everything — matching ``mx_any``
-    and ``mx_barrier`` — keeps all collectives on a single, proven-reliable
-    stream.
+    """Synchronize task lists across all ranks using only all_sum.
+
+    JACCL's ``all_gather`` deadlocks intermittently on 3-node pipelines —
+    the first all_gather in a sequence may succeed, but subsequent ones
+    hang.  ``all_sum`` is proven reliable (used by ``mx_any``,
+    ``mx_barrier``, and PipelineLastLayer decode broadcast without issue).
+
+    Strategy: use the same zero-contribution pattern as PipelineLastLayer.
+    Each rank fills its own slot in a ``[world_size, max_tasks, id_len]``
+    tensor with encoded task IDs and contributes zeros elsewhere.
+    ``all_sum`` merges them (non-overlapping slots ⇒ sum == gather).
     """
 
     def encode_task_id(task_id: TaskId) -> list[int]:
@@ -1222,36 +1223,52 @@ def decode_task_id(encoded_task_id: list[int]) -> TaskId:
 
     uuid_byte_length = 36
     cpu_stream = mx.default_stream(mx.Device(mx.cpu))
+    world_size: int = 1 if group is None else group.size()
+    rank: int = 0 if group is None else group.rank()
 
+    # --- Phase 1: exchange counts via all_sum ---
+    # Each rank contributes its count in its own slot of a [world_size]
+    # vector; other slots are zero.  all_sum merges them.
     n_tasks = len(tasks)
     logger.info(f"mx_all_gather_tasks: gathering counts (n_tasks={n_tasks})")
+    counts_vec = [0] * world_size
+    counts_vec[rank] = n_tasks
     all_counts = cast(
         list[int],
-        mx.distributed.all_gather(
-            mx.array([n_tasks]), group=group, stream=cpu_stream
+        mx.distributed.all_sum(
+            mx.array(counts_vec, dtype=mx.int32),
+            group=group,
+            stream=cpu_stream,
         ).tolist(),
     )
     logger.info(f"mx_all_gather_tasks: counts gathered: {all_counts}")
     max_tasks = max(all_counts)
-    world_size: int = 1 if group is None else group.size()
 
     if max_tasks == 0:
         return [], []
 
-    padded = [encode_task_id(task.task_id) for task in tasks] + [
+    # --- Phase 2: exchange task IDs via all_sum ---
+    # Build a [world_size, max_tasks, uuid_byte_length] tensor.  This rank
+    # fills its own slice; all other slices are zero.  all_sum merges them
+    # without overlap (each rank owns a unique slice).
+    padded_ids = [encode_task_id(task.task_id) for task in tasks] + [
         [0] * uuid_byte_length
     ] * (max_tasks - n_tasks)
+    assert all(len(eid) == uuid_byte_length for eid in padded_ids)
 
-    assert all(len(encoded_task_id) == uuid_byte_length for encoded_task_id in padded)
+    full = [[[0] * uuid_byte_length] * max_tasks for _ in range(world_size)]
+    full[rank] = padded_ids
 
+    gathered_flat = mx.distributed.all_sum(
+        mx.array(full, dtype=mx.int32),
+        group=group,
+        stream=cpu_stream,
+    )
     gathered = cast(
         list[list[list[int]]],
-        mx.distributed.all_gather(
-            mx.array(padded), group=group, stream=cpu_stream
-        )
-        .reshape(world_size, max_tasks, -1)
-        .tolist(),
+        gathered_flat.reshape(world_size, max_tasks, uuid_byte_length).tolist(),
     )
+
     all_task_ids: list[list[TaskId]] = [
         [decode_task_id(encoded_task_id) for encoded_task_id in rank_tasks[:count]]
         for rank_tasks, count in zip(gathered, all_counts, strict=True)
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
index 98be07d4c..31c2e1653 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
@@ -8,11 +8,17 @@
 
 
 class _FakeGroup:
+    def rank(self) -> int:
+        return 0
+
     def size(self) -> int:
         return 3
 
 
 class _SingleNodeGroup:
+    def rank(self) -> int:
+        return 0
+
     def size(self) -> int:
         return 1
 
@@ -48,8 +54,8 @@ def fake_mlx_generate(**_kwargs: object):
             yield None
         return
 
-    def fake_all_gather(array: object, *, group: object):
-        del group
+    def fake_all_sum(array: object, *, group: object, stream: object = None):
+        del group, stream
         return array
 
     def fake_log_request_shape(
@@ -69,7 +75,7 @@ def fake_log_request_shape(
     monkeypatch.setattr(generate_mod, "mx_barrier", fake_mx_barrier)
     monkeypatch.setattr(generate_mod, "mlx_generate", fake_mlx_generate)
     monkeypatch.setattr(generate_mod, "log_request_shape", fake_log_request_shape)
-    monkeypatch.setattr(generate_mod.mx.distributed, "all_gather", fake_all_gather)
+    monkeypatch.setattr(generate_mod.mx.distributed, "all_sum", fake_all_sum)
 
     check_every = generate_mod.warmup_inference(
         model=object(),  # type: ignore[arg-type]
@@ -120,8 +126,8 @@ def fake_mlx_generate(**_kwargs: object):
             yield None
         return
 
-    def fake_all_gather(array: object, *, group: object):
-        del group
+    def fake_all_sum(array: object, *, group: object, stream: object = None):
+        del group, stream
         return array
 
     _clear_warmup_env(monkeypatch)
@@ -129,7 +135,7 @@ def fake_all_gather(array: object, *, group: object):
     monkeypatch.setattr(generate_mod, "apply_chat_template", fake_apply_chat_template)
     monkeypatch.setattr(generate_mod, "mx_barrier", fake_mx_barrier)
     monkeypatch.setattr(generate_mod, "mlx_generate", fake_mlx_generate)
-    monkeypatch.setattr(generate_mod.mx.distributed, "all_gather", fake_all_gather)
+    monkeypatch.setattr(generate_mod.mx.distributed, "all_sum", fake_all_sum)
 
     generate_mod.warmup_inference(
         model=object(),  # type: ignore[arg-type]
@@ -163,8 +169,8 @@ def fake_mlx_generate(**_kwargs: object):
             yield None
         return
 
-    def fake_all_gather(array: object, *, group: object):
-        del group
+    def fake_all_sum(array: object, *, group: object, stream: object = None):
+        del group, stream
         return array
 
     _clear_warmup_env(monkeypatch)
@@ -172,7 +178,7 @@ def fake_all_gather(array: object, *, group: object):
     monkeypatch.setattr(generate_mod, "apply_chat_template", fake_apply_chat_template)
     monkeypatch.setattr(generate_mod, "mx_barrier", fake_mx_barrier)
     monkeypatch.setattr(generate_mod, "mlx_generate", fake_mlx_generate)
-    monkeypatch.setattr(generate_mod.mx.distributed, "all_gather", fake_all_gather)
+    monkeypatch.setattr(generate_mod.mx.distributed, "all_sum", fake_all_sum)
 
     generate_mod.warmup_inference(
         model=object(),  # type: ignore[arg-type]
@@ -206,8 +212,8 @@ def fake_mlx_generate(**_kwargs: object):
             yield None
         return
 
-    def fake_all_gather(array: object, *, group: object):
-        del group
+    def fake_all_sum(array: object, *, group: object, stream: object = None):
+        del group, stream
         return array
 
     _clear_warmup_env(monkeypatch)
@@ -216,7 +222,7 @@ def fake_all_gather(array: object, *, group: object):
     monkeypatch.setattr(generate_mod, "apply_chat_template", fake_apply_chat_template)
     monkeypatch.setattr(generate_mod, "mx_barrier", fake_mx_barrier)
     monkeypatch.setattr(generate_mod, "mlx_generate", fake_mlx_generate)
-    monkeypatch.setattr(generate_mod.mx.distributed, "all_gather", fake_all_gather)
+    monkeypatch.setattr(generate_mod.mx.distributed, "all_sum", fake_all_sum)
 
     generate_mod.warmup_inference(
         model=object(),  # type: ignore[arg-type]
@@ -254,15 +260,15 @@ def fake_mlx_generate(**_kwargs: object):
             generated_tokens += 1
             yield token
 
-    def fake_all_gather(array: object, *, group: object):
-        del group
+    def fake_all_sum(array: object, *, group: object, stream: object = None):
+        del group, stream
         return array
 
     _clear_warmup_env(monkeypatch)
     monkeypatch.setattr(generate_mod, "apply_chat_template", fake_apply_chat_template)
     monkeypatch.setattr(generate_mod, "mx_barrier", fake_mx_barrier)
     monkeypatch.setattr(generate_mod, "mlx_generate", fake_mlx_generate)
-    monkeypatch.setattr(generate_mod.mx.distributed, "all_gather", fake_all_gather)
+    monkeypatch.setattr(generate_mod.mx.distributed, "all_sum", fake_all_sum)
 
     check_every = generate_mod.warmup_inference(
         model=object(),  # type: ignore[arg-type]
@@ -306,15 +312,15 @@ def fake_mx_barrier(_group: object) -> None:
     def fake_mlx_generate(**_kwargs: object):
         yield "first"
 
-    def fake_all_gather(array: object, *, group: object):
-        del group
+    def fake_all_sum(array: object, *, group: object, stream: object = None):
+        del group, stream
         return array
 
     _clear_warmup_env(monkeypatch)
     monkeypatch.setattr(generate_mod, "apply_chat_template", fake_apply_chat_template)
     monkeypatch.setattr(generate_mod, "mx_barrier", fake_mx_barrier)
     monkeypatch.setattr(generate_mod, "mlx_generate", fake_mlx_generate)
-    monkeypatch.setattr(generate_mod.mx.distributed, "all_gather", fake_all_gather)
+    monkeypatch.setattr(generate_mod.mx.distributed, "all_sum", fake_all_sum)
     monkeypatch.setattr(generate_mod.time, "monotonic", lambda: next(monotonic_values))
 
     check_every = generate_mod.warmup_inference(

From 13a5f47c0391a1ee476c5de3beb25f5da818534f Mon Sep 17 00:00:00 2001
From: Thomas Tupper <thomastupper@kite2.local>
Date: Fri, 10 Apr 2026 00:49:49 -0500
Subject: [PATCH 16/21] Reduce empty distributed task polling during generation

---
 src/exo/worker/engines/mlx/utils_mlx.py       |  5 +-
 .../runner/llm_inference/batch_generator.py   | 73 +++++++++++----
 .../test_runner/test_event_ordering.py        | 92 ++++++++++++++++++-
 3 files changed, 147 insertions(+), 23 deletions(-)

diff --git a/src/exo/worker/engines/mlx/utils_mlx.py b/src/exo/worker/engines/mlx/utils_mlx.py
index 488c46368..7e135b4f2 100644
--- a/src/exo/worker/engines/mlx/utils_mlx.py
+++ b/src/exo/worker/engines/mlx/utils_mlx.py
@@ -1196,6 +1196,7 @@ def _parse_single_tool(text: str) -> dict[str, Any]:
 def mx_all_gather_tasks(
     tasks: list[TextGeneration],
     group: mx.distributed.Group | None,
+    label: str = "tasks",
 ) -> tuple[list[TextGeneration], list[TextGeneration]]:
     """Synchronize task lists across all ranks using only all_sum.
 
@@ -1230,7 +1231,7 @@ def decode_task_id(encoded_task_id: list[int]) -> TaskId:
     # Each rank contributes its count in its own slot of a [world_size]
     # vector; other slots are zero.  all_sum merges them.
     n_tasks = len(tasks)
-    logger.info(f"mx_all_gather_tasks: gathering counts (n_tasks={n_tasks})")
+    logger.debug(f"mx_all_gather_tasks[{label}]: gathering counts (n_tasks={n_tasks})")
     counts_vec = [0] * world_size
     counts_vec[rank] = n_tasks
     all_counts = cast(
@@ -1241,7 +1242,7 @@ def decode_task_id(encoded_task_id: list[int]) -> TaskId:
             stream=cpu_stream,
         ).tolist(),
     )
-    logger.info(f"mx_all_gather_tasks: counts gathered: {all_counts}")
+    logger.debug(f"mx_all_gather_tasks[{label}]: counts gathered: {all_counts}")
     max_tasks = max(all_counts)
 
     if max_tasks == 0:
diff --git a/src/exo/worker/runner/llm_inference/batch_generator.py b/src/exo/worker/runner/llm_inference/batch_generator.py
index cf146841c..5673ef175 100644
--- a/src/exo/worker/runner/llm_inference/batch_generator.py
+++ b/src/exo/worker/runner/llm_inference/batch_generator.py
@@ -93,6 +93,7 @@ def close(self) -> None: ...
 EXO_RUNNER_MUST_FAIL = "EXO RUNNER MUST FAIL"
 EXO_RUNNER_MUST_OOM = "EXO RUNNER MUST OOM"
 EXO_RUNNER_MUST_TIMEOUT = "EXO RUNNER MUST TIMEOUT"
+_DEFAULT_CANCEL_CHECK_INTERVAL = 50
 
 
 def _check_for_debug_prompts(task_params: TextGenerationTaskParams) -> None:
@@ -112,6 +113,18 @@ def _check_for_debug_prompts(task_params: TextGenerationTaskParams) -> None:
         time.sleep(100)
 
 
+def _safe_cancel_check_interval(interval: int) -> int:
+    """Normalize warmup-derived cancellation polling intervals for live generation."""
+    if interval > 0:
+        return interval
+
+    logger.warning(
+        "warmup returned a non-positive cancellation interval "
+        f"({interval}); using {_DEFAULT_CANCEL_CHECK_INTERVAL} tokens"
+    )
+    return _DEFAULT_CANCEL_CHECK_INTERVAL
+
+
 @dataclass(eq=False)
 class SequentialGenerator(InferenceGenerator):
     model: Model
@@ -147,12 +160,14 @@ class SequentialGenerator(InferenceGenerator):
     _logged_first_request_shape: bool = field(default=False, init=False)
 
     def warmup(self):
-        self.check_for_cancel_every = warmup_inference(
-            model=self.model,
-            tokenizer=self.tokenizer,
-            group=self.group,
-            model_id=self.model_id,
-            model_card=self.model_card,
+        self.check_for_cancel_every = _safe_cancel_check_interval(
+            warmup_inference(
+                model=self.model,
+                tokenizer=self.tokenizer,
+                group=self.group,
+                model_id=self.model_id,
+                model_card=self.model_card,
+            )
         )
 
     def submit(
@@ -165,10 +180,16 @@ def submit(
 
     def agree_on_tasks(self) -> None:
         """Agree between all ranks about the task ordering (some may have received in different order or not at all)."""
+        if not mx_any(bool(self._maybe_queue), self.group):
+            logger.debug("agree_on_tasks: no pending tasks on any rank")
+            return
+
         logger.info(
             f"agree_on_tasks: entering with {len(self._maybe_queue)} pending tasks"
         )
-        agreed, different = mx_all_gather_tasks(self._maybe_queue, self.group)
+        agreed, different = mx_all_gather_tasks(
+            self._maybe_queue, self.group, label="task-queue"
+        )
         logger.info(f"agree_on_tasks: {len(agreed)} agreed, {len(different)} different")
         self._queue.extend(task for task in self._maybe_queue if task in agreed)
         self._maybe_queue = [task for task in self._maybe_queue if task in different]
@@ -186,7 +207,13 @@ def agree_on_cancellations(self) -> None:
         if mx_any(has_cancel_all, self.group):
             self._cancelled_tasks.add(CANCEL_ALL_TASKS)
 
-        agreed, different = mx_all_gather_tasks(self._maybe_cancel, self.group)
+        if not mx_any(bool(self._maybe_cancel), self.group):
+            logger.debug("agree_on_cancellations: no pending cancellations on any rank")
+            return
+
+        agreed, different = mx_all_gather_tasks(
+            self._maybe_cancel, self.group, label="cancellations"
+        )
         self._cancelled_tasks.update(task.task_id for task in agreed)
         self._maybe_cancel = list(different)
 
@@ -381,12 +408,14 @@ def __post_init__(self) -> None:
         )
 
     def warmup(self):
-        self.check_for_cancel_every = warmup_inference(
-            model=self.model,
-            tokenizer=self.tokenizer,
-            group=self.group,
-            model_id=self.model_id,
-            model_card=self.model_card,
+        self.check_for_cancel_every = _safe_cancel_check_interval(
+            warmup_inference(
+                model=self.model,
+                tokenizer=self.tokenizer,
+                group=self.group,
+                model_id=self.model_id,
+                model_card=self.model_card,
+            )
         )
 
     def submit(
@@ -399,10 +428,16 @@ def submit(
 
     def agree_on_tasks(self) -> None:
         """Agree between all ranks about the task ordering (some may have received in different order or not at all)."""
+        if not mx_any(bool(self._maybe_queue), self.group):
+            logger.debug("agree_on_tasks: no pending tasks on any rank")
+            return
+
         logger.info(
             f"agree_on_tasks: entering with {len(self._maybe_queue)} pending tasks"
         )
-        agreed, different = mx_all_gather_tasks(self._maybe_queue, self.group)
+        agreed, different = mx_all_gather_tasks(
+            self._maybe_queue, self.group, label="task-queue"
+        )
         logger.info(f"agree_on_tasks: {len(agreed)} agreed, {len(different)} different")
         self._queue.extend(task for task in self._maybe_queue if task in agreed)
         self._maybe_queue = [task for task in self._maybe_queue if task in different]
@@ -420,7 +455,13 @@ def agree_on_cancellations(self) -> None:
         if mx_any(has_cancel_all, self.group):
             self._cancelled_tasks.add(CANCEL_ALL_TASKS)
 
-        agreed, different = mx_all_gather_tasks(self._maybe_cancel, self.group)
+        if not mx_any(bool(self._maybe_cancel), self.group):
+            logger.debug("agree_on_cancellations: no pending cancellations on any rank")
+            return
+
+        agreed, different = mx_all_gather_tasks(
+            self._maybe_cancel, self.group, label="cancellations"
+        )
         self._cancelled_tasks.update(task.task_id for task in agreed)
         self._maybe_cancel = list(different)
 
diff --git a/src/exo/worker/tests/unittests/test_runner/test_event_ordering.py b/src/exo/worker/tests/unittests/test_runner/test_event_ordering.py
index f95d7c819..22f340887 100644
--- a/src/exo/worker/tests/unittests/test_runner/test_event_ordering.py
+++ b/src/exo/worker/tests/unittests/test_runner/test_event_ordering.py
@@ -120,11 +120,16 @@ def patch_out_mlx(monkeypatch: pytest.MonkeyPatch):
     )
     monkeypatch.setattr(mlx_batch_generator, "warmup_inference", make_nothin(1))
     monkeypatch.setattr(mlx_batch_generator, "_check_for_debug_prompts", nothin)
-    monkeypatch.setattr(mlx_batch_generator, "mx_any", make_nothin(False))
+
+    def fake_mx_any(bool_: bool, _group: object) -> bool:
+        return bool_
+
+    monkeypatch.setattr(mlx_batch_generator, "mx_any", fake_mx_any)
 
     def fake_all_gather(
-        tasks: list[TextGeneration], group: object
+        tasks: list[TextGeneration], group: object, label: str = "tasks"
     ) -> tuple[list[TextGeneration], list[TextGeneration]]:
+        del label
         return (tasks, [])
 
     monkeypatch.setattr(mlx_batch_generator, "mx_all_gather_tasks", fake_all_gather)
@@ -232,6 +237,77 @@ def size(self) -> int:
         return 2
 
 
+class EmptyCancelReceiver:
+    def collect(self) -> list[TaskId]:
+        return []
+
+
+def _make_sequential_generator() -> mlx_batch_generator.SequentialGenerator:
+    return mlx_batch_generator.SequentialGenerator(
+        model=object(),  # type: ignore[arg-type]
+        tokenizer=MockTokenizer(),  # type: ignore[arg-type]
+        group=MockGroup(),  # type: ignore[arg-type]
+        kv_prefix_cache=None,
+        tool_parser=None,
+        model_card=None,
+        model_id=MODEL_A_ID,
+        device_rank=0,
+        cancel_receiver=EmptyCancelReceiver(),  # type: ignore[arg-type]
+        event_sender=EventCollector(),  # type: ignore[arg-type]
+    )
+
+
+def test_empty_task_agreement_skips_task_id_collective(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    mx_any_calls: list[bool] = []
+
+    def fake_mx_any(bool_: bool, _group: object) -> bool:
+        mx_any_calls.append(bool_)
+        return False
+
+    def fail_all_gather(*_args: object, **_kwargs: object) -> None:
+        raise AssertionError("empty task agreement should not gather task IDs")
+
+    monkeypatch.setattr(mlx_batch_generator, "mx_any", fake_mx_any)
+    monkeypatch.setattr(mlx_batch_generator, "mx_all_gather_tasks", fail_all_gather)
+
+    _make_sequential_generator().agree_on_tasks()
+
+    assert mx_any_calls == [False]
+
+
+def test_empty_cancellation_agreement_skips_task_id_collective(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    mx_any_calls: list[bool] = []
+
+    def fake_mx_any(bool_: bool, _group: object) -> bool:
+        mx_any_calls.append(bool_)
+        return False
+
+    def fail_all_gather(*_args: object, **_kwargs: object) -> None:
+        raise AssertionError("empty cancellation agreement should not gather task IDs")
+
+    monkeypatch.setattr(mlx_batch_generator, "mx_any", fake_mx_any)
+    monkeypatch.setattr(mlx_batch_generator, "mx_all_gather_tasks", fail_all_gather)
+
+    _make_sequential_generator().agree_on_cancellations()
+
+    assert mx_any_calls == [False, False]
+
+
+def test_zero_warmup_cancel_interval_falls_back_to_safe_default(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    generator = _make_sequential_generator()
+    monkeypatch.setattr(mlx_batch_generator, "warmup_inference", make_nothin(0))
+
+    generator.warmup()
+
+    assert generator.check_for_cancel_every == 50
+
+
 def _run(tasks: Iterable[Task], send_after_ready: list[Task] | None = None):
     bound_instance = get_bound_mlx_ring_instance(
         instance_id=INSTANCE_1_ID,
@@ -385,7 +461,9 @@ def test_distributed_warmup_skip_env_is_ignored(
     patch_out_mlx: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch
 ) -> None:
     monkeypatch.setenv("SKULK_SKIP_LLM_WARMUP", "1")
-    monkeypatch.setattr(mlx_runner, "initialize_mlx", make_nothin(MockDistributedGroup()))
+    monkeypatch.setattr(
+        mlx_runner, "initialize_mlx", make_nothin(MockDistributedGroup())
+    )
 
     warmup_calls = 0
 
@@ -421,9 +499,13 @@ def fake_log_request_shape(
     ) -> None:
         logged.append((label, task_params, prompt, extra))
 
-    monkeypatch.setattr(mlx_batch_generator, "log_request_shape", fake_log_request_shape)
+    monkeypatch.setattr(
+        mlx_batch_generator, "log_request_shape", fake_log_request_shape
+    )
 
-    _run([INIT_TASK, LOAD_TASK, WARMUP_TASK, CHAT_TASK], send_after_ready=[SHUTDOWN_TASK])
+    _run(
+        [INIT_TASK, LOAD_TASK, WARMUP_TASK, CHAT_TASK], send_after_ready=[SHUTDOWN_TASK]
+    )
 
     assert logged == [
         (

From 04893db6d8b7b65193551fd8db8746187a093c32 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <thomastupper@kite2.local>
Date: Fri, 10 Apr 2026 01:05:37 -0500
Subject: [PATCH 17/21] Use explicit pipeline prefill for short prompts

---
 .../worker/engines/mlx/generator/generate.py  | 43 ++++++-------------
 .../test_mlx/test_prefill_path_selection.py   | 42 +++++++++---------
 2 files changed, 36 insertions(+), 49 deletions(-)

diff --git a/src/exo/worker/engines/mlx/generator/generate.py b/src/exo/worker/engines/mlx/generator/generate.py
index d2b8e3f1b..f9d600e37 100644
--- a/src/exo/worker/engines/mlx/generator/generate.py
+++ b/src/exo/worker/engines/mlx/generator/generate.py
@@ -509,12 +509,12 @@ def combined_progress_callback(processed: int, total: int) -> None:
         prefill_step_size // min(4, group_size) if is_pipeline else prefill_step_size
     )
 
-    # Pipeline-parallel prefill can wedge when the prompt fits inside a single
-    # effective per-rank chunk (i.e. ``n_real == 1`` in the prefill loop).
     # Mirror pipeline_parallel_prefill's chunking math here by reserving the
     # final token for the post-loop pass before computing the real chunk count.
-    # That keeps multi-chunk prompts on pipeline_parallel_prefill while routing
-    # single-chunk short / warmup prompts through stream_generate.
+    # This value is primarily diagnostic now: all pipeline prompts use the
+    # explicit pipeline prefill path, but short one-chunk prompts intentionally
+    # suppress the distributed progress callback because there is no useful
+    # cancellation window inside a millisecond-scale prefill.
     real_prefill_tokens = max(num_tokens - 1, 0)
     pipeline_chunks = (
         (real_prefill_tokens + effective_prefill_step_size - 1)
@@ -522,7 +522,7 @@ def combined_progress_callback(processed: int, total: int) -> None:
         if effective_prefill_step_size > 0
         else 0
     )
-    use_pipeline_prefill = is_pipeline and pipeline_chunks >= 2
+    use_pipeline_prefill = is_pipeline
     logger.info(
         "Prefill path selected: "
         f"{'pipeline_parallel_prefill' if use_pipeline_prefill else 'stream_generate'} "
@@ -559,6 +559,9 @@ def combined_progress_callback(processed: int, total: int) -> None:
         if use_pipeline_prefill:
             set_pipeline_queue_sends(model, queue_sends=True)
             assert group is not None, "Pipeline prefill requires a distributed group"
+            pipeline_distributed_callback = (
+                distributed_prompt_progress_callback if pipeline_chunks >= 2 else None
+            )
             with _hang_debug_watch(
                 f"pipeline_parallel_prefill rank={rank} group_size={group_size}"
             ):
@@ -570,33 +573,15 @@ def combined_progress_callback(processed: int, total: int) -> None:
                     kv_group_size=KV_GROUP_SIZE,
                     kv_bits=KV_BITS,
                     prompt_progress_callback=progress_callback,
-                    distributed_prompt_progress_callback=distributed_prompt_progress_callback,
+                    distributed_prompt_progress_callback=pipeline_distributed_callback,
                     group=group,
                 )
         else:
-            # stream_generate is used for short prompts that fit in a single
-            # pipeline chunk.  We MUST NOT pass the distributed callback here:
-            #
-            # 1. mlx_lm's generate_step calls prompt_progress_callback inside
-            #    ``with mx.stream(generation_stream):``, and on the first decode
-            #    iteration it prefetches the next token via ``_step(y)`` with
-            #    ``mx.async_eval`` BEFORE firing the callback.  That prefetch
-            #    dispatches pipeline sends/recvs to generation_stream.  JACCL
-            #    cannot run an all_gather while sends/recvs are in-flight on
-            #    any stream — the collective deadlocks.
-            #
-            # 2. Warmup also uses stream_generate and abandons the generator
-            #    after one token.  The prefetched decode step leaves orphaned
-            #    sends/recvs on generation_stream.  The post-warmup barrier
-            #    synchronizes CPU stream but not generation_stream, so those
-            #    orphaned ops are still in-flight when real inference starts.
-            #
-            # 3. For short prompts the entire prefill takes milliseconds —
-            #    there is no meaningful window for task/cancellation checking.
-            #
-            # pipeline_parallel_prefill handles its own inter-chunk
-            # synchronization and safely calls the distributed callback
-            # between chunks, so it keeps the callback.
+            # Non-pipeline models can safely use upstream stream_generate for
+            # prefill. Pipeline models avoid it entirely: mlx-lm's generate_step
+            # prefetches one decode step before yielding even with max_tokens=1,
+            # which can leave hidden pipeline sends/recvs in flight and wedge
+            # the next collective.
             #
             # Use max_tokens=1 because max_tokens=0 does not work.
             # We just throw away the generated token - we only care about filling the cache
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py b/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py
index 25c8e1100..e5d79c219 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_prefill_path_selection.py
@@ -97,17 +97,15 @@ def _pipeline_parallel_prefill(**kwargs: object) -> None:
     assert prefill_mode_calls == [True, False]
 
 
-def test_prefill_uses_stream_generate_for_short_pipeline_prompts(
+def test_prefill_uses_pipeline_parallel_path_for_short_pipeline_prompts(
     monkeypatch: pytest.MonkeyPatch,
 ) -> None:
-    """Short prompts that fit in a single per-rank chunk must avoid pipeline prefill.
-
-    Regression for the Gemma 4 warmup wedge: pipeline_parallel_prefill hangs
-    when ``n_real == 1`` (the prompt fits inside a single per-rank chunk),
-    so any such prompt — including the 23-token warmup prompt — must route
-    through stream_generate even on pipeline-parallel models. PR #101's
-    pipeline-prefill widening removed this guard and reintroduced the hang
-    on Gemma 4; this test locks the guard back in.
+    """Short pipeline prompts must avoid upstream stream_generate prefill.
+
+    ``mlx_lm.generate_step`` prefetches a decode step before yielding even with
+    ``max_tokens=1``. In pipeline mode that hidden prefetch can leave sends/recvs
+    in flight, so short prompts use Skulk's explicit pipeline prefill path while
+    suppressing distributed progress callbacks.
     """
     calls: list[str] = []
     fake_cache = _FakeCache()
@@ -127,6 +125,12 @@ def test_prefill_uses_stream_generate_for_short_pipeline_prompts(
 
     def _pipeline_parallel_prefill(**kwargs: object) -> None:
         calls.append("pipeline_parallel_prefill")
+        assert kwargs["distributed_prompt_progress_callback"] is None
+        prompt_progress_callback = kwargs["prompt_progress_callback"]
+        assert callable(prompt_progress_callback)
+        prompt = kwargs["prompt"]
+        assert isinstance(prompt, list)
+        prompt_progress_callback(len(prompt), len(prompt))
 
     monkeypatch.setattr(
         generate_module,
@@ -140,6 +144,9 @@ def _stream_generate(*args: object, **kwargs: object) -> Iterator[object]:
 
     monkeypatch.setattr(generate_module, "stream_generate", _stream_generate)
 
+    def _fail_distributed_callback() -> None:
+        raise AssertionError("single-chunk pipeline prefill should not poll tasks")
+
     prefill_tps, prefill_tokens, snapshots = generate_module.prefill(
         model=_FakeModel(),
         tokenizer=object(),
@@ -148,25 +155,22 @@ def _stream_generate(*args: object, **kwargs: object) -> Iterator[object]:
         cache=[fake_cache],
         group=_FakeGroup(),
         on_prefill_progress=None,
-        distributed_prompt_progress_callback=None,
+        distributed_prompt_progress_callback=_fail_distributed_callback,
     )
 
-    assert calls == ["stream_generate"]
-    assert "pipeline_parallel_prefill" not in calls
+    assert calls == ["pipeline_parallel_prefill"]
+    assert "stream_generate" not in calls
     assert prefill_tokens == 23
     assert snapshots == []
     assert prefill_tps >= 0.0
     assert fake_cache.trim_calls == [2]
-    # First call sets is_prefill=True (pipeline models must run in prefill
-    # mode even on the stream_generate path to avoid all_gather deadlocks).
-    # Second call (in finally) resets to False for subsequent decode.
     assert prefill_mode_calls == [True, False]
 
 
-def test_prefill_uses_stream_generate_at_single_chunk_boundary(
+def test_prefill_uses_pipeline_parallel_path_at_single_chunk_boundary(
     monkeypatch: pytest.MonkeyPatch,
 ) -> None:
-    """Prompts with only one real pipeline chunk must stay off pipeline prefill."""
+    """Prompts with one real pipeline chunk still use explicit pipeline prefill."""
     calls: list[str] = []
     fake_cache = _FakeCache()
     prefill_mode_calls: list[bool] = []
@@ -205,13 +209,11 @@ def _stream_generate(*args: object, **kwargs: object) -> Iterator[object]:
         distributed_prompt_progress_callback=None,
     )
 
-    assert calls == ["stream_generate"]
+    assert calls == ["pipeline_parallel_prefill"]
     assert prefill_tokens == 1366
     assert snapshots == []
     assert prefill_tps >= 0.0
     assert fake_cache.trim_calls == [2]
-    # Pipeline models use is_prefill=True during stream_generate prefill,
-    # reset to False in finally.
     assert prefill_mode_calls == [True, False]
 
 

From 6c4ba6f3ddd924f3713f4041c0990a67555b5942 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <thomastupper@kite2.local>
Date: Fri, 10 Apr 2026 01:27:51 -0500
Subject: [PATCH 18/21] Gate experimental rotorquant backend

---
 README.md                                     |  9 ++-
 .../src/components/layout/SettingsPanel.tsx   | 16 ++---
 docs/kv-cache-backends.md                     | 27 +++++++-
 src/exo/api/main.py                           | 14 ++---
 src/exo/api/tests/test_config_api.py          | 29 ++++++++-
 src/exo/worker/engines/mlx/cache.py           | 32 ++++++++--
 src/exo/worker/engines/mlx/constants.py       | 62 +++++++++++++++----
 .../worker/engines/mlx/rotorquant/__init__.py |  5 +-
 .../worker/engines/mlx/rotorquant/cache.py    | 16 ++---
 .../test_mlx/test_kv_prefix_cache.py          | 20 ++++++
 website/docs/everything-different.md          |  2 +-
 website/docs/kv-cache-backends.md             | 50 ++++++++++-----
 12 files changed, 219 insertions(+), 63 deletions(-)

diff --git a/README.md b/README.md
index b70e81873..d0d484ec0 100644
--- a/README.md
+++ b/README.md
@@ -454,6 +454,7 @@ uv run exo --bootstrap-peers /ip4/192.168.1.20/tcp/5678/p2p/12D3KooW...
 | `EXO_NO_BATCH` | Force sequential generation | `false` |
 | `EXO_OPTIQ_BITS` | Bit width for `optiq` | `4` |
 | `EXO_OPTIQ_FP16_LAYERS` | Edge FP16 layers for `optiq` | `4` |
+| `SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT` | Enable experimental pure-MLX RotorQuant/IsoQuant cache backends | `false` |
 | `SKULK_ROTORQUANT_FP16_LAYERS` | Edge FP16 layers for `rotorquant_adaptive` | `4` |
 | `SKULK_ROTORQUANT_DEFER_PREFILL` | Set to `0` to disable deferred prefill (debugging only) | `1` |
 | `EXO_BOOTSTRAP_PEERS` | Comma-separated static peers to dial on startup | None |
@@ -465,9 +466,15 @@ Examples:
 EXO_OFFLINE=true uv run skulk
 EXO_ENABLE_IMAGE_MODELS=true uv run skulk
 EXO_KV_CACHE_BACKEND=optiq EXO_OPTIQ_BITS=4 EXO_OPTIQ_FP16_LAYERS=4 uv run skulk
-SKULK_KV_CACHE_BACKEND=rotorquant_adaptive SKULK_ROTORQUANT_FP16_LAYERS=4 uv run skulk
+SKULK_KV_CACHE_BACKEND=default SKULK_MLX_HANG_DEBUG=1 uv run skulk -vv
 ```
 
+The `rotorquant` and `rotorquant_adaptive` cache backends are experimental
+pure-MLX IsoQuant storage/dequant backends, not the fused RotorQuant+QJL
+implementation from the paper. They fall back to `default` unless
+`SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1` is also set, and should be used only
+for isolated cache experiments.
+
 ## RDMA on macOS
 
 RDMA is relevant only if you are building a multi-node Mac cluster on supported Thunderbolt 5 hardware.
diff --git a/dashboard-react/src/components/layout/SettingsPanel.tsx b/dashboard-react/src/components/layout/SettingsPanel.tsx
index c29860856..a66b93b23 100644
--- a/dashboard-react/src/components/layout/SettingsPanel.tsx
+++ b/dashboard-react/src/components/layout/SettingsPanel.tsx
@@ -449,20 +449,22 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
                 filled
                 content={
                   `• Default — No cache quantization. Best baseline quality, highest memory use.\n` +
-                  `• RotorQuant Adaptive — IsoQuant 3-bit with deferred prefill, FP16 edge layers. Recommended.\n` +
-                  `• RotorQuant — IsoQuant 3-bit on all KV layers. Most aggressive compression with deferred prefill.\n` +
                   `• OptiQ — Rotation-based quantization via mlx-optiq. Good long-context quality, no GQA support.\n` +
                   `• TurboQuant Adaptive — Quantizes middle KV layers, keeps edge layers in FP16. Proven stable.\n` +
-                  `• TurboQuant — Quantizes all KV layers. Most aggressive non-rotorquant compression.\n` +
+                  `• TurboQuant — Quantizes all KV layers. Most aggressive non-OptiQ compression.\n` +
                   `• MLX Quantized — MLX's built-in cache quantization.\n\n` +
-                  `Takes effect on next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.`
+                  `RotorQuant/IsoQuant is hidden from normal settings because it is experimental and must be enabled with SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1.`
                 }
               />
             </FieldLabel>
             <Select value={kvBackend} onChange={(e) => setKvBackend(e.target.value)} disabled={!!envOverride}>
               <option value="default">Default (no quantization)</option>
-              <option value="rotorquant_adaptive">RotorQuant Adaptive (recommended)</option>
-              <option value="rotorquant">RotorQuant</option>
+              {kvBackend === 'rotorquant_adaptive' ? (
+                <option value="rotorquant_adaptive">RotorQuant Adaptive (experimental, env-gated)</option>
+              ) : null}
+              {kvBackend === 'rotorquant' ? (
+                <option value="rotorquant">RotorQuant (experimental, env-gated)</option>
+              ) : null}
               <option value="optiq">OptiQ (rotation-based)</option>
               <option value="turboquant_adaptive">TurboQuant Adaptive</option>
               <option value="turboquant">TurboQuant</option>
@@ -471,7 +473,7 @@ export function SettingsPanel({ open, onClose }: SettingsPanelProps) {
             {envOverride ? (
               <HintText>Overridden by SKULK_KV_CACHE_BACKEND environment variable. Remove the env var to configure here.</HintText>
             ) : (
-              <HintText>Changes take effect on the next model launch. OptiQ falls back to Default for unsupported architectures; other backends will error on incompatible models.</HintText>
+              <HintText>Changes take effect on the next model launch. RotorQuant is experimental and intentionally unavailable from normal settings.</HintText>
             )}
           </Fieldset>
 
diff --git a/docs/kv-cache-backends.md b/docs/kv-cache-backends.md
index 1c21b8644..af7356af1 100644
--- a/docs/kv-cache-backends.md
+++ b/docs/kv-cache-backends.md
@@ -11,6 +11,8 @@ Skulk includes several opt-in KV cache backends for MLX text generation. These b
 - `turboquant`: correctness-first TurboQuant-inspired KV cache for standard `KVCache` layers
 - `turboquant_adaptive`: keeps outer KV layers in FP16 and applies TurboQuant to middle KV layers
 - `optiq`: **[NEW]** rotation-based KV cache via [mlx-optiq](https://mlx-optiq.pages.dev/) — uses randomized orthogonal rotations with Lloyd-Max quantization and rotated-space attention for superior long-context quality
+- `rotorquant`: **experimental, gated** pure-MLX IsoQuant-style storage/dequant cache. This is not the fused RotorQuant+QJL implementation from the RotorQuant paper.
+- `rotorquant_adaptive`: **experimental, gated** as above with FP16 protection on the first/last N attention layers.
 
 If `SKULK_KV_CACHE_BACKEND` is unset, or is set to `default`, Skulk behaves as before.
 
@@ -39,6 +41,23 @@ uv run skulk
 
 This mode keeps the first and last 4 KV layers in normal FP16-style cache and applies TurboQuant only to the middle KV layers. Proven stable across most models.
 
+### RotorQuant / IsoQuant (experimental only)
+
+The `rotorquant` names are intentionally gated behind an explicit opt-in:
+
+```bash
+SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1 \
+SKULK_KV_CACHE_BACKEND=rotorquant_adaptive \
+SKULK_ROTORQUANT_FP16_LAYERS=4 \
+uv run skulk
+```
+
+This backend is a pure-MLX IsoQuant-style cache that compresses storage and
+then returns fully dequantized fp16 K/V to normal MLX attention. It does not
+implement RotorQuant's fused Metal/CUDA attention path or QJL residual
+correction, so it should not be used as the default distributed inference
+baseline.
+
 ## Available Environment Variables
 
 | Variable | Backends | Default | Description |
@@ -50,6 +69,9 @@ This mode keeps the first and last 4 KV layers in normal FP16-style cache and ap
 | `SKULK_TQ_K_BITS` | `turboquant`, `turboquant_adaptive` | `3` | Key quantization bits |
 | `SKULK_TQ_V_BITS` | `turboquant`, `turboquant_adaptive` | `4` | Value quantization bits |
 | `SKULK_TQ_FP16_LAYERS` | `turboquant_adaptive` | `4` | Edge layers kept in FP16 |
+| `SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT` | `rotorquant`, `rotorquant_adaptive` | `0` | Required opt-in for experimental RotorQuant/IsoQuant backends |
+| `SKULK_ROTORQUANT_FP16_LAYERS` | `rotorquant_adaptive` | `4` | Edge layers kept in FP16 |
+| `SKULK_ROTORQUANT_DEFER_PREFILL` | `rotorquant`, `rotorquant_adaptive` | `1` | Set to `0` to disable deferred prefill while debugging |
 
 ## Invocation Examples
 
@@ -86,10 +108,12 @@ SKULK_KV_CACHE_BACKEND=turboquant_adaptive SKULK_TQ_K_BITS=3 SKULK_TQ_V_BITS=4 S
 | `turboquant_adaptive` | Low | Good | Moderate | Proven stable, Hadamard-based |
 | `turboquant` | Lowest | Variable | Moderate | Most aggressive compression |
 | `mlx_quantized` | Low | Good | Moderate | MLX built-in quantization |
+| `rotorquant_adaptive` | Low | Experimental | Experimental | Gated pure-MLX IsoQuant storage/dequant cache |
+| `rotorquant` | Lowest | Experimental | Experimental | Gated pure-MLX IsoQuant storage/dequant cache |
 
 ## Supported Cache Layouts
 
-All quantized backends (optiq, turboquant, mlx_quantized) compress only standard `KVCache` entries and preserve these cache types unchanged:
+All quantized backends (optiq, turboquant, mlx_quantized, and the gated rotorquant variants) compress only standard `KVCache` entries and preserve these cache types unchanged:
 
 - `ArraysCache`
 - `RotatingKVCache`
@@ -105,6 +129,7 @@ Mixed cache layouts are supported:
 - All quantized KV cache backends force sequential generation (no batch/history mode)
 - The optiq backend requires `mlx-optiq` to be installed (`pip install mlx-optiq`)
 - The optiq backend's `patch_attention()` monkey-patches MLX's SDPA — avoid switching between optiq and other backends within the same process lifetime without a restart
+- The rotorquant backends are disabled unless `SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1` is set. If selected without the gate, Skulk falls back to `default`.
 
 ## About mlx-optiq
 
diff --git a/src/exo/api/main.py b/src/exo/api/main.py
index 7e68b1f7f..af74fcb9a 100644
--- a/src/exo/api/main.py
+++ b/src/exo/api/main.py
@@ -204,6 +204,7 @@
 from exo.worker.engines.mlx.constants import (
     DEFAULT_KV_CACHE_BACKEND,
     VALID_KV_CACHE_BACKENDS,
+    resolve_kv_cache_backend,
 )
 
 if TYPE_CHECKING:
@@ -1197,8 +1198,7 @@ async def _send_text_generation_with_images(
             )
         for idx, h in cached_hashes.items():
             _log_image_transport(
-                f"TextGeneration cached image {idx}: "
-                f"b64_sha256={h[:12]}..."
+                f"TextGeneration cached image {idx}: b64_sha256={h[:12]}..."
             )
 
         if not new_images:
@@ -2214,9 +2214,7 @@ async def get_models(self, status: str | None = Query(default=None)) -> ModelLis
                         downloaded_model_ids.add(dl.shard_metadata.model_card.model_id)
             cards = [c for c in cards if c.model_id in downloaded_model_ids]
 
-        return ModelList(
-            data=[self._model_list_entry(card) for card in cards]
-        )
+        return ModelList(data=[self._model_list_entry(card) for card in cards])
 
     async def add_custom_model(self, payload: AddCustomModelParams) -> ModelListModel:
         """Fetch a model from HuggingFace and save as a custom model card, then sync across the cluster."""
@@ -2580,7 +2578,7 @@ def _effective_kv_cache_backend(self) -> str:
 
         if configured_backend not in VALID_KV_CACHE_BACKENDS:
             return DEFAULT_KV_CACHE_BACKEND
-        return configured_backend
+        return resolve_kv_cache_backend(configured_backend)
 
     async def get_config(self) -> JSONResponse:
         if not self._config_path.exists():
@@ -2678,7 +2676,9 @@ async def update_config(self, request: Request) -> JSONResponse:
             log_on = bool(logging_cfg_update.get("enabled", False)) and bool(
                 logging_cfg_update.get("ingest_url")
             )
-            set_structured_stdout(log_on, ingest_url=str(logging_cfg_update.get("ingest_url", "")))
+            set_structured_stdout(
+                log_on, ingest_url=str(logging_cfg_update.get("ingest_url", ""))
+            )
         # model_store changes still require restart; inference-only changes don't
         has_store_changes = "model_store" in config_data
         return JSONResponse(
diff --git a/src/exo/api/tests/test_config_api.py b/src/exo/api/tests/test_config_api.py
index 161ac5208..34e177c11 100644
--- a/src/exo/api/tests/test_config_api.py
+++ b/src/exo/api/tests/test_config_api.py
@@ -60,7 +60,9 @@ def test_get_config_treats_blank_skulk_kv_backend_as_default(
     tmp_path: Path, monkeypatch: pytest.MonkeyPatch
 ) -> None:
     config_path = tmp_path / "exo.yaml"
-    config_path.write_text("inference:\n  kv_cache_backend: default\n", encoding="utf-8")
+    config_path.write_text(
+        "inference:\n  kv_cache_backend: default\n", encoding="utf-8"
+    )
     monkeypatch.setenv("SKULK_KV_CACHE_BACKEND", "")
     monkeypatch.setenv("EXO_KV_CACHE_BACKEND", "optiq")
 
@@ -79,7 +81,9 @@ def test_get_config_treats_invalid_skulk_kv_backend_as_default(
     tmp_path: Path, monkeypatch: pytest.MonkeyPatch
 ) -> None:
     config_path = tmp_path / "exo.yaml"
-    config_path.write_text("inference:\n  kv_cache_backend: default\n", encoding="utf-8")
+    config_path.write_text(
+        "inference:\n  kv_cache_backend: default\n", encoding="utf-8"
+    )
     monkeypatch.setenv("SKULK_KV_CACHE_BACKEND", "typo-backend")
 
     api = _build_api()
@@ -91,3 +95,24 @@ def test_get_config_treats_invalid_skulk_kv_backend_as_default(
     assert response.status_code == 200
     data: dict[str, Any] = response.json()
     assert data["effective"]["kv_cache_backend"] == "default"
+
+
+def test_get_config_treats_ungated_rotorquant_as_default(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    config_path = tmp_path / "exo.yaml"
+    config_path.write_text(
+        "inference:\n  kv_cache_backend: default\n", encoding="utf-8"
+    )
+    monkeypatch.setenv("SKULK_KV_CACHE_BACKEND", "rotorquant_adaptive")
+    monkeypatch.setenv("SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT", "")
+
+    api = _build_api()
+    api._config_path = config_path  # pyright: ignore[reportPrivateUsage]
+    client = TestClient(api.app)
+
+    response = client.get("/config")
+
+    assert response.status_code == 200
+    data: dict[str, Any] = response.json()
+    assert data["effective"]["kv_cache_backend"] == "default"
diff --git a/src/exo/worker/engines/mlx/cache.py b/src/exo/worker/engines/mlx/cache.py
index 78da25272..8fc68c44c 100644
--- a/src/exo/worker/engines/mlx/cache.py
+++ b/src/exo/worker/engines/mlx/cache.py
@@ -20,6 +20,7 @@
     DEFAULT_KV_CACHE_BACKEND,
     DEFAULT_TURBOQUANT_K_BITS,
     DEFAULT_TURBOQUANT_V_BITS,
+    EXPERIMENTAL_ROTORQUANT_BACKENDS,
     KV_CACHE_BITS,
     OPTIQ_BITS,
     OPTIQ_FP16_LAYERS,
@@ -30,6 +31,8 @@
     TURBOQUANT_V_BITS,
     VALID_KV_CACHE_BACKENDS,
     KVCacheBackend,
+    experimental_rotorquant_enabled,
+    resolve_kv_cache_backend,
 )
 from exo.worker.engines.mlx.rotorquant import (
     make_rotorquant_adaptive_cache,
@@ -654,18 +657,35 @@ def get_kv_cache_backend() -> KVCacheBackend:
     it dynamically (instead of the frozen import-time constant) ensures
     that runtime updates are always visible to the runner.
     """
-    backend = cast(
-        KVCacheBackend,
+    configured_backend = (
         preferred_env_value(
-            "SKULK_KV_CACHE_BACKEND",
-            "EXO_KV_CACHE_BACKEND",
-            DEFAULT_KV_CACHE_BACKEND,
+            "SKULK_KV_CACHE_BACKEND", "EXO_KV_CACHE_BACKEND", DEFAULT_KV_CACHE_BACKEND
         )
-        or DEFAULT_KV_CACHE_BACKEND,
+        or DEFAULT_KV_CACHE_BACKEND
     )
+    backend = resolve_kv_cache_backend(configured_backend)
+
+    if configured_backend not in VALID_KV_CACHE_BACKENDS:
+        logger.warning(
+            f"Unknown KV_CACHE_BACKEND={configured_backend!r}; "
+            f"falling back to {DEFAULT_KV_CACHE_BACKEND!r}"
+        )
+        return backend
+
     if backend not in VALID_KV_CACHE_BACKENDS:
         logger.warning(
             f"Unknown KV_CACHE_BACKEND={backend!r}; falling back to {DEFAULT_KV_CACHE_BACKEND!r}"
         )
         return DEFAULT_KV_CACHE_BACKEND
+
+    if (
+        configured_backend in EXPERIMENTAL_ROTORQUANT_BACKENDS
+        and not experimental_rotorquant_enabled()
+    ):
+        logger.warning(
+            f"KV_CACHE_BACKEND={configured_backend!r} requested, but RotorQuant is "
+            "an experimental pure-MLX IsoQuant storage/dequant backend. Falling "
+            "back to 'default'. Set SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1 to "
+            "force this backend for isolated testing."
+        )
     return backend
diff --git a/src/exo/worker/engines/mlx/constants.py b/src/exo/worker/engines/mlx/constants.py
index 49b4c58e9..ee80c4621 100644
--- a/src/exo/worker/engines/mlx/constants.py
+++ b/src/exo/worker/engines/mlx/constants.py
@@ -1,5 +1,5 @@
 import os
-from typing import Literal, cast
+from typing import Literal
 
 from exo.shared.constants import preferred_env_value
 
@@ -36,15 +36,51 @@
     "rotorquant",
     "rotorquant_adaptive",
 )
+EXPERIMENTAL_ROTORQUANT_BACKENDS: tuple[KVCacheBackend, ...] = (
+    "rotorquant",
+    "rotorquant_adaptive",
+)
+
+
+def experimental_rotorquant_enabled() -> bool:
+    """Return whether the pure-MLX RotorQuant/IsoQuant backend is enabled.
+
+    The backend is intentionally gated because it is an experimental
+    storage/dequant cache, not the fused RotorQuant implementation described
+    in the paper. Keeping this behind an explicit flag prevents model-card or
+    dashboard config from putting unstable Metal cache code on the normal
+    inference path.
+    """
+    value = preferred_env_value(
+        "SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT",
+        "EXO_ENABLE_EXPERIMENTAL_ROTORQUANT",
+        "",
+    )
+    return str(value).strip().lower() in {"1", "true", "yes", "on"}
+
+
+def resolve_kv_cache_backend(raw_backend: str | None) -> KVCacheBackend:
+    """Normalize a configured KV cache backend into a safe runtime backend."""
+    backend = raw_backend or DEFAULT_KV_CACHE_BACKEND
+    if backend not in VALID_KV_CACHE_BACKENDS:
+        return DEFAULT_KV_CACHE_BACKEND
+
+    resolved: KVCacheBackend = backend
+    if (
+        resolved in EXPERIMENTAL_ROTORQUANT_BACKENDS
+        and not experimental_rotorquant_enabled()
+    ):
+        return DEFAULT_KV_CACHE_BACKEND
+
+    return resolved
+
+
 _kv_cache_backend_value = preferred_env_value(
     "SKULK_KV_CACHE_BACKEND",
     "EXO_KV_CACHE_BACKEND",
     DEFAULT_KV_CACHE_BACKEND,
 )
-KV_CACHE_BACKEND: KVCacheBackend = cast(
-    KVCacheBackend,
-    _kv_cache_backend_value if _kv_cache_backend_value else DEFAULT_KV_CACHE_BACKEND,
-)
+KV_CACHE_BACKEND: KVCacheBackend = resolve_kv_cache_backend(_kv_cache_backend_value)
 TURBOQUANT_K_BITS: int | None = (
     int(os.environ.get("SKULK_TQ_K_BITS", os.environ.get("EXO_TQ_K_BITS", "")))
     if os.environ.get("SKULK_TQ_K_BITS", os.environ.get("EXO_TQ_K_BITS"))
@@ -61,19 +97,19 @@
 OPTIQ_BITS: int = int(os.environ.get("EXO_OPTIQ_BITS", "4"))
 OPTIQ_FP16_LAYERS: int = int(os.environ.get("EXO_OPTIQ_FP16_LAYERS", "4"))
 ROTORQUANT_FP16_LAYERS: int = int(
-    os.environ.get("SKULK_ROTORQUANT_FP16_LAYERS", os.environ.get("EXO_ROTORQUANT_FP16_LAYERS", "4"))
+    os.environ.get(
+        "SKULK_ROTORQUANT_FP16_LAYERS",
+        os.environ.get("EXO_ROTORQUANT_FP16_LAYERS", "4"),
+    )
 )
 # Deferred prefill keeps K/V in fp16 during prompt processing and flushes
 # the buffer to compressed storage on the first decode token. It is the
 # load-bearing accuracy improvement of this backend; only disable for
 # debugging.
-ROTORQUANT_DEFER_PREFILL: bool = (
-    os.environ.get(
-        "SKULK_ROTORQUANT_DEFER_PREFILL",
-        os.environ.get("EXO_ROTORQUANT_DEFER_PREFILL", "1"),
-    )
-    not in ("0", "false", "False", "")
-)
+ROTORQUANT_DEFER_PREFILL: bool = os.environ.get(
+    "SKULK_ROTORQUANT_DEFER_PREFILL",
+    os.environ.get("EXO_ROTORQUANT_DEFER_PREFILL", "1"),
+) not in ("0", "false", "False", "")
 
 DEFAULT_TOP_LOGPROBS: int = 5
 
diff --git a/src/exo/worker/engines/mlx/rotorquant/__init__.py b/src/exo/worker/engines/mlx/rotorquant/__init__.py
index 9b81f5f46..608c8c48a 100644
--- a/src/exo/worker/engines/mlx/rotorquant/__init__.py
+++ b/src/exo/worker/engines/mlx/rotorquant/__init__.py
@@ -1,7 +1,10 @@
-"""RotorQuant KV cache backend (IsoQuant variant).
+"""Experimental RotorQuant-named KV cache backend (IsoQuant variant).
 
 Pure-MLX port of the IsoQuant 3-bit KV cache compression from
 johndpope/llama-cpp-turboquant (MIT) and scrya-com/rotorquant (MIT).
+This is not the fused RotorQuant+QJL attention implementation described in
+the RotorQuant paper; it is a storage/dequant cache used for isolated MLX
+experiments.
 
 Key properties vs the older TurboQuant native backend:
 - Block-diagonal 4D quaternion rotations instead of randomized Hadamard
diff --git a/src/exo/worker/engines/mlx/rotorquant/cache.py b/src/exo/worker/engines/mlx/rotorquant/cache.py
index d80d3d69a..cc4a09d3f 100644
--- a/src/exo/worker/engines/mlx/rotorquant/cache.py
+++ b/src/exo/worker/engines/mlx/rotorquant/cache.py
@@ -1,16 +1,16 @@
-"""RotorQuant KV cache class with optional deferred prefill.
+"""Experimental pure-MLX IsoQuant KV cache with optional deferred prefill.
 
-The deferred-prefill path is the load-bearing accuracy improvement that
-the upstream llama.cpp fork only ships on CUDA (``#ifdef GGML_USE_CUDA``
-in ``llama-context.cpp:1690``). On Metal/MLX it has not existed before
-this port.
+This module intentionally does not implement the fused RotorQuant+QJL
+attention path from the RotorQuant paper. It stores compressed K/V indices
+and norms, then materializes dequantized fp16 K/V for normal MLX attention.
+That makes it useful for isolated cache experiments, but not a production
+substitute for the upstream fused llama.cpp path.
 
 The idea: while the engine is processing a prompt (``update_and_fetch``
 called with ``num_steps > 1``), keep K and V in fp16. Quantization only
 happens once, on the first decode-shaped call (``num_steps == 1``).
-This avoids compounding centroid-roundtrip errors through every
-prefill attention chain and matches the published 5.3× prefill / PPL
-6.91 numbers from the llama.cpp fork.
+This avoids compounding centroid-roundtrip errors through every prefill
+attention chain.
 """
 
 from collections.abc import Sequence
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py b/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py
index 5f73a8177..d6738f757 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_kv_prefix_cache.py
@@ -113,6 +113,26 @@ def test_unknown_backend_falls_back_to_default(self):
         with patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "mystery"}):
             assert get_kv_cache_backend() == "default"
 
+    def test_rotorquant_backend_requires_experimental_gate(self):
+        with patch.dict(
+            os.environ,
+            {
+                "SKULK_KV_CACHE_BACKEND": "rotorquant_adaptive",
+                "SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT": "",
+            },
+        ):
+            assert get_kv_cache_backend() == "default"
+
+    def test_rotorquant_backend_can_be_explicitly_enabled(self):
+        with patch.dict(
+            os.environ,
+            {
+                "SKULK_KV_CACHE_BACKEND": "rotorquant",
+                "SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT": "1",
+            },
+        ):
+            assert get_kv_cache_backend() == "rotorquant"
+
     def test_make_kv_cache_default_backend(self):
         model = cast(Model, type("FakeModel", (), {"layers": [object(), object()]})())
         with patch.dict(os.environ, {"SKULK_KV_CACHE_BACKEND": "default"}):
diff --git a/website/docs/everything-different.md b/website/docs/everything-different.md
index fbf0fdb35..adab01a3a 100644
--- a/website/docs/everything-different.md
+++ b/website/docs/everything-different.md
@@ -36,7 +36,7 @@ If you are about to land a feature or fix that adds, removes, or materially chan
 - **OpenAI Responses API** — `/v1/responses` adapter alongside chat completions.
 - **Embeddings endpoint** for non-chat workloads.
 - **Model store endpoints** — search, add, download, capability resolution, optimization jobs, and registry management, all exposed under stable URLs and documented in the OpenAPI spec.
-- **Cluster-wide config endpoints** — `GET`/`POST` config that gossipsubs to every node and writes back to `skulk.yaml`. The KV cache backend selection (including the new `rotorquant` and `rotorquant_adaptive` values) goes through this same endpoint, so the dashboard Settings panel can switch backends cluster-wide without env vars.
+- **Cluster-wide config endpoints** — `GET`/`POST` config that gossipsubs to every node and writes back to `skulk.yaml`. Stable KV cache backend selection goes through this same endpoint, so the dashboard Settings panel can switch supported backends cluster-wide without env vars. Experimental `rotorquant` values remain env-gated and fall back to `default` unless explicitly enabled.
 - **Tracing, downloads, instance previews, and placement endpoints** — distributed-system observability and pre-launch placement inspection that upstream does not expose.
 
 ## Dashboard
diff --git a/website/docs/kv-cache-backends.md b/website/docs/kv-cache-backends.md
index abb059d4e..7956666ff 100644
--- a/website/docs/kv-cache-backends.md
+++ b/website/docs/kv-cache-backends.md
@@ -15,27 +15,26 @@ Skulk includes several opt-in KV cache backends for MLX text generation. These b
 - `turboquant`: correctness-first TurboQuant-inspired KV cache for standard `KVCache` layers
 - `turboquant_adaptive`: keeps outer KV layers in FP16 and applies TurboQuant to middle KV layers
 - `optiq`: rotation-based KV cache via [mlx-optiq](https://mlx-optiq.pages.dev/) — uses randomized orthogonal rotations with Lloyd-Max quantization and rotated-space attention. Falls back to default for GQA models.
-- `rotorquant`: **[NEW]** pure-MLX port of IsoQuant 3-bit from [scrya-com/rotorquant](https://github.com/scrya-com/rotorquant) and [johndpope/llama-cpp-turboquant](https://github.com/johndpope/llama-cpp-turboquant). Block-diagonal quaternion rotations + deferred prefill. GQA-native.
-- `rotorquant_adaptive`: **[NEW]** as above with FP16 protection on the first/last N attention layers. Recommended starting point.
+- `rotorquant`: **experimental, gated** pure-MLX IsoQuant-style storage/dequant cache. This is not the fused RotorQuant+QJL implementation from the RotorQuant paper.
+- `rotorquant_adaptive`: **experimental, gated** as above with FP16 protection on the first/last N attention layers.
 
 If `SKULK_KV_CACHE_BACKEND` is unset, or is set to `default`, Skulk behaves as before.
 
 ## Recommended Settings
 
-### RotorQuant Adaptive (recommended starting point)
+### Default Baseline (recommended while debugging)
 
 ```bash
-SKULK_KV_CACHE_BACKEND=rotorquant_adaptive \
-SKULK_ROTORQUANT_FP16_LAYERS=4 \
-uv run skulk
+SKULK_KV_CACHE_BACKEND=default \
+SKULK_MLX_HANG_DEBUG=1 \
+SKULK_MLX_HANG_DEBUG_INTERVAL_SECONDS=5 \
+uv run skulk -vv
 ```
 
-The rotorquant backend implements IsoQuant 3-bit compression with two key properties:
-
-- **Block-diagonal quaternion rotations**: each 4D group of a head dimension is rotated by a fixed unit quaternion, costing `O(d)` per token instead of the `O(d log d)` randomized Hadamard used by the legacy turboquant backend or the `O(d²)` random orthogonal matrix used by mlx-optiq. The quaternion table and 3-bit Lloyd-Max centroids are vendored verbatim from the llama.cpp fork so the math agrees with the upstream C reference.
-- **Deferred prefill**: K and V are kept in fp16 throughout prompt processing and quantized once on the first decode token. This eliminates the compounding centroid-roundtrip error that quantizing-on-insert introduces during prefill, and matches the published 5.3× prefill / PPL 6.91 numbers from the upstream llama.cpp fork. The deferred-prefill flush is the unique contribution of this MLX port — the upstream fork only ships it on CUDA.
-
-GQA models work natively (no fallback) because compression is per-(kv_head, token) and Q heads fan out at SDPA. The head dimension must be a multiple of 128 (the IsoQuant block size); for example, 128- and 256-d heads work directly, while 64-d heads do not satisfy this requirement and will fall back to the default backend.
+Use the default backend first when validating distributed pipeline behavior. It
+removes KV-cache compression from the failure surface so hangs can be attributed
+to pipeline scheduling, task agreement, or the model path rather than an
+experimental cache implementation.
 
 ### mlx-optiq (best quality)
 
@@ -60,6 +59,23 @@ uv run skulk
 
 This mode keeps the first and last 4 KV layers in normal FP16-style cache and applies TurboQuant only to the middle KV layers. Proven stable across most models.
 
+### RotorQuant / IsoQuant (experimental only)
+
+The `rotorquant` names are intentionally gated behind an explicit opt-in:
+
+```bash
+SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1 \
+SKULK_KV_CACHE_BACKEND=rotorquant_adaptive \
+SKULK_ROTORQUANT_FP16_LAYERS=4 \
+uv run skulk
+```
+
+This backend is a pure-MLX IsoQuant-style cache that compresses storage and
+then returns fully dequantized fp16 K/V to normal MLX attention. It does not
+implement RotorQuant's fused Metal/CUDA attention path or QJL residual
+correction, so it should not be used as the default distributed inference
+baseline.
+
 ## Available Environment Variables
 
 | Variable | Backends | Default | Description |
@@ -71,8 +87,9 @@ This mode keeps the first and last 4 KV layers in normal FP16-style cache and ap
 | `SKULK_TQ_K_BITS` | `turboquant`, `turboquant_adaptive` | `3` | Key quantization bits |
 | `SKULK_TQ_V_BITS` | `turboquant`, `turboquant_adaptive` | `4` | Value quantization bits |
 | `SKULK_TQ_FP16_LAYERS` | `turboquant_adaptive` | `4` | Edge layers kept in FP16 |
+| `SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT` | `rotorquant`, `rotorquant_adaptive` | `0` | Required opt-in for experimental RotorQuant/IsoQuant backends |
 | `SKULK_ROTORQUANT_FP16_LAYERS` | `rotorquant_adaptive` | `4` | Edge layers kept in FP16 |
-| `SKULK_ROTORQUANT_DEFER_PREFILL` | `rotorquant`, `rotorquant_adaptive` | `1` | Set to `0` to disable deferred prefill (debugging only) |
+| `SKULK_ROTORQUANT_DEFER_PREFILL` | `rotorquant`, `rotorquant_adaptive` | `1` | Set to `0` to disable deferred prefill while debugging |
 
 ## Invocation Examples
 
@@ -105,16 +122,16 @@ SKULK_KV_CACHE_BACKEND=turboquant_adaptive SKULK_TQ_K_BITS=3 SKULK_TQ_V_BITS=4 S
 | Backend | Memory | Quality | Speed | Notes |
 |---------|--------|---------|-------|-------|
 | `default` | Highest | Baseline | Fastest | No quantization |
-| `rotorquant_adaptive` | Low | Best quantized | Near-baseline | IsoQuant 3-bit + deferred prefill, GQA-native |
-| `rotorquant` | Lowest | Good | Near-baseline | All layers IsoQuant 3-bit + deferred prefill |
 | `optiq` | Low | Good | Near-baseline | Rotation-based, no GQA support |
 | `turboquant_adaptive` | Low | Good | Moderate | Proven stable, Hadamard-based |
 | `turboquant` | Low | Variable | Moderate | All layers, Hadamard-based |
 | `mlx_quantized` | Low | Good | Moderate | MLX built-in quantization |
+| `rotorquant_adaptive` | Low | Experimental | Experimental | Gated pure-MLX IsoQuant storage/dequant cache |
+| `rotorquant` | Lowest | Experimental | Experimental | Gated pure-MLX IsoQuant storage/dequant cache |
 
 ## Supported Cache Layouts
 
-All quantized backends (rotorquant, optiq, turboquant, mlx_quantized) compress only standard `KVCache` entries and preserve these cache types unchanged:
+All quantized backends (optiq, turboquant, mlx_quantized, and the gated rotorquant variants) compress only standard `KVCache` entries and preserve these cache types unchanged:
 
 - `ArraysCache`
 - `RotatingKVCache`
@@ -130,6 +147,7 @@ Mixed cache layouts are supported:
 - All quantized KV cache backends force sequential generation (no batch/history mode)
 - The optiq backend requires `mlx-optiq` to be installed (`pip install mlx-optiq`)
 - The optiq backend's `patch_attention()` monkey-patches MLX's SDPA — avoid switching between optiq and other backends within the same process lifetime without a restart
+- The rotorquant backends are disabled unless `SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1` is set. If selected without the gate, Skulk falls back to `default`.
 
 ## About mlx-optiq
 

From d612d1477575a6d5861517d0a9d16348861915c0 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <thomastupper@kite2.local>
Date: Fri, 10 Apr 2026 01:44:44 -0500
Subject: [PATCH 19/21] Restore Gemma 4 live prompt boundary

---
 src/exo/worker/engines/mlx/gemma4_prompt.py   | 58 ++++++++++++-------
 .../worker/engines/mlx/generator/generate.py  |  1 +
 src/exo/worker/engines/mlx/utils_mlx.py       |  2 +
 .../unittests/test_mlx/test_gemma4_prompt.py  | 15 ++++-
 .../unittests/test_mlx/test_warmup_request.py | 52 +++++++++++++----
 5 files changed, 95 insertions(+), 33 deletions(-)

diff --git a/src/exo/worker/engines/mlx/gemma4_prompt.py b/src/exo/worker/engines/mlx/gemma4_prompt.py
index 1cf44ce41..20e4270d3 100644
--- a/src/exo/worker/engines/mlx/gemma4_prompt.py
+++ b/src/exo/worker/engines/mlx/gemma4_prompt.py
@@ -2,13 +2,18 @@
 
 This module follows the Gemma 4 chat structure used by the reference Hugging
 Face template and Ollama's dedicated Gemma 4 renderer, while preserving one
-intentional divergence: when thinking is disabled we omit the empty synthetic
-thought-channel suffix because that shape wedges our distributed MLX warmup
-path. We keep a dedicated renderer so multimodal prompts stay under explicit
-repo control instead of relying on generic tokenizer chat templating.
+intentional warmup-only divergence: distributed warmup can suppress the empty
+synthetic thought-channel suffix because that shape has wedged our distributed
+MLX warmup path. Real inference keeps the reference suffix so generation begins
+at the expected visible assistant-content boundary. We keep a dedicated renderer
+so multimodal prompts stay under explicit repo control instead of relying on
+generic tokenizer chat templating.
 """
 
-from typing import Any
+from collections.abc import Mapping, Sequence
+from typing import TypeAlias, cast
+
+Gemma4Message: TypeAlias = Mapping[str, object]
 
 
 def strip_gemma4_thinking(text: str) -> str:
@@ -28,7 +33,7 @@ def strip_gemma4_thinking(text: str) -> str:
     return "".join(result).strip()
 
 
-def _render_gemma4_content(content: Any, role: str) -> str:
+def _render_gemma4_content(content: object, role: str) -> str:
     """Render one Gemma 4 message body."""
     if isinstance(content, str):
         return strip_gemma4_thinking(content) if role == "model" else content.strip()
@@ -36,14 +41,18 @@ def _render_gemma4_content(content: Any, role: str) -> str:
     if not isinstance(content, list):
         return str(content).strip()
 
+    content_items = cast(Sequence[object], content)
     parts: list[str] = []
-    for item in content:
-        if not isinstance(item, dict):
+    for item in content_items:
+        if not isinstance(item, Mapping):
             continue
-        item_type = str(item.get("type", ""))
+        item_mapping = cast(Mapping[str, object], item)
+        item_type = str(item_mapping.get("type", ""))
         if item_type == "text":
-            text = str(item.get("text", ""))
-            parts.append(strip_gemma4_thinking(text) if role == "model" else text.strip())
+            text = str(item_mapping.get("text", ""))
+            parts.append(
+                strip_gemma4_thinking(text) if role == "model" else text.strip()
+            )
         elif item_type == "image":
             parts.append("\n\n<|image|>\n\n")
         elif item_type == "audio":
@@ -54,10 +63,11 @@ def _render_gemma4_content(content: Any, role: str) -> str:
 
 
 def render_gemma4_prompt(
-    messages: list[dict[str, Any]],
+    messages: Sequence[Gemma4Message],
     *,
     add_generation_prompt: bool,
     enable_thinking: bool | None = None,
+    suppress_empty_thought_channel: bool = False,
 ) -> str:
     """Render a Gemma 4 prompt matching the reference chat template.
 
@@ -78,24 +88,32 @@ def render_gemma4_prompt(
         if enable_thinking:
             prompt_parts.append("<|think|>")
         if has_system_message:
-            prompt_parts.append(_render_gemma4_content(messages[0].get("content", ""), "system"))
+            prompt_parts.append(
+                _render_gemma4_content(messages[0].get("content", ""), "system")
+            )
             loop_messages = messages[1:]
         prompt_parts.append("<turn|>\n")
 
     for message in loop_messages:
-        role = "model" if str(message.get("role", "user")) == "assistant" else str(
-            message.get("role", "user")
+        role = (
+            "model"
+            if str(message.get("role", "user")) == "assistant"
+            else str(message.get("role", "user"))
         )
         prompt_parts.append(f"<|turn>{role}\n")
         prompt_parts.append(_render_gemma4_content(message.get("content", ""), role))
         prompt_parts.append("<turn|>\n")
 
     if add_generation_prompt:
-        # Gemma's published chat template appends an empty thought channel even
-        # when thinking is disabled. We intentionally omit that suffix here
-        # because it appears to trigger a pipeline warmup wedge in our MLX
-        # distributed path, and real traffic succeeds without this synthetic
-        # prefix.
         prompt_parts.append("<|turn>model\n")
+        if not enable_thinking and not suppress_empty_thought_channel:
+            # Gemma 4 expects the assistant turn to pass through the channel
+            # marker state. For normal inference we match the official
+            # template by adding an empty thought channel so generation begins
+            # in visible assistant content instead of at a raw ``<|channel>``
+            # boundary. Distributed warmup can suppress this suffix because it
+            # only validates prefill plus one decode step and historically
+            # wedged on richer synthetic prompt shapes.
+            prompt_parts.append("<|channel>thought\n<channel|>")
 
     return "".join(prompt_parts)
diff --git a/src/exo/worker/engines/mlx/generator/generate.py b/src/exo/worker/engines/mlx/generator/generate.py
index f9d600e37..5f38acd9d 100644
--- a/src/exo/worker/engines/mlx/generator/generate.py
+++ b/src/exo/worker/engines/mlx/generator/generate.py
@@ -687,6 +687,7 @@ def warmup_inference(
             tokenizer=tokenizer,
             task_params=warmup_task_params,
             model_card=model_card,
+            suppress_empty_gemma4_thought_channel=True,
         )
     logger.info(
         "Warmup prompt prepared "
diff --git a/src/exo/worker/engines/mlx/utils_mlx.py b/src/exo/worker/engines/mlx/utils_mlx.py
index 7e135b4f2..06fc8399b 100644
--- a/src/exo/worker/engines/mlx/utils_mlx.py
+++ b/src/exo/worker/engines/mlx/utils_mlx.py
@@ -841,6 +841,7 @@ def apply_chat_template(
     tokenizer: TokenizerWrapper,
     task_params: TextGenerationTaskParams,
     model_card: ModelCard | None = None,
+    suppress_empty_gemma4_thought_channel: bool = False,
 ) -> str:
     """Convert TextGenerationTaskParams to a chat template prompt.
 
@@ -902,6 +903,7 @@ def apply_chat_template(
             formatted_messages,
             add_generation_prompt=True,
             enable_thinking=task_params.enable_thinking,
+            suppress_empty_thought_channel=suppress_empty_gemma4_thought_channel,
         )
         if partial_assistant_content:
             prompt += partial_assistant_content
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_gemma4_prompt.py b/src/exo/worker/tests/unittests/test_mlx/test_gemma4_prompt.py
index 6bb0331c5..bdc9a97fb 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_gemma4_prompt.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_gemma4_prompt.py
@@ -1,7 +1,7 @@
 from exo.worker.engines.mlx.gemma4_prompt import render_gemma4_prompt
 
 
-def test_render_gemma4_prompt_does_not_append_empty_thought_channel_when_thinking_disabled():
+def test_render_gemma4_prompt_appends_empty_thought_channel_when_thinking_disabled():
     prompt = render_gemma4_prompt(
         [
             {"role": "system", "content": "You are helpful."},
@@ -20,7 +20,20 @@ def test_render_gemma4_prompt_does_not_append_empty_thought_channel_when_thinkin
         "Hello"
         "<turn|>\n"
         "<|turn>model\n"
+        "<|channel>thought\n"
+        "<channel|>"
     )
+
+
+def test_render_gemma4_prompt_can_suppress_empty_thought_channel_for_warmup():
+    prompt = render_gemma4_prompt(
+        [{"role": "user", "content": "Hello"}],
+        add_generation_prompt=True,
+        enable_thinking=False,
+        suppress_empty_thought_channel=True,
+    )
+
+    assert prompt.endswith("<|turn>model\n")
     assert "<|channel>thought" not in prompt
     assert "<channel|>" not in prompt
 
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
index 31c2e1653..3f7db19f2 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
@@ -40,10 +40,17 @@ def test_warmup_inference_uses_safe_default_user_content_without_instructions(
     captured: dict[str, object] = {}
 
     def fake_apply_chat_template(
-        *, tokenizer: object, task_params: object, model_card: object
+        *,
+        tokenizer: object,
+        task_params: object,
+        model_card: object,
+        suppress_empty_gemma4_thought_channel: bool = False,
     ):
         del tokenizer, model_card
         captured["task_params"] = task_params
+        captured["suppress_empty_gemma4_thought_channel"] = (
+            suppress_empty_gemma4_thought_channel
+        )
         return "warmup prompt"
 
     def fake_mx_barrier(_group: object) -> None:
@@ -95,6 +102,7 @@ def fake_log_request_shape(
     assert task_params.top_p == 1.0
     assert task_params.top_k == 0
     assert task_params.max_output_tokens == 1024
+    assert captured["suppress_empty_gemma4_thought_channel"] is True
     first_message = task_params.input[0]
     assert first_message.content == "hello"
     assert captured["label"] == "warmup"
@@ -112,9 +120,13 @@ def test_warmup_inference_ignores_repeat_count_override_for_pipeline_groups(
     captured: dict[str, object] = {}
 
     def fake_apply_chat_template(
-        *, tokenizer: object, task_params: object, model_card: object
+        *,
+        tokenizer: object,
+        task_params: object,
+        model_card: object,
+        suppress_empty_gemma4_thought_channel: bool = False,
     ):
-        del tokenizer, model_card
+        del tokenizer, model_card, suppress_empty_gemma4_thought_channel
         captured["task_params"] = task_params
         return "warmup prompt"
 
@@ -155,9 +167,13 @@ def test_warmup_inference_ignores_instruction_override_for_pipeline_groups(
     captured: dict[str, object] = {}
 
     def fake_apply_chat_template(
-        *, tokenizer: object, task_params: object, model_card: object
+        *,
+        tokenizer: object,
+        task_params: object,
+        model_card: object,
+        suppress_empty_gemma4_thought_channel: bool = False,
     ):
-        del tokenizer, model_card
+        del tokenizer, model_card, suppress_empty_gemma4_thought_channel
         captured["task_params"] = task_params
         return "warmup prompt"
 
@@ -198,9 +214,13 @@ def test_warmup_inference_honors_repeat_and_instruction_overrides_for_single_nod
     captured: dict[str, object] = {}
 
     def fake_apply_chat_template(
-        *, tokenizer: object, task_params: object, model_card: object
+        *,
+        tokenizer: object,
+        task_params: object,
+        model_card: object,
+        suppress_empty_gemma4_thought_channel: bool = False,
     ):
-        del tokenizer, model_card
+        del tokenizer, model_card, suppress_empty_gemma4_thought_channel
         captured["task_params"] = task_params
         return "warmup prompt"
 
@@ -246,9 +266,13 @@ def test_warmup_inference_stops_after_first_generated_token(
     generated_tokens = 0
 
     def fake_apply_chat_template(
-        *, tokenizer: object, task_params: object, model_card: object
+        *,
+        tokenizer: object,
+        task_params: object,
+        model_card: object,
+        suppress_empty_gemma4_thought_channel: bool = False,
     ):
-        del tokenizer, task_params, model_card
+        del tokenizer, task_params, model_card, suppress_empty_gemma4_thought_channel
         return "warmup prompt"
 
     def fake_mx_barrier(_group: object) -> None:
@@ -292,7 +316,7 @@ def test_warmup_helpers_prefer_blank_skulk_values_over_legacy_env(
     monkeypatch.setenv("EXO_DEBUG_WARMUP_INCLUDE_INSTRUCTIONS", "1")
 
     assert generate_mod._warmup_repeat_count() == 1  # pyright: ignore[reportPrivateUsage]
-    assert generate_mod._warmup_instructions(cast(object, _SingleNodeGroup())) is None  # pyright: ignore[reportPrivateUsage]
+    assert generate_mod._warmup_instructions(cast(object, _SingleNodeGroup())) is None  # pyright: ignore[reportPrivateUsage, reportArgumentType]
 
 
 def test_warmup_inference_enforces_minimum_cancel_check_interval_on_slow_start(
@@ -301,9 +325,13 @@ def test_warmup_inference_enforces_minimum_cancel_check_interval_on_slow_start(
     monotonic_values = iter([100.0, 112.0])
 
     def fake_apply_chat_template(
-        *, tokenizer: object, task_params: object, model_card: object
+        *,
+        tokenizer: object,
+        task_params: object,
+        model_card: object,
+        suppress_empty_gemma4_thought_channel: bool = False,
     ):
-        del tokenizer, task_params, model_card
+        del tokenizer, task_params, model_card, suppress_empty_gemma4_thought_channel
         return "warmup prompt"
 
     def fake_mx_barrier(_group: object) -> None:

From a50a53785e9dca39d675178146f69be1fad944c5 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <thomastupper@kite2.local>
Date: Fri, 10 Apr 2026 01:52:22 -0500
Subject: [PATCH 20/21] Let warmup finish one-token generation

---
 src/exo/worker/engines/mlx/generator/generate.py  |  9 ++++-----
 .../unittests/test_mlx/test_warmup_request.py     | 15 +++++++++------
 2 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/src/exo/worker/engines/mlx/generator/generate.py b/src/exo/worker/engines/mlx/generator/generate.py
index 5f38acd9d..ea1930d4c 100644
--- a/src/exo/worker/engines/mlx/generator/generate.py
+++ b/src/exo/worker/engines/mlx/generator/generate.py
@@ -675,7 +675,10 @@ def warmup_inference(
                 content=_warmup_user_content(group),
             )
         ],
-        max_output_tokens=1024,
+        # Warmup is a prefill + one-token decode sanity check. Asking the
+        # generator for exactly one token lets it take its normal terminal
+        # path instead of aborting a larger generation mid-stream.
+        max_output_tokens=1,
         enable_thinking=False,
         temperature=0.0 if is_distributed else 1.0,
         top_p=1.0 if is_distributed else 0.95,
@@ -722,10 +725,6 @@ def warmup_inference(
             group=group,
         ):
             tokens_generated += 1
-            # Warmup is only meant to validate prefill plus the first decode
-            # step. Stopping after the first token keeps runner readiness
-            # bounded even if the model would otherwise continue sampling.
-            break
 
     # The single-token warmup path intentionally samples only the first decode
     # step, so cold-start compile/prefill latency can dominate the elapsed
diff --git a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
index 3f7db19f2..9ae003ac7 100644
--- a/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
+++ b/src/exo/worker/tests/unittests/test_mlx/test_warmup_request.py
@@ -101,7 +101,7 @@ def fake_log_request_shape(
     assert task_params.temperature == 0.0
     assert task_params.top_p == 1.0
     assert task_params.top_k == 0
-    assert task_params.max_output_tokens == 1024
+    assert task_params.max_output_tokens == 1
     assert captured["suppress_empty_gemma4_thought_channel"] is True
     first_message = task_params.input[0]
     assert first_message.content == "hello"
@@ -260,10 +260,11 @@ def fake_all_sum(array: object, *, group: object, stream: object = None):
     )
 
 
-def test_warmup_inference_stops_after_first_generated_token(
+def test_warmup_inference_requests_one_token_and_finishes_normally(
     monkeypatch: pytest.MonkeyPatch,
 ) -> None:
     generated_tokens = 0
+    captured: dict[str, object] = {}
 
     def fake_apply_chat_template(
         *,
@@ -278,11 +279,12 @@ def fake_apply_chat_template(
     def fake_mx_barrier(_group: object) -> None:
         return None
 
-    def fake_mlx_generate(**_kwargs: object):
+    def fake_mlx_generate(**kwargs: object):
         nonlocal generated_tokens
-        for token in ("first", "second", "third"):
-            generated_tokens += 1
-            yield token
+        task = cast(TextGenerationTaskParams, kwargs["task"])
+        captured["max_output_tokens"] = task.max_output_tokens
+        generated_tokens += 1
+        yield "first"
 
     def fake_all_sum(array: object, *, group: object, stream: object = None):
         del group, stream
@@ -302,6 +304,7 @@ def fake_all_sum(array: object, *, group: object, stream: object = None):
         model_card=None,
     )
 
+    assert captured["max_output_tokens"] == 1
     assert generated_tokens == 1
     assert check_every == 100
 

From 856b30f7a8599f9044092fe716ff0ca8bff48913 Mon Sep 17 00:00:00 2001
From: Thomas Tupper <thomastupper@kite2.local>
Date: Fri, 10 Apr 2026 02:30:51 -0500
Subject: [PATCH 21/21] Force sequential generation for Gemma 4

---
 README.md                                     |  2 +-
 docs/kv-cache-backends.md                     |  1 +
 .../runner/llm_inference/batch_generator.py   |  1 +
 src/exo/worker/runner/llm_inference/runner.py | 54 +++++++++++++++++--
 .../test_turboquant_batch_guard.py            | 53 +++++++++++++++++-
 website/docs/kv-cache-backends.md             |  1 +
 6 files changed, 106 insertions(+), 6 deletions(-)

diff --git a/README.md b/README.md
index d0d484ec0..3b5c9ceb3 100644
--- a/README.md
+++ b/README.md
@@ -39,7 +39,7 @@ This is the running list of where Skulk diverges from upstream [exo](https://git
 - **OptiQ mixed-precision weight quantization pipeline** — async wrapper around `mlx-optiq`'s sensitivity analysis and KL-divergence per-layer bit allocation, exposed as a model-store optimization job.
 - **KV prefix cache with snapshot/restore** — LRU-evicted prompt-prefix cache that snapshots SSM and rotating-window cache states so prefix matches are reusable across conversation turns even for hybrid Mamba/Transformer architectures.
 - **Pipeline-parallel prefill for short prompts** — pipelined models now route every prefill through the pipeline path, fixing prior warmup hangs on Gemma-class models.
-- **Force-sequential fallback** — quantized backends transparently fall back to a sequential generator when batch/history mode is incompatible with their cache layout.
+- **Force-sequential fallback** — quantized backends and Gemma 4 transparently fall back to a sequential generator when batch/history mode is incompatible with their cache layout.
 
 ### Model capability system
 
diff --git a/docs/kv-cache-backends.md b/docs/kv-cache-backends.md
index af7356af1..d2fc3520d 100644
--- a/docs/kv-cache-backends.md
+++ b/docs/kv-cache-backends.md
@@ -127,6 +127,7 @@ Mixed cache layouts are supported:
 ## Current Limitations
 
 - All quantized KV cache backends force sequential generation (no batch/history mode)
+- Gemma 4 text generation also forces sequential generation for now because distributed BatchGenerator mode can produce degenerate repetition with its sliding-window cache layout
 - The optiq backend requires `mlx-optiq` to be installed (`pip install mlx-optiq`)
 - The optiq backend's `patch_attention()` monkey-patches MLX's SDPA — avoid switching between optiq and other backends within the same process lifetime without a restart
 - The rotorquant backends are disabled unless `SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1` is set. If selected without the gate, Skulk falls back to `default`.
diff --git a/src/exo/worker/runner/llm_inference/batch_generator.py b/src/exo/worker/runner/llm_inference/batch_generator.py
index 5673ef175..6065f75b2 100644
--- a/src/exo/worker/runner/llm_inference/batch_generator.py
+++ b/src/exo/worker/runner/llm_inference/batch_generator.py
@@ -62,6 +62,7 @@ def gen(self) -> Generator[T | None]:
 
 
 class InferenceGenerator(ABC):
+    group: mx.distributed.Group | None
     _cancelled_tasks: set[TaskId]
 
     def should_cancel(self, task_id: TaskId) -> bool:
diff --git a/src/exo/worker/runner/llm_inference/runner.py b/src/exo/worker/runner/llm_inference/runner.py
index 2e7822fcb..080cb0338 100644
--- a/src/exo/worker/runner/llm_inference/runner.py
+++ b/src/exo/worker/runner/llm_inference/runner.py
@@ -7,7 +7,13 @@
 from anyio import WouldBlock
 from mlx_lm.tokenizer_utils import TokenizerWrapper
 
-from exo.shared.models.model_cards import ModelCard, ModelTask
+from exo.shared.models.model_cards import (
+    ModelCard,
+    ModelTask,
+    OutputParserType,
+    PromptRendererType,
+    ToolCallFormat,
+)
 from exo.shared.types.chunks import (
     ErrorChunk,
     TokenChunk,
@@ -91,6 +97,28 @@ def _should_skip_llm_warmup(group_size: int) -> bool:
     return True
 
 
+def _is_gemma4_model(model_id: ModelId, model_card: ModelCard | None) -> bool:
+    """Return whether a model should be treated as a Gemma 4 runtime."""
+    normalized_model_id = str(model_id).lower().replace("_", "-")
+    if "gemma-4" in normalized_model_id or "gemma4" in normalized_model_id:
+        return True
+    if model_card is None:
+        return False
+
+    if model_card.vision is not None and model_card.vision.model_type == "gemma4":
+        return True
+
+    runtime = model_card.runtime
+    if runtime is not None and (
+        runtime.prompt_renderer == PromptRendererType.Gemma4
+        or runtime.output_parser == OutputParserType.Gemma4
+    ):
+        return True
+
+    tooling = model_card.tooling
+    return tooling is not None and tooling.tool_call_format == ToolCallFormat.Gemma4
+
+
 class ExitCode(str, Enum):
     AllTasksComplete = "AllTasksComplete"
     Shutdown = "Shutdown"
@@ -484,7 +512,7 @@ def build(
         kv_backend = get_kv_cache_backend()
         # TODO: Remove this forced sequential fallback once quantized KV cache
         # backends support BatchGenerator history merge/extract semantics.
-        force_sequential = kv_backend in (
+        force_sequential_for_kv_backend = kv_backend in (
             "mlx_quantized",
             "turboquant",
             "turboquant_adaptive",
@@ -492,14 +520,32 @@ def build(
             "rotorquant",
             "rotorquant_adaptive",
         )
-        if os.environ.get("EXO_NO_BATCH") or force_sequential:
-            if force_sequential and not os.environ.get("EXO_NO_BATCH"):
+        # Gemma 4 uses sliding-window RotatingKVCache entries. In distributed
+        # pipeline mode, the MLX BatchGenerator path currently produces
+        # degenerate token repetition after a valid prefill, while the
+        # sequential path is stable with the same prompt and default KV cache.
+        force_sequential_for_gemma4 = _is_gemma4_model(self.model_id, self.model_card)
+        no_batch_requested = os.environ.get("SKULK_NO_BATCH") or os.environ.get(
+            "EXO_NO_BATCH"
+        )
+        if (
+            no_batch_requested
+            or force_sequential_for_kv_backend
+            or force_sequential_for_gemma4
+        ):
+            if force_sequential_for_kv_backend and not no_batch_requested:
                 logger.warning(
                     "Quantized KV cache backend does not yet support "
                     "batch/history mode; forcing SequentialGenerator "
                     f"(kv_backend={kv_backend})"
                 )
                 logger.info(f"using SequentialGenerator (kv_backend={kv_backend})")
+            elif force_sequential_for_gemma4 and not no_batch_requested:
+                logger.warning(
+                    "Gemma 4 is not compatible with distributed BatchGenerator "
+                    "mode yet; forcing SequentialGenerator"
+                )
+                logger.info("using SequentialGenerator (model_family=gemma4)")
             else:
                 logger.info("using SequentialGenerator (batching disabled)")
             return SequentialGenerator(
diff --git a/src/exo/worker/tests/unittests/test_runner/test_turboquant_batch_guard.py b/src/exo/worker/tests/unittests/test_runner/test_turboquant_batch_guard.py
index a21cfa01f..cd470959a 100644
--- a/src/exo/worker/tests/unittests/test_runner/test_turboquant_batch_guard.py
+++ b/src/exo/worker/tests/unittests/test_runner/test_turboquant_batch_guard.py
@@ -4,8 +4,15 @@
 import pytest
 from mlx_lm.tokenizer_utils import TokenizerWrapper
 
+from exo.shared.models.model_cards import (
+    ModelCard,
+    ModelTask,
+    PromptRendererType,
+    RuntimeCapabilityCardConfig,
+)
 from exo.shared.types.common import ModelId
 from exo.shared.types.events import Event
+from exo.shared.types.memory import Memory
 from exo.shared.types.mlx import Model
 from exo.shared.types.tasks import TaskId
 from exo.utils.channels import mp_channel
@@ -29,7 +36,14 @@ class _FakeModel:
 
 @pytest.mark.parametrize(
     "kv_backend",
-    ["mlx_quantized", "turboquant", "turboquant_adaptive"],
+    [
+        "mlx_quantized",
+        "turboquant",
+        "turboquant_adaptive",
+        "optiq",
+        "rotorquant",
+        "rotorquant_adaptive",
+    ],
 )
 def test_builder_forces_sequential_for_quantized_kv_backends(
     monkeypatch: pytest.MonkeyPatch,
@@ -56,3 +70,40 @@ def test_builder_forces_sequential_for_quantized_kv_backends(
 
     assert isinstance(generator, SequentialGenerator)
     assert not isinstance(generator, BatchGenerator)
+
+
+def test_builder_forces_sequential_for_gemma4_runtime(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    _, cancel_recv = mp_channel[TaskId]()
+    event_send, _ = mp_channel[Event]()
+
+    monkeypatch.setattr(
+        "exo.worker.runner.llm_inference.runner.get_kv_cache_backend",
+        lambda: "default",
+    )
+
+    builder = Builder(
+        model_id=ModelId("local/custom-gemma-runtime"),
+        event_sender=event_send,
+        cancel_receiver=cancel_recv,
+        inference_model=cast(Model, cast(object, _FakeModel())),
+        tokenizer=cast(TokenizerWrapper, cast(object, _FakeTokenizer())),
+        group=cast(mx.distributed.Group | None, None),
+        model_card=ModelCard(
+            model_id=ModelId("local/custom-gemma-runtime"),
+            storage_size=Memory(),
+            n_layers=1,
+            hidden_size=1,
+            supports_tensor=False,
+            tasks=[ModelTask.TextGeneration],
+            runtime=RuntimeCapabilityCardConfig(
+                prompt_renderer=PromptRendererType.Gemma4
+            ),
+        ),
+    )
+
+    generator = builder.build()
+
+    assert isinstance(generator, SequentialGenerator)
+    assert not isinstance(generator, BatchGenerator)
diff --git a/website/docs/kv-cache-backends.md b/website/docs/kv-cache-backends.md
index 7956666ff..1636711d9 100644
--- a/website/docs/kv-cache-backends.md
+++ b/website/docs/kv-cache-backends.md
@@ -145,6 +145,7 @@ Mixed cache layouts are supported:
 ## Current Limitations
 
 - All quantized KV cache backends force sequential generation (no batch/history mode)
+- Gemma 4 text generation also forces sequential generation for now because distributed BatchGenerator mode can produce degenerate repetition with its sliding-window cache layout
 - The optiq backend requires `mlx-optiq` to be installed (`pip install mlx-optiq`)
 - The optiq backend's `patch_attention()` monkey-patches MLX's SDPA — avoid switching between optiq and other backends within the same process lifetime without a restart
 - The rotorquant backends are disabled unless `SKULK_ENABLE_EXPERIMENTAL_ROTORQUANT=1` is set. If selected without the gate, Skulk falls back to `default`.