Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,14 @@ Skulk now treats model capability handling as two layers:

This capability spine is the source of truth for model-aware reasoning defaults, prompt rendering, output parsing, tool-call handling, and additive `/v1/models` metadata consumed by the dashboard.

### Planned LARQL Slice Mode
Phase 1 LARQL ADRs live in `docs/adr/`. Phase 2 adds internal runner
supervision/readiness, but not slice placement:
- `LarqlRunner` is a Worker-managed runner subtype that supervises an upstream `larql serve` child process.
- The MLX runner remains the head runtime; LARQL peers serve cold FFN / expert slices.
- The MLX head never loads a vindex. Vindexes are consumed from HuggingFace and produced by the separate `skulk-vindex-publisher` repo.
- Phase 4 slice-placement code is blocked until the Phase 3 MLX FFN delegation spike confirms the design.

### Logging & Observability
Centralized logging uses a three-layer stack:
- **Structured JSON stdout**: When `logging.enabled` is `true` and `logging.ingest_url` is set in `skulk.yaml` (or dashboard Settings), skulk emits one JSON object per line on stdout (alongside human-readable stderr). Settings sync to all nodes via gossipsub. Configured in `src/exo/shared/logging.py`.
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ This starts a Vite dev server on port 3000 with hot reload. The dev server proxi
- `src/exo/worker/` — Worker node (inference, runner management, download coordination)
- `src/exo/store/` — Model store (registry, downloads, config, model optimizer)
- `src/exo/shared/` — Shared types, constants, topology
- `docs/adr/` — Architecture decision records for changes that affect runner taxonomy, placement, event-sourced state, APIs, or operator workflow
- `website/docs/` — Docusaurus documentation source, including API guide and model-capability docs

## Development Guidelines
Expand Down
62 changes: 62 additions & 0 deletions docs/adr/0001-larql-runner-type.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# ADR 0001: LARQL Runner Type

Status: Accepted for planning
Date: 2026-05-12
Parent roadmap: https://github.com/Foxlight-Foundation/Skulk/issues/173
Tracking issue: https://github.com/Foxlight-Foundation/Skulk/issues/153
Source plan: https://github.com/Foxlight-Foundation/Skulk/blob/claude/understand-larql-repo-sJqA1/docs/slice-placement-and-vindex-publisher-plan.md
Comment thread
ttupper92618 marked this conversation as resolved.

## Context

Skulk's current execution model is centered on worker-managed runner
subprocesses. A worker observes event-sourced placement state, downloads or
stages the assigned model artifacts, starts a runner subprocess, and reports
lifecycle transitions through the same event stream every node applies.

LARQL introduces a different execution role: a process that serves vindex-backed
FFN or expert slices over HTTP. That process still needs the same operational
properties Skulk expects from model runners: deterministic startup, supervised
shutdown, crash visibility, logging, readiness state, and eventual placement
metadata.

## Decision

Skulk will model LARQL as a first-class `LarqlRunner` runner type managed by
the worker. The worker will supervise a child `larql-server` process alongside
the existing MLX runner subprocesses.

The `LarqlRunner` will use Skulk's runner lifecycle conventions:

- worker-owned process supervision
- stdout/stderr forwarding into Skulk logging
- readiness and failure reporting through event-sourced state
- shutdown driven by instance and runner lifecycle events

The initial implementation will treat `larql-server` as an upstream binary
dependency, not as in-tree Rust or Python code.
Comment thread
ttupper92618 marked this conversation as resolved.

## Consequences

`LarqlRunner` becomes part of Skulk's runner taxonomy. Future implementation
work must add explicit runner metadata rather than overloading MLX shard
metadata or treating LARQL as an external sidecar.

Operators should be able to reason about LARQL-backed slices through the same
dashboard, state, diagnostics, and logging surfaces used for MLX runners.

Skulk remains insulated from LARQL internals. The integration boundary is the
LARQL server process and its HTTP contract.

## Rejected Alternatives

### Sidecar

A sidecar would be quick to prototype, but it moves process lifecycle,
readiness, logs, and crash recovery outside Skulk. That creates a second
operator workflow and makes slice placement harder to explain and diagnose.

### In-tree Port

Reimplementing LARQL's slice protocol inside Skulk would be a large, slow fork
of upstream LARQL. It would also make it harder to pick up future LARQL
improvements in vindex format, server behavior, and FFN/expert endpoints.
73 changes: 73 additions & 0 deletions docs/adr/0002-head-mlx-cold-larql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# ADR 0002: MLX Head With LARQL Cold Tier

Status: Accepted for planning; gated by Phase 3 feasibility
Comment thread
ttupper92618 marked this conversation as resolved.
Date: 2026-05-12
Parent roadmap: https://github.com/Foxlight-Foundation/Skulk/issues/173
Tracking issue: https://github.com/Foxlight-Foundation/Skulk/issues/154
Gate issues: https://github.com/Foxlight-Foundation/Skulk/issues/161 and https://github.com/Foxlight-Foundation/Skulk/issues/162
Source plan: https://github.com/Foxlight-Foundation/Skulk/blob/claude/understand-larql-repo-sJqA1/docs/slice-placement-and-vindex-publisher-plan.md
Comment thread
ttupper92618 marked this conversation as resolved.

## Context

Skulk's strongest runtime path is MLX on Apple Silicon. Slice placement should
extend that path instead of replacing it. The desired architecture lets a Mac
head node keep the hot attention/router path local while RAM-rich commodity
peers serve cold FFN or expert weights from LARQL vindexes.

This decision is expensive to reverse once placement state, runner assignment,
and API surfaces begin to encode slice responsibilities.

## Decision

The MLX runner remains the head runtime for slice mode. It owns the standard
MLX weights needed for the head role: embeddings, attention, norms, router, and
any locally assigned layers. It delegates selected per-layer FFN or expert work
to `LarqlRunner` peers over HTTP.

The MLX head never loads a vindex. Vindexes are cold-tier artifacts consumed by
LARQL peers.

The default wire format for delegated tensors is f16. i8 remains an explicit
future opt-in only where the LARQL contract supports it and Skulk can preserve
correctness.

## Feasibility Gate

This ADR is accepted for planning, not yet accepted for irreversible runtime
implementation. Phase 3 must prove that Skulk's MLX path can delegate a
per-layer FFN or expert step and continue generation with acceptable overhead.

If MLX does not expose usable hooks and a manual forward-pass split is too
fragile, too invasive, or too slow, this ADR must be superseded before Phase 4
slice-placement work starts.

## Consequences

Existing MLX single-node and MLX pipeline placement remain the default path for
models that fit on the selected head node.

Slice placement is additive. It is only considered when the normal MLX path
cannot fit the model or when the operator explicitly chooses a slice-mode flow
in future UI/API work.

The slice plan must identify which LARQL peer serves which preset, layer range,
expert range, and vindex URI so the MLX head can dispatch remote FFN/expert
calls deterministically.

## Rejected Alternatives

### Replace the Head Runtime With LARQL

Replacing MLX would discard Skulk's current strongest execution path and make
Apple Silicon performance dependent on a new serving stack.

### Load Vindexes on the Head

Loading vindexes on the head duplicates cold-tier storage and undermines the
purpose of using commodity RAM-rich peers for dormant weights.

### General Remote-Compute Abstraction

The v1 design targets LARQL's concrete FFN/expert server contract. A generic
remote execution abstraction would add surface area before Skulk has proven the
basic slice-mode value proposition.
55 changes: 55 additions & 0 deletions docs/adr/0003-vindex-provenance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# ADR 0003: Vindex Provenance

Status: Accepted for planning
Date: 2026-05-12
Parent roadmap: https://github.com/Foxlight-Foundation/Skulk/issues/173
Tracking issue: https://github.com/Foxlight-Foundation/Skulk/issues/155
Publisher issue: https://github.com/Foxlight-Foundation/Skulk/issues/156
Source plan: https://github.com/Foxlight-Foundation/Skulk/blob/claude/understand-larql-repo-sJqA1/docs/slice-placement-and-vindex-publisher-plan.md
Comment thread
ttupper92618 marked this conversation as resolved.

## Context

LARQL vindexes are directory-shaped artifacts derived from source model
weights. Extracting them can require substantial scratch disk, time, and
toolchain setup. Running that extraction on every Skulk user's machine would
make first use slow and operationally fragile.

Skulk already has model-store and download concepts for consuming artifacts.
The clean boundary is to make Skulk a vindex consumer and move extraction into
a dedicated publisher workflow.

## Decision

Skulk will consume vindexes from HuggingFace URIs such as `hf://...`. Skulk will
not extract vindexes in-tree.

Extraction, publication, manifest curation, and catalogue governance live in a
separate sibling repository: `skulk-vindex-publisher`.

The publisher repository owns scheduled LARQL extraction and publication jobs.
Skulk owns runtime consumption, local caching/staging, and placement metadata.

## Consequences

Skulk does not add a Rust toolchain or LARQL extraction dependency to its normal
runtime setup.

Future model-store work must support directory-shaped vindex artifacts, but it
does not need to know how to produce them.

The published vindex URI convention becomes part of the contract between the
publisher repo and Skulk's placement/runtime code.

## Rejected Alternatives

### Extract Inside Skulk

This would push heavyweight extraction work onto every operator machine,
including laptops that only need to run inference. It also expands Skulk's
runtime dependency surface for a build-time artifact-production task.

### No Curated Catalogue

Without a curated catalogue, users would need to find community vindexes or
produce their own before slice mode is useful. That weakens the operator
experience and makes supported-model behavior harder to reproduce.
27 changes: 27 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Architecture Decision Records

Architecture Decision Records capture choices that are expensive to reverse
once they affect Skulk's event-sourced state, runner taxonomy, placement
contracts, public APIs, or operator workflow.

## Format

Each ADR uses this structure:

- **Status:** Proposed, Accepted for planning, Accepted, Superseded, or Rejected.
- **Context:** The forces that made the decision necessary.
- **Decision:** The choice Skulk will implement.
- **Consequences:** Operational and implementation effects of the decision.
- **Rejected alternatives:** Options considered and why they were not chosen.

## Numbering

Use monotonically increasing four-digit filenames:

```text
0001-short-title.md
0002-short-title.md
```

Do not renumber existing ADRs. If a decision changes, add a new ADR that
supersedes the old one.
30 changes: 30 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,13 @@ This is where Skulk selects inference behavior such as:
- MLX execution path
- KV cache backend choice

Skulk's accepted LARQL planning ADRs add a runner subtype: `LarqlRunner`.
Phase 2 implements its internal worker-managed supervision path. It starts and
readiness-checks an upstream `larql serve` child process for directory-shaped
vindex artifacts, but placement does not create LARQL runners yet. Phase 3 of
the LARQL roadmap must first prove MLX can delegate per-layer FFN work with
acceptable overhead.

### API

The API server exposes:
Expand Down Expand Up @@ -153,6 +160,29 @@ Instead, it changes how model artifacts are sourced:

The rest of the system still uses the same master, worker, API, and placement model.

The LARQL roadmap extends this artifact model to vindexes. Skulk will consume
HuggingFace-hosted vindex directories, while extraction and publication live in
the separate `skulk-vindex-publisher` repository. Skulk does not extract
vindexes in-tree.

## Planned LARQL Slice Mode

The accepted planning ADRs in `docs/adr/` define the intended slice-mode shape:

- `LarqlRunner` is a first-class worker-managed runner type, not an unmanaged
sidecar.
- The MLX runner remains the head runtime. It owns the hot attention/router
path and delegates selected FFN or expert work to LARQL peers.
- The MLX head never loads a vindex; vindexes are cold-tier artifacts served by
LARQL runners.
- Slice mode is additive. Existing MLX placement remains unchanged for models
that fit on the selected head node.

Phase 2 implements internal `LarqlRunner` supervision and readiness state, but
slice mode is still not an operator-visible placement mode. Phase 4 placement
work is blocked until the Phase 3 MLX delegation spike confirms that the design
is viable.

## Where the Dashboard Fits

The dashboard is not a separate product or service.
Expand Down
30 changes: 26 additions & 4 deletions src/exo/download/coordinator.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@
from exo.download.download_utils import (
RepoDownloadProgress,
delete_model,
is_vindex_directory_complete,
map_repo_download_progress_to_download_progress_data,
resolve_model_in_path,
resolve_vindex_location,
)
from exo.download.shard_downloader import ShardDownloader
from exo.shared.constants import EXO_MODELS_DIR
Expand All @@ -40,7 +42,11 @@
DownloadPending,
DownloadProgress,
)
from exo.shared.types.worker.shards import PipelineShardMetadata, ShardMetadata
from exo.shared.types.worker.shards import (
LarqlShardMetadata,
PipelineShardMetadata,
ShardMetadata,
)
from exo.store.config import resolve_config_path
from exo.utils.channels import Receiver, Sender
from exo.utils.task_group import TaskGroup
Expand Down Expand Up @@ -342,7 +348,12 @@ async def _start_download(self, shard: ShardMetadata) -> None:
# clear the stale status and re-download from scratch.
if isinstance(status, DownloadCompleted) and status.model_directory:
model_dir = Path(status.model_directory)
if not model_dir.is_dir() or not (model_dir / "config.json").exists():
directory_is_complete = (
is_vindex_directory_complete(model_dir, shard.vindex_uri)
if isinstance(shard, LarqlShardMetadata)
else model_dir.is_dir() and (model_dir / "config.json").exists()
)
if not directory_is_complete:
logger.info(
f"DownloadCoordinator: {model_id} was DownloadCompleted but "
f"model directory {status.model_directory} no longer exists, re-downloading"
Expand All @@ -367,7 +378,16 @@ async def _start_download(self, shard: ShardMetadata) -> None:
return

# Check EXO_MODELS_PATH for pre-downloaded models
found_path = resolve_model_in_path(model_id)
vindex_location = (
resolve_vindex_location(model_id, shard.vindex_uri)
if isinstance(shard, LarqlShardMetadata)
else None
)
found_path = (
vindex_location.path
if vindex_location is not None
else resolve_model_in_path(model_id)
)
if found_path is not None:
logger.info(
f"DownloadCoordinator: Model {model_id} found in EXO_MODELS_PATH at {found_path}"
Expand All @@ -377,7 +397,9 @@ async def _start_download(self, shard: ShardMetadata) -> None:
node_id=self.node_id,
total=shard.model_card.storage_size,
model_directory=str(found_path),
read_only=True,
read_only=vindex_location.read_only
if vindex_location is not None
else True,
)
self.download_status[model_id] = completed
await self.event_sender.send(
Expand Down
Loading
Loading