Embeddings endpoint for local RAG and semantic search

[weklund-agent]

Summary

Serve a local embeddings model alongside chat models, exposed through the existing localhost:4000/v1/embeddings endpoint. This enables local RAG, semantic search, and agent memory retrieval without sending data to cloud embedding APIs.

Problem

Agent frameworks increasingly rely on embeddings for:

• RAG (Retrieval-Augmented Generation): Semantic search over knowledge bases, documentation, and past conversations
• Agent memory: Finding relevant past interactions beyond keyword matching ("find that AWS project from February" should work even when the query uses different words)
• Semantic deduplication: Detecting when an agent is generating near-identical outputs (useful for loop detection)

Without a local embeddings endpoint, users must either:

1. Call a cloud embedding API (OpenAI, Voyage, Cohere) — adds latency, ongoing cost, and sends potentially sensitive data off-machine. This directly conflicts with the privacy motivation for running local models.
2. Fall back to keyword search (FTS5, BM25) — misses semantic matches, which degrades agent memory and RAG quality significantly.
3. Run a separate embedding service (Ollama, sentence-transformers) — another process to manage, another port to configure, outside mlx-stack's watchdog and health monitoring.

Use Case

A user running Hermes Agent 24/7 on their Mac Mini has the agent processing client projects, writing code, and doing research. Over weeks, the agent accumulates a knowledge base of past work. When the agent encounters a new task, it should be able to semantically search past work for relevant context.

Today with FTS5 keyword search: Query "optimize API response time" won't find past work tagged as "reduce endpoint latency" — different words, same meaning.

With local embeddings: Both phrases map to nearby vectors. The agent finds relevant context automatically.

Proposed Solution

Architecture

Embedding models are tiny — nomic-embed-text is 137M parameters (~300MB in int8). On a 64GB machine, this is noise. The model can be served as a fourth process alongside the three chat tiers.

┌─────────────────────────────────────────┐
│        LiteLLM Proxy (:4000)            │
│  /v1/chat/completions → chat tiers      │
│  /v1/embeddings → embedding server      │
├─────────────────────────────────────────┤
│  vllm-mlx :8000 (standard)             │
│  vllm-mlx :8001 (fast)                 │
│  vllm-mlx :8002 (longctx)             │
│  embed-server :8003 (embeddings)        │
└─────────────────────────────────────────┘

Serving Layer

vllm-mlx is designed for generative models, not encoder-only embedding models. Options for the embedding server:

Option A (recommended): Minimal FastAPI wrapper around mlx-lm

A lightweight server (~100-150 lines) that:

• Loads an MLX embedding model at startup
• Exposes /v1/embeddings matching the OpenAI API spec
• Handles batched requests
• Returns normalized vectors

This is simple, purpose-built, and has no heavy dependencies beyond mlx and fastapi.

Option B: Wait for Ollama backend support

Once Ollama is supported as a backend (planned for v0.2), embeddings come for free — ollama pull nomic-embed-text and configure LiteLLM to route /v1/embeddings to Ollama's endpoint. Zero custom code needed.

Recommendation: If Ollama backend support ships first, use Option B. If embeddings are needed before Ollama support, use Option A as a bridge.

Catalog Extension

Add an embeddings section to the catalog schema:

nomic-embed-text:
  name: "Nomic Embed Text v1.5"
  type: embedding
  source: "mlx-community/nomic-embed-text-v1.5-4bit"
  params_m: 137
  dimensions: 768
  max_sequence_length: 8192
  mteb_score: 62.28
  memory_mb: 280
  quantizations:
    - id: int4
      source: "mlx-community/nomic-embed-text-v1.5-4bit"
      memory_mb: 140
    - id: bf16
      source: "mlx-community/nomic-embed-text-v1.5-bf16"
      memory_mb: 280

Stack Definition

Add an optional embeddings tier:

# stack.yaml
standard:
  model: qwen3-32b
  quant: int4
  port: 8000
fast:
  model: qwen3-8b
  quant: int4
  port: 8001
embeddings:              # new, optional
  model: nomic-embed-text
  quant: int4
  port: 8003
  backend: mlx-embed     # or "ollama" once supported

LiteLLM Config

model_list:
  # ... existing chat models ...
  - model_name: "local-embed"
    litellm_params:
      model: "openai/nomic-embed-text"
      api_base: "http://localhost:8003/v1"
      api_key: "mlx-stack"

Recommended Embedding Models

Model	Params	Memory	MTEB Score	Notes
nomic-embed-text v1.5	137M	~140MB (int4)	62.28	Best size/quality ratio
bge-small-en-v1.5	33M	~70MB	51.68	Ultra-lightweight
all-MiniLM-L6-v2	22M	~50MB	49.54	Classic, widely used
mxbai-embed-large	335M	~670MB	64.68	Best quality, still tiny

Process Management

The embedding server integrates into the existing process management:

• Managed by the same watchdog
• Health checks via /v1/models or /health
• Included in mlx-stack status output
• Logs rotated alongside chat model logs
• Started/stopped with mlx-stack up / mlx-stack down

CLI Changes

# Include embeddings in init
mlx-stack init --with-embeddings

# Or add to existing stack
mlx-stack init --add-embeddings nomic-embed-text

# Verify
curl http://localhost:4000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "test query", "model": "local-embed"}'

Cost Justification

While cloud embedding costs are low ($0.00002/1K tokens for OpenAI text-embedding-3-small), for a 24/7 agent doing 1,000+ embedding calls per day, the reasons to go local are:

1. Privacy: Client data, code, and documents stay on-machine. This is the primary motivation.
2. Latency: Local embeddings return in <10ms vs 50-200ms for cloud API calls. Matters for real-time RAG during agent reasoning.
3. Availability: No dependency on external API uptime for a critical agent capability.
4. Simplicity: One fewer external service to configure, authenticate, and monitor.

Sequencing

Recommended: v0.3, after Ollama backend support (v0.2)

If Ollama ships as a backend in v0.2, then embeddings become nearly free to add:

• Ollama handles the serving
• LiteLLM handles the routing
• mlx-stack just needs the catalog entries and stack definition additions

Building a custom MLX embedding server before Ollama support means maintaining a component that becomes redundant.

Acceptance Criteria

• At least one embedding model in the catalog
• Embeddings tier configurable in stack definition
• mlx-stack init --with-embeddings includes embedding model
• Embedding server managed by process manager and watchdog
• LiteLLM routes /v1/embeddings to the embedding server
• Health checks include the embedding server
• mlx-stack status shows embedding server state
• Documentation with example RAG integration

[weklund]

Research: User Demand, Workarounds, and Priority Assessment

Do Users Express a Need?

Yes, but it is a "table stakes" need, not a burning pain point. Embeddings are a prerequisite for RAG, which nearly every serious agent setup uses. Community evidence:

• Multiple r/LocalLLaMA threads about choosing local embedding models, running them via Ollama, and struggling with embedding performance for RAG pipelines
• HuggingFace MLX community discussion explicitly requesting "Embed + Rerank API for Apple Silicon" because users migrating from CUDA/Ollama homelabs found no native MLX embedding server
• Qwen3-Embedding launch thread (Jun 2025) got 472 upvotes -- strong interest in new embedding models
• Fully-local RAG setups are a frequent community project ("Spent months building a fully offline RAG system", "Fully local code indexing with Ollama", etc.)

However, unlike the reliability layer (#29) where users are actively in pain and abandoning local models, embeddings are something users have working workarounds for. This is more "nice to have unified under one umbrella" than "I cannot get my work done."

What Problem Does It Solve?

The issue frames three problems -- privacy, latency, and operational simplicity:

Problem	Strength	Reality Check
Privacy	Strong	Users running local chat models for privacy are forced to send documents to OpenAI's embedding API -- genuinely contradictory
Latency	Moderate	Local <10ms vs cloud 50-200ms is real, but embedding calls are typically batched at index time, not on the hot path of agent reasoning
Simplicity	Modest	Adding Ollama or sentence-transformers is a 5-minute task for most users

What Do Users Do Today as Workarounds?

1. Ollama -- By far the most common. ollama pull nomic-embed-text and point your RAG framework at http://localhost:11434/v1/embeddings. Works, well-documented, and is what most local-first RAG tutorials recommend.

2. sentence-transformers / FastEmbed -- Python library that loads a model directly. No server needed, just from sentence_transformers import SentenceTransformer. Used heavily in LangChain/LlamaIndex.

3. Cloud APIs -- OpenAI's text-embedding-3-small at ~$0.02/1M tokens. Cheap enough that many users accept the privacy tradeoff.

4. vllm-mlx's built-in --embedding-model flag -- Key finding: vllm-mlx already has an embeddings endpoint. The config docs show --embedding-model as a server option, and there is a dedicated embeddings guide. This mirrors what we found in Agent-aware prefix cache sharing for KV cache reuse across requests #30 (KV caching) -- the upstream dependency already provides the capability.

Should We Build This? Priority Assessment

Honest assessment: this is a low-priority "completeness" feature, not a differentiator.

1. vllm-mlx already supports embeddings via --embedding-model. mlx-stack's job may just be wiring the flag through build_vllm_flags() and adding a catalog entry -- not building a custom embedding server (Option A in the issue).

2. The workarounds are good enough. Ollama handles this well. sentence-transformers is trivial. Users are not blocked.

3. It does not strengthen the value proposition. mlx-stack's unique value is multi-tier local serving with cloud fallback for agent workloads. Embeddings are a commodity -- every framework supports them. Adding an embedding endpoint does not make anyone choose mlx-stack over the alternatives.

4. The issue already recommends deferring to v0.3 and waiting for Ollama backend support, which confirms this is not urgent even per the original proposal.

Recommendation

• Priority: Low. Ship after Agent reliability layer: circuit breakers, loop detection, and automatic model escalation #29 (reliability layer, the "wow factor" differentiator) and Agent-aware prefix cache sharing for KV cache reuse across requests #30 (KV cache passthrough, a quick win).
• Implementation: Thin integration, not a custom server. When the time comes, wire vllm-mlx's --embedding-model flag through the stack config + catalog, add LiteLLM routing for /v1/embeddings, and integrate into the process manager. This is days of work, not weeks.
• Skip Option A (custom FastAPI wrapper) entirely. vllm-mlx already covers it upstream.

[m13v]

totally agree with the research here. from experience running local embeddings for a knowledge extraction pipeline, the privacy argument is the real driver. processing browser data and docs locally then sending to OpenAI for embeddings kind of defeats the purpose. nomic-embed-text on M1/M2 via MLX handles ~300 embeddings/sec for short passages, plenty for most agent workloads. option B (wait for Ollama backend) is smart since ollama embedding support is solid now. one thing worth adding to the architecture: a hash-based cache for repeated queries. in practice about 40% of agent embedding calls are near-duplicates and a simple cache cuts load significantly.

[m13v]

our local embedding pipeline with caching and batch processing for knowledge extraction: https://github.com/m13v/ai-browser-profile/blob/main/ai_browser_profile/embeddings.py