contextlake

A Python toolkit for building semantic lakes over Markdown and plain-text files. Hybrid (vector + full-text) retrieval via LanceDB. Local-first multilingual embeddings via llama-cpp-python; OpenAI-compatible endpoints and Anthropic Claude work out of the box. Exposes a Python API, a Typer CLI, and an MCP server.

Highlights

• Multilingual by default — bge-m3 covers 100+ languages. English queries match Chinese / Japanese / Spanish content semantically.
• Hybrid retrieval — vector + BM25 fused with reciprocal-rank fusion (RRF), all through LanceDB's native APIs.
• RAG with parsed citations — Lake.ask() returns the answer plus Citation objects pointing back to the exact source file + line range.
• Local-first, but cloud is one env var away — same code path serves llama-cpp local GGUF, any OpenAI-compatible HTTP endpoint (OpenAI, Azure, Ollama, vLLM, LM Studio, DeepSeek, …), or Anthropic Claude.
• MCP server included — contextlake serve exposes 5 read-only tools (search / ask / get / multi-get / list-collections) over stdio or sse transport for Claude Desktop, Cursor, and other MCP clients.
• Config-only via env / .env — no TOML, no YAML, no surprises.

Install

uv sync --all-extras

Quick start

from contextlake import Lake

# Create a new lake at ./.contextlake
lake = Lake.create(path="./.contextlake")

# Register a collection. NOTE: relative root_path joins to the lake dir,
# not cwd. Use an absolute path for docs living outside the lake.
lake.add_collection(name="docs", root_path="/absolute/path/to/docs")

# Index the registered collections (incremental — SHA-256 diff under the hood)
report = lake.index()
print(f"indexed {report.total_chunks_written} chunks")

# Hybrid search (vector + BM25 + RRF). SearchHit is flat — no nested chunk.
hits = lake.search("how does indexing handle deletions?", top_k=5)
for h in hits:
    print(f"  {h.score:.3f}  {h.collection}/{h.source_path}  {h.heading_path}")

# Generate an answer with citations
answer = lake.ask("how do I install the package?")
print(answer.text)
for c in answer.citations:
    print(f"  [{c.n}] {c.collection}/{c.source_path}:{c.line_start}-{c.line_end}")

# Direct retrieval by #docid prefix (8-char SHA-256 with collision widening)
chunks = lake.get("#abcd1234")

# Or by path within a collection
chunks = lake.get("intro.md", collection="docs")

CLI

Once installed, the contextlake command is on your PATH:

# Lake lifecycle
contextlake init                              # create ./.contextlake
contextlake info                              # collections, chunks, contexts, model
contextlake config                            # effective config (api keys redacted)

# Collection management
contextlake collection add "$(pwd)/docs" --name docs
contextlake collection list                   # list collections
contextlake collection show docs              # one collection's metadata
contextlake collection rename docs guides     # hard relabel — moves chunks too
contextlake collection remove docs --yes      # drop chunks + collection row

# Indexing
contextlake index                             # incremental index of all collections
contextlake index docs                        # one collection only
contextlake reindex                           # drop chunks, re-index from scratch
contextlake update --pull                     # git pull in each collection root, then index
contextlake cleanup --yes                     # drop all chunks (keep collections)

# Search (three modes share -c, -n, --json, --files flags)
contextlake search "install"                  # BM25 keyword only
contextlake vsearch "install"                 # cosine vector only
contextlake query "install"                   # hybrid (RRF-fused) — recommended

# Retrieval
contextlake ask "how do I install?"           # RAG with citations
contextlake get "#abcd1234"                   # by docid prefix
contextlake get intro.md -c docs              # by path within a collection
contextlake multi-get "*.md" -c docs          # all chunks for a glob pattern
contextlake ls docs                           # list indexed files in a collection

# Context tree — human-written summaries attached to a collection/path-prefix
contextlake context add docs "Engineering docs root."
contextlake context list docs
contextlake context rm docs

# MCP server
contextlake serve --transport stdio           # for Claude Desktop, Cursor, …
contextlake serve --transport sse --port 8765 # network mode

Global options: --path <dir> overrides the default ./.contextlake location, --no-color strips ANSI codes, --no-emoji falls back to ASCII labels ([OK] / [WARN] / [ERR]). Search and listing commands accept --json for machine-readable output; search modes additionally accept --files for paths-only output.

Default models — first-run downloads

Trigger	Model	Size	Cached at
First contextlake init (embedding)	bge-m3 (Q8_0, 1024-dim, multilingual)	~660 MB	~/Library/Caches/contextlake/models/ (macOS)
First contextlake ask (LLM)	Qwen2.5-7B-Instruct (Q4_K_M)	~4.7 GB	same

index and search never touch the LLM. ask lazy-loads the LLM on first call and reuses it for the lifetime of the Lake instance. Both models fit comfortably on a Mac mini M4 16 GB.

Cloud / alternative providers

Switch via env vars — see .env.example for full commented templates.

# OpenAI-compatible cloud (also works for Together / DeepSeek / Moonshot)
export CONTEXTLAKE_EMBEDDING_PROVIDER=openai-compat
export CONTEXTLAKE_EMBEDDING_BASE_URL=https://api.openai.com/v1
export CONTEXTLAKE_EMBEDDING_MODEL_ID=text-embedding-3-small
export CONTEXTLAKE_EMBEDDING_DIM=1536
export OPENAI_API_KEY=sk-...

# Anthropic Claude for ask
export CONTEXTLAKE_LLM_PROVIDER=anthropic
export CONTEXTLAKE_LLM_MODEL_ID=claude-opus-4-7
export ANTHROPIC_API_KEY=sk-ant-...

# Local Ollama (no API key needed)
export CONTEXTLAKE_LLM_PROVIDER=openai-compat
export CONTEXTLAKE_LLM_BASE_URL=http://localhost:11434/v1
export CONTEXTLAKE_LLM_MODEL_ID=qwen2.5:7b

Architecture

Three layers, strict directionality:

contextlake/
├── core/         # pure logic — no langchain, no lancedb, no rich
│   ├── models.py   # value objects, Protocols, TypedDicts
│   ├── chunking.py # Chunker — markdown + text + line tracking
│   ├── indexing.py # Indexer — SHA-256 diff orchestration
│   ├── search.py   # Searcher — keyword / vector / hybrid + get / multi_get
│   └── ask.py      # Asker — RAG over SearcherProtocol + LLMProtocol
├── adapters/     # I/O wrappers — lancedb, langchain embeddings + LLMs
│   ├── store.py        # LakeDB (lancedb-backed StoreProtocol implementation)
│   ├── embeddings.py   # build_embeddings(settings) — llama-cpp + openai-compat
│   ├── llms.py         # build_llm(settings) — llama-cpp + openai-compat + anthropic
│   └── model_cache.py  # GGUF download / cache helpers
├── interfaces/   # outside-world surface
│   ├── cli.py        # Typer app, 21 subcommands
│   ├── tools.py      # build_tools(lake) — 5 LangChain @tool wrappers
│   ├── mcp_server.py # FastMCP server (serve_lake)
│   ├── _theme.py     # garden-themed icons + Rich color names
│   ├── _progress.py  # rich.progress wrapper
│   └── _serde.py     # SearchHit/Answer/Chunk → dict (shared CLI + MCP)
├── lake.py       # composition root — wires everything via settings
└── settings.py   # 7 frozen dataclasses, env-driven

core/ imports only from itself and settings.py. adapters/ may import from core/. interfaces/ may import from anywhere below it. Unit tests run against real LakeDB(tmp_path) instances rather than in-memory fakes — lancedb is local-file storage and create/destroy is cheap enough that maintaining a parallel FakeStore is net negative.

See ARCHITECTURE.md for the full architecture documentation and MCP.md for MCP server setup.

Running tests

uv run pytest tests/                              # all 355 tests
uv run pytest tests/unit                          # core/ pure-function tests
uv run pytest tests/integration                   # real LakeDB + cached GGUFs
uv run pytest tests/e2e                           # CLI via Typer CliRunner + MCP TestClient

Total suite: 355 passed. Integration tests that need a cached GGUF skip themselves cleanly when the corresponding model isn't on disk.