rust-rag

A local RAG (Retrieval-Augmented Generation) system that indexes your documents and lets you query them with context-aware LLM responses through LM Studio.

Why

You have a folder of documents -- PDFs, markdown files, source code, HTML pages. You want to ask questions about them using a local LLM without uploading anything to the cloud. rust-rag indexes those documents into a local vector database and automatically injects relevant passages into your prompts.

What it does

1. Scans a directory for supported files
2. Parses each file (extracts text from PDFs, strips HTML tags, reads plain text)
3. Chunks the text into overlapping segments
4. Embeds each chunk using a local embedding model via LM Studio
5. Stores embeddings in a local LanceDB vector database
6. Queries by embedding your question, finding the most similar chunks, and building a prompt with citations

Everything runs locally. No data leaves your machine.

Prerequisites

• Rust 1.85+ (2024 edition)
• LM Studio with models loaded:
  • An embedding model for indexing and search (e.g., nomic-embed-text, nomic-embed-text-v2-moe)
  • A chat model for interactive chat (e.g., qwen/qwen3.5-9b, llama-3.1-8b-instruct)

Setting up LM Studio

1. Install LM Studio
2. Download an embedding model (search for nomic-embed-text in the model browser)
3. Download a chat model (e.g., qwen3.5-9b)
4. Load both models in LM Studio (it can serve multiple models simultaneously)
5. Go to the Developer tab and start the server
6. Verify loaded models: curl http://localhost:1234/v1/models

Note the exact model IDs from the response — you'll need them for --embedding-model and --model flags.

Build

cargo build --release

Usage

Index your documents

rust-rag index --docs /path/to/documents --db /path/to/vectorstore

This scans the directory, parses all supported files, and stores their embeddings. An interactive TUI shows progress. Add --simple for a plain progress bar.

Chat (interactive TUI)

Launch a split-pane chat interface that shows retrieved source chunks alongside the model's streaming response.

rust-rag chat --db /path/to/vectorstore \
  --model qwen/qwen3.5-9b \
  --embedding-model text-embedding-nomic-embed-text-v2-moe

Both the embedding model and chat model must be loaded in LM Studio. Each question triggers:

1. An embedding call to search for relevant document chunks
2. A streaming chat completion to generate the answer with RAG context

+-- Rust RAG Chat -------------------------------------------+
| Model: qwen/qwen3.5-9b  |  DB: ./rag_db  |  History: [2/4]|
+--- Sources ---------------+--- Chat ----------------------+
| [0.95] guide.md           | You: How does allocation work?|
| "Use Box for heap..."     |                               |
|                            | AI: Based on the docs, Rust  |
| [0.87] types.md           | uses Box<T> for heap alloc...|
| "Smart pointers provide.."| (streaming...)               |
+----------------------------+-------------------------------+
| > Type your question here...                               |
+------------------------------------------------------------+

Keybindings:

Key	Action
Enter	Send message
Tab	Toggle focus between Sources and Chat panes
Up/Down	Scroll the focused pane
Esc	Cancel streaming / dismiss error
Ctrl+C	Quit

The chat keeps a sliding window of the last 4 exchanges for conversation continuity. Older messages are dropped to stay within the model's context window.

Flag	Default	Description
--model	llama3	Chat model name (must match LM Studio)
--limit	5	Number of source chunks to retrieve per query
--threshold	0.5	Minimum similarity score for chunks

Query (one-shot)

For scripting or single questions without the TUI:

rust-rag query "how does connection pooling work?" --db /path/to/vectorstore

Returns a formatted prompt with relevant citations from your indexed documents.

If the store is empty, pass --docs to auto-index first:

rust-rag query "explain the auth flow" --db ./db --docs /path/to/documents

Reindex

rust-rag reindex --docs /path/to/documents --db /path/to/vectorstore

Deletes the existing store and re-indexes everything. Add --force to also clear the record of previously failed files.

Stats

rust-rag stats --db /path/to/vectorstore

Shows total chunks and unique files in the store.

Options

Flag	Default	Description
--chunk-size	512	Words per chunk
--chunk-overlap	100	Overlapping words between chunks
--concurrency	4	Parallel file processing tasks
--embedding-model	nomic-embed-text	Model name as shown in LM Studio
--lm-studio-url	http://localhost:1234/v1	LM Studio server endpoint
--limit	5	Max results for queries
--threshold	0.5	Minimum similarity score (0-1)
--context-limit	0	Warn if prompt exceeds this many tokens (0 = no check)
--simple	false	Plain progress bar instead of TUI

Config file

Create rust-rag.toml in the current directory or ~/.config/rust-rag/rust-rag.toml to avoid repeating flags:

docs_dir = "/path/to/documents"
db_dir = "/path/to/vectorstore"
chunk_size = 512
chunk_overlap = 100
concurrency = 4
embedding_model = "text-embedding-nomic-embed-text-v1.5"
lm_studio_url = "http://localhost:1234/v1"
retrieval_limit = 5
retrieval_threshold = 0.5

Supported file formats

Category	Extensions
Text	.txt, .text, .log, .csv, .tsv
Markdown	.md, .markdown, .rst, .adoc
Source code	.rs, .ts, .js, .py, .go, .java, .c, .cpp, .h, .css, .sh, .sql, and more
Config	.json, .xml, .yaml, .yml, .toml, .ini, .cfg
PDF	.pdf
HTML	.html, .htm, .xhtml
EPUB	.epub

Project structure

crates/
  rust-rag-core/    Library: parsers, chunker, embedder, vector store, indexer, query engine, chat client
  rust-rag-cli/     Binary: CLI commands, indexing TUI, chat TUI, benchmarks

The library uses traits (Parser, Chunker, Embedder, VectorStore) so each component can be swapped or tested independently.

Important: embedding model matching. The --embedding-model used for chat and query must be the same model used during index. Different models produce different vector dimensions and the vector store search will fail if they don't match.

Development

cargo test --workspace    # Run tests
cargo clippy --workspace  # Lint
cargo fmt --all           # Format

License

ISC