datavault

Embedded knowledge base for RAG applications.

datavault is a self-contained Rust crate that packages a production-grade knowledge-base subsystem: ingest documents (PDF, markdown, image, office), chunk them, embed via OpenRouter (or any TEI-compatible endpoint), persist to SQLite + sqlite-vec, and retrieve with hybrid vector + BM25 search plus optional rerank. Ships as a library, a CLI (datavault), and an HTTP server (datavault-server).

Quickstart

# Install (once published to crates.io)
cargo install datavault

# Or build from source
cargo build --release

# Set your embedding key
export OPENROUTER_API_KEY=sk-or-v1-...

# Ingest a document
datavault ingest ./paper.pdf --category research

# Search
datavault search "what does the abstract say about attention?" --top 5

# List, get, delete
datavault list
datavault get <document_id>
datavault delete <document_id>           # soft-delete
datavault delete <document_id> --hard    # purge rows

# Maintenance
datavault drift           # which chunks were embedded with a stale model?
datavault re-embed        # re-embed every chunk with the current model

Features

• Storage — SQLite with the sqlite-vec extension for ANN search, plus FTS5 for BM25 lexical scoring. Single-file database; zero ops.
• Hybrid retrieval — reciprocal rank fusion combines vector and BM25 results. Both arms run concurrently.
• Pluggable embeddings — OpenRouter, plain TEI servers, or any HTTP endpoint with an OpenAI-compatible /embeddings shape.
• Pluggable rerankers — LLM reranker (any chat-completion model), Cohere /rerank endpoint, or a vLLM scorer.
• Extractors — pdf-extract (text-layer PDFs), MinerU (table-aware extraction service), vision-LLM (image-heavy PDFs), and a smart router that picks per page.
• Chunkers — recursive character chunker plus a "smart" markdown-aware chunker that preserves headings and code blocks.
• Query expansion + standalone-query rewriter + contextual prefixing — all opt-in.
• Office files (xlsx, docx, ods) behind the office feature.

Architecture

┌──────────────┐    ┌──────────────────┐    ┌──────────────────┐
│ datavault    │    │ datavault-server │    │ Your Rust app    │
│ CLI (TOON)   │    │ axum HTTP (JSON) │    │ (library API)    │
└──────┬───────┘    └────────┬─────────┘    └────────┬─────────┘
       └──────────────┬──────┴──────────────────────┘
                      ▼
          ┌──────────────────────────┐
          │ datavault::{retrieve,    │
          │   chunk, embed, extract, │
          │   rerank, maintenance}   │
          └──────────┬───────────────┘
                     ▼
          ┌──────────────────────────┐
          │ datavault::store::sqlite │
          │ (rusqlite + sqlite-vec)  │
          └──────────────────────────┘

Trait objects (KbStore, EmbeddingProvider, Reranker, Extractor) let you swap any layer without touching the rest of the pipeline.

Configuration

All runtime knobs are env-driven. Common ones:

Env var	Default	Purpose
DATAVAULT_DB_PATH	XDG data dir	SQLite database path
KB_DB_PATH	(alias)	Backward-compatible alias for DATAVAULT_DB_PATH
OPENROUTER_API_KEY	—	Shared fallback for embed / chat endpoints
KB_EMBEDDING_MODEL	qwen/qwen3-embedding-8b	Model identifier
KB_EMBEDDING_DIM	4096	Embedding dimensionality
KB_EMBEDDING_BASE_URL	https://openrouter.ai/api/v1/embeddings	Embedding endpoint
KB_RERANK_ENABLED	false	Enable the rerank stage
KB_RERANK_PROVIDER	(empty)	llm, cohere, or vllm
KB_HYBRID_BM25_ENABLED	true	Run BM25 alongside vector search
KB_QUERY_EXPANSION_ENABLED	false	Paraphrase the query before retrieval
KB_CONTEXTUAL_RETRIEVAL_ENABLED	false	Generate per-chunk LLM context lines at ingest

See docs/user-guide.md for the full reference.

HTTP API

datavault-server exposes /api/v1/kb/* (JSON). Mount it directly or behind a reverse proxy.

Method	Route	Purpose
POST	/api/v1/kb/search	Hybrid retrieval
POST	/api/v1/kb/documents	Ingest a file (multipart)
GET	/api/v1/kb/documents	List documents
GET	/api/v1/kb/documents/{id}	Fetch one document
DELETE	/api/v1/kb/documents/{id}	Soft- or hard-delete
GET	/api/v1/kb/drift	Stale-embedding report
POST	/api/v1/kb/re-embed	Re-embed every chunk

Bearer auth is opt-in:

DATAVAULT_AUTH_TOKENS=alpha,beta datavault-server --require-auth --bind 0.0.0.0:8080
curl -H "Authorization: Bearer alpha" http://host:8080/api/v1/kb/documents

Development

cargo build                       # debug
cargo build --release             # optimized
cargo build --features office     # +docx/xlsx/ods
cargo test                        # unit + integration
cargo clippy --all-targets -- -D warnings
cargo fmt -- --check
cargo bench --bench retrieval     # criterion bench

Integration tests live under tests/integration/; an umbrella file tests/integration.rs declares the per-module submodules. Tests that hit OpenRouter live behind #[ignore] and need OPENROUTER_API_KEY.

License

Apache-2.0. See LICENSE.

---

Built from the rantaiclaw KB port — see docs/user-guide.md.