A production-grade, fully on-premise RAG (Retrieval-Augmented Generation) stack for document understanding — built to showcase end-to-end AI solutioning skills and run on a single workstation, a private datacenter, or edge AI hardware like NVIDIA DGX Spark.
LLM-powered document workflows often touch regulated, confidential, or sovereign data — legal contracts, medical records, financial reports, internal policy, classified information. Sending that data to a third-party cloud API is frequently a non-starter.
This project demonstrates how to architect a complete RAG pipeline that never leaves your network:
- Models stay local — Ollama serves Qwen3-Coder 30B for analysis and
mxbai-embed-largefor embeddings, all on your GPU. - Storage stays local — PostgreSQL + pgvector for vectors, Elasticsearch for keyword search.
- Optional GPU acceleration — TensorRT-LLM profile for Blackwell-class hardware (DGX Spark).
- Zero outbound API dependency — air-gap friendly.
| Capability | What's demonstrated |
|---|---|
| Multi-agent RAG | ReAct-style analysis agent, query agent, and response normalizer agents working together |
| Hybrid retrieval | Vector (pgvector) + keyword (Elasticsearch BM25) fused with Reciprocal Rank Fusion (RRF) |
| Cross-encoder reranking | ms-marco-MiniLM-L-6-v2 re-scores top candidates for precision |
| Smart OCR | Tesseract first, low-confidence pages escalated to vision-LLM fallback (TensorRT-LLM optional) |
| Microservices architecture | Each AI capability isolated in its own container — independently scalable |
| GPU optimization | Flash Attention, model pinning, large context windows, FP4 quantization (Blackwell) |
| Production hygiene | Health checks, JWT auth, structured logging, CORS, env-driven config, profile-based deployment |
This is the kind of system you'd build for a customer who says "we love what GPT-4 can do, but our data can never leave our datacenter."
┌──────────────┐
Browser ──────────────────▶│ Frontend │ React + Vite + Tailwind
│ (port 3001) │
└──────┬───────┘
│ REST
▼
┌──────────────┐
│ Backend │ FastAPI orchestrator
│ (port 8001) │ • Auth (JWT)
└──────┬───────┘ • Document pipeline
│ • Multi-agent RAG
┌──────────────────────┼───────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Ollama │ │ Tesseract │ │ Embedding │
│ qwen3-coder │ │ OCR worker │ │ service │
│ + mxbai │ │ (parallel) │ │ (mxbai 1024) │
└──────────────┘ └──────────────┘ └──────────────┘
│ │
│ ┌──────────────┐ │
│ │ Reranker │◀─────────────┘
│ │ cross-encoder│
│ └──────────────┘
│
▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ PostgreSQL │ │Elasticsearch │ │ TensorRT-LLM │
│ + pgvector │ │ (BM25) │ │ (optional, │
│ │ │ │ │ Blackwell) │
└──────────────┘ └──────────────┘ └──────────────┘
Every box is a Docker container, wired together by docker-compose.yml.
- Docker 24+ and Docker Compose v2+
- NVIDIA GPU with
nvidia-container-toolkit(for the LLM tier)- For CPU-only experimentation, see CPU Mode
- ~50 GB free disk (most of which is model weights)
git clone https://github.com/<your-username>/onprem-document-intelligence.git
cd onprem-document-intelligence
# Optional: customise via env file
cp .env.example .env
# Start everything
docker compose up -d
# Watch the model pull on first run (5-15 min depending on bandwidth)
docker compose logs -f ollama-initdocker compose ps # all services should be healthy
curl http://localhost:8001/api/health # backend health
curl http://localhost:11434/api/tags # ollama models loadedVisit http://localhost:3001 and sign in with the default credentials:
| Username | admin |
| Password | changeme123 |
Change these immediately by editing
.env(APP_USERNAME,APP_PASSWORD,SECRET_KEY) and restarting the backend.
docker compose down # stop containers
docker compose down -v # also remove volumes (wipes models + data)The stack scales from a developer laptop to a flagship edge AI box.
| Tier | Hardware | Mode | Suggested env overrides |
|---|---|---|---|
| Edge / Workstation | RTX 4070/4080 (12-16 GB) | Default | OLLAMA_NUM_PARALLEL=2, smaller OLLAMA_MODEL (e.g. qwen2.5:7b) |
| Pro Workstation | RTX 4090 / A6000 (24-48 GB) | Default | OLLAMA_NUM_PARALLEL=4 (default) |
| Server-class | A100 / H100 (40-80 GB) | Default + --profile gpu |
OLLAMA_NUM_PARALLEL=8 |
| DGX Spark / Blackwell | GB10 / GB200 (128 GB+) | --profile trtllm --profile gpu |
OLLAMA_NUM_PARALLEL=16, TRTLLM_QUANTIZATION=fp4 |
# Enable both GPU preprocessing and TensorRT-LLM vision OCR
OLLAMA_NUM_PARALLEL=16 \
TESSERACT_MAX_WORKERS=64 \
TRTLLM_ENABLED=true \
docker compose --profile trtllm --profile gpu up -dThis pins models to VRAM, enables Flash Attention 2, and uses FP4 quantization on Blackwell tensor cores for the vision OCR fallback.
No GPU? You can still run the full stack — Ollama will fall back to CPU inference (slow, but functional).
# In .env, override the model to something small enough for CPU:
OLLAMA_MODEL=qwen2.5:3b
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
EMBEDDING_DIMENSION=768
# Comment out runtime: nvidia in docker-compose.yml under the ollama service,
# or simply remove --gpus from your daemon. Then:
docker compose up -dFor a hosted-API CPU mode, set LLM_PROVIDER=groq and supply GROQ_API_KEY (defeats the on-prem goal but useful for demos).
All knobs are environment variables. Copy .env.example to .env and adjust.
| Var | Default | Notes |
|---|---|---|
APP_USERNAME |
admin |
Login username |
APP_PASSWORD |
changeme123 |
Login password — change this |
SECRET_KEY |
please-change-... |
JWT signing key — change this |
| Var | Default |
|---|---|
POSTGRES_USER |
appuser |
POSTGRES_PASSWORD |
changeme123 |
POSTGRES_DB |
documents |
| Var | Default | Notes |
|---|---|---|
OLLAMA_MODEL |
qwen3-coder:30b |
Pulled on first start |
OLLAMA_EMBEDDING_MODEL |
mxbai-embed-large |
1024-dim |
OLLAMA_NUM_PARALLEL |
4 |
Concurrent decode requests |
OLLAMA_CONTEXT_LENGTH |
65536 |
Context window |
EMBEDDING_DIMENSION |
1024 |
Must match embedding model |
| Var | Default |
|---|---|
VECTOR_SEARCH_WEIGHT |
0.6 |
KEYWORD_SEARCH_WEIGHT |
0.4 |
RRF_K |
60 |
RERANKER_ENABLED |
true |
| Var | Default | Notes |
|---|---|---|
OCR_DEFAULT_LANG |
eng |
Tesseract language code(s) |
OCR_MAX_WORKERS |
16 |
Parallel page workers |
OCR_CONFIDENCE_THRESHOLD |
0.93 |
Below this, escalate to vision LLM |
TRTLLM_ENABLED |
false |
Set true with --profile trtllm |
See backend/app/config.py for the full list.
| Service | Port | Role |
|---|---|---|
frontend |
3001 | React SPA — upload, analyze, chat |
backend |
8001 | FastAPI — auth + RAG orchestrator |
ollama |
11434 | LLM + embedding inference |
tesseract |
8100 | OCR microservice |
embedding-service |
8200 | Cached embedding gateway |
reranker-service |
8300 | Cross-encoder reranker |
gpu-preprocess |
8400 | Optional GPU image preprocessing (--profile gpu) |
tensorrt-llm |
8355 | Optional vision OCR (--profile trtllm) |
postgres |
5432 | Metadata + vectors |
elasticsearch |
9200 | Keyword index |
The backend exposes an OpenAPI schema at http://localhost:8001/docs. Highlights:
# Login
curl -X POST http://localhost:8001/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "username=admin&password=changeme123"
# Upload a document
curl -X POST http://localhost:8001/api/documents/upload \
-H "Authorization: Bearer $TOKEN" \
-F "file=@/path/to/contract.pdf"
# List documents
curl http://localhost:8001/api/documents \
-H "Authorization: Bearer $TOKEN"
# Ask a question (RAG)
curl -X POST http://localhost:8001/api/chat \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"document_id": 1, "message": "What is the deadline mentioned in this document?"}'
# Trigger structured analysis
curl -X POST http://localhost:8001/api/documents/1/analyze \
-H "Authorization: Bearer $TOKEN".
├── backend/ FastAPI app, RAG orchestration, agents
│ ├── app/
│ │ ├── main.py API surface
│ │ ├── chat_service.py Multi-agent conversational RAG
│ │ ├── analysis_service.py ReAct-style structured analysis
│ │ ├── rag_service.py Hybrid retrieval + RRF
│ │ ├── acronym_service.py Domain-specific glossary agent (extensible)
│ │ ├── elasticsearch_service.py
│ │ └── config.py All env-driven settings
│ ├── Dockerfile Default backend image
│ └── Dockerfile.optimized Blackwell/CUDA-tuned image
├── frontend/ React + Vite + Tailwind SPA
├── embedding-service/ Embedding microservice (Ollama-backed)
├── reranker-service/ Cross-encoder reranker microservice
├── tesseract-service/ OCR microservice
├── gpu-preprocess-service/ Optional GPU image preprocessing
├── prompts/ Prompt templates for agents
├── scripts/start.sh Convenience launcher (default / optimized / cpu)
├── docker-compose.yml Orchestration
└── .env.example Configuration template
The system was built around a document-analysis use case but the architecture is domain-agnostic. To adapt it:
- Edit
prompts/document_analysis.txt— change the persona and the structured fields the analysis agent extracts (e.g. clauses, parties, financial terms for legal; symptoms, medications for clinical). - Tune
acronym_service.py— bundles a domain glossary agent. Replace the seed dictionary with terminology specific to your vertical (legal, medical, finance, military, etc.). - Adjust retrieval weights —
VECTOR_SEARCH_WEIGHTvsKEYWORD_SEARCH_WEIGHTdepending on whether your queries are semantic or keyword-heavy.
That's all most adaptations need — the rest of the pipeline is generic.
"CUDA out of memory" or Ollama keeps OOM-ing
Drop OLLAMA_NUM_PARALLEL to 1, switch OLLAMA_MODEL to a smaller model (e.g. qwen2.5:7b or qwen2.5:3b), and reduce OLLAMA_CONTEXT_LENGTH.
Backend can't reach Ollama
docker compose ps # is ollama healthy?
docker compose logs ollama --tail 50
docker exec onprem-ollama ollama list # are models pulled?OCR quality is poor
- Verify the document is actually a scanned image (not corrupted).
- For multi-language docs, set
OCR_DEFAULT_LANG=eng+deu(etc.). - Enable
--profile trtllmto fall back to a vision LLM on low-confidence pages.
Search returns nothing
curl http://localhost:9200/_cat/indices?v # is the document indexed?
docker compose logs backend | grep -i indexRe-upload the document if the index is empty.
GPU not detected inside containers
docker run --rm --gpus all nvidia/cuda:12.3.1-base-ubuntu22.04 nvidia-smiIf that fails, install nvidia-container-toolkit and restart the Docker daemon.
- Streaming responses end-to-end (token-by-token over SSE)
- Document-level access control / multi-tenant
- Semantic caching of frequent queries
- Native Triton Inference Server profile for non-Ollama deployments
- Eval harness with Ragas
MIT — free to fork, adapt, and use as a reference architecture for your own on-prem AI projects.
Built as a portfolio piece to demonstrate full-stack AI solutioning: model selection, retrieval architecture, microservice decomposition, GPU optimization, container orchestration, and developer experience — all stitched together into something that actually starts with docker compose up.
If you find it useful, a star is appreciated. PRs welcome.