fix(core): improve retrieval quality and pipeline observability

.claude/CLAUDE.md

@@ -2,6 +2,38 @@
 
 @../.s2s/CONTEXT.md
 
+## API interaction
+
+Before making **any** API call (curl, httpie, scripts), consult `docs/reference/api.md` or the live OpenAPI spec at `/openapi.json` to verify:
+- Authentication method and header format
+- Parameter names, types, and whether they are query, path, or body params
+- Request body field names (e.g. `question` vs `query`)
+- Correct endpoint for the task (e.g. `/api/v1/query` for RAG, `/api/v1/search` for raw vector search)
+
+Do not construct API calls from memory or guesswork.
+
+## Database investigation
+
+When querying Postgres directly (psql, DB inspection):
+- **Always run `\d table_name` first** to check column names and types. Common pitfalls: `namespace_id` (not `namespace`), `bytea` columns that need decryption, columns that don't exist.
+- **Conversation turns are encrypted**: `question` and `answer` are `pgp_sym_encrypt()`'d. To read them: `SELECT pgp_sym_decrypt(question, '<key>') FROM conversation_turns WHERE ...` using `VEKTRA_CONVERSATION_KEY` from `.env`.
+- **Postgres holds metadata/text, Qdrant holds vectors**: chunk text is in `document_chunks` (Postgres), but vector search runs against Qdrant (check `VEKTRA_QDRANT_URL` and `VEKTRA_QDRANT_COLLECTION` in config for host/port/collection). To inspect vectors or search results, query Qdrant REST API directly. The Qdrant payload uses `namespace_id` as the namespace filter field.
+
+## Query pipeline vs search endpoint
+
+These are fundamentally different:
+
+| | `/api/v1/search` | `/api/v1/query` |
+|--|--|--|
+| What it does | Raw vector search (Qdrant only) | Full RAG pipeline |
+| Reranker | No | Yes |
+| Threshold filter | No | Yes |
+| LLM call | No | Yes |
+| Field for question | `query` | `question` |
+| Response field | `results[].text_snippet` | `sources[].snippet` |
+
+When investigating **end-to-end answer quality** (retrieval + reranking + LLM), use `/api/v1/query`. When investigating **raw retrieval quality** (vector similarity only, no reranker), use `/api/v1/search`.
+
 ## Spec2Ship Commands
 
 - `/s2s:specs` - Define requirements via roundtable

.env.example

@@ -77,7 +77,7 @@
 # --------------------------------------------------------------------------
 
 # VEKTRA_QUERY_PIPELINE=advanced
-# VEKTRA_MIN_RELEVANCE_SCORE=0.3
+# VEKTRA_MIN_RELEVANCE_SCORE=0.15
 # VEKTRA_CHUNK_DEDUP_ENABLED=true
 # VEKTRA_RESPONSE_TOKEN_RESERVE=2048
 # VEKTRA_CONTEXT_CHUNK_RATIO=0.6
@@ -89,8 +89,8 @@
 
 # Reranking
 # VEKTRA_RERANK_ENABLED=true
-# VEKTRA_RERANK_PROVIDER=flashrank
-# VEKTRA_RERANK_MODEL=
+# VEKTRA_RERANK_PROVIDER=cross-encoder
+# VEKTRA_RERANK_MODEL=BAAI/bge-reranker-v2-m3
 # VEKTRA_RERANK_TOP_K=5
 
 # --------------------------------------------------------------------------
@@ -112,6 +112,7 @@
 # Fallback model when primary times out.
 # VEKTRA_LLM_FALLBACK_MODEL=
 # VEKTRA_LLM_FALLBACK_TIMEOUT_MS=60000
+# VEKTRA_LLM_CONTEXT_WINDOW=           # Required for models not in litellm registry (e.g. local vLLM). Example: 32768
 # VEKTRA_LLM_CONTEXT_ONLY_ENABLED=true
 
 # --------------------------------------------------------------------------

.github/workflows/integration.yml

@@ -10,7 +10,7 @@ jobs:
   integration:
     name: Integration tests + NFR gates
     runs-on: ubuntu-latest
-    timeout-minutes: 20
+    timeout-minutes: 30
 
     steps:
       - uses: actions/checkout@v6
@@ -72,8 +72,8 @@ jobs:
           VEKTRA_BOOTSTRAP_KEY: ci-bootstrap-key
           STARTUP_MS: ${{ steps.startup.outputs.startup_ms }}
 
-      - name: Query latency measurement (warn only)
-        if: always() && !cancelled()
+      - name: Query latency measurement (warn only, post-merge only)
+        if: always() && !cancelled() && github.event_name == 'push'
         continue-on-error: true
         run: |
           uv run pytest tests/nfr/test_performance.py \

.gitignore

@@ -53,6 +53,9 @@ vektra-learn/static/
 coverage/
 .nyc_output/
 
+# Evaluation results (generated by eval harness)
+tests/eval/results_*.jsonl
+
 # Temporary files
 tmp/
 temp/

Dockerfile

@@ -70,11 +70,10 @@ COPY vektra-app/src vektra-app/src
 # Include optional extras for Phase 2 vector store and sparse search support.
 # When INSTALL_UNSTRUCTURED=true, also install the Unstructured PDF extractor.
 ARG INSTALL_UNSTRUCTURED=false
-RUN uv sync --frozen --no-editable --no-dev \
-    && uv pip install 'qdrant-client==1.17.0' 'fastembed==0.7.4' \
-    && if [ "$INSTALL_UNSTRUCTURED" = "true" ]; then \
-       uv pip install 'torch' 'torchvision' --index-url https://download.pytorch.org/whl/cpu --reinstall \
-       && uv pip install 'unstructured[pdf]>=0.15' 'pi-heif' 'sentence-transformers' --reinstall; \
+RUN if [ "$INSTALL_UNSTRUCTURED" = "true" ]; then \
+       uv sync --frozen --no-editable --no-dev --extra sparse --extra qdrant --extra ocr; \
+    else \
+       uv sync --frozen --no-editable --no-dev --extra sparse --extra qdrant; \
     fi
 
 # --------------------------------------------------------------------------

Makefile

@@ -33,7 +33,7 @@ endif
 # Targets
 # --------------------------------------------------------------------------
 
-.PHONY: help up down health ingest query demo logs test lint reindex batch-ingest
+.PHONY: help up down health ingest query demo logs test lint reindex batch-ingest eval-retrieval eval-e2e
 
 help: ## Show available targets
 	@awk 'BEGIN {FS = ":.*##"} /^[a-zA-Z_-]+:.*##/ { \
@@ -98,3 +98,9 @@ reindex: ## Create reindex job (skeleton): make reindex VER=2 [NS=default]
 batch-ingest: ## Batch ingest files: make batch-ingest DIR=path/to/docs [NS=default]
 	$(if $(DIR),,$(error DIR is required. Usage: make batch-ingest DIR=path/to/docs))
 	@scripts/batch-ingest.sh "$(DIR)" "$(or $(NS),default)"
+
+eval-retrieval: ## Run retrieval-only evaluation (no LLM calls)
+	uv run python scripts/eval_retrieval.py $(EVAL_ARGS)
+
+eval-e2e: ## Run end-to-end RAG evaluation (with LLM calls)
+	uv run python scripts/eval_e2e.py $(EVAL_ARGS)

docs/reference/configuration.md

@@ -27,21 +27,33 @@ If using a cloud provider, also set the corresponding API key (see [LLM provider
 | `VEKTRA_LLM_PROVIDER` | str | (required) | LLM model in litellm format |
 | `VEKTRA_LLM_API_KEY` | str | - | API key for the LLM provider. Not needed for Ollama. |
 | `VEKTRA_LLM_API_BASE` | str | - | Custom API base URL for OpenAI-compatible providers (e.g., vLLM, LMStudio) |
+| `VEKTRA_LLM_EXTRA_BODY` | json | - | Extra JSON body passed to litellm (e.g. `{"chat_template_kwargs": {"enable_thinking": false}}` for vLLM thinking models) |
 | `VEKTRA_LLM_FALLBACK_MODEL` | str | - | Fallback model when the primary times out |
 | `VEKTRA_LLM_FALLBACK_TIMEOUT_MS` | int | `60000` | Milliseconds before switching to fallback model |
+| `VEKTRA_LLM_CONTEXT_WINDOW` | int | - | Context window size in tokens. Required for models not in litellm's registry (e.g. local vLLM). If unset, litellm lookup with 4096 fallback (warns on fallback) |
 | `VEKTRA_LLM_CONTEXT_ONLY_ENABLED` | bool | `true` | Return raw chunks without LLM synthesis when both models fail |
 | `VEKTRA_STARTUP_LLM_CHECK` | bool | `true` | Test LLM connectivity at startup (warning-only, not fatal) |
 
 ### LLM provider keys
 
-Set one based on your `VEKTRA_LLM_PROVIDER`:
+Use `VEKTRA_LLM_API_KEY` for any provider. It is passed directly to litellm and works with OpenAI, Anthropic, vLLM, and any OpenAI-compatible endpoint.
 
-| Variable | Provider |
-|----------|----------|
-| `OPENAI_API_KEY` | OpenAI (`openai/*`) |
-| `ANTHROPIC_API_KEY` | Anthropic (`anthropic/*`) |
+Provider-specific environment variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) are also recognized by litellm as a fallback, but `VEKTRA_LLM_API_KEY` takes precedence when set.
 
-These are standard provider environment variables recognized by litellm. `VEKTRA_LLM_API_KEY` can also be used as a generic alternative.
+### Using a local vLLM instance
+
+```bash
+VEKTRA_LLM_PROVIDER=openai//models/your-model-name
+VEKTRA_LLM_API_KEY=<vllm-api-key>
+VEKTRA_LLM_API_BASE=http://<vllm-host>:8000/v1
+VEKTRA_LLM_CONTEXT_WINDOW=32768
+# For models with thinking mode (e.g. Qwen3.5, DeepSeek-R1), disable it:
+VEKTRA_LLM_EXTRA_BODY={"chat_template_kwargs": {"enable_thinking": false}}
+```
+
+In `.env` files (Docker Compose), JSON values do not need quoting. In shell, use single quotes: `VEKTRA_LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'`.
+
+The model name must match the vLLM `--model` path exactly (e.g., `/models/qwen35-27b`).
 
 ## Embedding
 
@@ -67,7 +79,7 @@ These are standard provider environment variables recognized by litellm. `VEKTRA
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
 | `VEKTRA_QUERY_PIPELINE` | str | `advanced` | Pipeline implementation: `simple`, `advanced` |
-| `VEKTRA_MIN_RELEVANCE_SCORE` | float | `0.3` | Minimum cosine similarity for chunk inclusion (0.0-1.0). Chunks below this threshold are filtered out. |
+| `VEKTRA_MIN_RELEVANCE_SCORE` | float | `0.15` | Minimum relevance score for chunk inclusion (0.0-1.0). Safety net filter; top-k is the primary control. |
 | `VEKTRA_CHUNK_DEDUP_ENABLED` | bool | `true` | Deduplicate overlapping adjacent chunks from the same document |
 | `VEKTRA_RESPONSE_TOKEN_RESERVE` | int | `2048` | Tokens reserved for LLM response generation |
 | `VEKTRA_CONTEXT_CHUNK_RATIO` | float | `0.6` | Fraction of context window allocated to retrieved chunks (0.0-1.0) |
@@ -85,8 +97,8 @@ These are standard provider environment variables recognized by litellm. `VEKTRA
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
 | `VEKTRA_RERANK_ENABLED` | bool | `true` | Enable cross-encoder reranking after retrieval |
-| `VEKTRA_RERANK_PROVIDER` | str | `flashrank` | Reranking provider: `flashrank`, `cross-encoder`, `cohere` |
-| `VEKTRA_RERANK_MODEL` | str | - | Provider-specific reranking model name |
+| `VEKTRA_RERANK_PROVIDER` | str | `cross-encoder` | Reranking provider: `flashrank`, `cross-encoder`, `cohere` |
+| `VEKTRA_RERANK_MODEL` | str | `BAAI/bge-reranker-v2-m3` | Multilingual reranking model. For English-only lightweight deployments: provider=`flashrank`, model=`ms-marco-MiniLM-L-12-v2` |
 | `VEKTRA_RERANK_TOP_K` | int | `5` | Final top-k results after reranking |
 
 ## Ingestion