release: version 0.6.0 with RAG quality improvements

.env.example

@@ -64,6 +64,9 @@
 # Embedding provider and model.
 # VEKTRA_EMBEDDING_PROVIDER=sentence-transformers
 # VEKTRA_EMBEDDING_MODEL=paraphrase-multilingual-MiniLM-L12-v2
+# Remote embedding via Text Embeddings Inference (VEKTRA_EMBEDDING_PROVIDER=tei):
+# VEKTRA_TEI_URL=http://localhost:8080
+# VEKTRA_TEI_API_KEY=
 
 # --------------------------------------------------------------------------
 # Vector store
@@ -84,6 +87,9 @@
 # VEKTRA_QUERY_PIPELINE=advanced
 # VEKTRA_MIN_RELEVANCE_SCORE=0.15
 # VEKTRA_CHUNK_DEDUP_ENABLED=true
+# VEKTRA_RETRIEVAL_RESCUE_TOP_K=0
+# VEKTRA_RETRIEVAL_RESCUE_FLOOR=0.02
+# VEKTRA_PARENT_EXPANSION_ENABLED=false
 # VEKTRA_RESPONSE_TOKEN_RESERVE=2048
 # VEKTRA_CONTEXT_CHUNK_RATIO=0.6
 # VEKTRA_PROMPT_TEMPLATES_DIR=
@@ -103,6 +109,11 @@
 # VEKTRA_RERANK_PROVIDER=cross-encoder
 # VEKTRA_RERANK_MODEL=BAAI/bge-reranker-v2-m3
 # VEKTRA_RERANK_TOP_K=5
+# API key for API-based providers (cohere):
+# VEKTRA_RERANK_API_KEY=
+# Remote reranking via TEI (VEKTRA_RERANK_PROVIDER=tei, one instance per model):
+# VEKTRA_RERANK_TEI_URL=http://localhost:8080
+# VEKTRA_RERANK_TEI_API_KEY=
 
 # --------------------------------------------------------------------------
 # Ingestion

.github/workflows/docker-ocr-build.yml

@@ -0,0 +1,36 @@
+# Guard for the INSTALL_UNSTRUCTURED image variant (BUG-022).
+# The OCR extra pulls torch-adjacent wheels whose index pinning can silently
+# break the build; regular CI never builds with the flag, so this workflow
+# does, but only when the inputs that can break it change.
+name: docker-ocr-build
+
+on:
+  pull_request:
+    paths:
+      - "Dockerfile"
+      - "uv.lock"
+      - "pyproject.toml"
+      - "vektra-ingest/pyproject.toml"
+      - ".github/workflows/docker-ocr-build.yml"
+  push:
+    branches: [develop, main]
+    paths:
+      - "Dockerfile"
+      - "uv.lock"
+      - "pyproject.toml"
+      - "vektra-ingest/pyproject.toml"
+      - ".github/workflows/docker-ocr-build.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  build-ocr-image:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - name: Build image with INSTALL_UNSTRUCTURED=true
+        run: docker build --build-arg INSTALL_UNSTRUCTURED=true -t vektra-ocr-ci .

CHANGELOG.md

@@ -19,6 +19,28 @@ Convention (Keep a Changelog 1.1.0):
 
 <!-- Add entries under: Added, Changed, Deprecated, Removed, Fixed, Security -->
 
+## [0.6.0] - 2026-07-13
+
+RAG quality release: parent chunk expansion, retrieval-filter rescue, per-namespace citations, remote TEI providers.
+
+### Added
+
+- **rag**: remote embedding and reranking via HuggingFace Text Embeddings Inference (FEAT-024). `VEKTRA_EMBEDDING_PROVIDER=tei` embeds through a TEI instance (`VEKTRA_TEI_URL`/`VEKTRA_TEI_API_KEY`, native `/embed` API) instead of in-process sentence-transformers, enabling shared host inference and long-window models (bge-m3: 8192 tokens vs MiniLM's 128, which silently truncates 500-token chunks today). `VEKTRA_RERANK_PROVIDER=tei` reranks through TEI `/rerank` (`VEKTRA_RERANK_TEI_URL`/`VEKTRA_RERANK_TEI_API_KEY`). The Qdrant collection is now sized from the active embedding provider's dimensions instead of a hardcoded 384 (latent bug for any non-384 model), with a clear startup error on dimension mismatch against an existing collection. The `cohere` rerank option now actually passes its API key (`VEKTRA_RERANK_API_KEY`); it was dead as wired.
+- **rag**: optional retrieval-filter rescue for multi-part questions (TECH-007, `VEKTRA_RETRIEVAL_RESCUE_TOP_K` + `VEKTRA_RETRIEVAL_RESCUE_FLOOR`, default off). When `VEKTRA_MIN_RELEVANCE_SCORE` empties the candidate set, keep the top-N chunks above an absolute floor instead of refusing: the cross-encoder scores each partial-answer chunk of a comparative/multi-part question below the threshold (it answers only one part), so on the eval corpus 9/10 multi-chunk questions died at the filter with `before=5 after=0` despite 90% raw retrieval hit. With the rescue, borderline sets reach the LLM, which arbitrates via strict grounding. The `retrieval_filter` trace step now records a `rescued` count.
+- **rag**: optional per-namespace inline source citations (FEAT-021, `citations_enabled` in the namespace config JSONB via `PATCH /api/v1/admin/namespaces/{id}/config`, default off, advanced pipeline only). When enabled, the system prompt instructs the LLM to add inline `[n]` markers matching the `<source id>` elements, the context template carries a `title` attribute ("filename, p.N"), and each returned source includes a `title` field; the learn widget renders the markers as superscripts with a tooltip. Default-off renders byte-identical prompts; `prompt_version` changes anyway because the template files changed (trace comparability note).
+- **rag**: optional parent chunk expansion in the advanced query pipeline (FEAT-017, `VEKTRA_PARENT_EXPANSION_ENABLED`, default off). With `VEKTRA_CHUNKING_STRATEGY=dual`, retrieved child chunks are replaced with their parent chunk's text after the retrieval filter and before token budgeting; children of the same parent collapse into the highest-scored one. Parent-child linkage is now actually persisted (deterministic `uuid5(doc_id, position)` ids, `parent_id` in the Qdrant payload and in the pgvector column), a new `VectorStoreProvider.retrieve()` fetches chunks by id, and the trace records `children_expanded`/`siblings_merged`/`parents_fetched` in a `parent_expansion` step.
+
+### Changed
+
+- **index**: vector search now excludes parent-level chunks (`chunk_level=parent`) in both providers; parents are context material fetched by id during expansion, not retrieval targets. Only affects documents ingested with `dual` chunking, whose parents previously polluted search results.
+- **index**: pgvector `store()` honors caller-provided UUID chunk ids (deterministic ids from ingest) instead of always generating random ones; non-UUID ids still fall back to random.
+
+### Fixed
+
+- **docker**: the `INSTALL_UNSTRUCTURED=true` image variant builds again (BUG-022). torchvision (transitive via unstructured-inference) resolved from PyPI with CUDA-built wheels while torch is pinned to the CPU index, crashing the build with `operator torchvision::nms does not exist`. It is now declared in the `ocr` extra and pinned to the pytorch-cpu index; a new path-filtered CI workflow builds the OCR variant so it cannot silently regress.
+- **index**: `/api/v1/search` now resolves the embedding, sparse-embedding, and vector-store providers from the ProviderRegistry instead of hardcoding pgvector and reading a never-populated `app.state` attribute (BUG-021). In Qdrant deployments the endpoint returned zero results (it searched the empty `document_chunks` table) and hybrid mode always fell back to dense; the RAG pipeline (`/api/v1/query`) was unaffected. Found by the Sprint 3 baseline `make eval-retrieval` run.
+- **tests**: unit tests are now hermetic against the developer's local `.env` (DEBT-025). Importing litellm during pytest collection loads `.env` into the process environment, which made 4 default-assertion tests in `vektra-shared` fail on dev machines while CI stayed green. An autouse fixture in `vektra-shared/tests/conftest.py` scrubs ambient `VEKTRA_*` variables; production settings loading is unchanged. The fixture is replicated in `vektra-core` and `vektra-ingest` (FEAT-017 surfaced the same leak there: ambient `VEKTRA_CHUNKING_STRATEGY=dual` broke 12 chunker and pipeline-default tests).
+
 ## [0.5.1] - 2026-07-12
 
 Security hardening: DEBT-024 dependency sweep, workflow permissions, admin login hardening.

docs/reference/api.md

@@ -194,6 +194,7 @@ Request body (flat dict, one entry per config key):
 |-------|------|----------------|-------------|
 | `grounding_mode` | string or null | `"strict"`, `"hybrid"`, `null` | RAG grounding policy. `null` removes the key and falls back to `VEKTRA_PROMPT_GROUNDING_MODE`. |
 | `show_sources` | bool or null | `true`, `false`, `null` | Widget citation visibility (FEAT-014). `null` removes the key and falls back to `VEKTRA_LEARN_SHOW_SOURCES`. The API always returns the full sources list; the flag only instructs the widget whether to render them. Resolution chain: client `data-show-sources` attr > `namespaces.config.show_sources` > `VEKTRA_LEARN_SHOW_SOURCES` env > hardcoded `true`. |
+| `citations_enabled` | bool or null | `true`, `false`, `null` | Inline source citations (FEAT-021, advanced pipeline only). When `true`, the LLM is instructed to add `[n]` markers matching the context sources, and each returned source carries a `title` ("filename, p.N") for tooltip rendering. No env var: `null` (or absent) means disabled. Default-off leaves prompts unchanged. |
 
 Behavior:
 - **Partial update**: keys not present in the body are preserved.
@@ -340,6 +341,7 @@ Response:
 | `answer` | LLM-generated answer grounded in sources |
 | `sources` | Ranked list of source chunks |
 | `sources[].document_name` | Filename of the source document (e.g. `lecture-07.pdf`), or `null` when the document join returns no row. Soft-deleted documents (REQ-057) keep their citation with an `(archived)` suffix so traceability is preserved. |
+| `sources[].title` | FEAT-021: human-readable citation label ("filename, p.N") matching the `[n]` markers in the answer. Set only when the namespace has `citations_enabled`; `null` otherwise. |
 | `conversation_id` | Echoed back if provided in request |
 | `context_only` | `true` if LLM failed and raw sources returned |
 | `no_relevant_context` | `true` if no chunks exceeded relevance threshold |
@@ -522,6 +524,7 @@ Response (HTTP 200, JSON):
 |-------|-------------|
 | `show_sources` | Server-resolved citation-visibility hint for the widget (FEAT-014). The full `sources` list is always returned regardless; the widget uses the flag to decide whether to render the citations block. See the resolution chain in [Namespaces PATCH](#patch-apiv1adminnamespacesnamespace_idconfig). |
 | `sources[].document_name` | Filename of the source document. Soft-deleted documents (REQ-057) keep an `(archived)` suffix so traceability is preserved. The field is `null` when the document join returns no row. |
+| `sources[].title` | FEAT-021: citation label ("filename, p.N") for the `[n]` markers, `null` unless the namespace has `citations_enabled`. |
 
 #### Streaming
 

docs/reference/configuration.md

@@ -59,8 +59,10 @@ The model name must match the vLLM `--model` path exactly (e.g., `/models/qwen35
 
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
-| `VEKTRA_EMBEDDING_PROVIDER` | str | `sentence-transformers` | Embedding provider implementation |
-| `VEKTRA_EMBEDDING_MODEL` | str | `paraphrase-multilingual-MiniLM-L12-v2` | Model name within the selected provider |
+| `VEKTRA_EMBEDDING_PROVIDER` | str | `sentence-transformers` | Embedding provider implementation: `sentence-transformers` (in-process), `tei` (remote) |
+| `VEKTRA_EMBEDDING_MODEL` | str | `paraphrase-multilingual-MiniLM-L12-v2` | Model name within the selected provider (`sentence-transformers` only; a TEI instance serves one fixed model) |
+| `VEKTRA_TEI_URL` | str | `http://localhost:8080` | TEI server base URL (native API, no `/v1` suffix). Used when provider is `tei`. The Qdrant collection is sized from the served model's dimensions at startup |
+| `VEKTRA_TEI_API_KEY` | str | - | Bearer token for the TEI embedding server (`--api-key`). Optional |
 | `VEKTRA_SPARSE_EMBEDDING_PROVIDER` | str | - | Sparse embedding provider: `fastembed-bm25`, `splade` |
 | `VEKTRA_SPARSE_EMBEDDING_MODEL` | str | - | Sparse embedding model name |
 
@@ -81,6 +83,9 @@ The model name must match the vLLM `--model` path exactly (e.g., `/models/qwen35
 | `VEKTRA_QUERY_PIPELINE` | str | `advanced` | Pipeline implementation: `simple`, `advanced` |
 | `VEKTRA_MIN_RELEVANCE_SCORE` | float | `0.15` | Minimum relevance score for chunk inclusion (0.0-1.0). Safety net filter; top-k is the primary control. |
 | `VEKTRA_CHUNK_DEDUP_ENABLED` | bool | `true` | Deduplicate overlapping adjacent chunks from the same document |
+| `VEKTRA_RETRIEVAL_RESCUE_TOP_K` | int | `0` | When the `VEKTRA_MIN_RELEVANCE_SCORE` filter empties the candidate set, keep this many top-scored chunks above the rescue floor instead of refusing. Multi-part and comparative questions get uniformly low reranker scores (each chunk answers only one part), so with the rescue the LLM arbitrates via grounding instead of the query dying at the filter. `0` disables the rescue. Recommended starting point when enabling: `3`. |
+| `VEKTRA_RETRIEVAL_RESCUE_FLOOR` | float | `0.02` | Absolute minimum score for rescued chunks (0.0-1.0): candidates below this are never rescued. Only used when `VEKTRA_RETRIEVAL_RESCUE_TOP_K` > 0. Lower values rescue more multi-part questions but feed more irrelevant context to adversarial ones. |
+| `VEKTRA_PARENT_EXPANSION_ENABLED` | bool | `false` | Replace retrieved child chunks with their parent chunk text before prompt construction (advanced pipeline only). Requires documents ingested with `VEKTRA_CHUNKING_STRATEGY=dual`. |
 | `VEKTRA_RESPONSE_TOKEN_RESERVE` | int | `2048` | Tokens reserved for LLM response generation |
 | `VEKTRA_CONTEXT_CHUNK_RATIO` | float | `0.6` | Fraction of context window allocated to retrieved chunks (0.0-1.0) |
 | `VEKTRA_PROMPT_TEMPLATES_DIR` | str | - | Directory for custom Jinja2 prompt templates (`system.j2`, `context.j2`, `conversation.j2`). Uses built-in defaults if unset. |
@@ -98,9 +103,12 @@ The model name must match the vLLM `--model` path exactly (e.g., `/models/qwen35
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
 | `VEKTRA_RERANK_ENABLED` | bool | `true` | Enable cross-encoder reranking after retrieval |
-| `VEKTRA_RERANK_PROVIDER` | str | `cross-encoder` | Reranking provider: `flashrank`, `cross-encoder`, `cohere` |
+| `VEKTRA_RERANK_PROVIDER` | str | `cross-encoder` | Reranking provider: `flashrank`, `cross-encoder`, `cohere`, `tei` (remote) |
 | `VEKTRA_RERANK_MODEL` | str | `BAAI/bge-reranker-v2-m3` | Multilingual reranking model. For English-only lightweight deployments: provider=`flashrank`, model=`ms-marco-MiniLM-L-12-v2` |
 | `VEKTRA_RERANK_TOP_K` | int | `5` | Final top-k results after reranking |
+| `VEKTRA_RERANK_API_KEY` | str | - | API key for API-based providers (`cohere`) |
+| `VEKTRA_RERANK_TEI_URL` | str | `http://localhost:8080` | TEI reranker server base URL (one TEI instance per model, e.g. serving `BAAI/bge-reranker-v2-m3`). Used when provider is `tei`. Scores are sigmoid-normalized like the in-process path |
+| `VEKTRA_RERANK_TEI_API_KEY` | str | - | Bearer token for the TEI reranker server. Optional |
 
 ## Ingestion
 

tests/eval/README.md

@@ -0,0 +1,52 @@
+# RAG evaluation harness (TECH-002)
+
+Two-stage evaluation against a running Vektra stack. Both scripts are pure HTTP
+clients: they require `VEKTRA_API_URL` and `VEKTRA_API_KEY` in the environment.
+
+```bash
+make eval-retrieval    # /api/v1/search (no LLM) - hit rate, MRR, precision@k
+make eval-e2e          # /api/v1/query (full pipeline + LLM) - grounded rate, latency
+# extra args:
+make eval-retrieval EVAL_ARGS="--dataset tests/eval/dataset-full.jsonl --output tests/eval/results_retrieval_full.jsonl"
+```
+
+Results are written as JSONL next to the datasets (`results_*.jsonl`, gitignored:
+record aggregates in the active `.s2s/plans/` file and in vektra-internal).
+
+## Datasets
+
+| File | Namespace | Corpus |
+|------|-----------|--------|
+| `dataset.jsonl` | `default` | Excerpt corpus, 12 chunks: `costituzione_italiana.md` v2 (6), `udhr_excerpts.md` v2 (4), `sample.pdf` (2). Hit rate saturates here; useful for smoke/regression, not for tuning. |
+| `dataset-full.jsonl` | `eval-full` | Full-document corpus, 78 chunks: clean full Italian Constitution (72) + official UDHR English PDF (6). Same 55 questions, discriminative metrics. |
+
+Both files share the same 55 questions (21 factual, 15 reasoning, 10 multi-chunk,
+9 adversarial without ground truth; 37 IT / 18 EN). Entry shape:
+`{id, question, expected_keywords, namespace, category, language}`. Relevance is
+keyword-based (`expected_keywords`, diacritic-insensitive substring match), so it
+is chunking-independent and survives reingestion.
+
+## Corpus provenance (eval-full)
+
+- `costituzione-full-clean.md`: full text of the Italian Constitution converted
+  from Wikisource (`https://it.wikisource.org/api/rest_v1/page/html/Costituzione_della_Repubblica_italiana`,
+  CC BY-SA), metadata header stripped. Kept outside the repo at
+  `/mnt/ai/datasets/vektra-eval/` on the dev machine.
+- `udhr-en-ohchr.pdf`: official OHCHR English UDHR
+  (`https://www.ohchr.org/sites/default/files/UDHR/Documents/UDHR_Translations/eng.pdf`).
+- Do NOT use the senato.it combined PDF (`costituzione.pdf`, 506 pages): its print
+  layout breaks pdfplumber extraction (fused words like
+  `COSTITUZIONEDELLAREPUBBLICAITALIANA`, preserved hyphenation) and invalidates
+  keyword ground truth. Measured impact: IT hit rate 72% garbled vs 88% clean on
+  the same questions. Worth keeping in mind as a future "dirty extraction" test
+  case, but never as a retrieval-quality corpus.
+
+To rebuild the corpus: download the two sources, then
+`scripts/ingest.sh <file> eval-full` for each.
+
+## Adding questions
+
+Append JSONL entries with a unique `id`, the target `namespace`, and 1-3
+`expected_keywords` that only appear in the passages that truly answer the
+question. Adversarial entries (expected refusal) omit `expected_keywords` and are
+reported separately (`has_ground_truth: false`).