release: v0.4.0 - Observability & prompt quality

.claude/CLAUDE.md

@@ -2,6 +2,38 @@
 
 @../.s2s/CONTEXT.md
 
+## API interaction
+
+Before making **any** API call (curl, httpie, scripts), consult `docs/reference/api.md` or the live OpenAPI spec at `/openapi.json` to verify:
+- Authentication method and header format
+- Parameter names, types, and whether they are query, path, or body params
+- Request body field names (e.g. `question` vs `query`)
+- Correct endpoint for the task (e.g. `/api/v1/query` for RAG, `/api/v1/search` for raw vector search)
+
+Do not construct API calls from memory or guesswork.
+
+## Database investigation
+
+When querying Postgres directly (psql, DB inspection):
+- **Always run `\d table_name` first** to check column names and types. Common pitfalls: `namespace_id` (not `namespace`), `bytea` columns that need decryption, columns that don't exist.
+- **Conversation turns are encrypted**: `question` and `answer` are `pgp_sym_encrypt()`'d. To read them: `SELECT pgp_sym_decrypt(question, '<key>') FROM conversation_turns WHERE ...` using `VEKTRA_CONVERSATION_KEY` from `.env`.
+- **Postgres holds metadata/text, Qdrant holds vectors**: chunk text is in `document_chunks` (Postgres), but vector search runs against Qdrant (check `VEKTRA_QDRANT_URL` and `VEKTRA_QDRANT_COLLECTION` in config for host/port/collection). To inspect vectors or search results, query Qdrant REST API directly. The Qdrant payload uses `namespace_id` as the namespace filter field.
+
+## Query pipeline vs search endpoint
+
+These are fundamentally different:
+
+| | `/api/v1/search` | `/api/v1/query` |
+|--|--|--|
+| What it does | Raw vector search (Qdrant only) | Full RAG pipeline |
+| Reranker | No | Yes |
+| Threshold filter | No | Yes |
+| LLM call | No | Yes |
+| Field for question | `query` | `question` |
+| Response field | `results[].text_snippet` | `sources[].snippet` |
+
+When investigating **end-to-end answer quality** (retrieval + reranking + LLM), use `/api/v1/query`. When investigating **raw retrieval quality** (vector similarity only, no reranker), use `/api/v1/search`.
+
 ## Spec2Ship Commands
 
 - `/s2s:specs` - Define requirements via roundtable

.env.example

@@ -77,7 +77,7 @@
 # --------------------------------------------------------------------------
 
 # VEKTRA_QUERY_PIPELINE=advanced
-# VEKTRA_MIN_RELEVANCE_SCORE=0.3
+# VEKTRA_MIN_RELEVANCE_SCORE=0.15
 # VEKTRA_CHUNK_DEDUP_ENABLED=true
 # VEKTRA_RESPONSE_TOKEN_RESERVE=2048
 # VEKTRA_CONTEXT_CHUNK_RATIO=0.6
@@ -89,8 +89,8 @@
 
 # Reranking
 # VEKTRA_RERANK_ENABLED=true
-# VEKTRA_RERANK_PROVIDER=flashrank
-# VEKTRA_RERANK_MODEL=
+# VEKTRA_RERANK_PROVIDER=cross-encoder
+# VEKTRA_RERANK_MODEL=BAAI/bge-reranker-v2-m3
 # VEKTRA_RERANK_TOP_K=5
 
 # --------------------------------------------------------------------------
@@ -112,6 +112,13 @@
 # Fallback model when primary times out.
 # VEKTRA_LLM_FALLBACK_MODEL=
 # VEKTRA_LLM_FALLBACK_TIMEOUT_MS=60000
+# VEKTRA_LLM_CONTEXT_WINDOW=           # Required for models not in litellm registry (e.g. local vLLM).
+#   Set to the model's actual context length. Check your provider:
+#     vLLM: curl /v1/models → max_model_len
+#     Ollama: ollama show <model> → context_length
+#   If unset, litellm tries to look it up; for unknown models it falls back
+#   to 4096, which causes most retrieved chunks to be discarded (BUG-017).
+#   Examples: Qwen 3.5 27B → 262144, Llama 3 8B → 8192, GPT-4o → 128000
 # VEKTRA_LLM_CONTEXT_ONLY_ENABLED=true
 
 # --------------------------------------------------------------------------

.github/workflows/integration.yml

@@ -10,7 +10,7 @@ jobs:
   integration:
     name: Integration tests + NFR gates
     runs-on: ubuntu-latest
-    timeout-minutes: 20
+    timeout-minutes: 30
 
     steps:
       - uses: actions/checkout@v6
@@ -72,15 +72,9 @@ jobs:
           VEKTRA_BOOTSTRAP_KEY: ci-bootstrap-key
           STARTUP_MS: ${{ steps.startup.outputs.startup_ms }}
 
-      - name: Query latency measurement (warn only)
-        if: always() && !cancelled()
-        continue-on-error: true
-        run: |
-          uv run pytest tests/nfr/test_performance.py \
-            -v --tb=short --junitxml=test-results/performance.xml
-        env:
-          VEKTRA_API_URL: http://localhost:8000
-          VEKTRA_BOOTSTRAP_KEY: ci-bootstrap-key
+      # Query latency measurement removed from CI: requires a live LLM
+      # which is not available in GitHub Actions. Run manually on Kalypso:
+      #   uv run pytest tests/nfr/test_performance.py -v
 
       - name: Dump container logs on failure
         if: failure()

.gitignore

@@ -53,6 +53,9 @@ vektra-learn/static/
 coverage/
 .nyc_output/
 
+# Evaluation results (generated by eval harness)
+tests/eval/results_*.jsonl
+
 # Temporary files
 tmp/
 temp/