From ce55ddbf54541c7cfb9e5151ea1349400470bfba Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 18:43:20 +0100 Subject: [PATCH 01/25] docs: update specifications for Phase 2 completion - CONTEXT.md: component table, cross-cutting concerns, protocol interfaces updated to reflect Phase 2 as implemented (not planned) - requirements.md: scope section with Phase 2 deliverables, 7 EX-* exclusions resolved, 3 OQ-* resolved, version bumped to 1.6.0 - architecture.md: component table expanded with vektra-analytics and vektra-learn, protocol implementations updated (Qdrant, Unstructured, DualStrategyChunking, AdvancedQueryPipeline, FastEmbedBM25, LogEventEmitter), glossary and traceability corrected, version bumped to 2.0 - docker-compose.yml: profile comments updated (removed "Phase 2" labels) Co-Authored-By: Claude Opus 4.6 (1M context) --- .s2s/CONTEXT.md | 40 ++++++++++++++++++------------------- .s2s/architecture.md | 47 +++++++++++++++++++++++--------------------- .s2s/requirements.md | 46 ++++++++++++++++++++++++------------------- docker-compose.yml | 4 ++-- 4 files changed, 73 insertions(+), 64 deletions(-) diff --git a/.s2s/CONTEXT.md b/.s2s/CONTEXT.md index 09160264..16df3387 100644 --- a/.s2s/CONTEXT.md +++ b/.s2s/CONTEXT.md @@ -46,7 +46,7 @@ Split criteria defined in [ADR-0002](decisions/ADR-0002-repo-split-criteria.md). ## Cross-Cutting Concerns -- **Authentication**: API key authentication with argon2id hashing, scoped permissions (admin/ingest/query per REQ-031, multiple scopes per key). Single trust boundary at vektra-core gateway. Rate limiting slot in middleware (Phase 2 enforcement). See [ADR-0010](decisions/ADR-0010-authentication-gateway.md). +- **Authentication**: API key authentication with argon2id hashing, scoped permissions (admin/ingest/query per REQ-031, multiple scopes per key). Single trust boundary at vektra-core gateway. Rate limiting enforced via per-key RPM in middleware. See [ADR-0010](decisions/ADR-0010-authentication-gateway.md). - **Authorization**: Namespace isolation via PostgreSQL RLS policies. Application-level filtering for Phase 1, RLS binding via feature flag for multi-tenant activation. Namespace as first-class entity with metadata (ARCH-047). See [ADR-0009](decisions/ADR-0009-namespace-isolation-rls.md). - **Logging**: structlog with JSON output, PII redaction processors. Correlation ID propagation across sync calls and arq jobs. OpenTelemetry spans at module boundaries. QueryTrace (ARCH-041) for RAG-specific observability, separate from audit log. - **Monitoring**: Prometheus metrics on /metrics via starlette-prometheus. Hierarchical health endpoints (GET /health, GET /health/{component}). Memory observability via GET /health/memory. @@ -60,18 +60,18 @@ Split criteria defined in [ADR-0002](decisions/ADR-0002-repo-split-criteria.md). | Component | Role | Phase | |-----------|------|-------| -| vektra-core | RAG engine, LLM abstraction, conversation management, safeguards | 1 | -| vektra-ingest | Document processing pipeline (PDF, OCR, PPT, Word) | 1 | -| vektra-index | Vector store abstraction, embedding, semantic search | 1 | -| vektra-analytics | Metrics aggregation, reporting API, alerting | 2 | -| vektra-learn | E-learning vertical backend (LMS-agnostic) | 2 | -| vektra-admin | System administration interface | 1 (minimal), 2 (full) | +| vektra-core | RAG engine, LLM abstraction, conversation management, safeguards, advanced pipeline | 1, 2 | +| vektra-ingest | Document processing pipeline (PDF, OCR, PPT, Word), versioning, batch ops | 1, 2 | +| vektra-index | Vector store abstraction, embedding, hybrid search (dense+sparse) | 1, 2 | +| vektra-analytics | QueryTrace storage, metrics aggregation, reporting API | 2 (completed) | +| vektra-learn | E-learning vertical backend (LMS-agnostic), chatbot widget | 2 (completed) | +| vektra-admin | System administration interface (HTMX + Jinja2 dashboard) | 1 (minimal), 2 (full) | ### Separate repositories | Component | Role | Reason | Phase | |-----------|------|--------|-------| -| vektra-moodle | Moodle LMS adapter (PHP plugin) | PHP, Moodle Plugin Directory, different lifecycle | 2 | +| vektra-moodle | Moodle LMS adapter (PHP plugin) | PHP, Moodle Plugin Directory, different lifecycle | 2 (not started) | | vektra-sdk-py | Python SDK | Published to PyPI, independent versioning | 3 | | vektra-sdk-js | JavaScript/TypeScript SDK | Published to npm, independent versioning | 3 | @@ -103,12 +103,12 @@ vektra-moodle ──────────────── (integrates learn See [requirements.md](requirements.md) for the complete Software Requirements Specification. -**Key Phase 1 deliverables**: +**Key deliverables**: - 60 approved functional requirements (REQ-001 to REQ-065, some IDs unused) - 5 business rules - 13 non-functional requirements (8 HARD, 5 TARGET) -- 14 explicit exclusions (Phase 2/3 deferrals) -- 7 open questions (1 resolved, remainder deferred to design or Phase 2) +- 14 explicit exclusions (7 resolved in Phase 2, 7 remaining for Phase 3+) +- 7 open questions (4 resolved) **Primary user persona**: Platform Operator (DevOps/platform teams) **MVP exit criterion**: 30 minutes from git clone to successful query @@ -119,7 +119,7 @@ See [architecture.md](architecture.md) for complete architecture documentation. **Architectural style**: Modular monolith for Phase 1. Single deployable container with internal package boundaries. See [ADR-0003](decisions/ADR-0003-modular-monolith-phase1.md). -**Deployment**: Docker Compose stack: vektra + postgres (always), ollama (profile: local-llm), qdrant (profile: qdrant, Phase 2). See [ADR-0004](decisions/ADR-0004-minimal-docker-compose-stack.md), [ADR-0012](decisions/ADR-0012-docker-compose-spec.md). +**Deployment**: Docker Compose stack: vektra + postgres (always), ollama (profile: local-llm), qdrant (profile: qdrant). See [ADR-0004](decisions/ADR-0004-minimal-docker-compose-stack.md), [ADR-0012](decisions/ADR-0012-docker-compose-spec.md). **Key technology choices**: - Web framework: FastAPI 0.115+ with Pydantic v2 @@ -127,20 +127,20 @@ See [architecture.md](architecture.md) for complete architecture documentation. - Embeddings: sentence-transformers (all-MiniLM-L6-v2) via EmbeddingProvider Protocol - Vector store: pgvector (PostgreSQL extension) via VectorStoreProvider Protocol - Background tasks: arq with PostgreSQL job persistence -- PDF extraction: pdfplumber (Phase 1), Unstructured (Phase 2) via DocumentExtractor Protocol +- PDF extraction: pdfplumber (default), Unstructured (optional, INSTALL_UNSTRUCTURED=true) via DocumentExtractor Protocol - Content type detection: python-magic (magic bytes) - ORM: SQLAlchemy 2.0 async with asyncpg (ORM models internal to modules, Pydantic models as public API) **Protocol interfaces** (9 defined in vektra_shared): - LLMProvider: multi-provider LLM abstraction with graceful degradation - EmbeddingProvider: shared embedding generation with asymmetric model support -- SparseEmbeddingProvider: sparse vector generation for hybrid search (ARCH-053). Phase 1: not registered. Phase 2: BM25 or SPLADE via fastembed -- VectorStoreProvider: pluggable vector store with SearchMode, metadata filtering, index versioning, raw_filters escape hatch, full-store contract (ARCH-051), provider-specific atomicity (ARCH-052). Phase 2 candidate: Qdrant -- DocumentExtractor: PDF, Word, PowerPoint extraction with extended element classification (10 ElementType values) -- ChunkingStrategy: pluggable chunking (fixed-size Phase 1, dual-strategy Phase 2) -- QueryPipeline: RAG pipeline abstraction returning QueryResponse + QueryTrace. Phase 2: AdvancedQueryPipeline with query rewriting (ARCH-061), reranking, hybrid search +- SparseEmbeddingProvider: sparse vector generation for hybrid search (ARCH-053). Phase 1: not registered. Phase 2: FastEmbedBM25Provider via fastembed +- VectorStoreProvider: pluggable vector store with SearchMode, metadata filtering, index versioning, raw_filters escape hatch, full-store contract (ARCH-051), provider-specific atomicity (ARCH-052). Phase 2: QdrantVectorStoreProvider with native DENSE/SPARSE/HYBRID +- DocumentExtractor: PDF, Word, PowerPoint extraction with extended element classification (10 ElementType values). Phase 2: UnstructuredExtractor with OCR support +- ChunkingStrategy: pluggable chunking. FixedSizeChunking (Phase 1), DualStrategyChunking with semantic splitting (Phase 2) +- QueryPipeline: RAG pipeline abstraction returning QueryResponse + QueryTrace. Phase 2: AdvancedQueryPipeline with query rewriting (ARCH-061), reranking, hybrid search (implemented) - SafeguardHook: pre/post query safeguards (3 trust boundary points) with content modification support (ARCH-049) -- EventEmitter: internal event hooks (NoOp Phase 1, webhooks Phase 2) +- EventEmitter: internal event hooks. NoOpEventEmitter (Phase 1), LogEventEmitter (Phase 2) **Key decisions** (64 total, 25 ADRs): - [ADR-0003](decisions/ADR-0003-modular-monolith-phase1.md): Modular monolith for Phase 1 @@ -173,4 +173,4 @@ See [architecture.md](architecture.md) for complete architecture documentation. --- -*Last updated: 2026-03-01* +*Last updated: 2026-03-14* diff --git a/.s2s/architecture.md b/.s2s/architecture.md index fe4f03bc..b7b3d02e 100644 --- a/.s2s/architecture.md +++ b/.s2s/architecture.md @@ -429,10 +429,10 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource - **ARCH-035 - EmbeddingProvider Protocol**: Dedicated Protocol for embedding generation. Separates embed_documents() from embed_query() for asymmetric models. Single shared instance between ingest and core. Configurable provider and model via env vars. - **ARCH-036 - QueryPipeline Protocol**: Abstraction of RAG query pipeline. execute() returns (QueryResponse, QueryTrace). Phase 1: SimpleQueryPipeline (embed -> search -> relevance_filter -> build_prompt -> LLM, with graceful degradation). relevance_filter applies minimum score threshold and overlap deduplication (ARCH-056). build_prompt calculates token budget from model context window (ARCH-055), renders composable Jinja2 templates (ARCH-054). Phase 2: AdvancedQueryPipeline adds query_rewrite (ARCH-061), query_classify, rerank, confidence scoring steps. Selectable via VEKTRA_QUERY_PIPELINE. - **ARCH-037 - ChunkingStrategy Protocol**: Abstraction of text chunking. Receives DocumentChunk stream, returns chunked stream. Phase 1: FixedSizeChunking. Selectable via VEKTRA_CHUNKING_STRATEGY. -- **ARCH-038 - EventEmitter interface**: Internal event hooks with NoOp default. Emission points at document lifecycle, query completion, safeguard triggers, API key operations. Phase 2: WebhookEventEmitter with HMAC-SHA256. +- **ARCH-038 - EventEmitter interface**: Internal event hooks with NoOp default. Emission points at document lifecycle, query completion, safeguard triggers, API key operations. Phase 2 (implemented): LogEventEmitter. WebhookEventEmitter with HMAC-SHA256 deferred to Phase 3. - **ARCH-039 - ProviderRegistry pattern**: Unified registry for all Protocol implementations. Dict-based in Phase 1, extensible to entry_points plugin discovery in Phase 2. Consistent VEKTRA_* env var configuration pattern. - **ARCH-040 - Forward-compatible data model**: Phase 1 schema includes fields for Phase 2 features: response_id and citation_id for feedback loops, document version and supersedes_id for versioning, deleted_at and deletion_reason for soft delete, index_version for zero-downtime reindex. All fields nullable/defaulted, no behavioral change in Phase 1. -- **ARCH-041 - Audit/analytics separation**: QueryTrace (per-step timing, chunk refs, model info) emitted separately from audit log. QueryTrace does not contain query text or response content (REQ-051 compliance). Phase 1: structlog emission. Phase 2: dedicated storage. +- **ARCH-041 - Audit/analytics separation**: QueryTrace (per-step timing, chunk refs, model info) emitted separately from audit log. QueryTrace does not contain query text or response content (REQ-051 compliance). Phase 1: structlog emission. Phase 2 (implemented): dedicated query_traces table with reporting API (vektra-analytics). - **ARCH-042 - Content type detection**: Magic bytes validation before DocumentExtractor dispatch. python-magic for MIME detection. Logs warning on mismatch with declared type. - **ARCH-043 - Graceful degradation**: Two degradation paths. LLM degradation: primary model timeout triggers fallback model; both fail triggers context-only response (chunks without LLM synthesis). Retrieval degradation (ARCH-056): when no chunk passes the relevance threshold, QueryResponse.no_relevant_context=True; system returns explicit "no relevant information" instead of synthesizing from irrelevant context. Client always receives value (relevant sources, or empty sources with explanation, at minimum). - **ARCH-044 - Chunk metadata filtering**: JSONB metadata column on chunks with GIN index. SearchFilters parameter in VectorStoreProvider.search(). Generic filterable fields (content_type, language) plus arbitrary key-value pairs. Domain-specific fields (e.g., course_id for vektra-learn) defined by the vertical in Phase 2, stored in the same JSONB column. @@ -444,7 +444,7 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource - **ARCH-050 - RAG evaluation strategy**: Three-tier hybrid pattern for RAG quality evaluation that respects GDPR constraints (REQ-051). CI: synthetic test suite generated from ingested documents via RAGAS or DeepEval, used as regression gate. Staging: evaluation mode (VEKTRA_EVAL_MODE) enabling temporary text capture for batch evaluation, not active in production. Production: QueryTrace metrics only (timing, scores, chunk refs), feedback via response_id/citation_id (REQ-055). No query/response text persisted in production. - **ARCH-051 - VectorStoreProvider full-store contract**: VectorStoreProvider implementations own chunk text, metadata, and embeddings as a self-contained unit. store() persists all three; search() returns text_snippet directly from the provider without secondary lookups. This keeps the Protocol self-contained and avoids dual-query patterns (vector search + relational text lookup). PostgreSQL retains source_documents as canonical source of original documents; chunks in the vector store are derived, re-generable artifacts. When switching providers (e.g., pgvector to Qdrant), chunks are re-ingested into the new provider via the existing reindex mechanism (ARCH-045), not migrated. - **ARCH-052 - Provider-specific ingest atomicity**: Batch chunk ingest atomicity depends on provider capabilities. PgvectorProvider: multi-row INSERT in single SQL transaction (ARCH-010). Non-transactional providers (e.g., Qdrant, Milvus): batch upsert with best-effort durability (Qdrant: wait=true). On partial batch failure, the ingest job is marked as failed and a compensating delete by document_id removes partial writes. The IngestService treats atomicity as a provider contract, not a universal guarantee. -- **ARCH-053 - SparseEmbeddingProvider Protocol**: Dedicated Protocol for sparse vector generation, completing the hybrid search data path. SparseVector fields already exist in ChunkEmbedding and QueryEmbedding; this Protocol defines who populates them. Phase 1: not registered in ProviderRegistry (None). When absent, SearchMode.HYBRID is unavailable and sparse fields remain None. Phase 2: register one of FastEmbedBM25Provider (tokenization + TF, lightweight, Qdrant IDF modifier handles the rest server-side), SPLADEProvider (neural term expansion via fastembed, ~500 MB), or BM25sProvider (in-memory BM25 for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. +- **ARCH-053 - SparseEmbeddingProvider Protocol**: Dedicated Protocol for sparse vector generation, completing the hybrid search data path. SparseVector fields already exist in ChunkEmbedding and QueryEmbedding; this Protocol defines who populates them. Phase 1: not registered in ProviderRegistry (None). When absent, SearchMode.HYBRID is unavailable and sparse fields remain None. Phase 2 (implemented): FastEmbedBM25Provider registered (tokenization + TF, lightweight, Qdrant IDF modifier handles the rest server-side). Alternative options: SPLADEProvider (neural term expansion via fastembed, ~500 MB), or BM25sProvider (in-memory BM25 for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. - **ARCH-054 - Prompt template architecture**: Three composable Jinja2 template files for prompt construction (Phase 1), plus `rewrite.j2` for conversational query rewriting (Phase 2, ARCH-061): `system.j2` (LLM role and behavioral instructions), `context.j2` (retrieved chunk formatting), `conversation.j2` (multi-turn history formatting). Loaded from configurable directory (`VEKTRA_PROMPT_TEMPLATES_DIR`, default: built-in package defaults). Each template defines a typed variable contract: system receives `namespace` and `model_name`; context receives `chunks` (list of dicts with text, score, doc_id, element_type, metadata) and `num_chunks`; conversation receives `turns` (list of dicts with question and answer) and `num_turns`. Composition order is fixed: system + context (if chunks) + conversation (if conversation_id) + user question. prompt_version (ARCH-048) computed as SHA-256[:8] of concatenated template sources; per-template hashes recorded in StepTrace metadata for build_prompt. See ADR-0020. - **ARCH-056 - Retrieval quality controls**: Three controls in SimpleQueryPipeline between vector_search and build_prompt: (1) minimum relevance threshold (`VEKTRA_MIN_RELEVANCE_SCORE`, default 0.3) filters chunks below the score floor; (2) overlap deduplication removes redundant chunks from fixed-size chunking overlap (same document, adjacent positions) keeping the higher-scored chunk; (3) no-relevant-context detection sets QueryResponse.no_relevant_context=True when no chunk passes the threshold, triggering a dedicated response path instead of forcing synthesis from irrelevant context. All three controls are configurable and produce diagnostics in QueryTrace via a retrieval_filter StepTrace. See ADR-0021, REQ-066. @@ -496,10 +496,12 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource | Component | Responsibility | Technology | |-----------|---------------|------------| -| vektra-core | Orchestrates RAG query flow: receives queries, retrieves context, generates LLM responses, enforces safeguards | FastAPI, litellm, sentence-transformers | -| vektra-ingest | Processes documents into indexable chunks: format conversion, chunking, async job management | arq, pdfplumber, python-docx, python-pptx | -| vektra-index | Manages vector storage and semantic search | VectorStoreProvider (Phase 1: pgvector), sentence-transformers | -| vektra-admin | System administration: health monitoring, API key management, configuration | FastAPI | +| vektra-core | Orchestrates RAG query flow: receives queries, retrieves context, generates LLM responses, enforces safeguards. Phase 2: advanced pipeline, persistent conversations, query rewriting, reranking | FastAPI, litellm, sentence-transformers | +| vektra-ingest | Processes documents into indexable chunks: format conversion, chunking, async job management. Phase 2: OCR, dual chunking, document versioning, batch operations | arq, pdfplumber, python-docx, python-pptx, Unstructured (optional) | +| vektra-index | Manages vector storage and semantic search. Phase 2: hybrid search (dense+sparse), Qdrant provider | VectorStoreProvider (pgvector default, Qdrant optional), sentence-transformers | +| vektra-analytics | QueryTrace storage, metrics aggregation, reporting API (Phase 2) | FastAPI, SQLAlchemy | +| vektra-learn | E-learning vertical backend: enrollments, content ingestion, dashboard tokens, course-scoped query, chatbot widget (Phase 2) | FastAPI, PyJWT, esbuild | +| vektra-admin | System administration: health monitoring, API key management, configuration. Phase 2: HTMX + Jinja2 dashboard, RLS enforcement, rate limiting | FastAPI, HTMX, Jinja2 | | vektra_shared | Cross-cutting infrastructure: types, config schemas, error definitions, auth middleware, Protocol interfaces | Pydantic v2 | #### vektra-core @@ -508,7 +510,7 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource **State Owned**: - conversations (encrypted via pgcrypto) -- query_traces (Phase 1: structlog emission; Phase 2: dedicated storage) +- query_traces (Phase 1: structlog emission; Phase 2: dedicated storage table, implemented) **Interfaces**: - Provides: External REST API (POST /query, GET /providers, GET /health) @@ -542,7 +544,7 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource - Provides: REST routes via main FastAPI app (POST /search, POST /documents/{id}/chunks, DELETE /documents/{id}, GET /stats) + internal Protocol (callable by vektra-core) - Requires: VectorStoreProvider implementation (Phase 1: pgvector) -**Dependencies**: VectorStoreProvider (Phase 1: PostgreSQL with pgvector extension, Phase 2 candidate: Qdrant) +**Dependencies**: VectorStoreProvider (pgvector default, Qdrant optional via --profile qdrant) #### vektra-admin @@ -601,7 +603,7 @@ class EmbeddingProvider(Protocol): async def health_check() -> HealthStatus ``` -Phase 1: SentenceTransformersProvider (all-MiniLM-L6-v2, 384 dims, single shared instance). The distinction between embed_documents() and embed_query() supports asymmetric models (e.g., e5-large requires "query: ..." vs "passage: ..." prefixes). Config: `VEKTRA_EMBEDDING_PROVIDER`, `VEKTRA_EMBEDDING_MODEL`. Phase 2 option: TEI (Hugging Face Text Embeddings Inference) as external embedding server, freeing model RAM from the application container (~90 MB for MiniLM, ~1.2 GB for e5-large). Protocol supports this as a zero-change swap. +Phase 1: SentenceTransformersProvider (all-MiniLM-L6-v2, 384 dims, single shared instance). The distinction between embed_documents() and embed_query() supports asymmetric models (e.g., e5-large requires "query: ..." vs "passage: ..." prefixes). Config: `VEKTRA_EMBEDDING_PROVIDER`, `VEKTRA_EMBEDDING_MODEL`. Optional: TEI (Hugging Face Text Embeddings Inference) as external embedding server via --profile tei, freeing model RAM from the application container (~90 MB for MiniLM, ~1.2 GB for e5-large). Protocol supports this as a zero-change swap. #### SparseEmbeddingProvider (new - ARCH-053) @@ -612,7 +614,7 @@ class SparseEmbeddingProvider(Protocol): def vocab_size() -> int | None ``` -Phase 1: not registered (None in ProviderRegistry). When absent, the IngestService skips sparse vector generation (ChunkEmbedding.sparse = None) and the QueryPipeline skips sparse query embedding (QueryEmbedding.sparse = None), making SearchMode.HYBRID unavailable. Phase 2 candidates: FastEmbedBM25Provider (fastembed `Qdrant/bm25` model, tokenization + term frequencies, lightweight), SPLADEProvider (fastembed `Splade_PP_en_v1`, neural term expansion, ~500 MB, vocabulary ~30K tokens), BM25sProvider (bm25s library, ~100 KB, in-memory index for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. When using Qdrant with IDF modifier (`sparse_vectors_config.modifier = "idf"`), the provider sends only term frequencies and Qdrant computes IDF server-side, keeping the IDF in sync with the corpus automatically. +Phase 1: not registered (None in ProviderRegistry). When absent, the IngestService skips sparse vector generation (ChunkEmbedding.sparse = None) and the QueryPipeline skips sparse query embedding (QueryEmbedding.sparse = None), making SearchMode.HYBRID unavailable. Phase 2 (implemented): FastEmbedBM25Provider (fastembed `Qdrant/bm25` model, tokenization + term frequencies, lightweight). Alternatives: SPLADEProvider (fastembed `Splade_PP_en_v1`, neural term expansion, ~500 MB, vocabulary ~30K tokens), BM25sProvider (bm25s library, ~100 KB, in-memory index for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. When using Qdrant with IDF modifier (`sparse_vectors_config.modifier = "idf"`), the provider sends only term frequencies and Qdrant computes IDF server-side, keeping the IDF in sync with the corpus automatically. #### VectorStoreProvider (extended - REQ-050) @@ -650,7 +652,7 @@ class VectorStoreProvider(Protocol): **Full-store contract (ARCH-051)**: each VectorStoreProvider implementation owns chunk text, metadata, and embeddings. store() persists all three; search() returns text_snippet directly without secondary lookups. Switching provider means re-ingesting chunks (via ARCH-045 reindex), not migrating data. -Phase 1: PgvectorProvider, SearchMode.DENSE only (sparse ignored), filters applied as JSONB WHERE clause, GIN index on metadata column, index_version filter applied, raw_filters ignored. Phase 2 candidate: QdrantVectorStoreProvider with native DENSE/SPARSE/HYBRID support, payload-based namespace and index_version filtering, raw_filters for Qdrant-specific expressions (range, geo, full-text). Qdrant hybrid search: store() persists both dense and sparse vectors as named vectors in the same point; search(mode=HYBRID) maps to Qdrant's prefetch + fusion API (RRF or DBSF, server-side, single round trip). With Qdrant IDF modifier (`sparse_vectors_config.modifier = "idf"`), the server computes IDF from the corpus automatically, keeping BM25 scores in sync as documents are added/removed. The raw_filters parameter is an escape hatch for provider-specific filter expressions (e.g., Qdrant range/geo filters, Milvus boolean expressions, ChromaDB operator dicts). When both filters and raw_filters are provided, the implementation defines precedence. Batch ingest atomicity is provider-specific (ARCH-052). +Phase 1: PgvectorProvider, SearchMode.DENSE only (sparse ignored), filters applied as JSONB WHERE clause, GIN index on metadata column, index_version filter applied, raw_filters ignored. Phase 2 (implemented): QdrantVectorStoreProvider with native DENSE/SPARSE/HYBRID support, payload-based namespace and index_version filtering, raw_filters for Qdrant-specific expressions (range, geo, full-text). Qdrant hybrid search: store() persists both dense and sparse vectors as named vectors in the same point; search(mode=HYBRID) maps to Qdrant's prefetch + fusion API (RRF or DBSF, server-side, single round trip). With Qdrant IDF modifier (`sparse_vectors_config.modifier = "idf"`), the server computes IDF from the corpus automatically, keeping BM25 scores in sync as documents are added/removed. The raw_filters parameter is an escape hatch for provider-specific filter expressions (e.g., Qdrant range/geo filters, Milvus boolean expressions, ChromaDB operator dicts). When both filters and raw_filters are provided, the implementation defines precedence. Batch ingest atomicity is provider-specific (ARCH-052). #### DocumentExtractor @@ -661,7 +663,7 @@ class DocumentExtractor(Protocol): async def health_check() -> HealthStatus ``` -Phase 1: PdfplumberExtractor, WordExtractor, PowerPointExtractor. Content type detection via magic bytes (ARCH-042) before dispatch. Phase 2 option: Unstructured as local library or as external API (self-hosted Docker), offloading heavy dependencies (Tesseract, PyTorch for table detection) from the Vektra container. Same pattern as TEI for EmbeddingProvider. ExtractionRequest maps 1:1 with Unstructured's `partition()` parameters. +Phase 1: PdfplumberExtractor, WordExtractor, PowerPointExtractor. Content type detection via magic bytes (ARCH-042) before dispatch. Phase 2 (implemented): UnstructuredExtractor as local library (INSTALL_UNSTRUCTURED=true build arg), adding OCR support via Tesseract. Offloads heavy dependencies from the default image. ExtractionRequest maps 1:1 with Unstructured's `partition()` parameters. #### ChunkingStrategy (new - ARCH-037) @@ -672,7 +674,7 @@ class ChunkingStrategy(Protocol): ) -> AsyncIterator[DocumentChunk] ``` -Phase 1: FixedSizeChunking (1000 tokens, 200 overlap per REQ-016). Phase 2: DualStrategyChunking (text: split with overlap; tables: never split; parent-child: 2 levels). Config: `VEKTRA_CHUNKING_STRATEGY`. +Phase 1: FixedSizeChunking (1000 tokens, 200 overlap per REQ-016). Phase 2 (implemented): DualStrategyChunking (text: split with overlap; tables: never split; parent-child: 2 levels). Config: `VEKTRA_CHUNKING_STRATEGY`. #### QueryPipeline (new - ARCH-036) @@ -682,7 +684,7 @@ class QueryPipeline(Protocol): async def execute_stream(query: QueryRequest) -> AsyncIterator[QueryChunk] ``` -Phase 1: SimpleQueryPipeline (embed -> search -> relevance_filter -> build_prompt -> LLM, with graceful degradation). relevance_filter applies minimum score threshold, overlap deduplication, and no-relevant-context detection (ARCH-056). build_prompt calculates token budget from model context window (ARCH-055) and renders composable Jinja2 templates (ARCH-054). Phase 2: AdvancedQueryPipeline (classify -> retrieve -> rerank -> synthesize -> verify, implemented directly per ARCH-046). Reranking step recommended via `rerankers` library (Answer.AI): unified API for cross-encoder, FlashRank (~4 MB), Cohere, ColBERT, making the reranking backend swappable without touching pipeline code. Config: `VEKTRA_QUERY_PIPELINE`. +Phase 1: SimpleQueryPipeline (embed -> search -> relevance_filter -> build_prompt -> LLM, with graceful degradation). relevance_filter applies minimum score threshold, overlap deduplication, and no-relevant-context detection (ARCH-056). build_prompt calculates token budget from model context window (ARCH-055) and renders composable Jinja2 templates (ARCH-054). Phase 2 (implemented): AdvancedQueryPipeline (classify -> retrieve -> rerank -> synthesize -> verify, implemented directly per ARCH-046). Reranking step via `rerankers` library (Answer.AI): unified API for cross-encoder, FlashRank (~4 MB), Cohere, ColBERT, making the reranking backend swappable without touching pipeline code. Config: `VEKTRA_QUERY_PIPELINE`. #### SafeguardHook @@ -702,7 +704,7 @@ class EventEmitter(Protocol): async def emit(event_type: str, payload: dict) -> None ``` -Phase 1: NoOpEventEmitter. Emission points: document.indexed, document.failed, query.completed, safeguard.triggered, apikey.created, apikey.revoked. Phase 2: WebhookEventEmitter with HMAC-SHA256 signature. +Phase 1: NoOpEventEmitter. Emission points: document.indexed, document.failed, query.completed, safeguard.triggered, apikey.created, apikey.revoked. Phase 2 (implemented): LogEventEmitter (structlog-based event logging). WebhookEventEmitter with HMAC-SHA256 deferred to Phase 3. ### 8.3.1 Extended types @@ -1733,7 +1735,7 @@ Conventions: | Variable | Type | Default | Required | Description | Reference | |----------|------|---------|----------|-------------|-----------| -| `VEKTRA_VECTOR_STORE_PROVIDER` | string | `pgvector` | No | VectorStoreProvider implementation. Phase 2 option: `qdrant` | ARCH-039, ADR-0012 | +| `VEKTRA_VECTOR_STORE_PROVIDER` | string | `pgvector` | No | VectorStoreProvider implementation. Options: `pgvector` (default), `qdrant` | ARCH-039, ADR-0012 | | `VEKTRA_ACTIVE_INDEX_VERSION` | int | `1` | No | Active index version for search queries. Change to new version after re-embedding for zero-downtime reindex | ARCH-045, REQ-064 | ##### Pipeline and query @@ -2070,7 +2072,7 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali | **Query rewriting** | Pre-retrieval step that rewrites a conversational query into a self-contained form by resolving pronouns, demonstratives, and anaphoric references using conversation history (ARCH-061). | | **QueryPipeline** | Protocol abstracting the RAG query flow. Phase 1: SimpleQueryPipeline. Phase 2: AdvancedQueryPipeline with query rewriting (ARCH-061), reranking, and verification. | | **QueryTrace** | Structured per-step trace of a query execution (timing, chunk refs, model info). Separate from audit log. No query/response text. | -| **Qdrant** | Open-source vector database with native dense, sparse, and hybrid search. Phase 2 candidate for VectorStoreProvider (alternative to pgvector). Supports payload filtering, scalar quantization, and geo-search. | +| **Qdrant** | Open-source vector database with native dense, sparse, and hybrid search. Phase 2 (implemented): QdrantVectorStoreProvider (alternative to pgvector). Supports payload filtering, scalar quantization, and geo-search. | | **RAG** | Retrieval-Augmented Generation: technique combining document retrieval with LLM generation. | | **RAGAS** | Standalone RAG evaluation framework (Apache 2.0). Reference-free metrics: Faithfulness, Context Relevancy, Answer Relevancy, Context Recall, Context Precision. Supports synthetic test generation. Candidate for Phase 2 evaluation (ARCH-050). | | **raw_filters** | Provider-specific filter escape hatch on VectorStoreProvider.search(). Dict parameter for advanced filter expressions (Qdrant range/geo, Milvus boolean expressions, ChromaDB operator dicts) that SearchFilters cannot express. Phase 1: ignored. | @@ -2083,15 +2085,15 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali | **SearchFilters** | Typed metadata filters for vector search. Generic fields: content_type, language. Accepts arbitrary additional keys at runtime (JSONB is schema-free). Domain-specific fields (e.g., course_id for vektra-learn) defined by verticals in Phase 2. Each VectorStoreProvider translates to provider-specific syntax (Phase 1 pgvector: JSONB WHERE clause with GIN index; Qdrant: payload filter). For advanced filters beyond SearchFilters fields, see raw_filters parameter. | | **SearchMode** | Enum controlling vector search strategy: DENSE (Phase 1), SPARSE, HYBRID (Phase 2 with RRF fusion). | | **SparseEmbeddingProvider** | Protocol abstracting sparse vector generation for hybrid search. Converts text to SparseVector (token indices + weights). Phase 1: not registered (None). Phase 2: FastEmbedBM25Provider, SPLADEProvider, or BM25sProvider. | -| **SPLADE** | Sparse Lexical and Expansion Model: neural model that generates sparse vectors with term expansion. More powerful than BM25 (expands semantically related terms) but heavier (~500 MB). Phase 2 candidate for SparseEmbeddingProvider. | +| **SPLADE** | Sparse Lexical and Expansion Model: neural model that generates sparse vectors with term expansion. More powerful than BM25 (expands semantically related terms) but heavier (~500 MB). Alternative SparseEmbeddingProvider implementation. | | **Soft delete** | Deletion pattern marking records with deleted_at timestamp instead of removing them. Cleanup job removes after retention period. | | **SSE** | Server-Sent Events: streaming protocol for real-time query responses (Accept: text/event-stream). | | **Startup validation** | Ordered sequence of 8 checks executed before the application accepts requests (ARCH-057). Validates configuration, connectivity, schema, providers, and templates. Fails fast with structured plain-text error messages including remediation hints. | -| **TEI** | Text Embeddings Inference (Hugging Face): external embedding server with dynamic batching and Prometheus metrics. Phase 2 option for EmbeddingProvider, offloading model RAM from the application container. | +| **TEI** | Text Embeddings Inference (Hugging Face): external embedding server with dynamic batching and Prometheus metrics. Optional EmbeddingProvider via --profile tei, offloading model RAM from the application container. | | **TLS termination** | Decrypting HTTPS traffic at reverse proxy, forwarding HTTP to application container. | | **Token budget** | Allocation strategy (ARCH-055) that distributes model context window across prompt components (system prompt, question, response reserve, chunks, history) by priority order. Ensures prompt never exceeds model capacity. | | **Top-k** | Number of most relevant chunks retrieved for RAG context (default: 5). | -| **Vector store** | Database or service optimized for similarity search over embeddings. Phase 1: pgvector (PostgreSQL extension). Phase 2 candidate: Qdrant (dedicated vector database). | +| **Vector store** | Database or service optimized for similarity search over embeddings. Phase 1: pgvector (PostgreSQL extension). Phase 2 (implemented): Qdrant (dedicated vector database, optional). | | **VectorStoreProvider** | Protocol abstracting vector storage and similarity search. Supports SearchMode (DENSE/SPARSE/HYBRID), metadata filtering via SearchFilters, index versioning, and full-store contract (ARCH-051). Phase 1: PgvectorProvider with dense search only. | --- @@ -2131,7 +2133,7 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali | REQ-058 Content type detection | ARCH-042 Magic bytes | Reliable dispatch, mismatch warnings | | REQ-059 LLM graceful degradation | ARCH-043 Graceful degradation, ARCH-056 Retrieval quality | Fallback model, context-only response, no-relevant-context path | | REQ-060 QueryTrace | ARCH-041 Audit/analytics separation | Per-step RAG tracing, GDPR-safe | -| REQ-061 EventEmitter | ARCH-038 EventEmitter interface | NoOp Phase 1, webhook Phase 2 | +| REQ-061 EventEmitter | ARCH-038 EventEmitter interface | NoOp Phase 1, LogEventEmitter Phase 2 | | REQ-062 ProviderRegistry | ARCH-039 ProviderRegistry pattern | Unified provider configuration | | REQ-063 Metadata filtering | ARCH-044 Chunk metadata filtering | JSONB + GIN index, SearchFilters in search() | | REQ-064 Zero-downtime reindex | ARCH-045 Index version | Atomic version switch, no downtime | @@ -2215,4 +2217,5 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali *Version 1.8 - Configuration reference: ARCH-060 (37 env vars: 35 VEKTRA_* + 2 external, 5 newly named + VEKTRA_MAX_PDF_SIZE renamed to VEKTRA_MAX_FILE_SIZE_MB, startup validation mapping, .env.example)* *Version 1.9 - Quality scenarios: Section 10 expanded from 12 to 18 QS entries, quality tree restructured, 6 architecture-derived scenarios added (ARCH-043 degradation, ARCH-057 startup, ARCH-039 extensibility, ARCH-040 evolvability, NFR-008 retention, NFR-010 progress), validation scenario cross-reference (10.4)* *Version 1.9.1 - Conversational query rewriting: ARCH-061 (pre-retrieval query rewriting in AdvancedQueryPipeline), ADR-0023, rewrite.j2 template added to ARCH-054, ARCH-036 Phase 2 updated. Multilingual embedding note added to ADR-0013.* +*Version 2.0 - Phase 2 delivery annotations: all Phase 2 features marked as implemented (v0.2.0). Component table expanded with vektra-analytics and vektra-learn. Protocol implementations updated: QdrantVectorStoreProvider, UnstructuredExtractor, DualStrategyChunking, AdvancedQueryPipeline, FastEmbedBM25Provider, LogEventEmitter. WebhookEventEmitter deferred to Phase 3.* *Version 1.10 - OQ-018/OQ-019 resolution: ARCH-062 (admin UI server-side rendering, ADR-0024), ARCH-063 (learn chatbot widget, ADR-0025), ARCH-064 (Phase 2 hardware target 8GB/4CPU). Glossary: HTMX added. Deferred table: 3 entries added.* diff --git a/.s2s/requirements.md b/.s2s/requirements.md index dcf0cb04..83330c5c 100644 --- a/.s2s/requirements.md +++ b/.s2s/requirements.md @@ -3,10 +3,10 @@ # Software Requirements Specification **Project**: Vektra -**Version**: 1.5.0 +**Version**: 1.6.0 **Date**: 2026-02-18 **Sessions**: 20260129-specs-vektra (baseline), 20260201-specs-vektra-integration (merged) -**Corrections**: Manual gap closure v1.2 (REQ-048/049), v1.3 (REQ-050/051, EX-009 to EX-012, path fixes), v1.4 (architectural review: REQ-052 to REQ-065, EX-013/014, OQ-017 to OQ-019), v1.4.1 (consistency review: OQ-015 resolved, OQ-013 partial, REQ-056/057 edge cases, EX-014 ref), v1.5.0 (docs-008: REQ-066 no_relevant_context) +**Corrections**: Manual gap closure v1.2 (REQ-048/049), v1.3 (REQ-050/051, EX-009 to EX-012, path fixes), v1.4 (architectural review: REQ-052 to REQ-065, EX-013/014, OQ-017 to OQ-019), v1.4.1 (consistency review: OQ-015 resolved, OQ-013 partial, REQ-056/057 edge cases, EX-014 ref), v1.5.0 (docs-008: REQ-066 no_relevant_context), v1.6.0 (Phase 2 delivery status annotations) ## 1. Introduction @@ -18,14 +18,20 @@ Modular open-source platform for Retrieval-Augmented Generation (RAG) with speci **In scope (Phase 1)**: - vektra-core: RAG engine, LLM abstraction, conversation management, safeguards -- vektra-ingest: Document processing pipeline - PDF, Word, PPT (OCR deferred to Phase 2) +- vektra-ingest: Document processing pipeline - PDF, Word, PPT - vektra-index: Vector store abstraction, embedding, semantic search - vektra-admin: System administration interface - minimal Phase 1, full Phase 2 +**Delivered in Phase 2 (v0.2.0)**: +- vektra-core: Advanced pipeline, persistent conversations, query rewriting, reranking, safeguard hooks +- vektra-ingest: OCR via Unstructured, dual chunking, document versioning, batch operations +- vektra-index: Hybrid search (dense+sparse), Qdrant vector store provider +- vektra-analytics: QueryTrace storage, metrics aggregation, reporting API +- vektra-learn: E-learning vertical backend (LMS-agnostic), chatbot widget +- vektra-admin: Full dashboard (HTMX + Jinja2), RLS enforcement, rate limiting + **Out of scope**: -- vektra-analytics: Metrics aggregation, reporting API, alerting (Phase 2) -- vektra-learn: E-learning vertical backend, LMS-agnostic (Phase 2) -- vektra-moodle: PHP plugin - separate repo (Phase 2) +- vektra-moodle: PHP plugin - separate repo (Phase 2, not started) - vektra-sdk-py: Python SDK - separate repo (Phase 3) - vektra-sdk-js: JavaScript SDK - separate repo (Phase 3) @@ -161,7 +167,7 @@ Modular open-source platform for Retrieval-Augmented Generation (RAG) with speci ### REQ-016: Phase 1 input constraints - **Priority**: must -- **Description**: Explicit input constraints for Phase 1 APIs: file size: max 50MB (configurable via VEKTRA_MAX_FILE_SIZE_MB), query text: max 4000 characters, supported PDF types: text-based PDFs (scanned/OCR deferred to Phase 2), encrypted PDFs: rejected with ERR-INGEST-001, chunk size: 1000 tokens (configurable via VEKTRA_CHUNK_SIZE), chunk overlap: 200 tokens (configurable via VEKTRA_CHUNK_OVERLAP). Scanned PDF detection: PDF classified as scanned if text extraction yields < 100 characters per page (average of first 5 pages). Detection occurs before chunking (fail fast). Rejected with ERR-INGEST-003. +- **Description**: Explicit input constraints for Phase 1 APIs: file size: max 50MB (configurable via VEKTRA_MAX_FILE_SIZE_MB), query text: max 4000 characters, supported PDF types: text-based PDFs (Phase 1), scanned PDFs via OCR with UnstructuredExtractor (Phase 2), encrypted PDFs: rejected with ERR-INGEST-001, chunk size: 1000 tokens (configurable via VEKTRA_CHUNK_SIZE), chunk overlap: 200 tokens (configurable via VEKTRA_CHUNK_OVERLAP). Scanned PDF detection: PDF classified as scanned if text extraction yields < 100 characters per page (average of first 5 pages). Detection occurs before chunking (fail fast). Rejected with ERR-INGEST-003. - **Acceptance Criteria**: - [ ] Oversized PDF rejected with ERR-INGEST-002 and size limit in message - [ ] Long query rejected with ERR-QUERY-003 and limit in message @@ -235,9 +241,9 @@ Modular open-source platform for Retrieval-Augmented Generation (RAG) with speci - [ ] Key hash uses argon2id algorithm - [ ] Tokens may have multiple scopes (REQ-031) -### REQ-024: Phase 1 scope enforcement +### REQ-024: Scope enforcement - **Priority**: must -- **Description**: Phase 1 scope enforcement uses REQ-031 scope terminology (admin, ingest, query). Phase 1 enforcement behavior: 'admin' scope grants access to all endpoints (superset of all permissions), 'ingest' scope can be stored in key but enforcement deferred to Phase 2, 'query' scope can be stored in key but enforcement deferred to Phase 2, unknown scope rejected with 403. This is a deliberate simplification: Phase 1 has one user type (Platform Operator) who needs full access. Phase 2 will enforce granular scopes. +- **Description**: Scope enforcement uses REQ-031 scope terminology (admin, ingest, query). Phase 1 enforcement behavior: 'admin' scope grants access to all endpoints (superset of all permissions), 'ingest' and 'query' scopes stored but not enforced, unknown scope rejected with 403. Phase 2 (implemented): granular scope enforcement via require_scope() middleware — 'ingest' keys restricted to ingest endpoints, 'query' keys restricted to query endpoints. - **Acceptance Criteria**: - [ ] Enforcement logic uses 'admin', 'ingest', 'query' scopes per REQ-031 - [ ] All keys with 'admin' scope can access all endpoints @@ -805,17 +811,17 @@ Vector store and metadata database encryption delegated to PostgreSQL native enc ## 5. Out of Scope - **EX-001**: CLI deferred to Phase 2 - Dedicated CLI tool (vektra-cli or similar) is explicitly out of scope for Phase 1. CLI becomes a candidate when SDK users (secondary audience) become a priority. -- **EX-002**: OCR and scanned PDF support deferred to Phase 2 - Optical character recognition (OCR) for scanned PDFs is excluded. Text-extractable PDFs only. Detection via REQ-016 threshold (<100 chars/page average). +- **EX-002**: ~~OCR and scanned PDF support deferred to Phase 2~~ [Phase 2: implemented via UnstructuredExtractor with Tesseract OCR] - Optical character recognition (OCR) for scanned PDFs is excluded. Text-extractable PDFs only. Detection via REQ-016 threshold (<100 chars/page average). - **EX-003**: Automatic retry and circuit breakers deferred to Phase 2 - Phase 1 does not implement automatic retry, circuit breakers, or resilience patterns. Error detection and classification are in scope (BR-001, BR-002). Retry responsibility is on the caller. - **EX-004**: CI performance baselines deferred to Phase 2 - Automated CI pipeline performance regression detection is deferred. Phase 1 defines performance targets (REQ-017) but does not enforce them via automated baseline comparison. -- **EX-005**: Batch operations deferred to Phase 2 - Batch ingestion (ingest multiple documents in single request) and batch deletion are excluded. Single-document operations only. -- **EX-006**: Multi-tenant isolation deferred to Phase 2 - Phase 1 operates as single-tenant deployment. Multi-tenant data isolation, per-tenant namespacing, and tenant-scoped API keys are excluded. -- **EX-007**: Analytics and reporting deferred to Phase 2 - vektra-analytics component (query metrics, usage dashboards, alerting) is excluded. Audit logging (REQ-022) provides raw data; aggregation and visualization deferred. +- **EX-005**: ~~Batch operations deferred to Phase 2~~ [Phase 2: implemented - batch delete API] - Batch ingestion (ingest multiple documents in single request) and batch deletion are excluded. Single-document operations only. +- **EX-006**: ~~Multi-tenant isolation deferred to Phase 2~~ [Phase 2: implemented - RLS policies, namespace isolation, per-key scope enforcement] - Phase 1 operates as single-tenant deployment. Multi-tenant data isolation, per-tenant namespacing, and tenant-scoped API keys are excluded. +- **EX-007**: ~~Analytics and reporting deferred to Phase 2~~ [Phase 2: implemented - vektra-analytics component] - vektra-analytics component (query metrics, usage dashboards, alerting) is excluded. Audit logging (REQ-022) provides raw data; aggregation and visualization deferred. - **EX-008**: SDKs deferred to Phase 3 - Python SDK (vektra-sdk-py) and JavaScript SDK (vektra-sdk-js) are excluded from Phase 1 and Phase 2. SDK development targets Phase 3 when external developer adoption becomes priority. -- **EX-009**: Markdown ingestion deferred to Phase 2 - Markdown file extraction is excluded from Phase 1. PDF, Word, and PowerPoint cover primary use cases. Markdown support is low complexity and can be added in Phase 2 if needed. +- **EX-009**: ~~Markdown ingestion deferred to Phase 2~~ [Phase 2: implemented - markdown extractor] - Markdown file extraction is excluded from Phase 1. PDF, Word, and PowerPoint cover primary use cases. Markdown support is low complexity and can be added in Phase 2 if needed. - **EX-010**: Granular ingest APIs deferred to Phase 2 - Separate APIs for extract, clean, and chunk steps are excluded. Phase 1 provides atomic POST /ingest only. Granular APIs may be added in Phase 2 for advanced debugging and custom pipelines. -- **EX-011**: Ingest event emission deferred to Phase 2 - Event emission for monitoring and retry orchestration is excluded. Phase 1 uses polling (GET /ingest/jobs/{id}/status). Webhook/event-based notifications may be added in Phase 2 for n8n integration. -- **EX-012**: Multiple chunking strategies deferred to Phase 2 - Phase 1 supports only fixed-size chunking (REQ-016) behind ChunkingStrategy Protocol (REQ-054). Semantic chunking, dual-strategy chunking (text + table preservation), and parent-child hierarchy are deferred to Phase 2 as implementation swaps. +- **EX-011**: ~~Ingest event emission deferred to Phase 2~~ [Phase 2: implemented - LogEventEmitter] - Event emission for monitoring and retry orchestration is excluded. Phase 1 uses polling (GET /ingest/jobs/{id}/status). Webhook/event-based notifications may be added in Phase 2 for n8n integration. +- **EX-012**: ~~Multiple chunking strategies deferred to Phase 2~~ [Phase 2: implemented - DualStrategyChunking] - Phase 1 supports only fixed-size chunking (REQ-016) behind ChunkingStrategy Protocol (REQ-054). Semantic chunking, dual-strategy chunking (text + table preservation), and parent-child hierarchy are deferred to Phase 2 as implementation swaps. - **EX-013**: LlamaIndex not adopted for Phase 1-2 - RAG pipeline features (hybrid search, reranking, query routing) are implemented directly behind QueryPipeline Protocol (REQ-053). LlamaIndex not adopted due to version instability (v0.14 breaking changes), ~150-200 MB dependency footprint, debugging opacity, and abstraction mismatch with Vektra's Protocol-based design. RAG quality evaluation addressed by standalone frameworks (RAGAS or DeepEval) rather than LlamaIndex built-in evaluators. Reassessment for Phase 3+ if sub-question decomposition or agentic RAG features are needed. - **EX-014**: NeMo Guardrails excluded from Phase 2 baseline - NeMo Guardrails excluded from default SafeguardHook (REQ-044) implementation due to heavy footprint. Presidio for PII detection is accepted for Phase 2. For query and output guardrails, lighter alternatives (keyword filtering + embedding-based classification) should be evaluated before committing to NeMo. @@ -826,9 +832,9 @@ Vector store and metadata database encryption delegated to PostgreSQL native enc - **OQ-014**: Confidence scoring algorithm basis - Algorithm for response confidence scores needs research. Factors: chunk relevance scores, coverage of query terms, source diversity. Deferred to Phase 2 spike. - **OQ-015**: ~~Safeguard hook points specification~~ - **Resolved**: SafeguardHook Protocol fully defined in architecture.md section 8.3 with signatures for pre_query(), post_retrieval(), pre_response(). See REQ-044. - **OQ-016**: File storage encryption scope - Encryption at rest for ingested files depends on vektra-ingest storage architecture. Deferred to design phase. -- **OQ-017**: ORM/database layer selection - SQLAlchemy 2.0 async (with asyncpg) vs SQLModel. Both support repository pattern and Alembic migrations. Decision needed before implementation start, to be documented as ADR. -- **OQ-018**: UI architecture for admin and learn chatbot - Phase 2 requires admin UI and chatbot widget. Options: SPA (React/Vue) vs server-side (HTMX/Jinja2) for admin; standalone JS widget vs npm package for chatbot. Decision deferred to Phase 2 design. -- **OQ-019**: Phase 2 hardware minimum - Phase 2 full-featured (e5-large + cross-encoder + Presidio) estimated at ~2.9GB application + ~512MB PostgreSQL = ~3.4GB. Recommendation: document 8GB RAM / 4 CPU as Phase 2 target. Phase 2 with only hybrid search + reranking (without e5-large) can stay within 4GB. Phase 1 stays at 4GB / 2 CPU per NFR-006. +- ~~**OQ-017**: ORM/database layer selection~~ - **Resolved**: ADR-0022 (SQLAlchemy 2.0 async with asyncpg). +- ~~**OQ-018**: UI architecture for admin and learn chatbot~~ - **Resolved**: ADR-0024 (HTMX + Jinja2 server-side for admin), ADR-0025 (backend-served JS bundle for chatbot widget). +- ~~**OQ-019**: Phase 2 hardware minimum~~ - **Resolved**: ARCH-064 (8GB RAM / 4 CPU formalized as Phase 2 minimum). --- *Generated by Spec2Ship /s2s:specs* @@ -836,4 +842,4 @@ Vector store and metadata database encryption delegated to PostgreSQL native enc *Manual corrections: 2026-02-01 (REQ-048, REQ-049, citation format, path alignment)* *Architectural review: 2026-02-06 (REQ-052 to REQ-065, EX-013/014, OQ-017 to OQ-019, REQ-043/048/050 amended)* *Consistency review: 2026-02-06 (OQ-015 resolved, OQ-013 partial, REQ-056/057 edge cases clarified, EX-014 ref added)* -*Artifacts: 60 functional requirements, 13 NFRs, 5 business rules, 14 exclusions, 7 open questions (1 resolved)* +*Artifacts: 60 functional requirements, 13 NFRs, 5 business rules, 14 exclusions (7 resolved in Phase 2), 7 open questions (4 resolved)* diff --git a/docker-compose.yml b/docker-compose.yml index 9cd274e9..29f8b57a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -5,8 +5,8 @@ # Usage: # docker compose up -d # core stack # docker compose --profile local-llm up -d # + Ollama for local LLM -# docker compose --profile qdrant up -d # + Qdrant (Phase 2) -# docker compose --profile tei up -d # + TEI embeddings (Phase 2) +# docker compose --profile qdrant up -d # + Qdrant vector store +# docker compose --profile tei up -d # + TEI embeddings server # ========================================================================== services: From 8609ce9a4e17c9f54a417b13c1ae7fbfcfad0948 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 18:44:22 +0100 Subject: [PATCH 02/25] chore: bump version to 0.2.0 and update CHANGELOG - All 8 pyproject.toml: 0.2.0-dev -> 0.2.0 - CHANGELOG.md: add [0.2.0] section with Phase 2 features and fixes Co-Authored-By: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 29 +++++++++++++++++++++++++++++ vektra-admin/pyproject.toml | 2 +- vektra-analytics/pyproject.toml | 2 +- vektra-app/pyproject.toml | 2 +- vektra-core/pyproject.toml | 2 +- vektra-index/pyproject.toml | 2 +- vektra-ingest/pyproject.toml | 2 +- vektra-learn/pyproject.toml | 2 +- vektra-shared/pyproject.toml | 2 +- 9 files changed, 37 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b6f091c5..524c4c48 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,35 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [0.2.0] - 2026-03-14 + +Phase 2: advanced RAG features, e-learning vertical, and production hardening. + +### Added + +- **vektra-core**: AdvancedQueryPipeline with conversational query rewriting (ARCH-061), cross-encoder reranking via `rerankers` library, hybrid search orchestration. Persistent encrypted conversations (pgcrypto). Response feedback collection (response_id, citation_id). Presidio PII safeguard with content modification. Composable Jinja2 prompt templates (system, context, conversation, rewrite) +- **vektra-ingest**: OCR support via UnstructuredExtractor (Tesseract, optional INSTALL_UNSTRUCTURED build arg). DualStrategyChunking (text split with overlap, tables never split, parent-child hierarchy). Document versioning (re-ingest creates new version, soft-deletes old). Batch delete API. Markdown extractor. Granular pipeline APIs (extract, chunk, embed endpoints) +- **vektra-index**: QdrantVectorStoreProvider with native dense/sparse/hybrid search. FastEmbedBM25Provider for sparse embeddings. Hybrid search via Reciprocal Rank Fusion (RRF). Zero-downtime reindex API with index version management +- **vektra-analytics** (new component): QueryTrace dedicated storage, metrics aggregation (latency, retrieval scores, model distribution, throughput), reporting API with namespace filtering +- **vektra-learn** (new component): LMS-agnostic e-learning vertical with enrollment management, course-scoped content ingestion, JWT dashboard token generation, course-scoped RAG query endpoint. Backend-served chatbot widget (esbuild IIFE bundle) with streaming, source citations, light/dark themes, i18n (en/it) +- **vektra-admin**: HTMX + Jinja2 admin dashboard (health, keys, namespaces, audit, config pages). Per-key rate limiting (rate_limit_rpm). Granular scope enforcement (admin/ingest/query via require_scope middleware). Namespace quota management +- **Database**: 4 new migrations (Phase 2 tables, RLS policies, hybrid search indexes, learn tables). PostgreSQL RLS policies for namespace isolation. TOCTOU fix on API key creation +- **Infrastructure**: Docker multi-stage build with widget-builder stage. Qdrant Docker Compose profile (--profile qdrant). TEI embedding server profile (--profile tei). INSTALL_UNSTRUCTURED build arg for optional OCR. Torch CPU-only optimization +- **Protocols**: SparseEmbeddingProvider, extended VectorStoreProvider (SearchMode, full-store contract), ChunkingStrategy, extended SafeguardHook (content modification), LogEventEmitter +- **Configuration**: 11 new env vars (rewrite, rerank, webhook, ingest extensions, learn JWT). 48 total VEKTRA_* variables +- **ADRs**: ADR-0022 (SQLAlchemy async), ADR-0023 (query rewriting), ADR-0024 (admin UI server-side), ADR-0025 (chatbot widget) + +### Fixed + +- Qdrant point IDs: use uuid5(doc_id, chunk_index) for deterministic UUID generation +- Qdrant collection auto-creation at startup via ensure_collection() +- no_relevant_context detection: trigger on zero filtered results (not only when search returns results) +- Retrieval filter deduplication: preserve score-descending order +- Learn enrollment IntegrityError: detect FK violation via sqlstate/pgcode (not string matching) +- Learn content/ingest: SSRF mitigation with per-hop DNS re-validation on redirects +- Ingest namespace: auto-create namespace record on first ingest (pg upsert) +- Ingest status: return "new" (not "indexed") for successfully ingested documents + ## [0.1.0] - 2026-02-27 Phase 1 MVP: from `git clone` to first RAG query in under 30 minutes. diff --git a/vektra-admin/pyproject.toml b/vektra-admin/pyproject.toml index fd0feeab..86b39a40 100644 --- a/vektra-admin/pyproject.toml +++ b/vektra-admin/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-admin" -version = "0.2.0-dev" +version = "0.2.0" description = "System administration interface: health, API key management, audit log" readme = "README.md" requires-python = ">=3.12" diff --git a/vektra-analytics/pyproject.toml b/vektra-analytics/pyproject.toml index fbfcad4b..13d3cc2a 100644 --- a/vektra-analytics/pyproject.toml +++ b/vektra-analytics/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-analytics" -version = "0.2.0-dev" +version = "0.2.0" description = "QueryTrace storage, metrics aggregation, and reporting API" readme = "README.md" requires-python = ">=3.12" diff --git a/vektra-app/pyproject.toml b/vektra-app/pyproject.toml index 063dc4e4..bca43ccc 100644 --- a/vektra-app/pyproject.toml +++ b/vektra-app/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-app" -version = "0.2.0-dev" +version = "0.2.0" description = "FastAPI application assembly and startup validation (ARCH-015, ARCH-057)" readme = "README.md" requires-python = ">=3.12" diff --git a/vektra-core/pyproject.toml b/vektra-core/pyproject.toml index 28f091b1..766c2461 100644 --- a/vektra-core/pyproject.toml +++ b/vektra-core/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-core" -version = "0.2.0-dev" +version = "0.2.0" description = "RAG engine, LLM abstraction, and conversation management" readme = "README.md" requires-python = ">=3.12" diff --git a/vektra-index/pyproject.toml b/vektra-index/pyproject.toml index d1249189..28dcc93e 100644 --- a/vektra-index/pyproject.toml +++ b/vektra-index/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-index" -version = "0.2.0-dev" +version = "0.2.0" description = "Vector store abstraction and semantic search for the Vektra platform" readme = "README.md" requires-python = ">=3.12" diff --git a/vektra-ingest/pyproject.toml b/vektra-ingest/pyproject.toml index c99b0768..a6d2ad90 100644 --- a/vektra-ingest/pyproject.toml +++ b/vektra-ingest/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-ingest" -version = "0.2.0-dev" +version = "0.2.0" description = "Document processing pipeline for the Vektra platform" readme = "README.md" requires-python = ">=3.12" diff --git a/vektra-learn/pyproject.toml b/vektra-learn/pyproject.toml index 00f1c603..d9807e01 100644 --- a/vektra-learn/pyproject.toml +++ b/vektra-learn/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-learn" -version = "0.2.0-dev" +version = "0.2.0" description = "E-learning vertical: LMS-agnostic API and chatbot widget" readme = "README.md" requires-python = ">=3.12" diff --git a/vektra-shared/pyproject.toml b/vektra-shared/pyproject.toml index 3a4dfd25..bef4c3d4 100644 --- a/vektra-shared/pyproject.toml +++ b/vektra-shared/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vektra-shared" -version = "0.2.0-dev" +version = "0.2.0" description = "Shared protocols and types for the Vektra platform" readme = "README.md" requires-python = ">=3.12" From 00a6a21c0a35069cf8dfa9652e15a552c5edccc2 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:24:55 +0100 Subject: [PATCH 03/25] fix(admin): prevent namespace override via query param in write requests For POST/PUT/PATCH, read namespace from JSON body first. Query param is only used as fallback when body has no namespace field. Prevents a crafted ?namespace= from overriding the payload in multi-tenant mode. PR #38 review comment 2935607772 (CodeRabbit Critical). Co-Authored-By: Claude Opus 4.6 (1M context) --- vektra-admin/src/vektra_admin/rls.py | 18 +++++++++++------- vektra-admin/tests/test_rls.py | 28 ++++++++++++++++++++++++++-- 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/vektra-admin/src/vektra_admin/rls.py b/vektra-admin/src/vektra_admin/rls.py index af8d95ce..7f24ba08 100644 --- a/vektra-admin/src/vektra_admin/rls.py +++ b/vektra-admin/src/vektra_admin/rls.py @@ -55,13 +55,13 @@ async def dispatch( async def _resolve_namespace(request: Request) -> str | None: - """Extract namespace from request body or query params.""" - # Query param (for GET requests) - namespace = request.query_params.get("namespace") - if namespace: - return namespace + """Extract namespace from request body or query params. - # JSON body (for POST requests) + For write methods (POST/PUT/PATCH) the body takes precedence over + query params to prevent a crafted ``?namespace=`` from overriding + the payload namespace in multi-tenant mode. + """ + # JSON body first for write methods (prevents query-param override) if request.method in ("POST", "PUT", "PATCH"): try: body = await request.body() @@ -70,10 +70,14 @@ async def _resolve_namespace(request: Request) -> str | None: ns = data.get("namespace") if isinstance(ns, str): return ns - return None except (json.JSONDecodeError, UnicodeDecodeError): pass + # Query param (safe for GET; fallback for POST without body namespace) + namespace = request.query_params.get("namespace") + if namespace: + return namespace + return None diff --git a/vektra-admin/tests/test_rls.py b/vektra-admin/tests/test_rls.py index c67b83f8..641d483e 100644 --- a/vektra-admin/tests/test_rls.py +++ b/vektra-admin/tests/test_rls.py @@ -50,8 +50,8 @@ async def test_from_json_body(self): assert result == "ns-456" @pytest.mark.asyncio - async def test_query_param_takes_precedence(self): - """Query param is checked before body.""" + async def test_body_takes_precedence_over_query_param(self): + """For write methods, body namespace takes precedence over query param.""" import json body = json.dumps({"namespace": "body-ns"}).encode() @@ -61,8 +61,32 @@ async def test_query_param_takes_precedence(self): body=body, ) result = await _resolve_namespace(request) + assert result == "body-ns" + + @pytest.mark.asyncio + async def test_get_uses_query_param_over_body(self): + """For GET requests, query param is used (body not read).""" + request = _make_request( + method="GET", + query_params={"namespace": "query-ns"}, + ) + result = await _resolve_namespace(request) assert result == "query-ns" + @pytest.mark.asyncio + async def test_post_without_body_falls_back_to_query_param(self): + """POST without body namespace falls back to query param.""" + import json + + body = json.dumps({"other_field": "value"}).encode() + request = _make_request( + method="POST", + query_params={"namespace": "fallback-ns"}, + body=body, + ) + result = await _resolve_namespace(request) + assert result == "fallback-ns" + @pytest.mark.asyncio async def test_no_namespace_returns_none(self): """No namespace in request returns None.""" From ad1f0154c05e54fd4df34cdc9583f308aaf7a98f Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:25:03 +0100 Subject: [PATCH 04/25] fix(index): correct SQLAlchemy text().columns() usage in sparse search Use .columns(score=Float).c.score.label() instead of .columns(score=Float).label() which labels the TextAsFrom object instead of the score column, breaking sparse search at runtime. PR #38 review comment 2935607773 (CodeRabbit Critical). Co-Authored-By: Claude Opus 4.6 (1M context) --- vektra-index/src/vektra_index/providers/pgvector.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vektra-index/src/vektra_index/providers/pgvector.py b/vektra-index/src/vektra_index/providers/pgvector.py index 2132f44f..62f4cb03 100644 --- a/vektra-index/src/vektra_index/providers/pgvector.py +++ b/vektra-index/src/vektra_index/providers/pgvector.py @@ -220,7 +220,7 @@ async def _search_sparse( ), 0.0) """) - score_col = sparse_score_sql.columns(score=Float).label("score") + score_col = sparse_score_sql.columns(score=Float).c.score.label("score") stmt = ( select( From cb55c0e457ab604f1e7b10a76081c100e9758878 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:25:14 +0100 Subject: [PATCH 05/25] fix(index): skip compensating delete on Qdrant upsert timeout On TimeoutError, Qdrant may have already persisted the points server-side. Running compensating delete would cause data loss. Only run cleanup on non-timeout errors where Qdrant explicitly rejected the batch. Deterministic point IDs (uuid5) make re-ingest idempotent. PR #38 review comment 2935607776 (CodeRabbit Critical). Co-Authored-By: Claude Opus 4.6 (1M context) --- vektra-index/src/vektra_index/providers/qdrant.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/vektra-index/src/vektra_index/providers/qdrant.py b/vektra-index/src/vektra_index/providers/qdrant.py index d0912d80..1d0ef2f6 100644 --- a/vektra-index/src/vektra_index/providers/qdrant.py +++ b/vektra-index/src/vektra_index/providers/qdrant.py @@ -161,8 +161,19 @@ async def store( points=points, wait=True, ) + except TimeoutError: + # On timeout, Qdrant may have already persisted the points. + # Compensating delete would cause data loss. Log and re-raise + # so the caller can retry (idempotent via deterministic IDs). + logger.warning( + "qdrant_store_timeout, skipping compensating delete for %d points " + "(upsert may have succeeded server-side)", + len(point_ids), + ) + raise except Exception: - # Compensating delete on partial failure (ARCH-052) + # Non-timeout error: Qdrant explicitly rejected the batch. + # Safe to run compensating delete (ARCH-052). logger.warning( "qdrant_store_failed, executing compensating delete for %d points", len(point_ids), From 612a84f65c1eb542bcdc1eee2ba2153a25186930 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:25:29 +0100 Subject: [PATCH 06/25] chore: add migration 0006 for deletion_reason CHECK and fix __version__ - New migration 0006: ALTER CHECK constraint on source_documents to include 'pipeline_failure' (fixes modified-migration issue from 0001) - vektra-app/__init__.py: 0.2.0-dev -> 0.2.0 PR #38 review comments (Gemini Medium, CodeRabbit Major). Co-Authored-By: Claude Opus 4.6 (1M context) --- .../0006_add_pipeline_failure_reason.py | 42 +++++++++++++++++++ vektra-app/src/vektra_app/__init__.py | 2 +- 2 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 migrations/versions/0006_add_pipeline_failure_reason.py diff --git a/migrations/versions/0006_add_pipeline_failure_reason.py b/migrations/versions/0006_add_pipeline_failure_reason.py new file mode 100644 index 00000000..db897cdf --- /dev/null +++ b/migrations/versions/0006_add_pipeline_failure_reason.py @@ -0,0 +1,42 @@ +"""Add 'pipeline_failure' to source_documents deletion_reason CHECK. + +Environments where 0001 was already applied have the original CHECK +constraint without 'pipeline_failure'. This migration drops and recreates +the constraint to include the new value. + +Revision ID: 0006 +Revises: 0005 +""" + +from alembic import op + +revision = "0006" +down_revision = "0005" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.execute(""" + ALTER TABLE source_documents + DROP CONSTRAINT IF EXISTS ck_source_documents_deletion_reason + """) + op.execute(""" + ALTER TABLE source_documents + ADD CONSTRAINT ck_source_documents_deletion_reason + CHECK (deletion_reason IS NULL OR deletion_reason IN + ('user_request', 'superseded', 'expired', 'pipeline_failure')) + """) + + +def downgrade() -> None: + op.execute(""" + ALTER TABLE source_documents + DROP CONSTRAINT IF EXISTS ck_source_documents_deletion_reason + """) + op.execute(""" + ALTER TABLE source_documents + ADD CONSTRAINT ck_source_documents_deletion_reason + CHECK (deletion_reason IS NULL OR deletion_reason IN + ('user_request', 'superseded', 'expired')) + """) diff --git a/vektra-app/src/vektra_app/__init__.py b/vektra-app/src/vektra_app/__init__.py index c83c9d85..ee3c7198 100644 --- a/vektra-app/src/vektra_app/__init__.py +++ b/vektra-app/src/vektra_app/__init__.py @@ -1,3 +1,3 @@ # vektra-app: FastAPI assembly and startup validation (ARCH-015) -__version__ = "0.2.0-dev" +__version__ = "0.2.0" From e78359660bb23ec767ce276fa5798f5fa763fe45 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:53:05 +0100 Subject: [PATCH 07/25] fix(index): handle race condition in Qdrant collection creation Wrap create_collection in try/except and recheck existence on failure. Prevents startup crash when concurrent replicas both detect missing collection simultaneously. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../src/vektra_index/providers/qdrant.py | 35 +++++++++++-------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/vektra-index/src/vektra_index/providers/qdrant.py b/vektra-index/src/vektra_index/providers/qdrant.py index 1d0ef2f6..5b08b6f0 100644 --- a/vektra-index/src/vektra_index/providers/qdrant.py +++ b/vektra-index/src/vektra_index/providers/qdrant.py @@ -93,20 +93,27 @@ async def ensure_collection(self) -> None: if self._collection_name in existing: return - await self._client.create_collection( - collection_name=self._collection_name, - vectors_config={ - "dense": models.VectorParams( - size=self._dense_dimensions, - distance=models.Distance.COSINE, - ), - }, - sparse_vectors_config={ - "sparse": models.SparseVectorParams( - modifier=models.Modifier.IDF, - ), - }, - ) + try: + await self._client.create_collection( + collection_name=self._collection_name, + vectors_config={ + "dense": models.VectorParams( + size=self._dense_dimensions, + distance=models.Distance.COSINE, + ), + }, + sparse_vectors_config={ + "sparse": models.SparseVectorParams( + modifier=models.Modifier.IDF, + ), + }, + ) + except Exception: + # Race: another replica may have created it between our check and create. + collections = await self._client.get_collections() + if self._collection_name not in {c.name for c in collections.collections}: + raise + return logger.info("Created Qdrant collection: %s", self._collection_name) async def store( From 80623968c2e2d627c41b4816dd1ea8eb7f5d8576 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:53:15 +0100 Subject: [PATCH 08/25] fix(core): make conversation persistence best-effort Wrap add_turn() in try/except so DB or encryption failures do not turn a successful query into a 500. Applies to both execute() and _stream() paths in AdvancedQueryPipeline. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../src/vektra_core/advanced_pipeline.py | 27 +++++++++++++------ 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/vektra-core/src/vektra_core/advanced_pipeline.py b/vektra-core/src/vektra_core/advanced_pipeline.py index b2c71ca6..463e0f28 100644 --- a/vektra-core/src/vektra_core/advanced_pipeline.py +++ b/vektra-core/src/vektra_core/advanced_pipeline.py @@ -352,6 +352,10 @@ async def _run_pre_llm_steps( ) ) + # Recheck: post_retrieval safeguard may have filtered all chunks + if not no_relevant_context and not filtered: + no_relevant_context = True + return steps, filtered, no_relevant_context, effective_query, history def _build_prompt( @@ -520,11 +524,15 @@ async def execute( ) ) - # Save conversation turn + # Save conversation turn (best-effort: DB failure should not turn + # a successful query into a 500) if query.conversation_id is not None: - await self._conversation_store.add_turn( - query.conversation_id, query.question, answer - ) + try: + await self._conversation_store.add_turn( + query.conversation_id, query.question, answer + ) + except Exception as exc: + log.warning("conversation_turn_store_failed", error=str(exc)) total_ms = _elapsed_ms(t_total) trace = QueryTrace( @@ -682,11 +690,14 @@ async def _stream( ) ) - # Save conversation turn + # Save conversation turn (best-effort) if query.conversation_id is not None and full_answer: - await self._conversation_store.add_turn( - query.conversation_id, query.question, full_answer - ) + try: + await self._conversation_store.add_turn( + query.conversation_id, query.question, full_answer + ) + except Exception as exc: + log.warning("conversation_turn_store_failed", error=str(exc)) # Yield sources (only budget-selected chunks, not all filtered) sources_data = [ From e2654a3230ffa15843154ed2026d32419f382715 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:53:23 +0100 Subject: [PATCH 09/25] fix(admin): only cache successful key verifications Do not cache False results in TTLCache to prevent cache poisoning from a flood of invalid tokens evicting valid entries. Co-Authored-By: Claude Opus 4.6 (1M context) --- vektra-admin/src/vektra_admin/keys.py | 5 +++-- vektra-admin/tests/test_ttlcache.py | 7 +++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/vektra-admin/src/vektra_admin/keys.py b/vektra-admin/src/vektra_admin/keys.py index cc306d2c..ad564d6b 100644 --- a/vektra-admin/src/vektra_admin/keys.py +++ b/vektra-admin/src/vektra_admin/keys.py @@ -73,8 +73,9 @@ async def verify_key(plaintext: str, key_hash: str) -> bool: # Cache miss: run argon2id verification in a thread (CPU-bound) result = await asyncio.to_thread(_verify_sync, key_hash, plaintext) - with _cache_lock: - _verify_cache[cache_key] = result + if result: + with _cache_lock: + _verify_cache[cache_key] = True return result diff --git a/vektra-admin/tests/test_ttlcache.py b/vektra-admin/tests/test_ttlcache.py index c9392cdd..825e6f28 100644 --- a/vektra-admin/tests/test_ttlcache.py +++ b/vektra-admin/tests/test_ttlcache.py @@ -43,15 +43,14 @@ async def test_cache_populated_after_verify(self): assert _verify_cache[(key_hash, plaintext)] is True @pytest.mark.asyncio - async def test_cache_populated_for_failed_verify(self): - """Failed verifications should also be cached (False value).""" + async def test_failed_verify_not_cached(self): + """Failed verifications should NOT be cached (prevents cache poisoning).""" _, key_hash, _ = generate_key() wrong_token = "wrong_token_abc" await verify_key(wrong_token, key_hash) with _cache_lock: - assert (key_hash, wrong_token) in _verify_cache - assert _verify_cache[(key_hash, wrong_token)] is False + assert (key_hash, wrong_token) not in _verify_cache @pytest.mark.asyncio async def test_cache_hit_returns_same_result(self): From 45157210b2c75d89f2d725c6ca3db2b45bea7495 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:53:32 +0100 Subject: [PATCH 10/25] fix(core): fail closed on safeguard misconfiguration Raise ValueError for unknown safeguard modes and let ImportError propagate for presidio, instead of silently falling back to passthrough. Catches misconfiguration at startup. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../src/vektra_core/safeguards/__init__.py | 30 ++++++++----------- vektra-core/tests/test_safeguards.py | 13 ++++---- 2 files changed, 19 insertions(+), 24 deletions(-) diff --git a/vektra-core/src/vektra_core/safeguards/__init__.py b/vektra-core/src/vektra_core/safeguards/__init__.py index 57cffe10..a1a83be7 100644 --- a/vektra-core/src/vektra_core/safeguards/__init__.py +++ b/vektra-core/src/vektra_core/safeguards/__init__.py @@ -24,20 +24,16 @@ def create_safeguard(mode: str, *, pii_chunk_threshold: int = 3) -> SafeguardHoo Returns: A SafeguardHook implementation. """ - if mode == "presidio": - try: - from vektra_core.safeguards.presidio import PresidioPIISafeguard - - return PresidioPIISafeguard(pii_chunk_threshold=pii_chunk_threshold) - except Exception as exc: - log.warning( - "safeguard_presidio_unavailable", - error=str(exc), - fallback="passthrough", - ) - return PassthroughSafeguard() - - if mode != "passthrough": - log.warning("safeguard_unknown_mode", mode=mode, fallback="passthrough") - - return PassthroughSafeguard() + normalized = mode.strip().lower() + + if normalized == "passthrough": + return PassthroughSafeguard() + + if normalized == "presidio": + from vektra_core.safeguards.presidio import PresidioPIISafeguard + + return PresidioPIISafeguard(pii_chunk_threshold=pii_chunk_threshold) + + raise ValueError( + f"Unsupported safeguard mode: '{mode}'. Valid modes: 'passthrough', 'presidio'." + ) diff --git a/vektra-core/tests/test_safeguards.py b/vektra-core/tests/test_safeguards.py index 22a60cd7..c7b8bf0c 100644 --- a/vektra-core/tests/test_safeguards.py +++ b/vektra-core/tests/test_safeguards.py @@ -39,19 +39,18 @@ def test_create_presidio(): assert isinstance(sg, PresidioPIISafeguard) -def test_create_unknown_falls_back(): - sg = create_safeguard("nonexistent") - assert isinstance(sg, PassthroughSafeguard) +def test_create_unknown_raises(): + with pytest.raises(ValueError, match="Unsupported safeguard mode"): + create_safeguard("nonexistent") -def test_create_presidio_falls_back_on_import_error(): +def test_create_presidio_raises_on_import_error(): with patch( "vektra_core.safeguards.presidio.PresidioPIISafeguard", side_effect=ImportError("no presidio"), ): - # Force reimport by clearing the cached import - sg = create_safeguard("presidio") - assert isinstance(sg, PassthroughSafeguard) + with pytest.raises(ImportError): + create_safeguard("presidio") # --------------------------------------------------------------------------- From 35bc42efb929151d85303abe6c6d35628f710d66 Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:53:40 +0100 Subject: [PATCH 11/25] fix(admin): guard against non-object JSON in RLS namespace resolution Check isinstance(data, dict) before calling .get() to prevent AttributeError when request body is a JSON array or scalar. Co-Authored-By: Claude Opus 4.6 (1M context) --- vektra-admin/src/vektra_admin/rls.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/vektra-admin/src/vektra_admin/rls.py b/vektra-admin/src/vektra_admin/rls.py index 7f24ba08..f0018a46 100644 --- a/vektra-admin/src/vektra_admin/rls.py +++ b/vektra-admin/src/vektra_admin/rls.py @@ -67,9 +67,10 @@ async def _resolve_namespace(request: Request) -> str | None: body = await request.body() if body: data = json.loads(body) - ns = data.get("namespace") - if isinstance(ns, str): - return ns + if isinstance(data, dict): + ns = data.get("namespace") + if isinstance(ns, str): + return ns except (json.JSONDecodeError, UnicodeDecodeError): pass From a6a79dd5de4262ef0073fba193cf159df1285e0f Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:53:48 +0100 Subject: [PATCH 12/25] fix(core): recheck no_relevant_context after post_retrieval safeguard When post_retrieval filters all chunks, set no_relevant_context=True so the pipeline skips LLM instead of sending an empty-context prompt. Applies to both Simple and Advanced pipelines. Co-Authored-By: Claude Opus 4.6 (1M context) --- vektra-core/src/vektra_core/pipeline.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/vektra-core/src/vektra_core/pipeline.py b/vektra-core/src/vektra_core/pipeline.py index 07cb45a1..4ebed079 100644 --- a/vektra-core/src/vektra_core/pipeline.py +++ b/vektra-core/src/vektra_core/pipeline.py @@ -374,6 +374,10 @@ async def execute( ) ) + # Recheck: post_retrieval safeguard may have filtered all chunks + if not no_relevant_context and not filtered: + no_relevant_context = True + # No relevant context or safeguard blocked → skip LLM, return early if no_relevant_context or safeguard_blocked: trace = QueryTrace( From a900dcd8f3fff911e0e84c2a94ececc98523d27a Mon Sep 17 00:00:00 2001 From: Francesco Scalzo Date: Sat, 14 Mar 2026 19:54:18 +0100 Subject: [PATCH 13/25] fix(admin): authenticate logout, handle duplicate namespace, fix audit action - Add _require_admin_ui dependency to logout for audit completeness - Catch IntegrityError on namespace create, return flash error - Skip helper segments (create/revoke) in _derive_action to avoid "create_create" audit entries - Use strftime instead of isoformat in audit pagination to avoid unencoded +00:00 in URL Co-Authored-By: Claude Opus 4.6 (1M context) --- vektra-admin/src/vektra_admin/middleware.py | 7 +++++- .../templates/partials/audit_rows.html | 2 +- vektra-admin/src/vektra_admin/ui.py | 24 ++++++++++++++++--- 3 files changed, 28 insertions(+), 5 deletions(-) diff --git a/vektra-admin/src/vektra_admin/middleware.py b/vektra-admin/src/vektra_admin/middleware.py index d04f08b2..a6702547 100644 --- a/vektra-admin/src/vektra_admin/middleware.py +++ b/vektra-admin/src/vektra_admin/middleware.py @@ -105,11 +105,16 @@ def _derive_action(method: str, path: str) -> str: """Derive a human-readable action from HTTP method and path. Examples: "create_ingest", "read_health", "delete_api-keys". + Skips helper tails like /create, /revoke to avoid "create_create". """ verb = _METHOD_VERBS.get(method.upper(), method.lower()) # Use the last meaningful path segment (strip /api/v1/ prefix and IDs) segments = [s for s in path.strip("/").split("/") if s and not _is_uuid_like(s)] - resource = segments[-1] if segments else "unknown" + _helper_segments = {"create", "delete", "revoke", "update"} + if len(segments) >= 2 and segments[-1] in _helper_segments: + resource = segments[-2] + else: + resource = segments[-1] if segments else "unknown" return f"{verb}_{resource}" diff --git a/vektra-admin/src/vektra_admin/templates/partials/audit_rows.html b/vektra-admin/src/vektra_admin/templates/partials/audit_rows.html index 7c90d347..5449850b 100644 --- a/vektra-admin/src/vektra_admin/templates/partials/audit_rows.html +++ b/vektra-admin/src/vektra_admin/templates/partials/audit_rows.html @@ -15,7 +15,7 @@