diff --git a/.s2s/CONTEXT.md b/.s2s/CONTEXT.md index 09160264..16df3387 100644 --- a/.s2s/CONTEXT.md +++ b/.s2s/CONTEXT.md @@ -46,7 +46,7 @@ Split criteria defined in [ADR-0002](decisions/ADR-0002-repo-split-criteria.md). ## Cross-Cutting Concerns -- **Authentication**: API key authentication with argon2id hashing, scoped permissions (admin/ingest/query per REQ-031, multiple scopes per key). Single trust boundary at vektra-core gateway. Rate limiting slot in middleware (Phase 2 enforcement). See [ADR-0010](decisions/ADR-0010-authentication-gateway.md). +- **Authentication**: API key authentication with argon2id hashing, scoped permissions (admin/ingest/query per REQ-031, multiple scopes per key). Single trust boundary at vektra-core gateway. Rate limiting enforced via per-key RPM in middleware. See [ADR-0010](decisions/ADR-0010-authentication-gateway.md). - **Authorization**: Namespace isolation via PostgreSQL RLS policies. Application-level filtering for Phase 1, RLS binding via feature flag for multi-tenant activation. Namespace as first-class entity with metadata (ARCH-047). See [ADR-0009](decisions/ADR-0009-namespace-isolation-rls.md). - **Logging**: structlog with JSON output, PII redaction processors. Correlation ID propagation across sync calls and arq jobs. OpenTelemetry spans at module boundaries. QueryTrace (ARCH-041) for RAG-specific observability, separate from audit log. - **Monitoring**: Prometheus metrics on /metrics via starlette-prometheus. Hierarchical health endpoints (GET /health, GET /health/{component}). Memory observability via GET /health/memory. @@ -60,18 +60,18 @@ Split criteria defined in [ADR-0002](decisions/ADR-0002-repo-split-criteria.md). | Component | Role | Phase | |-----------|------|-------| -| vektra-core | RAG engine, LLM abstraction, conversation management, safeguards | 1 | -| vektra-ingest | Document processing pipeline (PDF, OCR, PPT, Word) | 1 | -| vektra-index | Vector store abstraction, embedding, semantic search | 1 | -| vektra-analytics | Metrics aggregation, reporting API, alerting | 2 | -| vektra-learn | E-learning vertical backend (LMS-agnostic) | 2 | -| vektra-admin | System administration interface | 1 (minimal), 2 (full) | +| vektra-core | RAG engine, LLM abstraction, conversation management, safeguards, advanced pipeline | 1, 2 | +| vektra-ingest | Document processing pipeline (PDF, OCR, PPT, Word), versioning, batch ops | 1, 2 | +| vektra-index | Vector store abstraction, embedding, hybrid search (dense+sparse) | 1, 2 | +| vektra-analytics | QueryTrace storage, metrics aggregation, reporting API | 2 (completed) | +| vektra-learn | E-learning vertical backend (LMS-agnostic), chatbot widget | 2 (completed) | +| vektra-admin | System administration interface (HTMX + Jinja2 dashboard) | 1 (minimal), 2 (full) | ### Separate repositories | Component | Role | Reason | Phase | |-----------|------|--------|-------| -| vektra-moodle | Moodle LMS adapter (PHP plugin) | PHP, Moodle Plugin Directory, different lifecycle | 2 | +| vektra-moodle | Moodle LMS adapter (PHP plugin) | PHP, Moodle Plugin Directory, different lifecycle | 2 (not started) | | vektra-sdk-py | Python SDK | Published to PyPI, independent versioning | 3 | | vektra-sdk-js | JavaScript/TypeScript SDK | Published to npm, independent versioning | 3 | @@ -103,12 +103,12 @@ vektra-moodle ──────────────── (integrates learn See [requirements.md](requirements.md) for the complete Software Requirements Specification. -**Key Phase 1 deliverables**: +**Key deliverables**: - 60 approved functional requirements (REQ-001 to REQ-065, some IDs unused) - 5 business rules - 13 non-functional requirements (8 HARD, 5 TARGET) -- 14 explicit exclusions (Phase 2/3 deferrals) -- 7 open questions (1 resolved, remainder deferred to design or Phase 2) +- 14 explicit exclusions (7 resolved in Phase 2, 7 remaining for Phase 3+) +- 7 open questions (4 resolved) **Primary user persona**: Platform Operator (DevOps/platform teams) **MVP exit criterion**: 30 minutes from git clone to successful query @@ -119,7 +119,7 @@ See [architecture.md](architecture.md) for complete architecture documentation. **Architectural style**: Modular monolith for Phase 1. Single deployable container with internal package boundaries. See [ADR-0003](decisions/ADR-0003-modular-monolith-phase1.md). -**Deployment**: Docker Compose stack: vektra + postgres (always), ollama (profile: local-llm), qdrant (profile: qdrant, Phase 2). See [ADR-0004](decisions/ADR-0004-minimal-docker-compose-stack.md), [ADR-0012](decisions/ADR-0012-docker-compose-spec.md). +**Deployment**: Docker Compose stack: vektra + postgres (always), ollama (profile: local-llm), qdrant (profile: qdrant). See [ADR-0004](decisions/ADR-0004-minimal-docker-compose-stack.md), [ADR-0012](decisions/ADR-0012-docker-compose-spec.md). **Key technology choices**: - Web framework: FastAPI 0.115+ with Pydantic v2 @@ -127,20 +127,20 @@ See [architecture.md](architecture.md) for complete architecture documentation. - Embeddings: sentence-transformers (all-MiniLM-L6-v2) via EmbeddingProvider Protocol - Vector store: pgvector (PostgreSQL extension) via VectorStoreProvider Protocol - Background tasks: arq with PostgreSQL job persistence -- PDF extraction: pdfplumber (Phase 1), Unstructured (Phase 2) via DocumentExtractor Protocol +- PDF extraction: pdfplumber (default), Unstructured (optional, INSTALL_UNSTRUCTURED=true) via DocumentExtractor Protocol - Content type detection: python-magic (magic bytes) - ORM: SQLAlchemy 2.0 async with asyncpg (ORM models internal to modules, Pydantic models as public API) **Protocol interfaces** (9 defined in vektra_shared): - LLMProvider: multi-provider LLM abstraction with graceful degradation - EmbeddingProvider: shared embedding generation with asymmetric model support -- SparseEmbeddingProvider: sparse vector generation for hybrid search (ARCH-053). Phase 1: not registered. Phase 2: BM25 or SPLADE via fastembed -- VectorStoreProvider: pluggable vector store with SearchMode, metadata filtering, index versioning, raw_filters escape hatch, full-store contract (ARCH-051), provider-specific atomicity (ARCH-052). Phase 2 candidate: Qdrant -- DocumentExtractor: PDF, Word, PowerPoint extraction with extended element classification (10 ElementType values) -- ChunkingStrategy: pluggable chunking (fixed-size Phase 1, dual-strategy Phase 2) -- QueryPipeline: RAG pipeline abstraction returning QueryResponse + QueryTrace. Phase 2: AdvancedQueryPipeline with query rewriting (ARCH-061), reranking, hybrid search +- SparseEmbeddingProvider: sparse vector generation for hybrid search (ARCH-053). Phase 1: not registered. Phase 2: FastEmbedBM25Provider via fastembed +- VectorStoreProvider: pluggable vector store with SearchMode, metadata filtering, index versioning, raw_filters escape hatch, full-store contract (ARCH-051), provider-specific atomicity (ARCH-052). Phase 2: QdrantVectorStoreProvider with native DENSE/SPARSE/HYBRID +- DocumentExtractor: PDF, Word, PowerPoint extraction with extended element classification (10 ElementType values). Phase 2: UnstructuredExtractor with OCR support +- ChunkingStrategy: pluggable chunking. FixedSizeChunking (Phase 1), DualStrategyChunking with semantic splitting (Phase 2) +- QueryPipeline: RAG pipeline abstraction returning QueryResponse + QueryTrace. Phase 2: AdvancedQueryPipeline with query rewriting (ARCH-061), reranking, hybrid search (implemented) - SafeguardHook: pre/post query safeguards (3 trust boundary points) with content modification support (ARCH-049) -- EventEmitter: internal event hooks (NoOp Phase 1, webhooks Phase 2) +- EventEmitter: internal event hooks. NoOpEventEmitter (Phase 1), LogEventEmitter (Phase 2) **Key decisions** (64 total, 25 ADRs): - [ADR-0003](decisions/ADR-0003-modular-monolith-phase1.md): Modular monolith for Phase 1 @@ -173,4 +173,4 @@ See [architecture.md](architecture.md) for complete architecture documentation. --- -*Last updated: 2026-03-01* +*Last updated: 2026-03-14* diff --git a/.s2s/architecture.md b/.s2s/architecture.md index fe4f03bc..18712808 100644 --- a/.s2s/architecture.md +++ b/.s2s/architecture.md @@ -429,10 +429,10 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource - **ARCH-035 - EmbeddingProvider Protocol**: Dedicated Protocol for embedding generation. Separates embed_documents() from embed_query() for asymmetric models. Single shared instance between ingest and core. Configurable provider and model via env vars. - **ARCH-036 - QueryPipeline Protocol**: Abstraction of RAG query pipeline. execute() returns (QueryResponse, QueryTrace). Phase 1: SimpleQueryPipeline (embed -> search -> relevance_filter -> build_prompt -> LLM, with graceful degradation). relevance_filter applies minimum score threshold and overlap deduplication (ARCH-056). build_prompt calculates token budget from model context window (ARCH-055), renders composable Jinja2 templates (ARCH-054). Phase 2: AdvancedQueryPipeline adds query_rewrite (ARCH-061), query_classify, rerank, confidence scoring steps. Selectable via VEKTRA_QUERY_PIPELINE. - **ARCH-037 - ChunkingStrategy Protocol**: Abstraction of text chunking. Receives DocumentChunk stream, returns chunked stream. Phase 1: FixedSizeChunking. Selectable via VEKTRA_CHUNKING_STRATEGY. -- **ARCH-038 - EventEmitter interface**: Internal event hooks with NoOp default. Emission points at document lifecycle, query completion, safeguard triggers, API key operations. Phase 2: WebhookEventEmitter with HMAC-SHA256. +- **ARCH-038 - EventEmitter interface**: Internal event hooks with NoOp default. Emission points at document lifecycle, query completion, safeguard triggers, API key operations. Phase 2 (implemented): LogEventEmitter. WebhookEventEmitter with HMAC-SHA256 deferred to Phase 3. - **ARCH-039 - ProviderRegistry pattern**: Unified registry for all Protocol implementations. Dict-based in Phase 1, extensible to entry_points plugin discovery in Phase 2. Consistent VEKTRA_* env var configuration pattern. - **ARCH-040 - Forward-compatible data model**: Phase 1 schema includes fields for Phase 2 features: response_id and citation_id for feedback loops, document version and supersedes_id for versioning, deleted_at and deletion_reason for soft delete, index_version for zero-downtime reindex. All fields nullable/defaulted, no behavioral change in Phase 1. -- **ARCH-041 - Audit/analytics separation**: QueryTrace (per-step timing, chunk refs, model info) emitted separately from audit log. QueryTrace does not contain query text or response content (REQ-051 compliance). Phase 1: structlog emission. Phase 2: dedicated storage. +- **ARCH-041 - Audit/analytics separation**: QueryTrace (per-step timing, chunk refs, model info) emitted separately from audit log. QueryTrace does not contain query text or response content (REQ-051 compliance). Phase 1: structlog emission. Phase 2 (implemented): dedicated query_traces table with reporting API (vektra-analytics). - **ARCH-042 - Content type detection**: Magic bytes validation before DocumentExtractor dispatch. python-magic for MIME detection. Logs warning on mismatch with declared type. - **ARCH-043 - Graceful degradation**: Two degradation paths. LLM degradation: primary model timeout triggers fallback model; both fail triggers context-only response (chunks without LLM synthesis). Retrieval degradation (ARCH-056): when no chunk passes the relevance threshold, QueryResponse.no_relevant_context=True; system returns explicit "no relevant information" instead of synthesizing from irrelevant context. Client always receives value (relevant sources, or empty sources with explanation, at minimum). - **ARCH-044 - Chunk metadata filtering**: JSONB metadata column on chunks with GIN index. SearchFilters parameter in VectorStoreProvider.search(). Generic filterable fields (content_type, language) plus arbitrary key-value pairs. Domain-specific fields (e.g., course_id for vektra-learn) defined by the vertical in Phase 2, stored in the same JSONB column. @@ -444,7 +444,7 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource - **ARCH-050 - RAG evaluation strategy**: Three-tier hybrid pattern for RAG quality evaluation that respects GDPR constraints (REQ-051). CI: synthetic test suite generated from ingested documents via RAGAS or DeepEval, used as regression gate. Staging: evaluation mode (VEKTRA_EVAL_MODE) enabling temporary text capture for batch evaluation, not active in production. Production: QueryTrace metrics only (timing, scores, chunk refs), feedback via response_id/citation_id (REQ-055). No query/response text persisted in production. - **ARCH-051 - VectorStoreProvider full-store contract**: VectorStoreProvider implementations own chunk text, metadata, and embeddings as a self-contained unit. store() persists all three; search() returns text_snippet directly from the provider without secondary lookups. This keeps the Protocol self-contained and avoids dual-query patterns (vector search + relational text lookup). PostgreSQL retains source_documents as canonical source of original documents; chunks in the vector store are derived, re-generable artifacts. When switching providers (e.g., pgvector to Qdrant), chunks are re-ingested into the new provider via the existing reindex mechanism (ARCH-045), not migrated. - **ARCH-052 - Provider-specific ingest atomicity**: Batch chunk ingest atomicity depends on provider capabilities. PgvectorProvider: multi-row INSERT in single SQL transaction (ARCH-010). Non-transactional providers (e.g., Qdrant, Milvus): batch upsert with best-effort durability (Qdrant: wait=true). On partial batch failure, the ingest job is marked as failed and a compensating delete by document_id removes partial writes. The IngestService treats atomicity as a provider contract, not a universal guarantee. -- **ARCH-053 - SparseEmbeddingProvider Protocol**: Dedicated Protocol for sparse vector generation, completing the hybrid search data path. SparseVector fields already exist in ChunkEmbedding and QueryEmbedding; this Protocol defines who populates them. Phase 1: not registered in ProviderRegistry (None). When absent, SearchMode.HYBRID is unavailable and sparse fields remain None. Phase 2: register one of FastEmbedBM25Provider (tokenization + TF, lightweight, Qdrant IDF modifier handles the rest server-side), SPLADEProvider (neural term expansion via fastembed, ~500 MB), or BM25sProvider (in-memory BM25 for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. +- **ARCH-053 - SparseEmbeddingProvider Protocol**: Dedicated Protocol for sparse vector generation, completing the hybrid search data path. SparseVector fields already exist in ChunkEmbedding and QueryEmbedding; this Protocol defines who populates them. Phase 1: not registered in ProviderRegistry (None). When absent, SearchMode.HYBRID is unavailable and sparse fields remain None. Phase 2 (implemented): FastEmbedBM25Provider registered (tokenization + TF, lightweight, Qdrant IDF modifier handles the rest server-side). Alternative options: SPLADEProvider (neural term expansion via fastembed, ~500 MB), or BM25sProvider (in-memory BM25 for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. - **ARCH-054 - Prompt template architecture**: Three composable Jinja2 template files for prompt construction (Phase 1), plus `rewrite.j2` for conversational query rewriting (Phase 2, ARCH-061): `system.j2` (LLM role and behavioral instructions), `context.j2` (retrieved chunk formatting), `conversation.j2` (multi-turn history formatting). Loaded from configurable directory (`VEKTRA_PROMPT_TEMPLATES_DIR`, default: built-in package defaults). Each template defines a typed variable contract: system receives `namespace` and `model_name`; context receives `chunks` (list of dicts with text, score, doc_id, element_type, metadata) and `num_chunks`; conversation receives `turns` (list of dicts with question and answer) and `num_turns`. Composition order is fixed: system + context (if chunks) + conversation (if conversation_id) + user question. prompt_version (ARCH-048) computed as SHA-256[:8] of concatenated template sources; per-template hashes recorded in StepTrace metadata for build_prompt. See ADR-0020. - **ARCH-056 - Retrieval quality controls**: Three controls in SimpleQueryPipeline between vector_search and build_prompt: (1) minimum relevance threshold (`VEKTRA_MIN_RELEVANCE_SCORE`, default 0.3) filters chunks below the score floor; (2) overlap deduplication removes redundant chunks from fixed-size chunking overlap (same document, adjacent positions) keeping the higher-scored chunk; (3) no-relevant-context detection sets QueryResponse.no_relevant_context=True when no chunk passes the threshold, triggering a dedicated response path instead of forcing synthesis from irrelevant context. All three controls are configurable and produce diagnostics in QueryTrace via a retrieval_filter StepTrace. See ADR-0021, REQ-066. @@ -496,10 +496,12 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource | Component | Responsibility | Technology | |-----------|---------------|------------| -| vektra-core | Orchestrates RAG query flow: receives queries, retrieves context, generates LLM responses, enforces safeguards | FastAPI, litellm, sentence-transformers | -| vektra-ingest | Processes documents into indexable chunks: format conversion, chunking, async job management | arq, pdfplumber, python-docx, python-pptx | -| vektra-index | Manages vector storage and semantic search | VectorStoreProvider (Phase 1: pgvector), sentence-transformers | -| vektra-admin | System administration: health monitoring, API key management, configuration | FastAPI | +| vektra-core | Orchestrates RAG query flow: receives queries, retrieves context, generates LLM responses, enforces safeguards. Phase 2: advanced pipeline, persistent conversations, query rewriting, reranking | FastAPI, litellm, sentence-transformers | +| vektra-ingest | Processes documents into indexable chunks: format conversion, chunking, async job management. Phase 2: OCR, dual chunking, document versioning, batch operations | arq, pdfplumber, python-docx, python-pptx, Unstructured (optional) | +| vektra-index | Manages vector storage and semantic search. Phase 2: hybrid search (dense+sparse), Qdrant provider | VectorStoreProvider (pgvector default, Qdrant optional), sentence-transformers | +| vektra-analytics | QueryTrace storage, metrics aggregation, reporting API (Phase 2) | FastAPI, SQLAlchemy | +| vektra-learn | E-learning vertical backend: enrollments, content ingestion, dashboard tokens, course-scoped query, chatbot widget (Phase 2) | FastAPI, PyJWT, esbuild | +| vektra-admin | System administration: health monitoring, API key management, configuration. Phase 2: HTMX + Jinja2 dashboard, RLS enforcement, rate limiting | FastAPI, HTMX, Jinja2 | | vektra_shared | Cross-cutting infrastructure: types, config schemas, error definitions, auth middleware, Protocol interfaces | Pydantic v2 | #### vektra-core @@ -508,7 +510,7 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource **State Owned**: - conversations (encrypted via pgcrypto) -- query_traces (Phase 1: structlog emission; Phase 2: dedicated storage) +- query_traces (Phase 1: structlog emission; Phase 2: dedicated storage table, implemented) **Interfaces**: - Provides: External REST API (POST /query, GET /providers, GET /health) @@ -542,7 +544,7 @@ See [section 8.4](#84-deployment) for Docker Compose specification and resource - Provides: REST routes via main FastAPI app (POST /search, POST /documents/{id}/chunks, DELETE /documents/{id}, GET /stats) + internal Protocol (callable by vektra-core) - Requires: VectorStoreProvider implementation (Phase 1: pgvector) -**Dependencies**: VectorStoreProvider (Phase 1: PostgreSQL with pgvector extension, Phase 2 candidate: Qdrant) +**Dependencies**: VectorStoreProvider (pgvector default, Qdrant optional via --profile qdrant) #### vektra-admin @@ -601,7 +603,7 @@ class EmbeddingProvider(Protocol): async def health_check() -> HealthStatus ``` -Phase 1: SentenceTransformersProvider (all-MiniLM-L6-v2, 384 dims, single shared instance). The distinction between embed_documents() and embed_query() supports asymmetric models (e.g., e5-large requires "query: ..." vs "passage: ..." prefixes). Config: `VEKTRA_EMBEDDING_PROVIDER`, `VEKTRA_EMBEDDING_MODEL`. Phase 2 option: TEI (Hugging Face Text Embeddings Inference) as external embedding server, freeing model RAM from the application container (~90 MB for MiniLM, ~1.2 GB for e5-large). Protocol supports this as a zero-change swap. +Phase 1: SentenceTransformersProvider (all-MiniLM-L6-v2, 384 dims, single shared instance). The distinction between embed_documents() and embed_query() supports asymmetric models (e.g., e5-large requires "query: ..." vs "passage: ..." prefixes). Config: `VEKTRA_EMBEDDING_PROVIDER`, `VEKTRA_EMBEDDING_MODEL`. Optional: TEI (Hugging Face Text Embeddings Inference) as external embedding server via --profile tei, freeing model RAM from the application container (~90 MB for MiniLM, ~1.2 GB for e5-large). Protocol supports this as a zero-change swap. #### SparseEmbeddingProvider (new - ARCH-053) @@ -612,7 +614,7 @@ class SparseEmbeddingProvider(Protocol): def vocab_size() -> int | None ``` -Phase 1: not registered (None in ProviderRegistry). When absent, the IngestService skips sparse vector generation (ChunkEmbedding.sparse = None) and the QueryPipeline skips sparse query embedding (QueryEmbedding.sparse = None), making SearchMode.HYBRID unavailable. Phase 2 candidates: FastEmbedBM25Provider (fastembed `Qdrant/bm25` model, tokenization + term frequencies, lightweight), SPLADEProvider (fastembed `Splade_PP_en_v1`, neural term expansion, ~500 MB, vocabulary ~30K tokens), BM25sProvider (bm25s library, ~100 KB, in-memory index for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. When using Qdrant with IDF modifier (`sparse_vectors_config.modifier = "idf"`), the provider sends only term frequencies and Qdrant computes IDF server-side, keeping the IDF in sync with the corpus automatically. +Phase 1: not registered (None in ProviderRegistry). When absent, the IngestService skips sparse vector generation (ChunkEmbedding.sparse = None) and the QueryPipeline skips sparse query embedding (QueryEmbedding.sparse = None), making SearchMode.HYBRID unavailable. Phase 2 (implemented): FastEmbedBM25Provider (fastembed `Qdrant/bm25` model, tokenization + term frequencies, lightweight). Alternatives: SPLADEProvider (fastembed `Splade_PP_en_v1`, neural term expansion, ~500 MB, vocabulary ~30K tokens), BM25sProvider (bm25s library, ~100 KB, in-memory index for pgvector hybrid search). Config: `VEKTRA_SPARSE_EMBEDDING_PROVIDER`, `VEKTRA_SPARSE_EMBEDDING_MODEL`. When using Qdrant with IDF modifier (`sparse_vectors_config.modifier = "idf"`), the provider sends only term frequencies and Qdrant computes IDF server-side, keeping the IDF in sync with the corpus automatically. #### VectorStoreProvider (extended - REQ-050) @@ -650,7 +652,7 @@ class VectorStoreProvider(Protocol): **Full-store contract (ARCH-051)**: each VectorStoreProvider implementation owns chunk text, metadata, and embeddings. store() persists all three; search() returns text_snippet directly without secondary lookups. Switching provider means re-ingesting chunks (via ARCH-045 reindex), not migrating data. -Phase 1: PgvectorProvider, SearchMode.DENSE only (sparse ignored), filters applied as JSONB WHERE clause, GIN index on metadata column, index_version filter applied, raw_filters ignored. Phase 2 candidate: QdrantVectorStoreProvider with native DENSE/SPARSE/HYBRID support, payload-based namespace and index_version filtering, raw_filters for Qdrant-specific expressions (range, geo, full-text). Qdrant hybrid search: store() persists both dense and sparse vectors as named vectors in the same point; search(mode=HYBRID) maps to Qdrant's prefetch + fusion API (RRF or DBSF, server-side, single round trip). With Qdrant IDF modifier (`sparse_vectors_config.modifier = "idf"`), the server computes IDF from the corpus automatically, keeping BM25 scores in sync as documents are added/removed. The raw_filters parameter is an escape hatch for provider-specific filter expressions (e.g., Qdrant range/geo filters, Milvus boolean expressions, ChromaDB operator dicts). When both filters and raw_filters are provided, the implementation defines precedence. Batch ingest atomicity is provider-specific (ARCH-052). +Phase 1: PgvectorProvider, SearchMode.DENSE only (sparse ignored), filters applied as JSONB WHERE clause, GIN index on metadata column, index_version filter applied, raw_filters ignored. Phase 2 (implemented): QdrantVectorStoreProvider with native DENSE/SPARSE/HYBRID support, payload-based namespace and index_version filtering, raw_filters for Qdrant-specific expressions (range, geo, full-text). Qdrant hybrid search: store() persists both dense and sparse vectors as named vectors in the same point; search(mode=HYBRID) maps to Qdrant's prefetch + fusion API (RRF or DBSF, server-side, single round trip). With Qdrant IDF modifier (`sparse_vectors_config.modifier = "idf"`), the server computes IDF from the corpus automatically, keeping BM25 scores in sync as documents are added/removed. The raw_filters parameter is an escape hatch for provider-specific filter expressions (e.g., Qdrant range/geo filters, Milvus boolean expressions, ChromaDB operator dicts). When both filters and raw_filters are provided, the implementation defines precedence. Batch ingest atomicity is provider-specific (ARCH-052). #### DocumentExtractor @@ -661,7 +663,7 @@ class DocumentExtractor(Protocol): async def health_check() -> HealthStatus ``` -Phase 1: PdfplumberExtractor, WordExtractor, PowerPointExtractor. Content type detection via magic bytes (ARCH-042) before dispatch. Phase 2 option: Unstructured as local library or as external API (self-hosted Docker), offloading heavy dependencies (Tesseract, PyTorch for table detection) from the Vektra container. Same pattern as TEI for EmbeddingProvider. ExtractionRequest maps 1:1 with Unstructured's `partition()` parameters. +Phase 1: PdfplumberExtractor, WordExtractor, PowerPointExtractor. Content type detection via magic bytes (ARCH-042) before dispatch. Phase 2 (implemented): UnstructuredExtractor as local library (INSTALL_UNSTRUCTURED=true build arg), adding OCR support via Tesseract. Offloads heavy dependencies from the default image. ExtractionRequest maps 1:1 with Unstructured's `partition()` parameters. #### ChunkingStrategy (new - ARCH-037) @@ -672,7 +674,7 @@ class ChunkingStrategy(Protocol): ) -> AsyncIterator[DocumentChunk] ``` -Phase 1: FixedSizeChunking (1000 tokens, 200 overlap per REQ-016). Phase 2: DualStrategyChunking (text: split with overlap; tables: never split; parent-child: 2 levels). Config: `VEKTRA_CHUNKING_STRATEGY`. +Phase 1: FixedSizeChunking (1000 tokens, 200 overlap per REQ-016). Phase 2 (implemented): DualStrategyChunking (text: split with overlap; tables: never split; parent-child: 2 levels). Config: `VEKTRA_CHUNKING_STRATEGY`. #### QueryPipeline (new - ARCH-036) @@ -682,7 +684,7 @@ class QueryPipeline(Protocol): async def execute_stream(query: QueryRequest) -> AsyncIterator[QueryChunk] ``` -Phase 1: SimpleQueryPipeline (embed -> search -> relevance_filter -> build_prompt -> LLM, with graceful degradation). relevance_filter applies minimum score threshold, overlap deduplication, and no-relevant-context detection (ARCH-056). build_prompt calculates token budget from model context window (ARCH-055) and renders composable Jinja2 templates (ARCH-054). Phase 2: AdvancedQueryPipeline (classify -> retrieve -> rerank -> synthesize -> verify, implemented directly per ARCH-046). Reranking step recommended via `rerankers` library (Answer.AI): unified API for cross-encoder, FlashRank (~4 MB), Cohere, ColBERT, making the reranking backend swappable without touching pipeline code. Config: `VEKTRA_QUERY_PIPELINE`. +Phase 1: SimpleQueryPipeline (embed -> search -> relevance_filter -> build_prompt -> LLM, with graceful degradation). relevance_filter applies minimum score threshold, overlap deduplication, and no-relevant-context detection (ARCH-056). build_prompt calculates token budget from model context window (ARCH-055) and renders composable Jinja2 templates (ARCH-054). Phase 2 (implemented): AdvancedQueryPipeline (classify -> retrieve -> rerank -> synthesize -> verify, implemented directly per ARCH-046). Reranking step via `rerankers` library (Answer.AI): unified API for cross-encoder, FlashRank (~4 MB), Cohere, ColBERT, making the reranking backend swappable without touching pipeline code. Config: `VEKTRA_QUERY_PIPELINE`. #### SafeguardHook @@ -702,7 +704,7 @@ class EventEmitter(Protocol): async def emit(event_type: str, payload: dict) -> None ``` -Phase 1: NoOpEventEmitter. Emission points: document.indexed, document.failed, query.completed, safeguard.triggered, apikey.created, apikey.revoked. Phase 2: WebhookEventEmitter with HMAC-SHA256 signature. +Phase 1: NoOpEventEmitter. Emission points: document.indexed, document.failed, query.completed, safeguard.triggered, apikey.created, apikey.revoked. Phase 2 (implemented): LogEventEmitter (structlog-based event logging). WebhookEventEmitter with HMAC-SHA256 deferred to Phase 3. ### 8.3.1 Extended types @@ -1733,7 +1735,7 @@ Conventions: | Variable | Type | Default | Required | Description | Reference | |----------|------|---------|----------|-------------|-----------| -| `VEKTRA_VECTOR_STORE_PROVIDER` | string | `pgvector` | No | VectorStoreProvider implementation. Phase 2 option: `qdrant` | ARCH-039, ADR-0012 | +| `VEKTRA_VECTOR_STORE_PROVIDER` | string | `pgvector` | No | VectorStoreProvider implementation. Options: `pgvector` (default), `qdrant` | ARCH-039, ADR-0012 | | `VEKTRA_ACTIVE_INDEX_VERSION` | int | `1` | No | Active index version for search queries. Change to new version after re-embedding for zero-downtime reindex | ARCH-045, REQ-064 | ##### Pipeline and query @@ -2070,7 +2072,7 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali | **Query rewriting** | Pre-retrieval step that rewrites a conversational query into a self-contained form by resolving pronouns, demonstratives, and anaphoric references using conversation history (ARCH-061). | | **QueryPipeline** | Protocol abstracting the RAG query flow. Phase 1: SimpleQueryPipeline. Phase 2: AdvancedQueryPipeline with query rewriting (ARCH-061), reranking, and verification. | | **QueryTrace** | Structured per-step trace of a query execution (timing, chunk refs, model info). Separate from audit log. No query/response text. | -| **Qdrant** | Open-source vector database with native dense, sparse, and hybrid search. Phase 2 candidate for VectorStoreProvider (alternative to pgvector). Supports payload filtering, scalar quantization, and geo-search. | +| **Qdrant** | Open-source vector database with native dense, sparse, and hybrid search. Phase 2 (implemented): QdrantVectorStoreProvider (alternative to pgvector). Supports payload filtering, scalar quantization, and geo-search. | | **RAG** | Retrieval-Augmented Generation: technique combining document retrieval with LLM generation. | | **RAGAS** | Standalone RAG evaluation framework (Apache 2.0). Reference-free metrics: Faithfulness, Context Relevancy, Answer Relevancy, Context Recall, Context Precision. Supports synthetic test generation. Candidate for Phase 2 evaluation (ARCH-050). | | **raw_filters** | Provider-specific filter escape hatch on VectorStoreProvider.search(). Dict parameter for advanced filter expressions (Qdrant range/geo, Milvus boolean expressions, ChromaDB operator dicts) that SearchFilters cannot express. Phase 1: ignored. | @@ -2083,15 +2085,15 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali | **SearchFilters** | Typed metadata filters for vector search. Generic fields: content_type, language. Accepts arbitrary additional keys at runtime (JSONB is schema-free). Domain-specific fields (e.g., course_id for vektra-learn) defined by verticals in Phase 2. Each VectorStoreProvider translates to provider-specific syntax (Phase 1 pgvector: JSONB WHERE clause with GIN index; Qdrant: payload filter). For advanced filters beyond SearchFilters fields, see raw_filters parameter. | | **SearchMode** | Enum controlling vector search strategy: DENSE (Phase 1), SPARSE, HYBRID (Phase 2 with RRF fusion). | | **SparseEmbeddingProvider** | Protocol abstracting sparse vector generation for hybrid search. Converts text to SparseVector (token indices + weights). Phase 1: not registered (None). Phase 2: FastEmbedBM25Provider, SPLADEProvider, or BM25sProvider. | -| **SPLADE** | Sparse Lexical and Expansion Model: neural model that generates sparse vectors with term expansion. More powerful than BM25 (expands semantically related terms) but heavier (~500 MB). Phase 2 candidate for SparseEmbeddingProvider. | +| **SPLADE** | Sparse Lexical and Expansion Model: neural model that generates sparse vectors with term expansion. More powerful than BM25 (expands semantically related terms) but heavier (~500 MB). Alternative SparseEmbeddingProvider implementation. | | **Soft delete** | Deletion pattern marking records with deleted_at timestamp instead of removing them. Cleanup job removes after retention period. | | **SSE** | Server-Sent Events: streaming protocol for real-time query responses (Accept: text/event-stream). | | **Startup validation** | Ordered sequence of 8 checks executed before the application accepts requests (ARCH-057). Validates configuration, connectivity, schema, providers, and templates. Fails fast with structured plain-text error messages including remediation hints. | -| **TEI** | Text Embeddings Inference (Hugging Face): external embedding server with dynamic batching and Prometheus metrics. Phase 2 option for EmbeddingProvider, offloading model RAM from the application container. | +| **TEI** | Text Embeddings Inference (Hugging Face): external embedding server with dynamic batching and Prometheus metrics. Optional EmbeddingProvider via --profile tei, offloading model RAM from the application container. | | **TLS termination** | Decrypting HTTPS traffic at reverse proxy, forwarding HTTP to application container. | | **Token budget** | Allocation strategy (ARCH-055) that distributes model context window across prompt components (system prompt, question, response reserve, chunks, history) by priority order. Ensures prompt never exceeds model capacity. | | **Top-k** | Number of most relevant chunks retrieved for RAG context (default: 5). | -| **Vector store** | Database or service optimized for similarity search over embeddings. Phase 1: pgvector (PostgreSQL extension). Phase 2 candidate: Qdrant (dedicated vector database). | +| **Vector store** | Database or service optimized for similarity search over embeddings. Phase 1: pgvector (PostgreSQL extension). Phase 2 (implemented): Qdrant (dedicated vector database, optional). | | **VectorStoreProvider** | Protocol abstracting vector storage and similarity search. Supports SearchMode (DENSE/SPARSE/HYBRID), metadata filtering via SearchFilters, index versioning, and full-store contract (ARCH-051). Phase 1: PgvectorProvider with dense search only. | --- @@ -2131,7 +2133,7 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali | REQ-058 Content type detection | ARCH-042 Magic bytes | Reliable dispatch, mismatch warnings | | REQ-059 LLM graceful degradation | ARCH-043 Graceful degradation, ARCH-056 Retrieval quality | Fallback model, context-only response, no-relevant-context path | | REQ-060 QueryTrace | ARCH-041 Audit/analytics separation | Per-step RAG tracing, GDPR-safe | -| REQ-061 EventEmitter | ARCH-038 EventEmitter interface | NoOp Phase 1, webhook Phase 2 | +| REQ-061 EventEmitter | ARCH-038 EventEmitter interface | NoOp Phase 1, LogEventEmitter Phase 2 | | REQ-062 ProviderRegistry | ARCH-039 ProviderRegistry pattern | Unified provider configuration | | REQ-063 Metadata filtering | ARCH-044 Chunk metadata filtering | JSONB + GIN index, SearchFilters in search() | | REQ-064 Zero-downtime reindex | ARCH-045 Index version | Atomic version switch, no downtime | @@ -2216,3 +2218,4 @@ Quality scenarios (QS-xx) define measurable targets. Validation scenarios ([vali *Version 1.9 - Quality scenarios: Section 10 expanded from 12 to 18 QS entries, quality tree restructured, 6 architecture-derived scenarios added (ARCH-043 degradation, ARCH-057 startup, ARCH-039 extensibility, ARCH-040 evolvability, NFR-008 retention, NFR-010 progress), validation scenario cross-reference (10.4)* *Version 1.9.1 - Conversational query rewriting: ARCH-061 (pre-retrieval query rewriting in AdvancedQueryPipeline), ADR-0023, rewrite.j2 template added to ARCH-054, ARCH-036 Phase 2 updated. Multilingual embedding note added to ADR-0013.* *Version 1.10 - OQ-018/OQ-019 resolution: ARCH-062 (admin UI server-side rendering, ADR-0024), ARCH-063 (learn chatbot widget, ADR-0025), ARCH-064 (Phase 2 hardware target 8GB/4CPU). Glossary: HTMX added. Deferred table: 3 entries added.* +*Version 2.0 - Phase 2 delivery annotations: all Phase 2 features marked as implemented (v0.2.0). Component table expanded with vektra-analytics and vektra-learn. Protocol implementations updated: QdrantVectorStoreProvider, UnstructuredExtractor, DualStrategyChunking, AdvancedQueryPipeline, FastEmbedBM25Provider, LogEventEmitter. WebhookEventEmitter deferred to Phase 3.* diff --git a/.s2s/requirements.md b/.s2s/requirements.md index dcf0cb04..83330c5c 100644 --- a/.s2s/requirements.md +++ b/.s2s/requirements.md @@ -3,10 +3,10 @@ # Software Requirements Specification **Project**: Vektra -**Version**: 1.5.0 +**Version**: 1.6.0 **Date**: 2026-02-18 **Sessions**: 20260129-specs-vektra (baseline), 20260201-specs-vektra-integration (merged) -**Corrections**: Manual gap closure v1.2 (REQ-048/049), v1.3 (REQ-050/051, EX-009 to EX-012, path fixes), v1.4 (architectural review: REQ-052 to REQ-065, EX-013/014, OQ-017 to OQ-019), v1.4.1 (consistency review: OQ-015 resolved, OQ-013 partial, REQ-056/057 edge cases, EX-014 ref), v1.5.0 (docs-008: REQ-066 no_relevant_context) +**Corrections**: Manual gap closure v1.2 (REQ-048/049), v1.3 (REQ-050/051, EX-009 to EX-012, path fixes), v1.4 (architectural review: REQ-052 to REQ-065, EX-013/014, OQ-017 to OQ-019), v1.4.1 (consistency review: OQ-015 resolved, OQ-013 partial, REQ-056/057 edge cases, EX-014 ref), v1.5.0 (docs-008: REQ-066 no_relevant_context), v1.6.0 (Phase 2 delivery status annotations) ## 1. Introduction @@ -18,14 +18,20 @@ Modular open-source platform for Retrieval-Augmented Generation (RAG) with speci **In scope (Phase 1)**: - vektra-core: RAG engine, LLM abstraction, conversation management, safeguards -- vektra-ingest: Document processing pipeline - PDF, Word, PPT (OCR deferred to Phase 2) +- vektra-ingest: Document processing pipeline - PDF, Word, PPT - vektra-index: Vector store abstraction, embedding, semantic search - vektra-admin: System administration interface - minimal Phase 1, full Phase 2 +**Delivered in Phase 2 (v0.2.0)**: +- vektra-core: Advanced pipeline, persistent conversations, query rewriting, reranking, safeguard hooks +- vektra-ingest: OCR via Unstructured, dual chunking, document versioning, batch operations +- vektra-index: Hybrid search (dense+sparse), Qdrant vector store provider +- vektra-analytics: QueryTrace storage, metrics aggregation, reporting API +- vektra-learn: E-learning vertical backend (LMS-agnostic), chatbot widget +- vektra-admin: Full dashboard (HTMX + Jinja2), RLS enforcement, rate limiting + **Out of scope**: -- vektra-analytics: Metrics aggregation, reporting API, alerting (Phase 2) -- vektra-learn: E-learning vertical backend, LMS-agnostic (Phase 2) -- vektra-moodle: PHP plugin - separate repo (Phase 2) +- vektra-moodle: PHP plugin - separate repo (Phase 2, not started) - vektra-sdk-py: Python SDK - separate repo (Phase 3) - vektra-sdk-js: JavaScript SDK - separate repo (Phase 3) @@ -161,7 +167,7 @@ Modular open-source platform for Retrieval-Augmented Generation (RAG) with speci ### REQ-016: Phase 1 input constraints - **Priority**: must -- **Description**: Explicit input constraints for Phase 1 APIs: file size: max 50MB (configurable via VEKTRA_MAX_FILE_SIZE_MB), query text: max 4000 characters, supported PDF types: text-based PDFs (scanned/OCR deferred to Phase 2), encrypted PDFs: rejected with ERR-INGEST-001, chunk size: 1000 tokens (configurable via VEKTRA_CHUNK_SIZE), chunk overlap: 200 tokens (configurable via VEKTRA_CHUNK_OVERLAP). Scanned PDF detection: PDF classified as scanned if text extraction yields < 100 characters per page (average of first 5 pages). Detection occurs before chunking (fail fast). Rejected with ERR-INGEST-003. +- **Description**: Explicit input constraints for Phase 1 APIs: file size: max 50MB (configurable via VEKTRA_MAX_FILE_SIZE_MB), query text: max 4000 characters, supported PDF types: text-based PDFs (Phase 1), scanned PDFs via OCR with UnstructuredExtractor (Phase 2), encrypted PDFs: rejected with ERR-INGEST-001, chunk size: 1000 tokens (configurable via VEKTRA_CHUNK_SIZE), chunk overlap: 200 tokens (configurable via VEKTRA_CHUNK_OVERLAP). Scanned PDF detection: PDF classified as scanned if text extraction yields < 100 characters per page (average of first 5 pages). Detection occurs before chunking (fail fast). Rejected with ERR-INGEST-003. - **Acceptance Criteria**: - [ ] Oversized PDF rejected with ERR-INGEST-002 and size limit in message - [ ] Long query rejected with ERR-QUERY-003 and limit in message @@ -235,9 +241,9 @@ Modular open-source platform for Retrieval-Augmented Generation (RAG) with speci - [ ] Key hash uses argon2id algorithm - [ ] Tokens may have multiple scopes (REQ-031) -### REQ-024: Phase 1 scope enforcement +### REQ-024: Scope enforcement - **Priority**: must -- **Description**: Phase 1 scope enforcement uses REQ-031 scope terminology (admin, ingest, query). Phase 1 enforcement behavior: 'admin' scope grants access to all endpoints (superset of all permissions), 'ingest' scope can be stored in key but enforcement deferred to Phase 2, 'query' scope can be stored in key but enforcement deferred to Phase 2, unknown scope rejected with 403. This is a deliberate simplification: Phase 1 has one user type (Platform Operator) who needs full access. Phase 2 will enforce granular scopes. +- **Description**: Scope enforcement uses REQ-031 scope terminology (admin, ingest, query). Phase 1 enforcement behavior: 'admin' scope grants access to all endpoints (superset of all permissions), 'ingest' and 'query' scopes stored but not enforced, unknown scope rejected with 403. Phase 2 (implemented): granular scope enforcement via require_scope() middleware — 'ingest' keys restricted to ingest endpoints, 'query' keys restricted to query endpoints. - **Acceptance Criteria**: - [ ] Enforcement logic uses 'admin', 'ingest', 'query' scopes per REQ-031 - [ ] All keys with 'admin' scope can access all endpoints @@ -805,17 +811,17 @@ Vector store and metadata database encryption delegated to PostgreSQL native enc ## 5. Out of Scope - **EX-001**: CLI deferred to Phase 2 - Dedicated CLI tool (vektra-cli or similar) is explicitly out of scope for Phase 1. CLI becomes a candidate when SDK users (secondary audience) become a priority. -- **EX-002**: OCR and scanned PDF support deferred to Phase 2 - Optical character recognition (OCR) for scanned PDFs is excluded. Text-extractable PDFs only. Detection via REQ-016 threshold (<100 chars/page average). +- **EX-002**: ~~OCR and scanned PDF support deferred to Phase 2~~ [Phase 2: implemented via UnstructuredExtractor with Tesseract OCR] - Optical character recognition (OCR) for scanned PDFs is excluded. Text-extractable PDFs only. Detection via REQ-016 threshold (<100 chars/page average). - **EX-003**: Automatic retry and circuit breakers deferred to Phase 2 - Phase 1 does not implement automatic retry, circuit breakers, or resilience patterns. Error detection and classification are in scope (BR-001, BR-002). Retry responsibility is on the caller. - **EX-004**: CI performance baselines deferred to Phase 2 - Automated CI pipeline performance regression detection is deferred. Phase 1 defines performance targets (REQ-017) but does not enforce them via automated baseline comparison. -- **EX-005**: Batch operations deferred to Phase 2 - Batch ingestion (ingest multiple documents in single request) and batch deletion are excluded. Single-document operations only. -- **EX-006**: Multi-tenant isolation deferred to Phase 2 - Phase 1 operates as single-tenant deployment. Multi-tenant data isolation, per-tenant namespacing, and tenant-scoped API keys are excluded. -- **EX-007**: Analytics and reporting deferred to Phase 2 - vektra-analytics component (query metrics, usage dashboards, alerting) is excluded. Audit logging (REQ-022) provides raw data; aggregation and visualization deferred. +- **EX-005**: ~~Batch operations deferred to Phase 2~~ [Phase 2: implemented - batch delete API] - Batch ingestion (ingest multiple documents in single request) and batch deletion are excluded. Single-document operations only. +- **EX-006**: ~~Multi-tenant isolation deferred to Phase 2~~ [Phase 2: implemented - RLS policies, namespace isolation, per-key scope enforcement] - Phase 1 operates as single-tenant deployment. Multi-tenant data isolation, per-tenant namespacing, and tenant-scoped API keys are excluded. +- **EX-007**: ~~Analytics and reporting deferred to Phase 2~~ [Phase 2: implemented - vektra-analytics component] - vektra-analytics component (query metrics, usage dashboards, alerting) is excluded. Audit logging (REQ-022) provides raw data; aggregation and visualization deferred. - **EX-008**: SDKs deferred to Phase 3 - Python SDK (vektra-sdk-py) and JavaScript SDK (vektra-sdk-js) are excluded from Phase 1 and Phase 2. SDK development targets Phase 3 when external developer adoption becomes priority. -- **EX-009**: Markdown ingestion deferred to Phase 2 - Markdown file extraction is excluded from Phase 1. PDF, Word, and PowerPoint cover primary use cases. Markdown support is low complexity and can be added in Phase 2 if needed. +- **EX-009**: ~~Markdown ingestion deferred to Phase 2~~ [Phase 2: implemented - markdown extractor] - Markdown file extraction is excluded from Phase 1. PDF, Word, and PowerPoint cover primary use cases. Markdown support is low complexity and can be added in Phase 2 if needed. - **EX-010**: Granular ingest APIs deferred to Phase 2 - Separate APIs for extract, clean, and chunk steps are excluded. Phase 1 provides atomic POST /ingest only. Granular APIs may be added in Phase 2 for advanced debugging and custom pipelines. -- **EX-011**: Ingest event emission deferred to Phase 2 - Event emission for monitoring and retry orchestration is excluded. Phase 1 uses polling (GET /ingest/jobs/{id}/status). Webhook/event-based notifications may be added in Phase 2 for n8n integration. -- **EX-012**: Multiple chunking strategies deferred to Phase 2 - Phase 1 supports only fixed-size chunking (REQ-016) behind ChunkingStrategy Protocol (REQ-054). Semantic chunking, dual-strategy chunking (text + table preservation), and parent-child hierarchy are deferred to Phase 2 as implementation swaps. +- **EX-011**: ~~Ingest event emission deferred to Phase 2~~ [Phase 2: implemented - LogEventEmitter] - Event emission for monitoring and retry orchestration is excluded. Phase 1 uses polling (GET /ingest/jobs/{id}/status). Webhook/event-based notifications may be added in Phase 2 for n8n integration. +- **EX-012**: ~~Multiple chunking strategies deferred to Phase 2~~ [Phase 2: implemented - DualStrategyChunking] - Phase 1 supports only fixed-size chunking (REQ-016) behind ChunkingStrategy Protocol (REQ-054). Semantic chunking, dual-strategy chunking (text + table preservation), and parent-child hierarchy are deferred to Phase 2 as implementation swaps. - **EX-013**: LlamaIndex not adopted for Phase 1-2 - RAG pipeline features (hybrid search, reranking, query routing) are implemented directly behind QueryPipeline Protocol (REQ-053). LlamaIndex not adopted due to version instability (v0.14 breaking changes), ~150-200 MB dependency footprint, debugging opacity, and abstraction mismatch with Vektra's Protocol-based design. RAG quality evaluation addressed by standalone frameworks (RAGAS or DeepEval) rather than LlamaIndex built-in evaluators. Reassessment for Phase 3+ if sub-question decomposition or agentic RAG features are needed. - **EX-014**: NeMo Guardrails excluded from Phase 2 baseline - NeMo Guardrails excluded from default SafeguardHook (REQ-044) implementation due to heavy footprint. Presidio for PII detection is accepted for Phase 2. For query and output guardrails, lighter alternatives (keyword filtering + embedding-based classification) should be evaluated before committing to NeMo. @@ -826,9 +832,9 @@ Vector store and metadata database encryption delegated to PostgreSQL native enc - **OQ-014**: Confidence scoring algorithm basis - Algorithm for response confidence scores needs research. Factors: chunk relevance scores, coverage of query terms, source diversity. Deferred to Phase 2 spike. - **OQ-015**: ~~Safeguard hook points specification~~ - **Resolved**: SafeguardHook Protocol fully defined in architecture.md section 8.3 with signatures for pre_query(), post_retrieval(), pre_response(). See REQ-044. - **OQ-016**: File storage encryption scope - Encryption at rest for ingested files depends on vektra-ingest storage architecture. Deferred to design phase. -- **OQ-017**: ORM/database layer selection - SQLAlchemy 2.0 async (with asyncpg) vs SQLModel. Both support repository pattern and Alembic migrations. Decision needed before implementation start, to be documented as ADR. -- **OQ-018**: UI architecture for admin and learn chatbot - Phase 2 requires admin UI and chatbot widget. Options: SPA (React/Vue) vs server-side (HTMX/Jinja2) for admin; standalone JS widget vs npm package for chatbot. Decision deferred to Phase 2 design. -- **OQ-019**: Phase 2 hardware minimum - Phase 2 full-featured (e5-large + cross-encoder + Presidio) estimated at ~2.9GB application + ~512MB PostgreSQL = ~3.4GB. Recommendation: document 8GB RAM / 4 CPU as Phase 2 target. Phase 2 with only hybrid search + reranking (without e5-large) can stay within 4GB. Phase 1 stays at 4GB / 2 CPU per NFR-006. +- ~~**OQ-017**: ORM/database layer selection~~ - **Resolved**: ADR-0022 (SQLAlchemy 2.0 async with asyncpg). +- ~~**OQ-018**: UI architecture for admin and learn chatbot~~ - **Resolved**: ADR-0024 (HTMX + Jinja2 server-side for admin), ADR-0025 (backend-served JS bundle for chatbot widget). +- ~~**OQ-019**: Phase 2 hardware minimum~~ - **Resolved**: ARCH-064 (8GB RAM / 4 CPU formalized as Phase 2 minimum). --- *Generated by Spec2Ship /s2s:specs* @@ -836,4 +842,4 @@ Vector store and metadata database encryption delegated to PostgreSQL native enc *Manual corrections: 2026-02-01 (REQ-048, REQ-049, citation format, path alignment)* *Architectural review: 2026-02-06 (REQ-052 to REQ-065, EX-013/014, OQ-017 to OQ-019, REQ-043/048/050 amended)* *Consistency review: 2026-02-06 (OQ-015 resolved, OQ-013 partial, REQ-056/057 edge cases clarified, EX-014 ref added)* -*Artifacts: 60 functional requirements, 13 NFRs, 5 business rules, 14 exclusions, 7 open questions (1 resolved)* +*Artifacts: 60 functional requirements, 13 NFRs, 5 business rules, 14 exclusions (7 resolved in Phase 2), 7 open questions (4 resolved)* diff --git a/Makefile b/Makefile index ce6de321..78768caf 100644 --- a/Makefile +++ b/Makefile @@ -91,7 +91,7 @@ lint: ## Run linters (ruff + mypy + import-linter) vektra-learn/src/vektra_learn uv run lint-imports -reindex: ## Trigger zero-downtime reindex: make reindex VER=2 [NS=default] +reindex: ## Create reindex job (skeleton): make reindex VER=2 [NS=default] $(if $(VER),,$(error VER is required. Usage: make reindex VER=2 [NS=default])) @scripts/reindex.sh "$(VER)" "$(or $(NS),default)" diff --git a/docker-compose.yml b/docker-compose.yml index d2a8b52a..39c56f4a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -5,8 +5,8 @@ # Usage: # docker compose up -d # core stack # docker compose --profile local-llm up -d # + Ollama for local LLM -# docker compose --profile qdrant up -d # + Qdrant (Phase 2) -# docker compose --profile tei up -d # + TEI embeddings (Phase 2) +# docker compose --profile qdrant up -d # + Qdrant vector store +# docker compose --profile tei up -d # + TEI embeddings server # ========================================================================== services: diff --git a/migrations/versions/0006_add_pipeline_failure_reason.py b/migrations/versions/0006_add_pipeline_failure_reason.py new file mode 100644 index 00000000..db897cdf --- /dev/null +++ b/migrations/versions/0006_add_pipeline_failure_reason.py @@ -0,0 +1,42 @@ +"""Add 'pipeline_failure' to source_documents deletion_reason CHECK. + +Environments where 0001 was already applied have the original CHECK +constraint without 'pipeline_failure'. This migration drops and recreates +the constraint to include the new value. + +Revision ID: 0006 +Revises: 0005 +""" + +from alembic import op + +revision = "0006" +down_revision = "0005" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.execute(""" + ALTER TABLE source_documents + DROP CONSTRAINT IF EXISTS ck_source_documents_deletion_reason + """) + op.execute(""" + ALTER TABLE source_documents + ADD CONSTRAINT ck_source_documents_deletion_reason + CHECK (deletion_reason IS NULL OR deletion_reason IN + ('user_request', 'superseded', 'expired', 'pipeline_failure')) + """) + + +def downgrade() -> None: + op.execute(""" + ALTER TABLE source_documents + DROP CONSTRAINT IF EXISTS ck_source_documents_deletion_reason + """) + op.execute(""" + ALTER TABLE source_documents + ADD CONSTRAINT ck_source_documents_deletion_reason + CHECK (deletion_reason IS NULL OR deletion_reason IN + ('user_request', 'superseded', 'expired')) + """) diff --git a/scripts/batch-ingest.sh b/scripts/batch-ingest.sh index f5c44e95..360e4814 100755 --- a/scripts/batch-ingest.sh +++ b/scripts/batch-ingest.sh @@ -21,7 +21,7 @@ for arg in "$@"; do echo " DIR Directory containing documents (required)" echo " NAMESPACE Target namespace (default: 'default')" echo "" - echo "Supported formats: .pdf, .docx, .pptx" + echo "Supported formats: .pdf, .docx, .pptx, .md" echo "" echo "Environment:" echo " VEKTRA_API_URL Base URL (default: http://localhost:8000)" @@ -61,10 +61,10 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FILES=() while IFS= read -r -d '' file; do FILES+=("$file") -done < <(find "$DIR" -maxdepth 1 -type f \( -name "*.pdf" -o -name "*.docx" -o -name "*.pptx" \) -print0 | sort -z) +done < <(find "$DIR" -maxdepth 1 -type f \( -name "*.pdf" -o -name "*.docx" -o -name "*.pptx" -o -name "*.md" \) -print0 | sort -z) if [ ${#FILES[@]} -eq 0 ]; then - echo "No supported files (.pdf, .docx, .pptx) found in $DIR" + echo "No supported files (.pdf, .docx, .pptx, .md) found in $DIR" exit 0 fi diff --git a/scripts/ingest.sh b/scripts/ingest.sh index 72ac86a8..46b2060d 100755 --- a/scripts/ingest.sh +++ b/scripts/ingest.sh @@ -62,6 +62,7 @@ case "${FILENAME##*.}" in pdf) CONTENT_TYPE="application/pdf" ;; docx) CONTENT_TYPE="application/vnd.openxmlformats-officedocument.wordprocessingml.document" ;; pptx) CONTENT_TYPE="application/vnd.openxmlformats-officedocument.presentationml.presentation" ;; + md) CONTENT_TYPE="text/markdown" ;; *) CONTENT_TYPE="application/octet-stream" ;; esac diff --git a/vektra-admin/src/vektra_admin/keys.py b/vektra-admin/src/vektra_admin/keys.py index cc306d2c..a5e005e0 100644 --- a/vektra-admin/src/vektra_admin/keys.py +++ b/vektra-admin/src/vektra_admin/keys.py @@ -22,7 +22,7 @@ # Single PasswordHasher instance (argon2id, default time_cost=3, memory_cost=65536) _ph = PasswordHasher() -# TTLCache: maps (key_hash, plaintext_key) -> True (verified) or False (failed). +# TTLCache: maps (key_hash, plaintext_key) -> True (verified only). # Max 300s TTL limits plaintext key exposure in memory (DEBT-008). # Cache size 512 is generous for a single-process deployment. _CACHE_SIZE = 512 @@ -73,8 +73,9 @@ async def verify_key(plaintext: str, key_hash: str) -> bool: # Cache miss: run argon2id verification in a thread (CPU-bound) result = await asyncio.to_thread(_verify_sync, key_hash, plaintext) - with _cache_lock: - _verify_cache[cache_key] = result + if result: + with _cache_lock: + _verify_cache[cache_key] = True return result diff --git a/vektra-admin/src/vektra_admin/middleware.py b/vektra-admin/src/vektra_admin/middleware.py index d04f08b2..66cac981 100644 --- a/vektra-admin/src/vektra_admin/middleware.py +++ b/vektra-admin/src/vektra_admin/middleware.py @@ -100,16 +100,24 @@ async def dispatch( "DELETE": "delete", } +_HELPER_TAILS = frozenset({"create", "delete", "revoke", "update"}) + def _derive_action(method: str, path: str) -> str: """Derive a human-readable action from HTTP method and path. - Examples: "create_ingest", "read_health", "delete_api-keys". + Examples: "create_ingest", "read_health", "delete_api-keys", "revoke_api-keys". + When the path ends with a helper tail (/create, /revoke, /delete, /update), + use the tail as the verb instead of the HTTP method to preserve intent. """ verb = _METHOD_VERBS.get(method.upper(), method.lower()) # Use the last meaningful path segment (strip /api/v1/ prefix and IDs) segments = [s for s in path.strip("/").split("/") if s and not _is_uuid_like(s)] - resource = segments[-1] if segments else "unknown" + if len(segments) >= 2 and segments[-1] in _HELPER_TAILS: + verb = segments[-1] + resource = segments[-2] + else: + resource = segments[-1] if segments else "unknown" return f"{verb}_{resource}" diff --git a/vektra-admin/src/vektra_admin/rls.py b/vektra-admin/src/vektra_admin/rls.py index af8d95ce..f0018a46 100644 --- a/vektra-admin/src/vektra_admin/rls.py +++ b/vektra-admin/src/vektra_admin/rls.py @@ -55,25 +55,30 @@ async def dispatch( async def _resolve_namespace(request: Request) -> str | None: - """Extract namespace from request body or query params.""" - # Query param (for GET requests) - namespace = request.query_params.get("namespace") - if namespace: - return namespace + """Extract namespace from request body or query params. - # JSON body (for POST requests) + For write methods (POST/PUT/PATCH) the body takes precedence over + query params to prevent a crafted ``?namespace=`` from overriding + the payload namespace in multi-tenant mode. + """ + # JSON body first for write methods (prevents query-param override) if request.method in ("POST", "PUT", "PATCH"): try: body = await request.body() if body: data = json.loads(body) - ns = data.get("namespace") - if isinstance(ns, str): - return ns - return None + if isinstance(data, dict): + ns = data.get("namespace") + if isinstance(ns, str): + return ns except (json.JSONDecodeError, UnicodeDecodeError): pass + # Query param (safe for GET; fallback for POST without body namespace) + namespace = request.query_params.get("namespace") + if namespace: + return namespace + return None diff --git a/vektra-admin/src/vektra_admin/ui.py b/vektra-admin/src/vektra_admin/ui.py index b7f143b6..18564444 100644 --- a/vektra-admin/src/vektra_admin/ui.py +++ b/vektra-admin/src/vektra_admin/ui.py @@ -183,8 +183,11 @@ async def login_submit(request: Request) -> Response: @ui_router.get("/logout") -async def logout(request: Request) -> Response: - """Clear cookie and redirect to login.""" +async def logout( + request: Request, + _auth: ApiKeyInfo = Depends(_require_admin_ui), +) -> Response: + """Clear cookie and redirect to login (authenticated for audit trail).""" response = RedirectResponse(url="/admin/login", status_code=303) response.delete_cookie(_COOKIE_NAME) return response @@ -436,7 +439,25 @@ async def namespaces_create( ns = NamespaceOrm(id=name, display_name=display_name) session.add(ns) - await session.commit() + try: + await session.commit() + except IntegrityError as exc: + await session.rollback() + pgcode = getattr(getattr(exc, "orig", None), "pgcode", None) + if pgcode != "23505": + raise + namespaces = await _load_namespaces(session) + return templates.TemplateResponse( + request=request, + name="partials/namespaces_table.html", + context={ + "namespaces": namespaces, + "flash": { + "type": "error", + "message": f"Namespace '{name}' already exists.", + }, + }, + ) namespaces = await _load_namespaces(session) return templates.TemplateResponse( diff --git a/vektra-admin/tests/test_rls.py b/vektra-admin/tests/test_rls.py index c67b83f8..406a1f80 100644 --- a/vektra-admin/tests/test_rls.py +++ b/vektra-admin/tests/test_rls.py @@ -50,8 +50,8 @@ async def test_from_json_body(self): assert result == "ns-456" @pytest.mark.asyncio - async def test_query_param_takes_precedence(self): - """Query param is checked before body.""" + async def test_body_takes_precedence_over_query_param(self): + """For write methods, body namespace takes precedence over query param.""" import json body = json.dumps({"namespace": "body-ns"}).encode() @@ -61,8 +61,36 @@ async def test_query_param_takes_precedence(self): body=body, ) result = await _resolve_namespace(request) + assert result == "body-ns" + + @pytest.mark.asyncio + async def test_get_uses_query_param_over_body(self): + """For GET requests, query param is used (body not read).""" + import json + + body = json.dumps({"namespace": "body-ns"}).encode() + request = _make_request( + method="GET", + query_params={"namespace": "query-ns"}, + body=body, + ) + result = await _resolve_namespace(request) assert result == "query-ns" + @pytest.mark.asyncio + async def test_post_without_body_falls_back_to_query_param(self): + """POST without body namespace falls back to query param.""" + import json + + body = json.dumps({"other_field": "value"}).encode() + request = _make_request( + method="POST", + query_params={"namespace": "fallback-ns"}, + body=body, + ) + result = await _resolve_namespace(request) + assert result == "fallback-ns" + @pytest.mark.asyncio async def test_no_namespace_returns_none(self): """No namespace in request returns None.""" diff --git a/vektra-admin/tests/test_ttlcache.py b/vektra-admin/tests/test_ttlcache.py index c9392cdd..825e6f28 100644 --- a/vektra-admin/tests/test_ttlcache.py +++ b/vektra-admin/tests/test_ttlcache.py @@ -43,15 +43,14 @@ async def test_cache_populated_after_verify(self): assert _verify_cache[(key_hash, plaintext)] is True @pytest.mark.asyncio - async def test_cache_populated_for_failed_verify(self): - """Failed verifications should also be cached (False value).""" + async def test_failed_verify_not_cached(self): + """Failed verifications should NOT be cached (prevents cache poisoning).""" _, key_hash, _ = generate_key() wrong_token = "wrong_token_abc" await verify_key(wrong_token, key_hash) with _cache_lock: - assert (key_hash, wrong_token) in _verify_cache - assert _verify_cache[(key_hash, wrong_token)] is False + assert (key_hash, wrong_token) not in _verify_cache @pytest.mark.asyncio async def test_cache_hit_returns_same_result(self): diff --git a/vektra-core/src/vektra_core/advanced_pipeline.py b/vektra-core/src/vektra_core/advanced_pipeline.py index b2c71ca6..463e0f28 100644 --- a/vektra-core/src/vektra_core/advanced_pipeline.py +++ b/vektra-core/src/vektra_core/advanced_pipeline.py @@ -352,6 +352,10 @@ async def _run_pre_llm_steps( ) ) + # Recheck: post_retrieval safeguard may have filtered all chunks + if not no_relevant_context and not filtered: + no_relevant_context = True + return steps, filtered, no_relevant_context, effective_query, history def _build_prompt( @@ -520,11 +524,15 @@ async def execute( ) ) - # Save conversation turn + # Save conversation turn (best-effort: DB failure should not turn + # a successful query into a 500) if query.conversation_id is not None: - await self._conversation_store.add_turn( - query.conversation_id, query.question, answer - ) + try: + await self._conversation_store.add_turn( + query.conversation_id, query.question, answer + ) + except Exception as exc: + log.warning("conversation_turn_store_failed", error=str(exc)) total_ms = _elapsed_ms(t_total) trace = QueryTrace( @@ -682,11 +690,14 @@ async def _stream( ) ) - # Save conversation turn + # Save conversation turn (best-effort) if query.conversation_id is not None and full_answer: - await self._conversation_store.add_turn( - query.conversation_id, query.question, full_answer - ) + try: + await self._conversation_store.add_turn( + query.conversation_id, query.question, full_answer + ) + except Exception as exc: + log.warning("conversation_turn_store_failed", error=str(exc)) # Yield sources (only budget-selected chunks, not all filtered) sources_data = [ diff --git a/vektra-core/src/vektra_core/pipeline.py b/vektra-core/src/vektra_core/pipeline.py index 07cb45a1..b9ecd92b 100644 --- a/vektra-core/src/vektra_core/pipeline.py +++ b/vektra-core/src/vektra_core/pipeline.py @@ -374,6 +374,10 @@ async def execute( ) ) + # Recheck: post_retrieval safeguard may have filtered all chunks + if not no_relevant_context and not filtered: + no_relevant_context = True + # No relevant context or safeguard blocked → skip LLM, return early if no_relevant_context or safeguard_blocked: trace = QueryTrace( @@ -673,8 +677,12 @@ async def _stream(self, query: QueryRequest) -> AsyncGenerator[QueryChunk, None] ) ) + # Recheck: post_retrieval safeguard may have filtered all chunks + if not no_relevant_context and not filtered: + no_relevant_context = True + # Safeguard blocked all results → early return - if safeguard_blocked: + if safeguard_blocked or no_relevant_context: yield QueryChunk(type="sources", data=[]) trace = QueryTrace( response_id=response_id, diff --git a/vektra-core/src/vektra_core/safeguards/__init__.py b/vektra-core/src/vektra_core/safeguards/__init__.py index 57cffe10..8ea802a2 100644 --- a/vektra-core/src/vektra_core/safeguards/__init__.py +++ b/vektra-core/src/vektra_core/safeguards/__init__.py @@ -1,7 +1,7 @@ """Safeguard factory for selecting SafeguardHook implementations. -Phase 1: PassthroughSafeguard (no-op, from vektra_shared). -Phase 2: PresidioPIISafeguard (Presidio-based PII anonymization). +PassthroughSafeguard (no-op, from vektra_shared). +PresidioPIISafeguard (Presidio-based PII anonymization). """ from __future__ import annotations diff --git a/vektra-index/src/vektra_index/providers/pgvector.py b/vektra-index/src/vektra_index/providers/pgvector.py index 2132f44f..62f4cb03 100644 --- a/vektra-index/src/vektra_index/providers/pgvector.py +++ b/vektra-index/src/vektra_index/providers/pgvector.py @@ -220,7 +220,7 @@ async def _search_sparse( ), 0.0) """) - score_col = sparse_score_sql.columns(score=Float).label("score") + score_col = sparse_score_sql.columns(score=Float).c.score.label("score") stmt = ( select( diff --git a/vektra-index/src/vektra_index/providers/qdrant.py b/vektra-index/src/vektra_index/providers/qdrant.py index d0912d80..bb27d8ae 100644 --- a/vektra-index/src/vektra_index/providers/qdrant.py +++ b/vektra-index/src/vektra_index/providers/qdrant.py @@ -35,6 +35,28 @@ _RRF_K = 60 +def _is_timeout(exc: BaseException) -> bool: + """Detect timeout-like exceptions from qdrant-client (HTTP or gRPC). + + qdrant-client wraps transport exceptions: + - HTTP: ResponseHandlingException with httpx.TimeoutException as .source + - gRPC: grpc.aio.AioRpcError with DEADLINE_EXCEEDED status + Python's built-in TimeoutError is also caught as a safety net. + """ + if isinstance(exc, TimeoutError): + return True + # HTTP transport: ResponseHandlingException wrapping httpx timeout + exc_type = type(exc).__name__ + if exc_type == "ResponseHandlingException": + source = getattr(exc, "source", None) + return source is not None and "timeout" in type(source).__name__.lower() + # gRPC transport: AioRpcError with DEADLINE_EXCEEDED + if exc_type == "AioRpcError": + code = getattr(exc, "code", lambda: None)() + return code is not None and "DEADLINE_EXCEEDED" in str(code) + return False + + def _import_qdrant() -> Any: """Import qdrant_client with a clear error message if missing.""" try: @@ -93,20 +115,31 @@ async def ensure_collection(self) -> None: if self._collection_name in existing: return - await self._client.create_collection( - collection_name=self._collection_name, - vectors_config={ - "dense": models.VectorParams( - size=self._dense_dimensions, - distance=models.Distance.COSINE, - ), - }, - sparse_vectors_config={ - "sparse": models.SparseVectorParams( - modifier=models.Modifier.IDF, - ), - }, - ) + try: + await self._client.create_collection( + collection_name=self._collection_name, + vectors_config={ + "dense": models.VectorParams( + size=self._dense_dimensions, + distance=models.Distance.COSINE, + ), + }, + sparse_vectors_config={ + "sparse": models.SparseVectorParams( + modifier=models.Modifier.IDF, + ), + }, + ) + except Exception: + # Race: another replica may have created it between our check and create. + collections = await self._client.get_collections() + if self._collection_name not in {c.name for c in collections.collections}: + raise + logger.debug( + "Qdrant collection %s already exists (race resolved)", + self._collection_name, + ) + return logger.info("Created Qdrant collection: %s", self._collection_name) async def store( @@ -161,8 +194,20 @@ async def store( points=points, wait=True, ) - except Exception: - # Compensating delete on partial failure (ARCH-052) + except Exception as exc: + if _is_timeout(exc): + # On timeout, Qdrant may have already persisted the points. + # Compensating delete would cause data loss. Log and re-raise + # so the caller can retry (idempotent via deterministic IDs). + logger.warning( + "qdrant_store_timeout, skipping compensating delete for %d points " + "(upsert may have succeeded server-side)", + len(point_ids), + ) + raise + + # Non-timeout error: Qdrant explicitly rejected the batch. + # Safe to run compensating delete (ARCH-052). logger.warning( "qdrant_store_failed, executing compensating delete for %d points", len(point_ids),