diff --git a/REFERENCES.md b/REFERENCES.md index 0b31741..d8c4754 100644 --- a/REFERENCES.md +++ b/REFERENCES.md @@ -94,7 +94,7 @@ This recursive approach demonstrates the power of RAG (Retrieval-Augmented Gener - **Structured Error Handling** - Consistent error propagation and handling **Files using these concepts:** -- `src/domain/exceptions.ts` - Exception hierarchy implementation +- `src/domain/exceptions/` - Modular exception hierarchy (base `ConceptRAGError` with domain-specific subtypes) ## Testing Patterns diff --git a/USAGE.md b/USAGE.md index 8d52efb..e361dca 100644 --- a/USAGE.md +++ b/USAGE.md @@ -533,7 +533,7 @@ Default: `~/.concept_rag` Custom: ```bash -export CONCEPT_RAG_DB="/custom/path" +export DATABASE_URL="/custom/path" ``` ### MCP Server Configuration diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 212368e..896c031 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -14,7 +14,7 @@ This section covers the technical architecture of Concept-RAG. | `src/tools/` | MCP tool implementations (10 tools) | | `src/wordnet/` | WordNet integration and strategies | | `docs/` | MkDocs documentation site | -| `docs/architecture/` | Architecture Decision Records | +| `docs/architecture/` | Architecture deep-dives and decision records (ADRs) | | `scripts/` | Maintenance and diagnostic utilities | | `prompts/` | LLM prompt templates | @@ -26,4 +26,6 @@ This section covers the technical architecture of Concept-RAG. - **[Database Schema](../database-schema.md)** - LanceDB table structures - **[Stage Cache](../stage-cache-structure.md)** - Intermediate processing cache -For architectural decisions and their rationale, see the **[ADRs](../architecture/adr0001-typescript-nodejs-runtime.md)** section. +Architectural decisions and their rationale are recorded as Architecture Decision +Records (ADRs) in the repository under `docs/architecture/`. ADRs are point-in-time +engineering artifacts and are not published to this site. diff --git a/docs/architecture/bm25-keywords.md b/docs/architecture/bm25-keywords.md index 7fc1a53..05367ef 100644 --- a/docs/architecture/bm25-keywords.md +++ b/docs/architecture/bm25-keywords.md @@ -114,15 +114,16 @@ flowchart TB ## Integration in Hybrid Search -BM25 is one of four signals in the hybrid ranking system: +BM25 is one of five signals in the hybrid ranking system: ```mermaid flowchart TB subgraph Signals["Search Signals"] - Vector["Vector Similarity
35%"] - BM25["BM25 Keywords
30-35%"] - Concept["Concept Matching
15-20%"] - WN["WordNet Expansion
15%"] + Vector["Vector Similarity
30-35%"] + BM25["BM25 Keywords
20-30%"] + Title["Title / Name Match
0-40%"] + Concept["Concept Matching
0-20%"] + WN["WordNet Expansion
10-15%"] end subgraph Combined["Hybrid Score"] @@ -132,6 +133,7 @@ flowchart TB Vector --> Combine BM25 --> Combine + Title --> Combine Concept --> Combine WN --> Combine Combine --> Results @@ -249,7 +251,8 @@ The hybrid approach preserves these benefits while adding keyword precision. ## Related Documentation -- [ADR-0006: Hybrid Search Strategy](adr0006-hybrid-search-strategy.md) — Design decision - [WordNet Enrichment](wordnet-enrichment.md) — Synonym expansion -- [ADR-0010: Query Expansion](adr0010-query-expansion.md) — Term weighting + +The hybrid-search and query-expansion design decisions are recorded in ADRs +(ADR-0006, ADR-0010), kept in the repository under `docs/architecture/`. diff --git a/docs/architecture/seeding-architecture.md b/docs/architecture/seeding-architecture.md index 67acc8b..adf68a2 100644 --- a/docs/architecture/seeding-architecture.md +++ b/docs/architecture/seeding-architecture.md @@ -135,6 +135,7 @@ flowchart TB - [Getting Started](../getting-started.md) — Quick start guide with seeding commands - [Stage Cache](../stage-cache-structure.md) — Intermediate caching during seeding -- [ADR-0013: Incremental Seeding](adr0013-incremental-seeding.md) — Design decision for incremental processing -- [ADR-0044: Seeding Modularization](adr0044-seeding-script-modularization.md) — Script architecture + +The incremental-seeding and seeding-modularization design decisions are recorded +in ADRs (ADR-0013, ADR-0044), kept in the repository under `docs/architecture/`. diff --git a/docs/architecture/wordnet-enrichment.md b/docs/architecture/wordnet-enrichment.md index b38e2e6..bcf3d51 100644 --- a/docs/architecture/wordnet-enrichment.md +++ b/docs/architecture/wordnet-enrichment.md @@ -83,15 +83,16 @@ Expanded terms receive lower weights than original terms to maintain search prec ### Integration in Hybrid Search -WordNet contributes to the 4-signal hybrid search ranking: +WordNet contributes to the 5-signal hybrid search ranking: ```mermaid flowchart TB subgraph Signals["Search Signals"] - Vector["Vector Similarity
35%"] - BM25["BM25 Keywords
35%"] - Concept["Concept Matching
15%"] - WN["WordNet Expansion
15%"] + Vector["Vector Similarity
30-35%"] + BM25["BM25 Keywords
20-30%"] + Title["Title / Name Match
0-40%"] + Concept["Concept Matching
0-20%"] + WN["WordNet Expansion
10-15%"] end subgraph Score["Final Ranking"] @@ -101,6 +102,7 @@ flowchart TB Vector --> Combine BM25 --> Combine + Title --> Combine Concept --> Combine WN --> Combine Combine --> Results @@ -223,7 +225,7 @@ python3 -c "from nltk.corpus import wordnet as wn; print(f'✅ WordNet: {len(lis ## Related Documentation -- [ADR-0008: WordNet Integration](adr0008-wordnet-integration.md) — Design decision -- [ADR-0010: Query Expansion](adr0010-query-expansion.md) — Query expansion strategy -- [ADR-0006: Hybrid Search](adr0006-hybrid-search-strategy.md) — Multi-signal ranking +The WordNet-integration, query-expansion, and hybrid-search design decisions are +recorded in ADRs (ADR-0008, ADR-0010, ADR-0006), kept in the repository under +`docs/architecture/`. diff --git a/docs/database-schema.md b/docs/database-schema.md index f9b5b00..2f0bca4 100644 --- a/docs/database-schema.md +++ b/docs/database-schema.md @@ -577,12 +577,11 @@ These performance caches remain for CPU/DB optimization: ## Related Documentation -- [ADR-0002: LanceDB Vector Storage](architecture/adr0002-lancedb-vector-storage.md) -- [ADR-0009: Three-Table Architecture](architecture/adr0009-three-table-architecture.md) -- [ADR-0027: Hash-Based Integer IDs](architecture/adr0027-hash-based-integer-ids.md) -- [ADR-0043: Schema Normalization](architecture/adr0043-schema-normalization.md) - Domain Models: `src/domain/models/` - Schema Validators: `src/infrastructure/lancedb/utils/schema-validators.ts` + +Schema design decisions are recorded in ADRs (ADR-0002, ADR-0009, ADR-0027, +ADR-0043), kept in the repository under `docs/architecture/`. - Lexical Linking: `scripts/link_related_concepts.ts` - Derived Fields Regeneration: `scripts/rebuild_derived_names.ts` - Seeding Script: `hybrid_fast_seed.ts` (supports `--parallel N` for parallel concept extraction) diff --git a/docs/development.md b/docs/development.md index c28efec..de196e0 100644 --- a/docs/development.md +++ b/docs/development.md @@ -8,7 +8,7 @@ This fork extends the original [lance-mcp](https://github.com/adiom-data/lance-m - 📚 **Formal concept model** - Rigorous definition ensuring semantic matching and disambiguation - 🧠 **Enhanced concept extraction** - 80-150+ concepts per document (Claude Sonnet 4.5) - 🌐 **WordNet semantic enrichment** - Synonym expansion and hierarchical navigation -- 🔍 **Multi-signal hybrid ranking** - Vector + BM25 + title + concept + WordNet (4-signal scoring) +- 🔍 **Multi-signal hybrid ranking** - Vector + BM25 + title + concept + WordNet (5-signal scoring) - 📖 **Large document support** - Multi-pass extraction for >100k token documents - ⚡ **Parallel concept extraction** - Process up to 25 documents concurrently with shared rate limiting - 🔁 **Resumable seeding** - Checkpoint-based recovery from interrupted runs @@ -31,7 +31,7 @@ src/ ├── infrastructure/ # External integrations │ ├── lancedb/ # Database adapters (normalized schema v7) │ ├── embeddings/ # Embedding service -│ ├── search/ # Hybrid search with 4-signal scoring +│ ├── search/ # Hybrid search with 5-signal scoring │ ├── resilience/ # Circuit breaker, bulkhead, timeout patterns │ ├── checkpoint/ # Resumable seeding with progress tracking │ ├── cli/ # Progress bar display utilities @@ -67,7 +67,7 @@ Catalog Chunks Concepts Categories └─────────┴─────────┴─────────┘ ↓ Hybrid Search Engine - (Vector + BM25 + Concepts + WordNet) + (Vector + BM25 + Title + Concepts + WordNet) ``` ### Four-Table Normalized Schema diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 32b29a7..35e3d94 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -28,6 +28,7 @@ flowchart TB subgraph Search["🔍 Hybrid Search"] Vector[Vector Similarity] BM25[BM25 Keywords] + Title[Title / Name Match] ConceptMatch[Concept Matching] WordNet[WordNet Expansion] Rank[Weighted Ranking] @@ -52,11 +53,13 @@ flowchart TB Extract --> Categories Catalog --> Vector + Catalog --> Title Chunks --> Vector Concepts --> ConceptMatch Vector --> Rank BM25 --> Rank + Title --> Rank ConceptMatch --> Rank WordNet --> Rank @@ -98,16 +101,20 @@ Vector embeddings are generated for: ### 5. Hybrid Search -Queries are scored using four signals: +Queries are scored by combining five signals: -| Signal | Weight | Purpose | -|--------|--------|---------| -| Vector Similarity | 35% | Semantic meaning match | -| BM25 Keywords | 35% | Exact term matching | -| Concept Matching | 15% | Extracted concept overlap | -| WordNet Expansion | 15% | Synonym and hypernym matching | +| Signal | Purpose | +|--------|---------| +| Vector Similarity | Semantic meaning match | +| BM25 Keywords | Exact term matching | +| Title / Name Match | Document title or concept-name relevance | +| Concept Matching | Extracted concept overlap | +| WordNet Expansion | Synonym and hypernym matching | -Results are combined using weighted ranking for optimal retrieval accuracy. +Signal weights are tuned per search type — for example, document (catalog) search +weights them 30% / 25% / 20% / 15% / 10%, while passage and concept searches use +different profiles. See the [API Reference](api-reference.md#scoring-weights) for the +full per-tool weighting table. ### 6. Gap Detection (Elbow Method) diff --git a/docs/stage-cache-structure.md b/docs/stage-cache-structure.md index 59d7465..9fc5f37 100644 --- a/docs/stage-cache-structure.md +++ b/docs/stage-cache-structure.md @@ -262,6 +262,8 @@ This ensures cache files are never partially written, even if the process is kil ## Related Documentation -- [ADR-0048: Stage Caching](architecture/adr0048-stage-caching.md) - Architecture decision record - [Database Schema](database-schema.md) - LanceDB table structures +The stage-caching design decision is recorded in ADR-0048, kept in the repository +under `docs/architecture/`. + diff --git a/mkdocs.yml b/mkdocs.yml index 2409a39..42d768f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -105,67 +105,20 @@ nav: - Seeding Architecture: architecture/seeding-architecture.md - BM25 Keywords: architecture/bm25-keywords.md - WordNet Enrichment: architecture/wordnet-enrichment.md + - Activity & Skill Interface: architecture/skills-interface.md - Database Schema: database-schema.md - Stage Cache: stage-cache-structure.md - - ADRs: - - "ADR-0001: TypeScript Runtime": architecture/adr0001-typescript-nodejs-runtime.md - - "ADR-0002: LanceDB Storage": architecture/adr0002-lancedb-vector-storage.md - - "ADR-0003: MCP Protocol": architecture/adr0003-mcp-protocol.md - - "ADR-0004: RAG Architecture": architecture/adr0004-rag-architecture.md - - "ADR-0005: PDF Processing": architecture/adr0005-pdf-document-processing.md - - "ADR-0006: Hybrid Search": architecture/adr0006-hybrid-search-strategy.md - - "ADR-0007: Concept Extraction": architecture/adr0007-concept-extraction-llm.md - - "ADR-0008: WordNet Integration": architecture/adr0008-wordnet-integration.md - - "ADR-0009: Three-Table Architecture": architecture/adr0009-three-table-architecture.md - - "ADR-0010: Query Expansion": architecture/adr0010-query-expansion.md - - "ADR-0011: Multi-Model Strategy": architecture/adr0011-multi-model-strategy.md - - "ADR-0012: OCR Fallback": architecture/adr0012-ocr-fallback-tesseract.md - - "ADR-0013: Incremental Seeding": architecture/adr0013-incremental-seeding.md - - "ADR-0014: Multi-Pass Extraction": architecture/adr0014-multi-pass-extraction.md - - "ADR-0015: Formal Concept Model": architecture/adr0015-formal-concept-model.md - - "ADR-0016: Layered Architecture": architecture/adr0016-layered-architecture-refactoring.md - - "ADR-0017: Repository Pattern": architecture/adr0017-repository-pattern.md - - "ADR-0018: Dependency Injection": architecture/adr0018-dependency-injection-container.md - - "ADR-0019: Vitest Framework": architecture/adr0019-vitest-testing-framework.md - - "ADR-0020: TypeScript Strict Mode": architecture/adr0020-typescript-strict-mode.md - - "ADR-0021: Performance Optimization": architecture/adr0021-performance-optimization-vector-search.md - - "ADR-0022: HybridSearchService": architecture/adr0022-hybrid-search-service-extraction.md - - "ADR-0023: SQL Injection Prevention": architecture/adr0023-sql-injection-prevention.md - - "ADR-0024: Multi-Provider Embeddings": architecture/adr0024-multi-provider-embeddings.md - - "ADR-0025: Document Loader Factory": architecture/adr0025-document-loader-factory.md - - "ADR-0026: EPUB Support": architecture/adr0026-epub-format-support.md - - "ADR-0027: Hash-Based IDs": architecture/adr0027-hash-based-integer-ids.md - - "ADR-0028: Category Storage": architecture/adr0028-category-storage-strategy.md - - "ADR-0029: Category Search Tools": architecture/adr0029-category-search-tools.md - - "ADR-0030: Auto-Extracted Categories": architecture/adr0030-auto-extracted-categories.md - - "ADR-0031: Eight Tools Strategy": architecture/adr0031-eight-specialized-tools-strategy.md - - "ADR-0032: Tool Selection Guide": architecture/adr0032-tool-selection-guide.md - - "ADR-0033: BaseTool Abstraction": architecture/adr0033-basetool-abstraction.md - - "ADR-0034: Error Handling": architecture/adr0034-comprehensive-error-handling.md - - "ADR-0035: Test Suite Expansion": architecture/adr0035-test-suite-expansion.md - - "ADR-0036: Configuration Centralization": architecture/adr0036-configuration-centralization.md - - "ADR-0037: Functional Validation": architecture/adr0037-functional-validation-layer.md - - "ADR-0038: Dependency Rules": architecture/adr0038-dependency-rules-enforcement.md - - "ADR-0039: Observability": architecture/adr0039-observability-infrastructure.md - - "ADR-0040: Result/Option Types": architecture/adr0040-result-option-types.md - - "ADR-0041: Advanced Caching": architecture/adr0041-advanced-caching.md - - "ADR-0042: System Resilience": architecture/adr0042-system-resilience-patterns.md - - "ADR-0043: Schema Normalization": architecture/adr0043-schema-normalization.md - - "ADR-0044: Seeding Modularization": architecture/adr0044-seeding-script-modularization.md - - "ADR-0045: API Key Preflight": architecture/adr0045-api-key-preflight-check.md - - "ADR-0046: Document Classification": architecture/adr0046-document-type-classification.md - - "ADR-0047: Architecture Review": architecture/adr0047-architecture-review-dec-2025.md - - "ADR-0048: Stage Caching": architecture/adr0048-stage-caching.md - - "ADR-0049: Incremental Summaries": architecture/adr0049-incremental-category-summaries.md - - "ADR-0050: MkDocs Site": architecture/adr0050-mkdocs-material-documentation-site.md - - "ADR-0051: API Documentation": architecture/adr0051-api-documentation-consolidation.md - - "ADR-0052: Docs Restructure": architecture/adr0052-documentation-site-restructure.md - - "ADR-0053: Meta Content Detection": architecture/adr0053-meta-content-detection.md - - "ADR-0054: Debug Flag Deprecation": architecture/adr0054-deprecate-debug-input-schema.md - - FAQ: faq.md +# Engineering artifacts kept in the repo but excluded from the published site. +# ADRs are point-in-time decision records, not user documentation. development.md +# is the README-linked developer guide and uses repo-relative links that only +# resolve when browsed on GitHub, not on the rendered site. +exclude_docs: | + architecture/adr*.md + development.md + # Plugins plugins: - search