feat: Auto-heal for silent semantic search failure

[PatrickSys]

Description

This PR implements an auto-heal mechanism to resolve the 'Silent Semantic Search Failure' issue.

Changes

• Adds automatic detection and recovery when the vector index becomes corrupted
• Implements new \IndexCorruptedError\ custom error type
• Triggers automatic re-indexing when critical index failures occur
• Falls back to keyword search for non-critical errors
• Updates version to v1.3.1

Testing

• Verified in real-world scenarios
• New test scripts validate the auto-heal behavior

[greptile-apps]

Greptile Summary

Implements automatic corruption detection and healing for LanceDB semantic search failures. When the vector database schema is corrupted (missing vector column), the system now throws IndexCorruptedError which triggers synchronous re-indexing and automatic retry instead of silently falling back to keyword-only search.

Key Changes

• New Error Type: IndexCorruptedError signals schema corruption requiring re-indexing
• Detection Points: Corruption detected during initialization, schema validation, and search operations in lancedb.ts
• Propagation: Errors bubble through search.ts to index.ts handler without silent catches
• Auto-Heal: MCP server catches IndexCorruptedError, triggers performIndexing(), and retries search with fresh CodebaseSearcher instance
• Test Coverage: Three new test suites validate corruption detection, error propagation, and E2E auto-heal flow
• Embedding Model: Changed from bge-base-en-v1.5 (768d) to bge-small-en-v1.5 (384d) - requires re-indexing for existing users

Potential Issues

• Broad error catching in lancedb.ts:192-193 may trigger false-positive auto-heals for transient errors (see inline comment)
• Embedding dimension change is breaking but not highlighted in changelog as requiring manual re-index

Confidence Score: 4/5

• Safe to merge with one notable issue around overly broad error catching
• Score reflects solid implementation of auto-heal mechanism with comprehensive test coverage, but the broad error catching pattern in lancedb.ts could cause unnecessary re-indexing for non-corruption errors. The embedding model change is also breaking for existing users but handled by auto-heal.
• Pay close attention to src/storage/lancedb.ts - the catch-all error handling at line 192 may need refinement to avoid false positives

Important Files Changed

Filename	Overview
src/errors/index.ts	New custom error class for LanceDB corruption detection - clean implementation
src/storage/lancedb.ts	Implements corruption detection with IndexCorruptedError throws - overly broad error catching may trigger false positives
src/core/search.ts	Propagates IndexCorruptedError through initialization and semantic search - clean pass-through logic
src/index.ts	Implements auto-heal mechanism with synchronous re-indexing and retry - comprehensive error handling
src/core/indexer.ts	Changed embedding model from bge-base (768d) to bge-small (384d) and formatting fix - requires re-indexing

Sequence Diagram

sequenceDiagram
    participant User
    participant MCP as MCP Server
    participant Searcher as CodebaseSearcher
    participant Storage as LanceDBStorage
    participant Indexer as CodebaseIndexer

    User->>MCP: search_codebase(query)
    MCP->>Searcher: search(query, limit, filters)
    
    Searcher->>Searcher: initialize()
    Searcher->>Storage: initialize(storagePath)
    
    alt Table exists but schema invalid
        Storage->>Storage: Check schema for vector column
        Storage-->>Storage: Missing vector column detected
        Storage->>Storage: dropTable('code_chunks')
        Storage-->>Searcher: throw IndexCorruptedError
        Searcher-->>MCP: throw IndexCorruptedError
    else Table missing
        Storage->>Searcher: search() called with no table
        Storage-->>Searcher: throw IndexCorruptedError
        Searcher-->>MCP: throw IndexCorruptedError
    end
    
    MCP->>MCP: Catch IndexCorruptedError
    MCP->>MCP: Log "[Auto-Heal] Index corrupted..."
    MCP->>Indexer: performIndexing()
    Indexer->>Indexer: index() - full re-index
    Indexer->>Storage: store(chunks with embeddings)
    Storage->>Storage: createTable('code_chunks')
    Indexer-->>MCP: indexing complete
    
    alt Re-indexing successful
        MCP->>MCP: Check indexState.status === 'ready'
        MCP->>Searcher: new CodebaseSearcher(ROOT_PATH)
        MCP->>Searcher: search(query, limit, filters)
        Searcher->>Storage: search(queryVector, limit)
        Storage-->>Searcher: results
        Searcher-->>MCP: SearchResult[]
        MCP-->>User: Success response with results
    else Re-indexing failed
        MCP-->>User: Error response (auto-heal failed)
    end

Loading

[greptile-apps]

_{11 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: Overly broad error catching - any error matching "LanceDB" or "Arrow" triggers auto-heal

This catch-all may cause unnecessary re-indexing for transient network issues, OOM errors, or other non-corruption problems. Consider narrowing to specific error types or adding additional validation before throwing IndexCorruptedError.

Suggested change

	if (error instanceof Error && (error.message.includes('LanceDB') || error.message.includes('Arrow'))) {
	throw new IndexCorruptedError(`LanceDB runtime error: ${error.message}`);
	if (error instanceof Error && error.message.includes('No vector column')) {
	throw new IndexCorruptedError('LanceDB index corrupted: missing vector column');
	}

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/storage/lancedb.ts
Line: 192:193

Comment:
**logic:** Overly broad error catching - any error matching "LanceDB" or "Arrow" triggers auto-heal

This catch-all may cause unnecessary re-indexing for transient network issues, OOM errors, or other non-corruption problems. Consider narrowing to specific error types or adding additional validation before throwing `IndexCorruptedError`.

```suggestion
      if (error instanceof Error && error.message.includes('No vector column')) {
        throw new IndexCorruptedError('LanceDB index corrupted: missing vector column');
      }
```

How can I resolve this? If you propose a fix, please make it concise.

[PatrickSys]

Closing in favor of a new PR with proper branch naming. See the new PR for this feature.