Skip to content

Fix FTS search degradation diagnostics - #2773

Merged
Widthdom merged 4 commits into
mainfrom
fix-issue1458-1606-1776
May 31, 2026
Merged

Fix FTS search degradation diagnostics#2773
Widthdom merged 4 commits into
mainfrom
fix-issue1458-1606-1776

Conversation

@Widthdom

Copy link
Copy Markdown
Owner

Summary

  • Add zero-result JSON/MCP diagnostics when every literal FTS token exceeds SQLite FTS5 unicode61's 1000-character token cap.
  • Report indexed files containing overlong unbroken FTS tokens as fts_token_too_long validation issues.
  • Normalize literal FTS search input to Unicode NFC and keep Latin-diacritic explicit-prefix search covered by regression tests.

Note: #1458's stop-word premise does not reproduce with this repository's fts5(content, content='chunks', content_rowid='id') schema because default SQLite FTS5 has no stop-word filter. The PR keeps diagnostics to actual token filtering/degradation cases.

Validation

  • dotnet build CodeIndex.sln -c Release -p:UseSharedCompilation=false
  • dotnet test CodeIndex.sln -c Release -p:UseSharedCompilation=false
  • dotnet test tests/CodeIndex.Tests/CodeIndex.Tests.csproj -c Release -p:UseSharedCompilation=false --filter "FullyQualifiedName~DbReaderTests|FullyQualifiedName~FileIndexerTests|FullyQualifiedName~McpServerTests|FullyQualifiedName~QueryCommandRunnerTests"
  • dotnet run --project tools/CodeIndex.Changelog -- check
  • dotnet ./src/CodeIndex/bin/Debug/net8.0/cdidx.dll status --check --json

Documentation / Changelog

  • Updated USER_GUIDE.md.
  • Added changelog.d/unreleased/1458.fixed.md.

Follow-up Candidates

  • None.

Fixes #1458
Fixes #1606
Fixes #1776

@Widthdom
Widthdom merged commit a07cddd into main May 31, 2026
10 checks passed
@Widthdom
Widthdom deleted the fix-issue1458-1606-1776 branch May 31, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant