fix(search): preserve CJK relevance signals

[w-up]

Summary / 摘要

Fixes #230.

The core ranker used ASCII-only tokenization for lexical alignment and rare-term phrase signals. CJK queries were reduced to an empty token set, so topically relevant results and unrelated fresh calendar/weather pages could receive the same zero lexical score.

核心 ranking 的 lexical alignment 与 rare-term phrase 使用 ASCII-only tokenizer，CJK 查询会被清空，导致主题相关结果无法通过文本相关性压过黄历、天气等无关新内容。

Changes / 改动

• Add a shared Unicode-aware ranking tokenizer.
• Preserve existing lowercase Latin/digit token behavior.
• Convert contiguous Han, Hiragana, Katakana, and Hangul runs into overlapping bigrams.
• Use Script_Extensions so Japanese shared marks such as ー stay attached:
  • AIニュース → ai, ニュ, ュー, ース
• Bound processed input to 4096 code units and output to 256 tokens before expensive intermediate work.
• Reuse the tokenizer in:
  • lexicalAlignment()
  • CJK concept-phrase detection
  • rare-term document phrase matching
• Keep compound-token detection (sqlite-vec, vec0, vec_distance) unchanged.

Regression coverage / 回归测试

• Relevant Chinese AI result scores above unrelated calendar/weather content.
• CJK queries produce bounded rare-term concept phrases.
• rareTermFactor() and isRareTermMiss() distinguish CJK topic hits from misses.
• Mixed Latin/CJK/digit text preserves each signal.
• Japanese prolonged sound marks remain inside Katakana bigrams.
• Hangul text produces deterministic bigrams.
• A 10,000,000-character pathological input completes without regex stack overflow and returns at most 256 tokens.
• Existing Latin, punctuation, compound, ranking, and rerank-fold suites stay green.

Verification / 验证

24 test files passed
236 tests passed

npm run lint
npm run build
git diff --check

Scope / 范围

• No new dependencies.
• No public API or schema changes.
• No search-engine integration changes.
• Contributor agrees to CONTRIBUTING.md and its CLA terms.

Summary by CodeRabbit

• New Features
  • Improved search understanding for Chinese, Japanese, and Korean queries, including unsegmented text.
  • Added language-aware matching for multilingual titles and snippets.
  • Improved handling of Japanese, Korean, Latin, numeric, and mixed-language content.
  • Added more consistent tokenization for punctuation, Japanese sound marks, and long inputs.
• Bug Fixes
  • Improved relevance scoring for topic-aligned CJK content.
  • Reduced missed matches when search terms appear as continuous character sequences.
• Tests
  • Added coverage for multilingual matching, CJK tokenization, and large inputs.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b3909f23-2623-4818-b592-de0738873bac

📥 Commits

Reviewing files that changed from the base of the PR and between ee91666 and 3acd577.

📒 Files selected for processing (2)

• src/search/core/text-tokenizer.ts
• tests/unit/search/core/text-tokenizer.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/unit/search/core/text-tokenizer.test.ts

---

📝 Walkthrough

Walkthrough

The change adds a bounded Unicode-aware tokenizer and uses it in lexical alignment and rare-term scoring. Tests cover Latin, numeric, Japanese, Chinese, Korean, and large unsegmented inputs.

Changes

CJK ranking signals

Layer / File(s)	Summary
Shared Unicode tokenizer src/search/core/text-tokenizer.ts, tests/unit/search/core/text-tokenizer.test.ts	Adds bounded tokenization for Latin, numeric, Han, Hiragana, Katakana, and Hangul text. It emits overlapping CJK bigrams and preserves Japanese prolonged-sound marks.
Lexical alignment integration src/search/core/lexical-alignment.ts, tests/unit/search/core/lexical-alignment.test.ts	Uses the shared tokenizer while preserving short-token and stopword filtering. Tests verify aligned CJK scoring.
Rare-term token integration src/search/core/rare-terms.ts, tests/unit/search/core/rare-terms.test.ts	Uses shared tokens for concept phrases, contiguous document runs, rare-term factors, and miss detection. Tests cover unsegmented CJK queries and documents.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Query
  participant lexicalAlignment
  participant tokenizeRankingText
  participant rareTermFunctions
  Query->>lexicalAlignment: tokenize query, title, and snippet
  lexicalAlignment->>tokenizeRankingText: tokenize ranking text
  tokenizeRankingText-->>lexicalAlignment: return Latin or CJK tokens
  Query->>rareTermFunctions: build rare-term signals
  rareTermFunctions->>tokenizeRankingText: tokenize query and document text
  tokenizeRankingText-->>rareTermFunctions: return bounded token streams

Loading

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The CJK fix meets the main goals, but rare-term paths no longer preserve prior short-token and stopword filtering required by issue [#230].	Restore equivalent short-token and stopword filtering in rare-term token paths, then rerun existing Latin and stopword ranking tests.
Docstring Coverage	⚠️ Warning	Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: preserving CJK relevance signals in search.
Out of Scope Changes check	✅ Passed	All implementation and test changes support the linked issue by adding shared CJK tokenization and regression coverage.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[squariaa-dev]

qqq

[KnockOutEZ]

Nice work — verified locally that this introduces zero new test failures (the two answer-synthesis failures reproduce on main and are unrelated), and the Unicode-property tokenizer with CJK bigrams, Script_Extensions handling, and bounded input/token caps is well thought through. The existing ef_search / Next.js / compound-token behavior stays intact.

One thing before merge: the comments in src/search/core/text-tokenizer.ts are written in Chinese. The rest of the codebase comments in English, so could you translate them so the module stays consistent and maintainable for all contributors? The logic itself is good as-is.

I'll approve CI on the branch so you get the cross-platform run.

[w-up]

Thanks for the review. I translated all Chinese comments and JSDoc in src/search/core/text-tokenizer.ts to English, keeping the implementation unchanged.

Verification:

• 28 targeted search-core tests passed
• npm run lint passed
• npm run build passed
• git diff --check passed

The update is in commit ee916669.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/search/core/text-tokenizer.ts`:
- Around line 3-6: Update the CJK_RUN pattern used by TOKEN_PART_RE to use base
Script properties for Han, Hiragana, Katakana, and Hangul, explicitly allowing
only intended non-letter marks such as \u30FC; do not use Script_Extensions so
punctuation like 、, 。, and ・ cannot join CJK runs. Add regression coverage
verifying these punctuation characters are excluded from token parts and
bigrams.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4be0ee1f-1639-487f-af60-b1a123848360

📥 Commits

Reviewing files that changed from the base of the PR and between 51dee55 and ee91666.

📒 Files selected for processing (1)

• src/search/core/text-tokenizer.ts

[w-up]

Follow-up: CodeRabbit identified that Script_Extensions also admitted CJK punctuation into token runs. I reproduced it with a failing test and fixed it in 3acd5775 by using base Script properties plus explicit U+30FC support. Regression coverage now verifies that 、, 。, and ・ split CJK runs while AIニュース keeps ー inside Katakana bigrams.

[w-up]

CI follow-up: all lint/build/unit jobs, the Ubuntu full test suite, and 9 of 10 clean-machine smoke jobs passed. The only failure was clean-machine smoke (windows-latest, node 24), where the unrelated ML reranker warmup failed to load the downloaded ONNX model with system error number 13; the same Windows smoke passed on Node 20 and Node 22. I cannot rerun the upstream workflow from the fork account because GitHub requires repository admin rights.