feat: normalize Traditional Chinese to Simplified for BM25 cross-match

[gdccyuen]

What

Normalizes Traditional Chinese (繁體) to Simplified (简体) before jieba tokenization, so BM25 cross-matches 繁/简 variants. A document ingested with "營收" becomes findable by a query for "营收", and vice versa.

Why

Knowhere's BM25 search uses jieba (a Simplified Chinese tokenizer) + PostgreSQL 'simple' tsvector. Without normalization, 繁/简 variants produce different tokens:

• 營收 → jieba cuts ["營收"] → stored as 營收 in content_search_text
• 营收 → jieba cuts ["营收"] → query tsquery is 营收
• 營收 @@ 营收 → no match

With OpenCC t2s applied before jieba:

• Both 營收 and 营收 → OpenCC → 营收 → jieba → ["营收"] → match

Changes

Dependency (packages/shared-python/pyproject.toml)

Added opencc-python-reimplemented>=0.1.7 — pure Python, no native deps. Only converts CJK characters; English, numbers, and punctuation are untouched.

Normalization (packages/shared-python/shared/utils/text_utils.py)

New _normalize_cjk_to_simplified(text) helper with a module-level lazy OpenCC('t2s') instance. Applied at two chokepoints:

1. _tokenize_cjk_segment — used by tokenize_for_retrieval → tokenize_contents_for_retrieval, which powers:
   • Ingest: build_content_search_text, build_path_search_text, build_lexical_text (in lexical_text.py)
   • Query: lexical_ranker.py
2. tokenize2stw_remove — used by docx, atlas, excel, markdown parsers for chunk tokens metadata

Both ingest and query go through the same normalization, so symmetry is automatic — no need to change PostgreSQL config or the 'simple' tsvector.

Tests (packages/shared-python/shared/tests/test_text_utils_cjk.py)

11 unit tests:

• _normalize_cjk_to_simplified: 繁→简 conversion, English unchanged, punctuation unchanged, empty string, mixed 繁/简
• tokenize_for_retrieval: 繁 and 简 produce identical tokens, mixed 繁/简+English produces identical tokens, English still tokenized correctly
• tokenize_contents_for_retrieval: 繁 and 简 produce identical token strings
• tokenize2stw_remove: 繁 and 简 produce identical token strings
• Cross-match regression: a 繁體 document ("台積電營收創歷史新高") is findable by both 繁體 and 简体 queries ("台積電營收" / "台积电营收")

Backward compatibility

• Existing content_search_tsv rows are pre-normalization. They continue to work for matching queries (same pre-normalization tokens on both sides). Full 繁↔简 cross-match requires re-ingest. No migration needed — new ingestions produce normalized tokens going forward.
• The 'simple' PostgreSQL tsvector config is unchanged. All normalization happens in the Python application layer before text reaches the database.

Verification

uv run pytest packages/shared-python/shared/tests/test_text_utils_cjk.py -q
11 passed in 0.27s

3 pre-existing failures in test_page_memory_vlm_limiter.py are unrelated (VLM AI service TypeError — confirmed present on main without this change).

Design decision: OpenCC vs zhparser

This implements the ADR-recommended "OpenCC + jieba + simple tsvector" stack rather than PostgreSQL zhparser:

	OpenCC (this PR)	zhparser
繁↔简 cross-match	✅	❌ (different problem layer)
Synonym expansion	❌	✅
PostgreSQL changes	None	Install extension, change tsvector config
Python-layer changes	1 helper + 2 call sites	Remove jieba pre-tokenization
Deployment complexity	None (pure Python dep)	Medium (compile PG extension)

繁↔简 is the higher-priority gap; synonyms can be addressed separately if needed.