feat: full-text search inverted index (jieba/CJK), indexing only#237
Merged
Conversation
adds the RocksDB-native index that POST /search will use: a `search_index` CF keyed room+token+stream_pos, populated on the event write path and dropped on redaction, tokenized with jieba so Chinese text is searchable word-by-word (你好世界 -> 你好/世界) where synapse and conduwuit only whitespace-split (and so can't match CJK at all). - vela-store `search` module: shared lazy jieba tokenizer, key/value codec, searchable-field extraction (body/name/topic) - index-on-persist inside persist_event_kind (same WriteBatch, atomic), de-index inside mark_redacted_by, both gated on a runtime flag - search_room_token: bounded newest-first reverse-scan for the query path - !reindex-search admin command + [search] enabled config (default on), jieba warmed at boot so the first message doesn't eat the dict load the query path still uses the existing substring scan; switching it to the index is the next change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of a short series that gives vela real server-side search (
POST /v3/search), closing the one genuine CS-API gap and the long-deferred CJK blocker.This PR adds the indexing infrastructure only — the query path still uses the existing substring scan, so no user-facing change yet.
Why jieba + our own index. Synapse (English
tsvector/porter) and conduwuit (whitespace split) can't segment Chinese, so CJK search doesn't work on either. Dendrite works via a separate Bleve index. vela keeps it in RocksDB (one store, one consistency/backup model) and uses jieba for real word segmentation — 你好世界 → 你好 / 世界.What's here
vela-store::search: shared lazy jieba tokenizer + key/value codec; extracts body/name/topic per the spec's searchable keys.search_indexCF, keyroom_nid ++ token ++ 0xFF ++ stream_pos, valueevent_nid ++ field ++ tf.0xFFis a safe separator (never valid UTF-8); stream_pos suffix gives free recency ordering.persist_event_kind(same WriteBatch → atomic), de-index inmark_redacted_by; both gated on a runtime flag.search_room_token: bounded, newest-first reverse-scan (no load-then-sort) for the query path.!reindex-searchadmin command to backfill history;[search] enabledconfig (default on); jieba warmed at boot.Tests: tokenizer (CJK segmentation, mixed, punctuation, term-frequency, over-long cap) + DB-backed round-trip (persist → prefix-scan → redact → gone, cross-room isolation, off-by-default + reindex).
Next: rewrite
/searchto use the index (typed request, tokenized AND query, rank/recent ordering, per-event history-visibility, skip E2EE).