feat: index-backed /search (jieba/CJK, ranked, visibility-filtered)#239
Merged
Conversation
rewrites POST /search to match against the inverted index instead of a substring scan over the last 1000 timeline events. the query is jieba- tokenized the same way the index is, so CJK searches work word-by-word; multi-term queries AND-intersect postings; order_by rank sums term frequency; the keys and filter.senders filters are honored. every hit AND its before/after context events pass the per-event history-visibility gate (the substring version gated neither, so context could leak pre-join / post-leave events); redacted and E2EE-room events are excluded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 2 of the full-text search feature. PR 1 (#237) added the inverted index; this rewrites
POST /v3/searchto actually use it, replacing the substring scan.What changes
order_by: ranksums term frequency;recentsorts by time.keys(body/name/topic) andfilter.senders/filter.roomsare honored.TimelineReadGate::permits). The old substring version gated neither the hit nor the context, so context could leak pre-join (underjoinedvisibility) or post-leave events — closed here. Redacted and E2EE-room events excluded.Tests (
tests/search_index_query.rs): body + CJK + multi-term AND +keysfilter, rank-by-term-frequency ordering, and a negative-space test that a pre-join message does not leak into a hit's context underjoinedvisibility.Known limitation: the per-token candidate cap (1000, newest-first) means a
common + rareAND query can miss an old match whose common token falls outside the recent window — same recency-window bound the old scan had. A rarest-token-first intersection is a later optimization.A reviewer pass flagged the context-visibility gap (now fixed) and confirmed the AND-intersection, keys filter, pagination safety, and filter.rooms handling are correct.