perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (#858)#861
Merged
Merged
Conversation
…NSW/Flat/IVF (#858) Second half of the Issue #633 field-name dictionary: the three vector index types now write version-3 segments whose records carry a u16 field_id instead of the inline name, and read v1/v2/v3 uniformly. - readers: the 8 record-parse branches collapse into the shared read_record_field helper; record strides become version-dependent (10 vs 12 + name bytes); the Flat/IVF quant matches go by-ref so the parsed header (version + dictionary) survives to the parse loops - fixes a latent bug: the Flat and HNSW lazy paths computed the record seek by rebuilding a fresh header instead of measuring the parsed one, which would have silently omitted the v3 dictionary - OnDemand offsets now point at the record payload, so VectorStorage::get seeks straight to the vector data instead of re-parsing the variable-width name prefix on every access - writers: build the per-segment dictionary in first-appearance order over the exact emission order (HNSW doc_id-sorted, Flat buffer order, IVF cluster order), stamp VERSION_FIELD_DICT, emit u16 ids; the load() re-parsers translate ids back for the doc_id-keyed in-memory state, so compaction/merge upgrade v1/v2 segments in place - tests: v1->v3 upgrade-on-rewrite proof (version bytes asserted), v3 hand-built fixture load + search, out-of-range field_id and empty-dict-with-records rejection, and byte-exact file-size assertions (per-type formula + the v1-vs-v3 delta formula) - docs: module layout docs + persistence.md + vector_indexing.md (en/ja) describe the v3 record prefix and dictionary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second half of the #633 field-name dictionary (PR-A2+A3 of the plan there; the format layer landed in #860). The three vector index types now write LVS version-3 segments whose records carry a
u16 field_idinto the per-segment dictionary instead of the inlineu32 len + UTF-8 name, and read v1/v2/v3 uniformly.Readers
read_record_field(); record strides become version-dependent viarecord_prefix_size().VectorStorage::getseeks straight to the vector data instead of re-reading the variable-width name prefix on every access (net-negative LOC in storage.rs; a structural win for the mmap-default path).Writers
u16ids; HNSW's five header stamps go 2→3 and Flat/IVF stamp a version for the first time.load()re-parsers translate ids back for the doc_id-keyed in-memory state, so compaction/merge upgrade v1/v2 segments to v3 in place.Compatibility
v1/v2 read paths are byte-identical (all pre-existing fixture tests pass unmodified). Older builds reading v3 get the clean range-gate
IncompatibleFormat— the same deliberate forward break as #853/#854, documented in persistence.md.Testing
n·(k+2) − dictbytes saved; −18.9%/record at the issue's k=32 example).field_idand empty-dict-with-records rejected as corruption; multi-field and empty-segment cases.Closes #858
Refs #633