perf(vector/format): LVS v3 header with per-segment field-name dictionary (#857)#860
Merged
Conversation
…nary (#857) Add the format-layer half of the Issue #633 field-name dictionary (Lucene fnm precedent). Inert on its own: no writer stamps version 3 yet, so no bytes on disk change in this commit. - VERSION_FIELD_DICT = 3 / MAX_SUPPORTED_VERSION = 3; the version ladder (1 = base, >=2 = HNSW ordinal graph, >=3 = dict records) is documented on CURRENT_VERSION - VectorSegmentHeader gains field_dict: Vec<String> (+ with_field_dict); serialized_size / write_to / read_from carry the dictionary block only at version >= 3, so v1/v2 headers stay byte-identical - dict allocations are self-bounded by the u16 length domain (#806 property without file-size plumbing); writers reject > u16::MAX entries or names - shared helpers for the reader/writer migration (#858): read_record_field (v3 id + bounds check / legacy inline name), record_prefix_size (10 vs 12), build_field_dict (first-appearance order)
This was referenced Jul 12, 2026
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Format-layer half of the #633 per-segment field-name dictionary (PR-A1 of the plan posted there). Inert by design: no writer stamps version 3 yet, so no bytes on disk change — this PR only teaches the shared
VectorSegmentHeaderthe v3 vocabulary so the reader/writer migration (#858) can land as a focused follow-up.VERSION_FIELD_DICT = 3/MAX_SUPPORTED_VERSION = 3; the monotone version ladder (1 = base, ≥2 = HNSW ordinal graph from perf(vector/hnsw): segment-local u32 ordinal graph + LVS1 v2 format (sub-issue of #686) #853, ≥3 = dictionary records) is documented onCURRENT_VERSION.VectorSegmentHeader.field_dict: Vec<String>+with_field_dict();serialized_size/write_to/read_fromcarry the dictionary block only at version ≥ 3 — v1/v2 headers stay byte-identical (asserted by test).InvalidOperation.#[allow(dead_code)]until then):read_record_field()(v3 = u16 id + dictionary bounds check →LaurusError::indexon corruption; legacy = the existing inline u32+UTF-8 parse),record_prefix_size()(10 vs 12),build_field_dict()(first-appearance order).Testing
Io, oversized-name write rejection,read_record_fieldv3 resolve + out-of-range + legacy inline path,record_prefix_sizeladder,build_field_dictfirst-appearance order.cargo checkwasm32 all clean.Closes #857
Refs #633