Skip to content

perf(vector/format): LVS v3 header with per-segment field-name dictionary (#857)#860

Merged
mosuka merged 1 commit into
mainfrom
perf/633-field-dict-format
Jul 12, 2026
Merged

perf(vector/format): LVS v3 header with per-segment field-name dictionary (#857)#860
mosuka merged 1 commit into
mainfrom
perf/633-field-dict-format

Conversation

@mosuka

@mosuka mosuka commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

Format-layer half of the #633 per-segment field-name dictionary (PR-A1 of the plan posted there). Inert by design: no writer stamps version 3 yet, so no bytes on disk change — this PR only teaches the shared VectorSegmentHeader the v3 vocabulary so the reader/writer migration (#858) can land as a focused follow-up.

Testing

  • 8 new format tests: v3 roundtrip with 0/1/3-entry dicts (serialized_size == written bytes), v1 stays 24 bytes with an empty dict, truncated dict → Io, oversized-name write rejection, read_record_field v3 resolve + out-of-range + legacy inline path, record_prefix_size ladder, build_field_dict first-appearance order.
  • All 16 pre-existing format tests pass unmodified; lib 1212/1212; fmt / clippy (1.95 and 1.97 parity, workspace + wasm target) / cargo check wasm32 all clean.

Closes #857
Refs #633

…nary (#857)

Add the format-layer half of the Issue #633 field-name dictionary
(Lucene fnm precedent). Inert on its own: no writer stamps version 3
yet, so no bytes on disk change in this commit.

- VERSION_FIELD_DICT = 3 / MAX_SUPPORTED_VERSION = 3; the version
  ladder (1 = base, >=2 = HNSW ordinal graph, >=3 = dict records) is
  documented on CURRENT_VERSION
- VectorSegmentHeader gains field_dict: Vec<String> (+ with_field_dict);
  serialized_size / write_to / read_from carry the dictionary block
  only at version >= 3, so v1/v2 headers stay byte-identical
- dict allocations are self-bounded by the u16 length domain (#806
  property without file-size plumbing); writers reject > u16::MAX
  entries or names
- shared helpers for the reader/writer migration (#858):
  read_record_field (v3 id + bounds check / legacy inline name),
  record_prefix_size (10 vs 12), build_field_dict (first-appearance
  order)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(vector/format): LVS v3 header with per-segment field-name dictionary — format layer (sub-issue of #633)

1 participant