Skip to content

perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (#858)#861

Merged
mosuka merged 1 commit into
mainfrom
perf/633-field-dict-readers-writers
Jul 12, 2026
Merged

perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (#858)#861
mosuka merged 1 commit into
mainfrom
perf/633-field-dict-readers-writers

Conversation

@mosuka

@mosuka mosuka commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

Second half of the #633 field-name dictionary (PR-A2+A3 of the plan there; the format layer landed in #860). The three vector index types now write LVS version-3 segments whose records carry a u16 field_id into the per-segment dictionary instead of the inline u32 len + UTF-8 name, and read v1/v2/v3 uniformly.

Readers

  • The 8 record-parse branches (HNSW SQ eager/lazy, PQ, FastScan; Flat eager/lazy; IVF eager/lazy) collapse into the shared read_record_field(); record strides become version-dependent via record_prefix_size().
  • Flat/IVF quant matches go by-ref — previously a partial move meant the parsed header did not survive to the parse loops (latent, harmless at v1/v2, load-bearing at v3).
  • Latent bug fixed: the Flat and HNSW lazy paths computed the record-section seek by rebuilding a fresh header instead of measuring the parsed one — that reconstruction would have silently omitted the v3 dictionary.
  • OnDemand offsets now point at the record payload, so VectorStorage::get seeks straight to the vector data instead of re-reading the variable-width name prefix on every access (net-negative LOC in storage.rs; a structural win for the mmap-default path).

Writers

  • Per-segment dictionary built in first-appearance order over the exact emission order (HNSW doc_id-sorted, Flat buffer order, IVF cluster order — its existing flatten pass provides the walk for free); records emit u16 ids; HNSW's five header stamps go 2→3 and Flat/IVF stamp a version for the first time.
  • The three load() re-parsers translate ids back for the doc_id-keyed in-memory state, so compaction/merge upgrade v1/v2 segments to v3 in place.

Compatibility

v1/v2 read paths are byte-identical (all pre-existing fixture tests pass unmodified). Older builds reading v3 get the clean range-gate IncompatibleFormat — the same deliberate forward break as #853/#854, documented in persistence.md.

Testing

  • Byte-exact file-size assertion (the deterministic headline gate): total-size formula per component plus the v1-vs-v3 delta formula (n·(k+2) − dict bytes saved; −18.9%/record at the issue's k=32 example).
  • v1→v3 upgrade-on-rewrite proof (writer loads a hand-built v1 fixture, rewrites, version bytes asserted, reader loads, search results identical); v3 hand-built fixture load + search (eager and lazy); out-of-range field_id and empty-dict-with-records rejected as corruption; multi-field and empty-segment cases.
  • lib 1216/1216 (+ pq-fastscan 1222/1222); all 23 vector/hnsw/filter integration test files green; fmt / clippy (1.95 and 1.97 parity, workspace + wasm target) / wasm32 check clean; markdownlint clean.
  • Eager gate bench neutral as required (deletion bench del0 33.2µs / del10 31.6µs — inside the post-perf(vector/hnsw): segment-local u32 ordinal graph + LVS1 v2 format (#853) #854 cross-session envelope 33.0–37.4 / 29.0–31.7µs; the eager hot loop never touches names).
  • Docs: module layout docs ×4, persistence.md and vector_indexing.md (en/ja).

Closes #858
Refs #633

…NSW/Flat/IVF (#858)

Second half of the Issue #633 field-name dictionary: the three vector
index types now write version-3 segments whose records carry a u16
field_id instead of the inline name, and read v1/v2/v3 uniformly.

- readers: the 8 record-parse branches collapse into the shared
  read_record_field helper; record strides become version-dependent
  (10 vs 12 + name bytes); the Flat/IVF quant matches go by-ref so the
  parsed header (version + dictionary) survives to the parse loops
- fixes a latent bug: the Flat and HNSW lazy paths computed the record
  seek by rebuilding a fresh header instead of measuring the parsed
  one, which would have silently omitted the v3 dictionary
- OnDemand offsets now point at the record payload, so
  VectorStorage::get seeks straight to the vector data instead of
  re-parsing the variable-width name prefix on every access
- writers: build the per-segment dictionary in first-appearance order
  over the exact emission order (HNSW doc_id-sorted, Flat buffer
  order, IVF cluster order), stamp VERSION_FIELD_DICT, emit u16 ids;
  the load() re-parsers translate ids back for the doc_id-keyed
  in-memory state, so compaction/merge upgrade v1/v2 segments in place
- tests: v1->v3 upgrade-on-rewrite proof (version bytes asserted), v3
  hand-built fixture load + search, out-of-range field_id and
  empty-dict-with-records rejection, and byte-exact file-size
  assertions (per-type formula + the v1-vs-v3 delta formula)
- docs: module layout docs + persistence.md + vector_indexing.md
  (en/ja) describe the v3 record prefix and dictionary
@mosuka mosuka merged commit 59c2f1b into main Jul 12, 2026
22 checks passed
@mosuka mosuka deleted the perf/633-field-dict-readers-writers branch July 12, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (sub-issue of #633)

1 participant