perf(vector/index): intern field names in memory as (u64, u16) + per-segment dictionary (#859)#862
Merged
Merged
Conversation
…segment dictionary (#859) Final piece of the Issue #633 field-name work: readers keep one shared Arc<[Arc<str>]> dictionary per segment and (doc_id, u16) pairs per record instead of one heap String per record per retained structure. - FieldInterner (format.rs): v3 segments use the header dictionary (fixed, ids validated); v1/v2 segments synthesize it at load in first-appearance order, so legacy files get the same interned representation with no version branching afterwards - readers: vector_ids becomes Vec<(u64, u16)>; IVF cluster_to_vectors likewise (its eager path used to retain 3 String copies per record); per-field caches and pool build inputs keep their String shapes (transient clones only, nothing per-record retained) - VectorStorage::get/contains take (doc_id, &str) and the OnDemand arm resolves the name against the segment dictionary by linear scan — the per-candidate field_name.to_string() on the mmap-default f32 paths is gone structurally; OnDemand offsets are keyed (u64, u16) - IVF searcher: probe results stay (u64, u16), the field filter resolves once per search and compares u16 per candidate, and names rehydrate only for emitted top-k hits - public trait signatures unchanged: vector_ids(), iterators, and the other boundary methods rehydrate from the dictionary at return time RAM accounting (structural): a retained per-record String costs ~24B header + heap + allocator overhead, x2 retained copies (HNSW/Flat) or x3 (IVF); the interned pair costs 16B inline with zero per-record heap. Eager gate bench neutral (del0 33.4us / del10 29.3us, inside the post-#854 envelope).
This was referenced Jul 13, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR-B, the final piece of #633 (the on-disk dictionary landed in #860/#861): readers now keep one shared
Arc<[Arc<str>]>field-name dictionary per segment and(doc_id, u16)pairs per record, instead of one heapStringper record per retained structure.FieldInterner(format.rs): v3 segments use the header dictionary (fixed, ids validated); v1/v2 segments synthesize it at load in first-appearance order — legacy files get the same interned representation, and nothing branches on version after load.vector_ids: Vec<(u64, u16)>; IVFcluster_to_vectorslikewise (its eager path used to retain three String copies per record — the worst case found in the Phase 0 lifecycle sweep). Per-field caches and pool build inputs keep their String shapes (transient clones only; nothing per-record is retained).VectorStorage::get/containstake(doc_id, &str), and the OnDemand arm resolves the name against the segment dictionary by linear scan (1–3 entries in practice) — the per-candidatefield_name.to_string()on the mmap-default f32 paths is gone structurally; OnDemand offsets are keyed(u64, u16).(u64, u16), the field filter resolves once per search and compares u16 per candidate; names rehydrate only for emitted top-k hits.vector_ids(), iterators, and the other boundary methods rehydrate from the dictionary at return time. TheVectorStoragemethod signatures changed (pubbut not part of the reader/writer traits; no external callers).RAM accounting (structural, deterministic)
A retained per-record
Stringcosts ~24B header + heap payload + allocator overhead, ×2 retained copies (HNSW/Flat) or ×3 (IVF eager); the interned pair costs 16B inline with zero per-record heap. At 1M records with short names that is roughly 90–135MB → a few MB of id-bearing structures.Deviation from the plan, recorded honestly: pool build inputs keep the
(u64, String, …)shape — the per-record Strings there are transient (consumed by per-field grouping, not retained), so converting the pipelines was all churn for no retained-memory gain.Testing
FieldInterner/resolve_field_idunit tests; fmt / clippy (1.95 and 1.97 parity, workspace + wasm target) / wasm32 check clean.Deletion Searchdel0 33.4µs / del10 29.3µs — inside the post-perf(vector/hnsw): segment-local u32 ordinal graph + LVS1 v2 format (#853) #854 cross-session envelope (33.0–37.4 / 29.0–31.7µs). The eager hot loop never touched these maps, so neutrality is the expected and required result; the structural wins live on the mmap/OnDemand paths (the ±6–19% bench class — claimed structurally per the perf(vector/hnsw): segment-local u32 ordinal graph + LVS1 v2 format (sub-issue of #686) #853/chore(vector/bench): resolve the mmap/OnDemand HNSW 50k bench class verdict after #853 #856 measurement policy).Closes #859
Refs #633