perf(vector/index): tighten on-disk record by promoting `field_name` to a per-segment dictionary

[mosuka]

Round-3 perf push sub-issue (tracked under umbrella #535).

[M] Tighten on-disk record by promoting field_name to a per-segment dictionary

• Where: laurus/src/vector/index/quantized_io.rs:14-23,
  :74-85; HNSW write loop (hnsw/writer.rs:1210-1218); Flat / IVF
  equivalents.

• Current behaviour: every vector record stores its field_name
  inline (u32 len + UTF-8 bytes). Most segments have one field; this
  is pure overhead.

• Why it might be a bottleneck: For 10 M vectors with a 32-byte
  field name ("text_embedding_e5_large_v2_768d"), that's 320 MB of
  duplicated UTF-8 alongside the int8 payload.

• Reference precedent: Lucene stores field names in a per-segment
  field-info dictionary (fnm) and references them by ordinal in every
  vector record.

• Current layout:

  per record: [doc_id u64][name_len u32][field_name UTF-8][int8 dim][meta 8B]
  

• Proposed layout:

  segment header: field_dict_len u16, field_dict (u16 len + UTF-8)*
  per record:    [doc_id u64][field_id u16][int8 dim][meta 8B]
  ─ field_id indexes into the dictionary above ─
  

  For the single-field case field_id can be elided entirely (encode
  as field_dict_len = 0 → implicit field_id = 0).

• Suggested direction: extend VectorSegmentHeader (or move to
  LVS2) to carry the field dictionary; readers translate field_id back
  to the string on first access (cached in a SmallVec<String> per
  segment).

• Risk / scope: format break (must bump LVS version); pairs with
  the columnar restructure (Add parallel index #2).

---

---

ID: VI-15 — see ~/.claude/tasks/laurus/20260523_perf_round3_audit/task_list.md for the full Round-3 issue list.

Task list (2026-07-12 start)

Phase 0: Investigation

• In-memory field-name lifecycle map (vector_ids, per-field caches, pool map keys, OnDemand offset keys, iterators, hit/result types, per-call String allocations; merge / NRT-active / writer buffers)
• Design constraints (version-gate matrix on top of perf(vector/hnsw): segment-local u32 ordinal graph + LVS1 v2 format (sub-issue of #686) #853's v2; empty segments; legacy multi-field; WAL/manifest/bindings impact; dictionary placement)
• Opportunity ceiling (on-disk bytes + load time + RAM, with fire-condition line evidence)
• Post investigation summary as an issue comment

Phase 1: Plan

• Scope decision: A+B full, staged as three sub-issue PRs → plan posted as an issue comment

Phase 2+: Implementation (sub-issues)

• perf(vector/format): LVS v3 header with per-segment field-name dictionary — format layer (sub-issue of #633) #857 PR-A1: format layer — PR perf(vector/format): LVS v3 header with per-segment field-name dictionary (#857) #860 merged
• perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (sub-issue of #633) #858 PR-A2+A3: readers + writers emit/read v3 — PR perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (#858) #861 merged
• perf(vector/index): intern field names in memory as (u64, u16) + per-segment dictionary (sub-issue of #633) #859 PR-B: in-memory interning — PR perf(vector/index): intern field names in memory as (u64, u16) + per-segment dictionary (#859) #862 merged

[mosuka]

Phase 0 investigation report

Method: two parallel code sweeps (in-memory field-name lifecycle / format-design constraints), plus reuse of the #686 on-disk inventory (which already pinned every record parse/emit site at line level).

Current state

On disk: all three index types prefix every record with [doc_id u64][field_name_len u32][field_name UTF-8]; the 12 + payload stride constant is spread across the three readers, the writers' load() re-parsers, and the #806 alloc bounds. With a dictionary the prefix becomes [doc_id u64][field_id u16] — constant 10 bytes.

At load, each record materializes 2–4 heap Strings: HNSW/Flat eager = base + 1 clone (retained ×2); Lazy paths also key the OnDemand offsets: HashMap<(u64, String), u64> map; IVF eager is the worst at base + 3 clones/record (one of them allocated then immediately dropped), retained ×3 (pool records / cluster_to_vectors / vector_ids). IVF's writer adds 2 more clones/record (load + build_inverted_lists).

Per query — the important find: VectorStorage::get demands an owned &(u64, String) key (storage.rs:191), so get_vector allocates field_name.to_string() per call — and that call is per-candidate on every f32 fallback path (HNSW OnDemand graph traversal, brute fallbacks, Flat/IVF f32). Since use_mmap defaults to true on Linux, this is the production file-backed path. OnDemand get() additionally re-reads and skips the variable-width name bytes on every access (storage.rs:242-254) — a u16 prefix collapses that to a constant skip. The eager quantized hot path (#853 ordinals) is already string-free, so the #853 gate benches should be exactly neutral under this change. IVF probe_clusters also does a per-candidate String == field filter (→ u16 compare). Dead surface noted: VectorIterator::position/skip_to and VectorIndexReader::get_vectors have zero non-test callers.

Opportunity ceiling (deterministic — no bench-noise gate needed)

item	value
disk, k=5 ("field"), SQ dim=128	−4.6%/record
disk, k=32 (the issue's example)	−18.9%/record → −340MB at 10M vectors
load allocations	2–4M Strings eliminated per 1M records
resident RAM (if interning is in scope)	~90–135MB → ~4–6MB per 1M records (k=5, ×2–3 retained copies)
mmap-path per-candidate to_string + variable name skip	removed structurally (claimed structurally, not numerically — that bench class is the ±6–19% one)

Verification design: exact file-size assertions in tests (deterministic), eager gate benches for neutrality, load-time as a secondary informal measurement.

Design constraints (settled)

1. Version ladder is clean: no production == version checks exist — only the range gate and two >= VERSION_ORDINAL_GRAPH dispatches. So v3 = "dict records (all types) + (for HNSW, ≥2 ordinal graph)" works as a monotone ladder; Flat/IVF writers start stamping v3 via with_version (they stamp bare v1 today), MAX_SUPPORTED_VERSION → 3.
2. Dictionary placement: extend VectorSegmentHeader (present iff version ≥ 3, after the quant metadata). Trap found: the Flat and HNSW Lazy paths compute record offsets by reconstructing a fresh header and taking its serialized_size() (flat/reader.rs:196, hnsw/reader.rs:640) — they would silently omit the dict; both must switch to the actually-parsed header's size (a latent fragility worth fixing anyway).
3. Parsers to touch: 8 reader branches, 5 writer emit loops, 3 writer load() re-parsers, the OnDemand per-access skip. Merge engines, the rerank sidecar (positional, no names), and the AoS quantized segment are unaffected.
4. Edge cases: empty dict coincides exactly with zero records (fields only exist attached to records) — no ambiguity; field_id >= dict_len → LaurusError::index (v2 corruption-check precedent).
5. Touchpoints: ~16 format.rs unit tests, the hand-built fixture helpers in hnsw/reader.rs tests (sq_record builds exactly the old prefix), quantized_segment header-size tests; docs: persistence.md + vector_indexing.md (en/ja) + 5 module-doc layout comments. No bindings document the layout; public trait signatures stay (rehydrate Strings from the dictionary at the boundary).

Recommendation

GO. The headline wins (disk bytes, allocation counts, resident RAM) are deterministically verifiable — no noise-floor gamble; compat rides the established version-ladder precedent. Phase 1 (plan) will decide the scope split: (A) on-disk dictionary only, vs (B) A + in-memory interning ((u64, u16) internally, rehydrate at trait boundaries — also subsumes #682/#672 and removes the per-candidate to_string on the mmap path), likely as staged PRs. Plan follows.

[mosuka]

Phase 1: approved implementation plan (scope: A+B full)

v3 format spec

Header (version >= 3 only, right after the quant metadata):
  [field_dict_count u16][ per entry: name_len u16 + UTF-8 ]
Record prefix (all three index types): [doc_id u64][field_id u16]   -- stride 12 → 10

• VERSION_FIELD_DICT = 3, MAX_SUPPORTED_VERSION = 3; monotone ladder (>=2 = HNSW ordinal graph, >=3 = dict records). HNSW's five stamp sites go 2→3; Flat/IVF stamp a version for the first time.
• field_id = first-appearance order over the emitted record sequence (HNSW = doc_id-sorted; Flat = buffer order; IVF = cluster order — its existing flatten pass walks all lists before the header is written, so the dict is complete in time for free).
• The issue's single-field field_id elision is rejected: it makes the stride dict-dependent across ~15 checked_capacity sites and the OnDemand offsets, to save 2 bytes/record. field_dict_count = 0 ⟺ 0 records stays the unambiguous empty encoding.
• Corruption: field_id >= dict_count → LaurusError::index; dict name_len is self-bounded by the u16 domain (reliability(vector/index): bound reader allocations sized from unverified on-disk header counts (generalize #791) #806 property without new plumbing); writers error on > u16::MAX names/entries. v1/v2 read paths stay byte-identical; old builds reading v3 get the clean range-gate IncompatibleFormat (deliberate forward break, perf(vector/hnsw): segment-local u32 ordinal graph + LVS1 v2 format (sub-issue of #686) #853 precedent).

PR structure (sub-issues under this tracking issue)

1. PR-A1 — format layer (inert, ~280 LOC): constants; field_dict: Vec<String> on VectorSegmentHeader (+ with_field_dict); version-gated serialized_size/write_to/read_from; shared helpers read_record_field() (collapses the 11 record-parse sites), record_prefix_size(version), build_field_dict(). Nothing emits v3 yet.
2. PR-A2+A3 — readers + writers (~850 LOC, commits ordered readers-first for bisect safety): readers accept v3 (by-ref quant-match refactor for flat/ivf — the parsed header currently does NOT survive to the flat lazy seek due to a partial move; fix the two lazy seeks that rebuild a fresh header; store payload-start offsets and delete the per-access name skip in storage.rs); writers build the dict at the natural pre-header points and emit u16 ids; writer load() re-parsers translate back; docs (4 module layout docs + persistence.md + vector_indexing.md, en/ja). Tests: v3 fixtures, corruption cases, exact file-size assertions (byte formulas per index type + the v1-delta formula — the deterministic headline gate), v1→v3 upgrade-on-rewrite, multi-field, empty segment.
3. PR-B — in-memory interning (~650 LOC, independent once A lands): internal (u64, u16) + per-segment Arc<[Arc<str>]> (v1/v2 files get the dict synthesized at load — legacy files benefit too); vector_ids/IVF cluster_to_vectors/OnDemand+Owned storage keys/pool pipelines; VectorStorage::get(doc_id, &str, dim) + get_by_field_id() with once-per-search resolution — structurally removes the per-candidate to_string() on the mmap-default path; boundary rehydration keeps every public trait signature unchanged. Substantially subsumes perf(vector/search): HNSW reader keeps vector_ids: Vec<(u64, String)> alongside vector_ids_by_field #682 and the hot-path half of perf(vector/search): vector_ids() clones full Vec<(u64, String)> per call #672 (to be recorded there on completion).

Verification

Primary gate = exact file-size assertions (deterministic, no bench noise). Eager gate benches (deletion / sparse_ids / Flat int8 kernel) must be neutral — the eager hot loop never touches names, so any delta is a bug. mmap/OnDemand improvements are claimed structurally (that class is the ±6–19% one). fmt / clippy 1.95+1.97 parity / full tests + pq-fastscan / wasm check / markdownlint.