perf(vector/index): replace `HashMap<(u64, String), u32>` field/doc->position index with field-id + dense ordinal lookup

[mosuka]

Round-3 perf push sub-issue (tracked under umbrella #535).

[M] Replace HashMap<(u64, String), u32> field/doc->position index with field-id + dense ordinal lookup

• Where:
  laurus/src/vector/index/quantized_storage.rs:45,77-93 and
  laurus/src/vector/index/pq_storage.rs:43,72-93.

• Current behaviour: field_index: HashMap<String, Arc<HashMap<u64, u32>>>. Looking up a candidate's position requires hashing the
  String field name once per search (cached, OK) then hashing u64
  doc_id per candidate. Hot loop calls record_at(pos); the inner map
  is HashMap<u64, u32> which is ~28 B per entry plus rehash growth.

• Why it might be a bottleneck: HashMap<u64, u32> is 3-4x larger
  than the underlying ids, and dirties cache lines that the SIMD
  distance kernel just brought in. For a 10 M segment, ~280 MB of map
  overhead.

• Reference precedent: Lucene uses dense int ordinals (a int[]
  indexed by docID gives the vector ordinal). Qdrant uses dense
  IdTracker with Vec<PointOffset> keyed by internal id.

• Current data structure:

  field_index: HashMap<String, Arc<HashMap<u64, u32>>>
                 │            │
                 │            └── per-(doc_id) entry: 12 B + bucket pad
                 └── per-field heap allocation
  

• Proposed layout:

  field_dict   : Vec<String>             // small, sorted, ~10 entries
  field_id_of  : SmallMap<String, u16>   // built once
  doc_to_pos   : Vec<u32>                // length = max_internal_id + 1
                                         // (or RoaringBitmap → range coded)
  ─ doc_to_pos[ord(field_id, doc_id)] = position in payload column ─
  

  Use a perfect-hash or sorted (field_id, doc_id) → u32 array for
  multi-field. For single-field (the common case) the index collapses
  to Vec<u32> indexed directly by internal id.

• Suggested direction: combine with Issue Add parallel index #2 columnar layout so the
  reader builds this dense map from the doc_ids + field_ids columns
  in one O(N) pass. Hot loop becomes one indexed load (cache friendly)
  instead of a hash lookup.

• Risk / scope: needs internal-id allocation strategy (mapping
  external u64 doc_ids to dense u32 ordinals); a no-op when external
  IDs are already dense.

---

---

ID: VI-08 — see ~/.claude/tasks/laurus/20260523_perf_round3_audit/task_list.md for the full Round-3 issue list.

[mosuka]

Substantially superseded by #853 (PR #854) and #862: the HNSW search hot path no longer performs any per-candidate doc->position map probe at all — the ordinal traversal indexes pool records directly (identity or one array read), and the remaining per-field position maps (HashMap<String, Arc<HashMap<u64, u32>>>) are resolved once per search, off the per-record path this issue targeted. The dense-ordinal idea this issue proposed is exactly what the ordinal design delivered.