perf(vector/hnsw): segment-local u32 ordinal graph + LVS1 v2 format (#853)

[mosuka]

Summary

Implements #853: the HNSW read side moves from external u64 doc ids to segment-local u32 ordinals (rank in the ascending unique record doc-id sequence), removing every per-candidate hash probe from the search hot loop, and introduces the LVS1 header version 2 graph block that encodes those ordinals on disk.

Why this works where #709 (width-only narrowing) failed: the win is not the width — it is that ordinals turn the per-candidate SipHash HashMap<u64,u32> probe (quantized distance lookup), the per-hop AHashMap probe (get_neighbors), and the doc_id-keyed prefetch probe into plain array indexing, and let the per-query visited bitmap be sized by the segment's node count instead of the global doc-id space.

Design

• Reader: new OrdinalHnswGraph (validated at load); version-dispatched graph parse — v1 = legacy parse + rank translation (warn-and-drop for corrupt danglers), v2 = direct HashMap-free build with hard validation. ord_to_pos is identity for single-field segments (every current producer); legacy multi-field segments get an explicit table with u32::MAX = absent, preserving the perf(vector/search): VectorSearchParams.fields (field selector) ignored — no per-field routing #676 field-routing semantics.
• Searcher: ordinal traversal with 8-byte frontier candidates, BitVec(node_count) visited set, probe-free distance_ord, direct-address int8 prefetch. ResultCandidate, rerank, and the perf(vector/search): cardinality-driven mode selection for filtered HNSW (follow-up of #645) #738 brute-force mode stay u64; the perf(vector/search): no filter-aware HNSW graph traversal — allowed_ids is post-filter only #645 pristine/bookkeeping codegen split is preserved. Pass-1 prefetch loops are gated (prefetch_enabled) so no-prefetch storages skip them entirely — a small real regression caught and fixed during A/B.
• Writer: build internals stay doc_id-keyed (active-segment search is a flat scan); write() emits v2 with guards that refuse to serialize a corrupt block (node_count < u32::MAX, graph/record set equality); load() reads both versions, so compaction/merge upgrade v1 segments in place.
• Compat: Flat/IVF keep stamping v1 — older builds keep reading them; only new HNSW segments require this build. .delmap-style version dispatch precedent followed; the v2 graph block is ~2x smaller on disk.

Measured (A = main ×2 runs, B = this branch, medians, worst-pairing)

store-level gate benches (eager int8, ±0.06–3.5% instrument)	A	B	Δ
Deletion Search/top10/del0pct	89.1–89.7µs	34.6µs	−61%
Deletion Search/top10/del10pct	66.1–70.4µs	29.0µs	−56%
HNSW Graph Search sparse_ids/top10/10000 (new bench; ids from 50M — churned-store shape)	264.9µs	36.6µs	−86%
HNSW Multi-field Search (non-identity ord_to_pos guard)	159.7–162.0µs	147.9µs	−7%

B-side same-code spread on the deletion cases: ±1.7%. The sparse case is the visited-bitmap structural fix (the #647 premise): v1 zeroed ~6.3MB per query there, ordinals ~1.25KB.

Honest caveat: the mmap/OnDemand f32 benches (historical ±6–19% noise class) are mixed at 50k (+17% plain vs −0.2%/+20% on the same-traversal rerank case across two runs of the same/lighter code) — that class cannot resolve ±20% on this host and is flagged for post-merge observation. Dense 1k/5k, ef_16/ef_128 and multi-field all improved. Full A/B tables and the Phase 0a/0b gate record are in #853.

Testing

• 6 new migration tests, including the repo's first hand-built legacy-format fixture (v1 read, v1→v2 upgrade with version-byte assert, v2 corruption cases, empty graph, multi-field routing).
• lib 1204/1204; all 21 vector integration test files green (recall, filter-aware, deletion-aware, append, compaction, checksum, merge).
• fmt; clippy clean on 1.95 and 1.97 (workspace --all-targets --features embeddings-all + wasm target); cargo check clean for wasm32 and --features pq-fastscan.
• Docs: persistence.md (en/ja) documents the versioned segment headers; markdownlint clean.

Closes #853
Refs #686