perf: FieldId(u16) registry across crates (drop HashMap<String, _> from hot paths)

[mosuka]

Cross-cutting data-structure rewrite tracked under #537.

Current state

Field names are kept as String everywhere on the hot path:

• laurus/src/lexical/index/inverted/writer.rs:59: fields: HashMap<String, FieldOption>.
• laurus/src/lexical/index/inverted/writer.rs:621: format!("{field}:{term}") per token.
• laurus/src/vector/index/quantized_storage.rs:45: field_index: HashMap<String, Arc<HashMap<u64, u32>>>.
• laurus/src/vector/index/hnsw/reader.rs:64,560: prefetch_index: HashMap<String, _>, vector_ids: Vec<(u64, String)>.
• Per-vector field_name UTF-8 inline in segment records (quantized_io.rs:14-23).

Typical schemas have 1-10 fields per index; the String cost on every probe is pure waste.

Proposed direction

• Introduce FieldId(u16) and FieldRegistry { id_by_name: AHashMap<String, FieldId>, names: Vec<String> } at segment/index open.
• Lexical writer: Vec<(SmolStr, FieldOption)> indexed by FieldId.
• Vector storage pools: SmallVec<[Arc<DocIdMap>; 4]> indexed by FieldId.0 as usize.
• Per-segment field dictionary on disk (one string per id, varint-coded) so per-record payload references field by id only.
• String field names retained only at the public API boundary.

Acceptance

• Lexical writer hot path no longer calls format!.
• Vector storage records drop the inline field_name UTF-8 (replaced by field_id: u16).
• Per-query hashing on field name eliminated in HNSW/Flat/IVF search paths.

References

• Lucene FieldInfos (interns to int fieldNumber).
• Tantivy FieldId(u32).
• Qdrant payload schema ids.

[mosuka]

Phase 0 premise gate result: NO-GO (data-backed)

Before implementing, I profiled the claimed hot paths with perf (sample-weighted self%, rebuilt with CARGO_PROFILE_BENCH_DEBUG=2 RUSTFLAGS="-C force-frame-pointers=yes" so symbols resolve; perf record -g --call-graph=fp, perf report --no-children).

Lexical ingest (lexical_indexing_bench add_documents/10000):

symbol	self%
__memmove_evex_unaligned_erms (libc)	31.8%
add_analyzed_document_to_index (contains the per-token format!)	2.5%
malloc/free family	~2.7%
format! / fmt::	not in top 25

The per-token format!("{field}:{term}") this issue targets is <3% of ingest — dwarfed by memmove (Vec growth, ~32%). It fails the ">5% to be worth a perf change" bar.

Vector search (vector_search_bench): field-name hashing is per-query and ~0.01%; the ~3.5% SipHash is per-candidate u64 doc-id aggregation (the domain of #654), not field names.

On-disk phases: an adversarial design review found that the field-id-prefixed FST key (lexical) breaks FST sort order / find_prefix/find_range, and per-record field_id (vector) breaks OnDemand offsets keyed by (doc_id, String). Removing those hazards leaves only "embed a self-describing registry block" — which has no measurable perf win on its own.

Conclusion: in a back-compat-safe, correctness-preserving form, FieldId(u16) here is a <3% micro-optimization. Pursuing the real win would require trading correctness (FST ordering, merge field-id stability, OnDemand resolution) for a sub-3% gain — a bad trade. Recommend deferring / closing this as a low-ROI item, or rescoping it to a pure code-cleanliness change (not a perf change).

Higher-ROI target found by the gate: lexical ingest is dominated by __memmove (~32%) — posting-list Vec growth/reallocation — not field-name handling. That is the real ingest bottleneck and a better perf target (will be investigated separately).