perf(engine): retain vector writer across commits + lexical segment-meta cache (#864)

[mosuka]

Summary

Removes the two verified per-doc write-path offenders (#864, PR-2 of the #551 campaign) — this is what makes #572's commit batching real:

1. Vector writer retention across commits. VectorStore::commit no longer take()s the cached writer when the index opts in via the new VectorIndex::retain_writer_after_commit() (default false; true for HnswIndex only — Flat/IVF keep today's drop-on-commit until audited). Pre-fix, the first upsert after every commit reloaded + dequantized the entire .hnsw; post-commit the writer's in-memory state equals the file it just wrote (idempotent finalize(), incremental graph append, non-consuming write(&self)), so the reload is pure waste. The writer-cache lock is now held across the commit ladder so a concurrent upsert cannot interleave.

The two paths that rewrite the index behind the writer both invalidate the cache — without this, a stale retained writer would write physically reclaimed vectors back with no deletion bitmap marking them:

• auto-compaction firing inside commit (maybe_auto_compact()? == true),
• VectorStore::optimize() (now async; commits the cached writer first when it holds buffered docs so nothing is lost, then always drops the cache). Only two test call sites needed .await; no other callers exist.

2. Lexical segment-metadata cache + cached DeletionManager (closes #559 and #571). Pre-fix, every upsert listed the storage and JSON-parsed every *.meta file (find_segments_for_doc), and every overwrite of a committed doc built a fresh DeletionManager (reloading every .delmap). Now InvertedIndexWriter keeps:

• segment_ranges + max_committed_doc_id — seeded by the constructor's existing recovery scan (zero extra I/O), extended in place by both flush paths. Fresh WAL doc ids exceed every committed max, so steady-state ingest answers "is this doc committed?" with one integer compare; overwrites resolve from memory.
• a lazily-created deletion_manager reused across overwrites (built on first need only — pure-ingest writers never pay the bitmap load).
• invalidate_segment_cache() (new default-no-op trait hook), called by LexicalStore::optimize on the live writer — force-merge is the only path that rewrites segments behind a live writer (commit drops the writer before merging). Without it, a post-optimize overwrite would mark its deletion in a ghost segment and the old version would resurface: the test pins hits("alpha") == 0.

Hardening from the pre-commit adversarial review

A 3-dimension × adversarial-verify review of the diff confirmed five defect classes in the first draft; all are fixed and regression-pinned in this PR:

1. Commit error path retained a stale writer (resurrection class): the cache-clear ran after fallible calls, so a failed maybe_auto_compact (which may have partially rewritten the index and cleared the delmap before erroring) left the stale writer cached — and made Flat/IVF flush failures sticky. Now any mid-ladder error drops the cache (the pre-retention behavior on every path). Pinned by failed_commit_drops_retained_writer (fault-injected create_output failure).
2. optimize()'s pending_docs() > 0 gate discarded deletions: a dirty writer whose buffer was emptied by deletions has pending_docs() == 0 but holds an uncommitted delete-everything mutation. New VectorIndexWriter::has_pending_changes() (default true; HNSW: !is_finalized) is the correct gate. Pinned by optimize_flushes_writer_emptied_by_deletions.
3. Every commit rewrote the .hnsw from the retained writer even with zero changes — a perf regression for no-op commits. The same has_pending_changes() gate skips the flush; pinned by noop_commit_does_not_rewrite_hnsw (create-count stays flat over back-to-back commits, rises again on a real change).
4. Lexical lock windows: LexicalStore::optimize ran the force-merge outside the writer lock (a concurrent upsert could mark deletions in segments being deleted), and commit released the lock before maybe_merge (a writer constructed mid-merge would seed its cache from a racing, inconsistent .meta scan and never be invalidated). Both now hold the writer-cache lock across the merge, mirroring the vector side; optimize also invalidates even when the merge errs partway, and drops the writer if the rebuild itself fails.
5. invalidate_segment_cache was not failure-atomic: it cleared the cache before the fallible rescan, so a list_files error left a live writer with an empty cache silently skipping every subsequent deletion. Now it builds the new view first and swaps on success. Pinned by invalidate_failure_preserves_old_cache (fault-injected list_files failure).

Verification (deterministic counters; fault-injection-proven)

• CountingStorage decorators: .hnsw open_input count stays flat across post-commit upserts and follow-up commits; 50 fresh-id upserts perform zero list_files; the first overwrite pays exactly the one-time DeletionManager scan, subsequent overwrites zero.
• Corruption tests fail without their fixes (verified by temporarily disabling each invalidation): auto-compaction resurrection flips document_count 61 → 101; the lexical optimize test resurrects the pre-merge doc version.
• Correctness: 3 retained-writer commit cycles preserve all 30 records live + after cold reopen; delmap lands only in the merged segment; existing regression net unchanged and green (soft-delete, auto-compaction, hnsw_append, upsert-dedup, optimize, auto-merge, deletion-logic, unified-engine, batch-ingest, wal-recovery, vector-deletion, merge-bkd, json-mirror — 44 integration tests incl. the 12 new counter/corruption tests, + 1229 lib tests).
• fmt / clippy 1.95 + 1.97 (--features laurus/embeddings-all) clean / laurus-wasm builds / markdownlint clean (en+ja docs).

Discovered while testing: pre-existing HNSW reachability bug → #868

The retention tests initially asserted exact search hit sets and flaked: incremental HNSW appends can nondeterministically leave nodes unreachable on layer 0. Reproduced on unmodified main (27eed28) — up to 6 of 30 docs missing after 3 commit cycles, varying per run, persisting across reopen, with all records present on disk. Filed as #868 with full evidence; the tests here gate on stats().document_count (record-level, exact) instead, and can be tightened back once #868 is fixed.

API note

VectorStore::optimize() is now async (it must take the writer-cache tokio mutex). Approved in the campaign plan; no callers outside the store besides two integration tests.

Docs

docs/{src,ja/src}/concepts/indexing/vector_indexing.md: new "Writer retention across commits" section.

Closes #864, closes #572, closes #559, closes #571. Refs #551, #868.