perf(lexical/index): batch ingestion API (`add_documents` / `commit_batch`)

[mosuka]

Round-3 perf push sub-issue (tracked under umbrella #533).

[L] Batch ingestion API (add_documents / commit_batch) (parity:Tantivy)

• Where: laurus/src/lexical/store.rs:145-151 (single-doc upsert_document), laurus/src/engine.rs:170-272 (only single-doc put).
• Current behavior: Every external ingest call takes one document, walks the full stack (apply_dynamic_schema → log.append → store_document → lexical.upsert_document → vector.upsert_document), and acquires the writer mutex once per doc.
• Why it's a bottleneck/risk: Per-doc Mutex-acquire is a real cost at high QPS. The dynamic-schema policy resnapshots the schema's RwLock<Schema> once per doc. WAL fsync also happens once per doc.
• Reference precedent: Tantivy IndexWriter::add_document is the API but is intended to be called from N threads with a single shared writer; the commit is one fsync per batch. Lucene IndexWriter.addDocuments(Iterable) exists.
• Suggested direction: Add LexicalStore::upsert_documents(&self, batch: Vec<(u64, Document)>) -> Result<()> and a HybridEngine::put_batch. Inside, snapshot schema once, hold writer mutex once, WAL-batch all records under group-commit.
• Risk / scope: M. Pure addition; no on-disk format change.

---

---

ID: LI-14 — see ~/.claude/tasks/laurus/20260523_perf_round3_audit/task_list.md for the full Round-3 issue list.

Task list (2026-07-13 campaign start, jointly with #572)

Phase 0: Investigation

• Full ingestion/commit flow map (Engine put/add → WAL append incl. the perf(lexical/index): replace JSON WAL with length-prefixed binary records + group commit #542 group-commit layer → lexical/vector upserts → commit(): segment-creation conditions, files written, fsync count, WAL truncation, merge/compaction triggers; verify perf(lexical/index): no commit batching — LexicalStore::commit always writes a new segment #572's premise at line level)
• API surface survey (every existing add/put entry point across engine/stores/bindings; insertion point for a batch API; existing ingestion benches/tests; Tantivy/Lucene batch-API shapes)
• Opportunity ceiling (files/fsyncs/segments per commit today vs batched — deterministic metrics)
• Post investigation summary as an issue comment

Phase 1: Plan

• Scope/split decision (perf(lexical/index): batch ingestion API (add_documents / commit_batch) #551 API / perf(lexical/index): no commit batching — LexicalStore::commit always writes a new segment #572 batching / possible subsumption of perf(lexical/index): storage sync() on every commit even for in-memory backend #573, perf(lexical/index): inverted_index = TermPostingIndex::new() re-allocates AHashMap per flush #575) → plan comment. Decision: batch API only; commit_batch/CommitPolicy deferred until perf(vector/index): commit pipeline: write payload incrementally instead of one giant create_output at flush #634 (O(N²/c) .hnsw rewrite footgun). Sub-issues filed: feat(engine): batch ingestion API put_documents / add_documents with one WAL fsync per batch (#551 PR-1) #863 / perf(engine): retain vector writer across commits + lexical segment-meta cache (#551 PR-2) #864 / feat(server,cli): bulk document ingestion — gRPC PutDocuments/AddDocuments, HTTP bulk, MCP tool, CLI JSONL (#551 PR-3) #865 / feat(bindings): put_documents / add_documents across python, nodejs, wasm, ruby, php (#551 PR-4) #866

Phase 2: Implementation

• PR-1 (feat(engine): batch ingestion API put_documents / add_documents with one WAL fsync per batch (#551 PR-1) #863): Engine::put_documents / add_documents + fail-fast BatchIngest error + WAL deferred-fsync scope + tests/bench/docs — merged (PR feat(engine): batch ingestion API put_documents / add_documents with one WAL fsync per batch (#863) #867, main 771c6f80; fsync N→1 asserted by a cfg(test) counter; FileStorage bench ~36x at n=1000, 144→5,250 docs/s; also fixed a global-deferral hazard so concurrent singular writes keep per-record durability)
• PR-2 (perf(engine): retain vector writer across commits + lexical segment-meta cache (#551 PR-2) #864): vector writer retention across commits + lexical segment-meta cache + cached DeletionManager — merged (PR perf(engine): retain vector writer across commits + lexical segment-meta cache (#864) #869, main 0d95906c; closed perf(lexical/index): no commit batching — LexicalStore::commit always writes a new segment #572, perf(lexical/index): InvertedIndexWriter::find_segments_for_doc scans all .meta per upsert #559, perf(lexical/index): mark_persisted_doc_deleted constructs a fresh DeletionManager per call #571; .hnsw reload per commit cycle → 0, per-upsert .meta scans → 0, deterministic counter + fault-injection tests; a pre-commit adversarial review hardened 5 defect classes incl. error-path invalidation and a has_pending_changes() gate that also eliminates the full .hnsw rewrite on no-change commits; discovered pre-existing HNSW reachability bug filed as bug(vector/hnsw): incremental graph append can leave layer-0 nodes unreachable (nondeterministic recall loss across commit cycles) #868)
• PR-3 (feat(server,cli): bulk document ingestion — gRPC PutDocuments/AddDocuments, HTTP bulk, MCP tool, CLI JSONL (#551 PR-3) #865): gRPC PutDocuments/AddDocuments + HTTP bulk + MCP tool + CLI JSONL ingest
• PR-4 (feat(bindings): put_documents / add_documents across python, nodejs, wasm, ruby, php (#551 PR-4) #866): bindings ×5 + docs (closes perf(lexical/index): batch ingestion API (add_documents / commit_batch) #551)

Wrap-up

• Comment on perf(lexical/index): single fused write call instead of 7 separate per-segment files #554 (workload impact shrinks; structural issue remains) and close perf(lexical/index): storage sync() on every commit even for in-memory backend #573 (weak premise: memory-backend sync() is a no-op)

[mosuka]

Phase 0 investigation report (campaign with #572)

Method: two parallel line-level code sweeps (ingestion/commit flow; API surface + bindings + config + docs).

Premises verified

• perf(lexical/index): no commit batching — LexicalStore::commit always writes a new segment #572 holds: any commit with ≥1 buffered doc writes exactly one new lexical segment (inverted/writer.rs:1306-1308). Correction to the framing: segments do not grow unboundedly — the commit-path auto-merge caps them at max_segments = 100 (default), so the steady state under tiny commits is oscillation at the cap = constant merge pressure, proven by lexical_auto_merge_test.rs.
• perf(lexical/index): single fused write call instead of 7 separate per-segment files #554 holds: a lexical commit writes .post/.dict/.docs/.lens/.fstats/.dv/.meta + per-field .bkd + index.meta + metadata.json; on the file backend every file close is an fsync, plus ≥4 directory syncs across Engine::commit.
• perf(lexical/index): InvertedIndexWriter::find_segments_for_doc scans all .meta per upsert #559/perf(lexical/index): LexicalStore::find_doc_ids_by_term does a full-scan search per upsert #567 hold: every single upsert_document lists and JSON-parses every *.meta file (writer.rs:1504-1530) — a per-document cost on the hot ingest path (a fresh monotonic id matches nothing, but the scan still runs); a fresh DeletionManager is built per call (:1469).
• perf(lexical/index): storage sync() on every commit even for in-memory backend #573 is weak: the directory sync() is a no-op on the memory backend (memory.rs:351-355) — the calls are issued but free. Close candidate.

New finding — the largest write-side cost

The vector store is single-segment and (a) every commit rewrites the entire .hnsw (all vectors + graph, hnsw/writer.rs:1341-1631), and (b) the writer cache is take()n on commit (vector/store.rs:348), so the first upsert after every commit reloads and dequantizes the whole .hnsw (writer.rs:247-257 → 474-695). Under a commit-per-few-docs workload that is O(index size) per cycle. The full rewrite belongs to #634; the reload is independently fixable by retaining the writer across commits (invariants to be checked in the plan).

Deterministic opportunity ceiling (N docs, 1 commit vs N commits)

segments N→1; files written 10N→10; fsyncs 12N→12; .hnsw rewrites N→1; .hnsw reloads N−1→0 (0 forever with writer retention); WAL truncations N→1. All verifiable by counting files/calls — no bench-noise gate needed (the #633 deterministic-gate pattern applies). Ingestion benches already exist (lexical_indexing_bench bulk_ingest/multi_segment_commit, vector_indexing_bench, mutation_bench).

API surface facts

No plural write API exists at any layer (engine / stores / 5 bindings / gRPC / HTTP / MCP / CLI). The all-layer precedent for a plural method is search_batch (engine.rs:1745 empty-guard + per-item delegation; proto repeated; per-binding array marshalling). Commit is explicit-only; WalSyncPolicy (#542) is the policy-enum + builder-knob precedent; VectorIndexWriterConfig already carries unused auto-flush knobs. Docs to touch: persistence/engine/api_reference/quickstart/concepts + 5 binding pages + server/MCP/CLI (en + ja mirrors).

Candidate scope (to be fixed in the plan)

1. perf(lexical/index): batch ingestion API (add_documents / commit_batch) #551 core: Engine::put_documents/add_documents (search_batch shape, per-doc via index_internal) + tests; commit_batch as sugar vs a CommitPolicy (threshold auto-commit, Tantivy RAM-budget precedent) is the main design decision.
2. perf(lexical/index): no commit batching — LexicalStore::commit always writes a new segment #572 made real: vector writer retention across commits (reloads → 0) + a cached segment-meta view to kill the per-doc .meta scan (subsumes perf(lexical/index): InvertedIndexWriter::find_segments_for_doc scans all .meta per upsert #559; invalidate on commit/merge).
3. Fan-out (bindings ×5, gRPC AddDocuments, HTTP, MCP, CLI JSONL ingest, docs en/ja) as a separate PR.
4. perf(lexical/index): storage sync() on every commit even for in-memory backend #573 → close candidate (weak premise); perf(lexical/index): inverted_index = TermPostingIndex::new() re-allocates AHashMap per flush #575 small/optional; the .hnsw full-rewrite itself stays with perf(vector/index): commit pipeline: write payload incrementally instead of one giant create_output at flush #634.

Recommendation: GO — deterministic verification, low API risk via the established plural-method precedent, and the two real per-doc offenders identified. Plan (scope/PR split) follows.

[mosuka]

Implementation plan (Phase 1 complete — approved)

Scope decision: batch API only. commit_batch sugar and a CommitPolicy auto-commit knob are deferred until #634: every commit currently rewrites the entire .hnsw, so auto-committing every c docs during an N-doc ingest costs O(N²/c) vector bytes — a footgun until vector commits are incremental. The CLI's --commit-every (PR-3) prototypes the policy shape client-side.

Verified invariants the design rests on

1. Vector writer retention is sound: commit() = idempotent finalize() + non-consuming write(&self); build_hnsw_graph appends incrementally to the existing graph; post-commit writer memory state equals the file it just wrote. delete_document only invalidates the graph when a vector was actually removed, so fresh monotonic ids keep the append path.
2. Resurrection hazard: maybe_auto_compact() → optimize() loads a fresh writer, drops deletions, rewrites .hnsw, and clears the delmap — a retained writer would re-write deleted vectors with no delmap. maybe_auto_compact() returning Ok(bool) is the exact invalidation signal; VectorStore::optimize() must also clear the cache.
3. WAL deferral works with the perf(lexical/index): replace JSON WAL with length-prefixed binary records + group commit #542 split (append_record_bytes without fsync + flush_writer): a deferral scope suppresses per-record fsync during a batch; one flush_wal() at batch end (success and error paths). A literal append_batch is wrong — put's delete-first interleaves Delete records, and hoisting appends breaks replay order for in-batch duplicate ids.
4. Replay is per-doc idempotent (recorded doc_id overwrite), so a crash mid-batch recovers exactly the fsync'd prefix. Fail-fast error semantics (BatchIngest { failed_index, failed_id, applied, source }) with no batch rollback — same durability contract as N singular puts, documented explicitly.
5. Sequential processing (not search_batch's try_join_all): WAL doc_id/seq allocation order is the replay order, in-batch dedup requires program order, and both stores serialize behind mutexes anyway.

PR ladder (sub-issues filed)

PR	Issue	Scope	Est.
PR-1	#863	Engine::put_documents/add_documents + fail-fast BatchIngest + WAL deferred-fsync scope + tests/bench/docs	~550-700
PR-2	#864	Vector writer retention across commits + lexical segment-meta cache + cached DeletionManager (closes #572, #559, #571)	~500-650
PR-3	#865	gRPC PutDocuments/AddDocuments (unary repeated) + HTTP bulk + MCP tool + CLI JSONL ingest	~550
PR-4	#866	Bindings ×5 + docs (closes #551)	~600-750

Verification (deterministic counters as the primary gate, per the #633 pattern)

CountingStorage test decorator (per-file open_input/create_output/list_files counts): .hnsw opens must not grow across commit cycles; list_files = 0 per fresh-id upsert. A cfg(test) WAL fsync counter asserts the N→1 amortization exactly. Existing regression net (lexical_upsert_dedup / hnsw_append / vector_hnsw_auto_compaction / vector_hnsw_soft_delete) stays green unchanged. Benches (FileStorage backend) provide docs/sec as supporting evidence only.

Adjacent issues

#567 stays open (per-upsert find_doc_ids_by_term search is a different path, not removed by batching); #554 stays open (per-commit file count unchanged structurally; batching shrinks the workload impact); #573 recommended close at campaign end (weak premise — directory sync() is a no-op on the memory backend); #575 stays open; #634 stays open (PR-2 removes only the reload, not the rewrite).