feat(engine): batch ingestion API put_documents / add_documents with one WAL fsync per batch (#863)#867
Merged
Merged
Conversation
…one WAL fsync per batch (#863) Add Engine::put_documents / add_documents, batched forms of the singular write methods. Documents are applied sequentially in input order (WAL doc_id/seq allocation order is the replay order; in-batch duplicate-id dedup requires program order) and fail fast with the new LaurusError::BatchIngest { failed_index, failed_id, applied, source }. There is no batch rollback: the applied prefix stays in the WAL and NRT buffers, so retrying the batch or its suffix is idempotent. Under the default WalSyncPolicy::PerRecord the whole batch is made durable with a single WAL fsync at batch end instead of one per record: DocumentLog::defer_sync() opens an RAII sync-deferral scope that suppresses the per-record fsync, and the engine calls flush_wal() once on both the success and error paths. Group policy thresholds keep firing mid-batch, preserving that policy's bounded loss window. The deferral flag is global, so singular writes acknowledged during a concurrent batch would silently lose their per-record durability; put_document / add_document / delete_documents now re-assert it via DocumentLog::ensure_per_record_durability() before returning (a no-op in the common no-batch path). Verification: a cfg(test) fsync counter asserts the N->1 amortization exactly; integration tests cover uncommitted-batch recovery after reopen and applied-prefix recovery of a failed batch. On the FileStorage bench backend, ingest/bulk_batch_api/1000 runs ~36x faster than the per-record ingest/bulk/1000 (190 ms vs 6.9 s, fsync-bound). Refs #551, #572
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the batch ingestion API (#551 PR-1 of the campaign, jointly addressing #572's workload impact):
Engine::put_documents(docs: Vec<(String, Document)>)/add_documents(...)— batched forms of the singular methods, applied sequentially in input order (WAL doc_id/seq allocation order is the replay order; in-batch duplicate-id dedup requires program order; both stores serialize writes behind mutexes, sosearch_batch-style concurrency would buy nothing and cost determinism).WalSyncPolicy::PerRecord: a new RAII sync-deferral scope onDocumentLog(defer_sync()) suppresses the per-record fsync while a batch is in flight; the engine callsflush_wal()once at batch end on both the success and the error path.Grouppolicy thresholds keep firing mid-batch, so its bounded loss window is preserved.LaurusError::BatchIngest { failed_index, failed_id, applied, source }. No batch rollback — the applied prefix stays in WAL + NRT buffers (searchable immediately, durable at next commit, replayed per-doc on crash), so retrying the batch or its suffix is idempotent under put semantics.put_document/add_document/delete_documents) now re-assert durability viaDocumentLog::ensure_per_record_durability()before acknowledging — a no-op in the common (no-batch) path, the load-bearing fsync during a concurrent batch. Regression-tested.Not included by design (user decision on #551):
commit_batchsugar and aCommitPolicyauto-commit knob — deferred until #634, because every commit currently rewrites the entire.hnsw, so auto-committing every c docs during an N-doc ingest costs O(N²/c) vector bytes.Verification (deterministic gates first)
cfg(test)fsync counter on the WAL writer (wal_sync_count()); a 50-doc batch increments it by exactly 1 (test_put_documents_single_wal_fsync_per_batch).test_put_document_dedupes_duplicate_ids_in_batch) / same-id chunks viaadd_documents/ fail-fast index+applied reporting / singular-write durability during a concurrent batch deferral.engine_batch_ingest_test.rs): acknowledged-uncommitted batch fully recovers after reopen / failed batch recovers exactly its applied prefix /add_documentschunks replay as chunks.wal_recovery_test,wal_crash_injection_test,dynamic_schema_test,engine_search_batch_test,lexical_upsert_dedup_test).--features laurus/embeddings-all) clean /laurus-wasmbuilds / markdownlint clean (en+ja docs).Throughput evidence (supporting only; the fsync counter is the gate)
ingest/bulkvsingest/bulk_batch_apiat n=1000,LAURUS_BENCH_DISK=1(temp-dir FileStorage on ext4/NVMe; comparative use only perbenches/common.rsguidance):ingest/bulk/1000(per-docadd_document+ commit)ingest/bulk_batch_api/1000(add_documents+ commit)The per-record path is fsync-bound (~6.9 ms/record on this host's ext4); the batch path pays one fsync + one commit. On the default MemoryStorage bench backend the two variants are expected to be equal (fsync is a no-op there).
Docs
docs/src/laurus/{engine.md, api_reference.md, persistence.md}+ ja mirrors: new Batch Ingestion section in persistence.md (ordering, fail-fast/no-rollback, durability, sizing guidance, Group interplay, concurrent singular writes).Closes #863. Refs #551, #572.