GigaMap concurrency handling improvements and eventual indexing (#564)

* Bump jvector version to 4.0.0-rc.8 in pom.xml

* Add parallel on-disk write support to VectorIndex

* Introduce BackgroundIndexingManager for eventual consistency in VectorIndex

- Added `BackgroundIndexingManager` interface to manage background graph indexing.
- Implemented a default asynchronous queue-based manager for deferred HNSW operations.
- Added unit tests to validate eventual indexing behavior across add, update, remove, and bulk operations.

* Add concurrent stress tests for VectorIndex thread safety

- Introduced stress test cases to validate VectorIndex under heavy concurrent operations.
- Tests cover various configurations, including in-memory and on-disk setups with/without PQ compression and background tasks.
- Ensured thread safety via assertions for no exceptions or deadlocks.
- Included targeted eventual indexing and heavy load scenarios for robustness.

* Add performance tests for synchronous vs eventual indexing in VectorIndex

- Introduced test cases to evaluate insertion performance (single and batch adds) between synchronous and eventual indexing modes.
- Verified search quality to ensure correctness of deferred graph indexing.
- Included detailed metrics on caller-visible speedup and indexing throughput.

* Refactor BackgroundIndexingManager to delegate operation handling via execute()

- Replaced the applyOperation method with per-operation execute() implementations in IndexingOperation types.
- Simplified indexing logic by encapsulating operation-specific behavior within each record.
- Improved maintainability and reduced duplication in BackgroundIndexingManager.

* Update default value for parallel on-disk write to false in VectorIndexConfiguration

* Update tests for parallel on-disk write default change to false in VectorIndexConfiguration

* Refactor VectorIndex synchronization and builder operation handling

- Replaced `persistenceLock` with `builderLock` for unified read/write access control over builder operations.
- Introduced deferred operation handling for sync-mode mutations during cleanup phases.
- Improved thread-safety for concurrent graph updates by coordinating via builder read/write locks.
- Added `cleanupInProgress` flag and `deferredBuilderOps` queue to manage in-flight operations during cleanup and persistence tasks.
- Removed redundant `synchronized(parentMap)` calls to avoid lock-ordering issues.

* Documented `eventualIndexing` and `parallelOnDiskWrite` features in VectorIndexConfiguration with examples and configuration details.

* Document eventual indexing and parallel on-disk write features with examples and configuration guidance in VectorIndex JavaDoc.

* Remove redundant default case check for unsupported similarity function in VectorIndex and add comment

* Add braces to conditional statements in VectorEntry equals() for consistency

* Consolidate BackgroundIndexingManager, BackgroundOptimizationManager, and BackgroundPersistenceManager into unified BackgroundTaskManager.

* Gigamap jvector update tests updade (#569)

* move configuration tests to VICT file, remove duplicite tests.

* remove test, duplicate of testBackgroundOptimizationTriggersAfterIntervalAndThreshold

* test refactoring

* change indents - to see diff in PR

* reverts original formats

* add vector indices unit tests

* remove dulicite tests

---------

Co-authored-by: Zdenek Jonas <z.jonas@microstream.one>

Author: fh-ms

docs/modules/gigamap/pages/indexing/jvector/configuration.adoc

@@ -141,6 +141,10 @@ For datasets that exceed available memory, enable on-disk storage to use memory-
 |`pqSubspaces`
 |`0`
 |Number of PQ subspaces (0 = auto: dimension/4).
+
+|`parallelOnDiskWrite`
+|`false`
+|Use parallel direct buffers and multiple worker threads for on-disk index writing. Speeds up persistence for large indices but uses more resources. Only applies when `onDisk=true`.
 |===
 
 === Example
@@ -157,6 +161,55 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
     .build();
 ----
 
+== Eventual Indexing
+
+Enable eventual indexing to defer expensive HNSW graph mutations to a background thread, reducing mutation latency at the cost of eventual search consistency.
+
+[options="header",cols="1,1,3"]
+|===
+|Parameter |Default |Description
+
+|`eventualIndexing`
+|`false`
+|Defer HNSW graph mutations (add, update, remove) to a background thread. The vector store is updated synchronously, but graph construction happens asynchronously. Search results may not immediately reflect the most recent mutations.
+|===
+
+When enabled:
+
+* The vector store is always updated synchronously (no data loss).
+* HNSW graph mutations are queued and applied by a single background worker thread.
+* The queue is automatically drained before `optimize()`, `persistToDisk()`, and `close()`.
+
+=== Example
+
+[source, java]
+----
+VectorIndexConfiguration config = VectorIndexConfiguration.builder()
+    .dimension(768)
+    .similarityFunction(VectorSimilarityFunction.COSINE)
+    .eventualIndexing(true)
+    .build();
+----
+
+== Parallel On-Disk Writes
+
+When on-disk storage is enabled, persistence can optionally use parallel direct buffers and multiple worker threads (one per available processor) to write the index concurrently. This can significantly speed up persistence for large indices.
+
+This is disabled by default, as sequential single-threaded writing is preferred in resource-constrained environments or for smaller indices.
+
+=== Example
+
+[source, java]
+----
+VectorIndexConfiguration config = VectorIndexConfiguration.builder()
+    .dimension(768)
+    .similarityFunction(VectorSimilarityFunction.COSINE)
+    .onDisk(true)
+    .indexDirectory(Path.of("/data/vectors"))
+    .parallelOnDiskWrite(true)
+    .build();
+----
+
 == Background Persistence
 
 Enable automatic asynchronous persistence to avoid blocking operations during writes.

gigamap/jvector/README.md

@@ -10,6 +10,8 @@ A Java library that integrates [JVector](https://github.com/datastax/jvector) (h
 - **PQ Compression**: Product Quantization for reduced memory footprint
 - **Background Persistence**: Automatic asynchronous persistence at configurable intervals
 - **Background Optimization**: Periodic graph cleanup for improved query performance
+- **Eventual Indexing**: Deferred graph mutations via background thread for reduced write latency
+- **Parallel On-Disk Writes**: Multi-threaded index persistence for large on-disk indices
 - **Lazy Entity Access**: Search results provide direct access to entities without additional lookups
 - **Stream API**: Java Stream support for search results
 - **GigaMap Integration**: Seamlessly integrates with GigaMap's index system
@@ -163,6 +165,13 @@ List<Document> topDocs = result.stream()
 | `indexDirectory` | `null` | Directory for index files (required if `onDisk=true`) |
 | `enablePqCompression` | `false` | Enable Product Quantization compression |
 | `pqSubspaces` | `0` | Number of PQ subspaces (0 = auto: dimension/4) |
+| `parallelOnDiskWrite` | `false` | Use parallel direct buffers and multiple worker threads for on-disk index writing. Speeds up persistence for large indices but uses more resources. Only applies when `onDisk=true` |
+
+### Eventual Indexing
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `eventualIndexing` | `false` | Defer HNSW graph mutations to a background thread. The vector store is updated synchronously, but graph construction happens asynchronously. Reduces mutation latency at the cost of eventual search consistency |
 
 ### Background Persistence
 
@@ -223,6 +232,38 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
     .build();
 ```
 
+### Eventual Indexing
+
+For high-throughput systems where mutation latency matters more than immediate search consistency:
+
+```java
+VectorIndexConfiguration config = VectorIndexConfiguration.builder()
+    .dimension(768)
+    .similarityFunction(VectorSimilarityFunction.COSINE)
+    // Eventual indexing (graph mutations deferred to background thread)
+    .eventualIndexing(true)
+    .build();
+```
+
+When enabled, the vector store is always updated synchronously (no data loss), but expensive HNSW graph mutations are queued and applied by a background worker thread. Search results may not immediately reflect the most recent mutations. The queue is automatically drained before `optimize()`, `persistToDisk()`, and `close()`.
+
+### Parallel On-Disk Writes
+
+For large on-disk indices where persistence speed is critical:
+
+```java
+VectorIndexConfiguration config = VectorIndexConfiguration.builder()
+    .dimension(768)
+    .similarityFunction(VectorSimilarityFunction.COSINE)
+    .onDisk(true)
+    .indexDirectory(Path.of("/data/vectors"))
+    // Parallel on-disk writing (multiple worker threads)
+    .parallelOnDiskWrite(true)
+    .build();
+```
+
+When enabled, the on-disk graph writer uses parallel direct buffers and multiple worker threads (one per available processor) to write the index concurrently. This is disabled by default as sequential writing is preferred in resource-constrained environments or for smaller indices.
+
 ### Manual Optimization and Persistence
 
 ```java

gigamap/jvector/pom.xml

@@ -19,7 +19,7 @@
     <url>https://projects.eclipse.org/projects/technology.store</url>
 
     <properties>
-        <jvector.version>4.0.0-rc.7</jvector.version>
+        <jvector.version>4.0.0-rc.8</jvector.version>
     </properties>
 
     <dependencies>
@@ -44,6 +44,12 @@
             <artifactId>junit-jupiter-engine</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.awaitility</groupId>
+            <artifactId>awaitility</artifactId>
+            <version>4.2.2</version>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
 
     <build>