GigaMap concurrency handling improvements and eventual indexing#564
Open
GigaMap concurrency handling improvements and eventual indexing#564
Conversation
…rIndex - Added `BackgroundIndexingManager` interface to manage background graph indexing. - Implemented a default asynchronous queue-based manager for deferred HNSW operations. - Added unit tests to validate eventual indexing behavior across add, update, remove, and bulk operations.
- Introduced stress test cases to validate VectorIndex under heavy concurrent operations. - Tests cover various configurations, including in-memory and on-disk setups with/without PQ compression and background tasks. - Ensured thread safety via assertions for no exceptions or deadlocks. - Included targeted eventual indexing and heavy load scenarios for robustness.
…ndex - Introduced test cases to evaluate insertion performance (single and batch adds) between synchronous and eventual indexing modes. - Verified search quality to ensure correctness of deferred graph indexing. - Included detailed metrics on caller-visible speedup and indexing throughput.
… execute() - Replaced the applyOperation method with per-operation execute() implementations in IndexingOperation types. - Simplified indexing logic by encapsulating operation-specific behavior within each record. - Improved maintainability and reduced duplication in BackgroundIndexingManager.
…ctorIndexConfiguration
- Replaced `persistenceLock` with `builderLock` for unified read/write access control over builder operations. - Introduced deferred operation handling for sync-mode mutations during cleanup phases. - Improved thread-safety for concurrent graph updates by coordinating via builder read/write locks. - Added `cleanupInProgress` flag and `deferredBuilderOps` queue to manage in-flight operations during cleanup and persistence tasks. - Removed redundant `synchronized(parentMap)` calls to avoid lock-ordering issues.
…ectorIndexConfiguration with examples and configuration details.
…xamples and configuration guidance in VectorIndex JavaDoc.
…on in VectorIndex and add comment
… and BackgroundPersistenceManager into unified BackgroundTaskManager.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BackgroundTaskManagerfor simplified task handling.VectorIndex.VectorEntryfor consistency and removed redundant checks inVectorIndex.4.0.0-rc.8.