feat(faiss): best-effort per-file embedding statuses - #2762
Open
greglum wants to merge 2 commits into
Open
Conversation
Embed each vector CSV independently: a document that fails is rolled back (artifacts deleted, in-memory vectors restored) and reported as a FAILED documentStatuses entry while the remaining documents still embed. Artifacts removed by post-write validation now downgrade their document to PARTIAL or FAILED instead of silently shrinking createdDocuments. FaissDatabaseEngine builds per-file FileEmbeddingStatus entries from documentStatuses, falling back to the previous index-level datasetsLoaded check when the response omits them.
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
documentStatuseslist, with inserted/failed/total record counts and an error message on failurecreatedDocumentsFileEmbeddingStatusentries inFaissDatabaseEnginefromdocumentStatuses, keeping the previous index-leveldatasetsLoaded()check as a fallback when the response omits themWhy
#2205 made document embedding best-effort: every file is attempted and the reactor reports per-file FAILED/PARTIAL statuses. The FAISS engine bypasses that contract on its batched path — a single bad CSV raises out of Python and fails the entire batch, and when the batch completes the engine stamps every file SUCCESS or every file FAILED from one index-level boolean. Separately, artifacts that fail validation after writing are removed from
createdDocumentswithout any signal to the caller. This PR extends the #2205 contract to FAISS.Behavior change
Per-file embedding failures in the FAISS batch path now return FAILED statuses instead of raising — the same deliberate shift #2205 made for the documents path. Batch-level guards ahead of the loop are unchanged and still raise.
Depends on
Builds on #2759 (its first commit); only the top commit here is new. Review #2759 first.
Verification
test_faiss_best_effort_status.py; 5 intest_faiss_batch_vector_csv.py, of which one was updated because it asserted the old fail-fast behavior)mvn compilesucceeds on the full projectfaiss_client.py