Skip to content

feat(faiss): best-effort per-file embedding statuses - #2762

Open
greglum wants to merge 2 commits into
SEMOSS:devfrom
greglum:feature/faiss-best-effort-embedding-status
Open

feat(faiss): best-effort per-file embedding statuses#2762
greglum wants to merge 2 commits into
SEMOSS:devfrom
greglum:feature/faiss-best-effort-embedding-status

Conversation

@greglum

@greglum greglum commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • embed each vector CSV independently so one bad document no longer fails the whole batch
  • report a per-file SUCCESS/PARTIAL/FAILED entry in a new documentStatuses list, with inserted/failed/total record counts and an error message on failure
  • roll a failed document back fully: its already-written per-Source artifacts are deleted and the in-memory vector state is restored, so no partially-indexed data survives under a FAILED label
  • downgrade a document to PARTIAL or FAILED when post-write validation removes its artifacts, instead of silently shrinking createdDocuments
  • build real per-file FileEmbeddingStatus entries in FaissDatabaseEngine from documentStatuses, keeping the previous index-level datasetsLoaded() check as a fallback when the response omits them

Why

#2205 made document embedding best-effort: every file is attempted and the reactor reports per-file FAILED/PARTIAL statuses. The FAISS engine bypasses that contract on its batched path — a single bad CSV raises out of Python and fails the entire batch, and when the batch completes the engine stamps every file SUCCESS or every file FAILED from one index-level boolean. Separately, artifacts that fail validation after writing are removed from createdDocuments without any signal to the caller. This PR extends the #2205 contract to FAISS.

Behavior change

Per-file embedding failures in the FAISS batch path now return FAILED statuses instead of raising — the same deliberate shift #2205 made for the documents path. Batch-level guards ahead of the loop are unchanged and still raise.

Depends on

Builds on #2759 (its first commit); only the top commit here is new. Review #2759 first.

Verification

  • 12 Python unit tests pass (7 new in test_faiss_best_effort_status.py; 5 in test_faiss_batch_vector_csv.py, of which one was updated because it asserted the old fail-fast behavior)
  • mvn compile succeeds on the full project
  • Black clean on changed files; Ruff reports only the six pre-existing diagnostics in faiss_client.py

greglum added 2 commits July 16, 2026 09:29
Embed each vector CSV independently: a document that fails is rolled
back (artifacts deleted, in-memory vectors restored) and reported as a
FAILED documentStatuses entry while the remaining documents still
embed. Artifacts removed by post-write validation now downgrade their
document to PARTIAL or FAILED instead of silently shrinking
createdDocuments. FaissDatabaseEngine builds per-file
FileEmbeddingStatus entries from documentStatuses, falling back to the
previous index-level datasetsLoaded check when the response omits them.
@greglum
greglum requested a review from a team as a code owner July 16, 2026 21:33
@snyk-io

snyk-io Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant