-
Notifications
You must be signed in to change notification settings - Fork 26
fix(health): rebuild_index must write embeddings to embedding_index #173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -183,7 +183,7 @@ def rebuild_index(store: KBStore) -> dict: | |
|
|
||
| def _rebuild_embeddings(store: KBStore) -> None: | ||
| try: | ||
| from .embeddings import get_embedder | ||
| from .embeddings import content_hash, get_embedder | ||
| embedder = get_embedder() | ||
| except Exception: | ||
| return | ||
|
|
@@ -201,9 +201,20 @@ def _rebuild_embeddings(store: KBStore) -> None: | |
| for i in range(0, len(texts), batch_size): | ||
| batch = texts[i:i + batch_size] | ||
| vecs = embedder.encode_batch([t[2] for t in batch]) | ||
| for (kind, eid, _), row in zip(batch, vecs, strict=True): | ||
| index_db.index_embedding(conn, kind=kind, id=eid, | ||
| vec=row.tolist()) | ||
| for (kind, eid, text), row in zip(batch, vecs, strict=True): | ||
| # Write to `embedding_index` — the table `search_embedding()` | ||
| # reads and `reset()` (called at the top of rebuild) clears. | ||
| # `index_embedding()` targeted the legacy `embeddings` table, | ||
| # which no reader queries, so every rebuild/reindex silently | ||
| # left semantic search empty until each artifact was next | ||
| # re-written. Mirror the live `KBStore._embed_and_store` | ||
| # metadata so the per-content/model skip-check stays consistent. | ||
| index_db.put_embedding( | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Useful? React with 👍 / 👎. |
||
| conn, kind=kind, id=eid, vec=row.tolist(), | ||
| content_hash=content_hash(text), | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For rebuilt pages/entities, this stores a hash for the Useful? React with 👍 / 👎. |
||
| model=embedder.name, model_version=embedder.version, | ||
| dim=embedder.dim, | ||
| ) | ||
|
|
||
|
|
||
| # --- helpers used by `vouch discover` (CLI) ------------------------------- | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reset()has already deletedembedding_index, but this newput_embeddingloop only writes the claim/page/entity tuples assembled above. The live write path andsearch_embedding()supportsource,relation, andevidenceembeddings, so after a plainkb.index_rebuild/import-apply those rows are lost and semantic-only searches over those artifact types return nothing until a separatekb.reindex_embeddings/rewrite. Please rebuild all artifact kinds that_embed_and_storecreates.Useful? React with 👍 / 👎.