fix(scan): exclude per-scan extractedAt and warnings from enrichment stage hashes#352
Open
sricursion wants to merge 1 commit into
Open
fix(scan): exclude per-scan extractedAt and warnings from enrichment stage hashes#352sricursion wants to merge 1 commit into
sricursion wants to merge 1 commit into
Conversation
…stage hashes Every connector stamps a fresh extractedAt on each introspection, and the snapshot may carry per-run warnings. Both were folded into the descriptions, embeddings, and relationships stage hashes, so a rescan of an unchanged schema never hit the resume caches and re-sent every table to the LLM. Hash only schema-identifying snapshot content, mirroring what stableLiveDatabaseHashContent already does for the ingest work-unit cache. Fixes Kaelio#347 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@sricursion is attempting to deploy a commit to the Kaelio Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #347
Makes rescans of an unchanged schema hit the enrichment resume caches instead of re-sending every table and column description to the LLM.
Every connector stamps a fresh
extractedAton each introspection, and that timestamp (plus any per-runwarnings) was folded into the descriptions/embeddings/relationships stage hashes via the whole-snapshot hash. A rescan therefore always produced a newinputHash, both persistent caches (the SQLite stage cache andenrichment-progress/descriptions.json) missed, and enrichment restarted from zero — the repeated ~4-minute description runs in the issue.Approach
enrichment-state.tsnow hashes only schema-identifying snapshot content: a sharedstableSnapshotHashContenthelper stripsextractedAt(fresh wall-clock per scan) andwarnings(per-run diagnostics, e.g. one transient failed sample would otherwise re-key every stage) before hashing, and all three stage-hash functions route through it. This mirrors whatstableLiveDatabaseHashContentinlocal-stage-ingest.tsalready does for the ingest work-unit cache. No enrichment stage reads either field, so the hash loses no discriminating content, and a real schema change (added/renamed column) still re-keys.Existing caches re-key once after upgrading (the hashed content shape changed); rescans are stable from then on.
Tests
extractedAt/warningshash identically for all three stages, and an added column still re-keys all three. Fails onmain, passes here.distfrommain, a freshextractedAtre-keyed all three stages; with this change all three hashes are stable.type-check, CLI test suite, anddead-code(biome + knip default + production) pass. I developed on Windows, where ~60 test files fail onmainfor pre-existing environment reasons (sqliteEBUSYon temp-dir cleanup, doubledZ:\Z:\drive prefix in benchmark fixture paths); I diffed failing-file sets betweenmainand this branch on the same machine to confirm no new failures. Happy to file those Windows issues separately.Out of scope
Per-table checkpointing so a scope/schema change only regenerates changed tables — tracked in #348.