Skip to content

fix(scan): exclude per-scan extractedAt and warnings from enrichment stage hashes#352

Open
sricursion wants to merge 1 commit into
Kaelio:mainfrom
sricursion:fix/347-extracted-at-stage-hash
Open

fix(scan): exclude per-scan extractedAt and warnings from enrichment stage hashes#352
sricursion wants to merge 1 commit into
Kaelio:mainfrom
sricursion:fix/347-extracted-at-stage-hash

Conversation

@sricursion

@sricursion sricursion commented Jul 12, 2026

Copy link
Copy Markdown

Fixes #347

Makes rescans of an unchanged schema hit the enrichment resume caches instead of re-sending every table and column description to the LLM.

Every connector stamps a fresh extractedAt on each introspection, and that timestamp (plus any per-run warnings) was folded into the descriptions/embeddings/relationships stage hashes via the whole-snapshot hash. A rescan therefore always produced a new inputHash, both persistent caches (the SQLite stage cache and enrichment-progress/descriptions.json) missed, and enrichment restarted from zero — the repeated ~4-minute description runs in the issue.

Approach

enrichment-state.ts now hashes only schema-identifying snapshot content: a shared stableSnapshotHashContent helper strips extractedAt (fresh wall-clock per scan) and warnings (per-run diagnostics, e.g. one transient failed sample would otherwise re-key every stage) before hashing, and all three stage-hash functions route through it. This mirrors what stableLiveDatabaseHashContent in local-stage-ingest.ts already does for the ingest work-unit cache. No enrichment stage reads either field, so the hash loses no discriminating content, and a real schema change (added/renamed column) still re-keys.

Existing caches re-key once after upgrading (the hashed content shape changed); rescans are stable from then on.

Tests

  • New regression test: two snapshots differing only in extractedAt/warnings hash identically for all three stages, and an added column still re-keys all three. Fails on main, passes here.
  • Before/after check against the built package: with dist from main, a fresh extractedAt re-keyed all three stages; with this change all three hashes are stable.
  • type-check, CLI test suite, and dead-code (biome + knip default + production) pass. I developed on Windows, where ~60 test files fail on main for pre-existing environment reasons (sqlite EBUSY on temp-dir cleanup, doubled Z:\Z:\ drive prefix in benchmark fixture paths); I diffed failing-file sets between main and this branch on the same machine to confirm no new failures. Happy to file those Windows issues separately.

Out of scope

Per-table checkpointing so a scope/schema change only regenerates changed tables — tracked in #348.

…stage hashes

Every connector stamps a fresh extractedAt on each introspection, and the
snapshot may carry per-run warnings. Both were folded into the descriptions,
embeddings, and relationships stage hashes, so a rescan of an unchanged
schema never hit the resume caches and re-sent every table to the LLM.

Hash only schema-identifying snapshot content, mirroring what
stableLiveDatabaseHashContent already does for the ingest work-unit cache.

Fixes Kaelio#347

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 12, 2026

Copy link
Copy Markdown

@sricursion is attempting to deploy a commit to the Kaelio Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: ktx ingest regenerates every AI description on each scan — the scan timestamp extractedAt is inside the resume hash

1 participant