Skip to content

feat: verify dataset compatibility and integrity#2

Merged
xr843 merged 13 commits into
masterfrom
agent/data-verify
Jul 10, 2026
Merged

feat: verify dataset compatibility and integrity#2
xr843 merged 13 commits into
masterfrom
agent/data-verify

Conversation

@xr843

@xr843 xr843 commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • validate dataset version, normalization protocol, required query schema, FTS5 trigram declaration, SQLite integrity, and external-content FTS consistency
  • add fojin data verify with human and JSON output while keeping normal query connections read-only
  • make data update download and fully validate an isolated candidate before atomically replacing live data
  • preserve the live database or a validated Windows recovery candidate across every download, validation, and replacement failure path
  • enforce the Rust 1.77 MSRV, locked installation, Windows data tests, and the four release targets in CI

Why

Previously, incompatible or internally inconsistent local data could fail only at query time, and data update removed the current database before a replacement was proven usable. The CLI also had no explicit health-check command.

User impact

fojin data verify now performs protocol, SQLite, and FTS checks without downloading data. Queries reject incompatible data before use. Updates preserve the current working dataset until a downloaded candidate passes all checks.

Validation

  • cargo fmt --all --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test --all --locked (95 tests)
  • cargo +1.77.0 test --all --locked (95 tests)
  • python3 -m pytest tests/ -q
  • cargo build --release --locked
  • Rust 1.77 locked cargo install --path .
  • real 587,984,896-byte dataset: full SQLite + FTS verification passed
  • independent whole-branch review: approved with no findings

@xr843
xr843 merged commit 0d533b5 into master Jul 10, 2026
17 checks passed
@xr843
xr843 deleted the agent/data-verify branch July 10, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant