Skip to content

store: convert every pre-convention table to SQLite STRICT (v27)#183

Merged
mbertschler merged 2 commits into
mainfrom
claude/issue-148-9o6mmn
Jul 25, 2026
Merged

store: convert every pre-convention table to SQLite STRICT (v27)#183
mbertschler merged 2 commits into
mainfrom
claude/issue-148-9o6mmn

Conversation

@mbertschler

Copy link
Copy Markdown
Owner

Closes #148.

Both halves of the issue in one PR: the new-table convention (already in AGENTS.md and honoured since v25's destination_alarms) is now enforced by a test, and the eighteen tables that predated it are converted in bulk by a dedicated migration.

What STRICT buys

STRICT rejects any value whose storage class can't be losslessly converted to the column's declared type, instead of coercing it through type affinity. The schema was already STRICT-compatible — only INTEGER/TEXT/BLOB columns, ns-integer timestamps, INTEGER … CHECK (x IN (0,1)) booleans, fixed-length BLOB hashes — so no row in a healthy database changes. What changes is the failure mode of a future mishap (a stray string bound into size_bytes, a concatenated query, a reflection accident): a hard error at the write instead of a wrong-storage-class row that reads back as garbage. Belt-and-suspenders over the existing CHECK / NOT NULL / append-only-trigger discipline, in the same integrity-first spirit — not a fix for a known bug.

The migration (store/migrate_v27.go)

STRICT can't be switched on with ALTER, so each table is rebuilt with the standard recipe: create a STRICT copy, INSERT … SELECT every row, drop the original, rename the copy into place, recreate its indexes and triggers. Table bodies are transcribed verbatim from v26 with STRICT appended and nothing else changed — same columns in the same order (the ALTER-appended ones keep their trailing positions), same CHECKs, same FKs, same defaults.

Safety properties:

  • One transaction with foreign keys off — off because the drops and renames would otherwise trip the dense FK graph mid-rebuild, and because a rename with FKs enabled would rewrite other tables' REFERENCES clauses to point at the scratch name. PRAGMA foreign_key_check runs before commit, exactly as the v4→v5 and v24→v25 rebuilds do.
  • Per-table row counts compared across the copy, so a short copy fails the migration instead of quietly losing observations.
  • Every rebuild is self-contained (create → copy → drop → rename → recreate objects), so the rename — which makes SQLite reparse every trigger in the schema — never sees a half-converted database.
  • A table the chain never materialised is skipped rather than diagnosed here, which is what the older minimal migration fixtures in this repo rely on.
  • No history is touched: ids are preserved, contents keeps both append-only triggers, files keeps uniq_files_live_per_path, and runs is copied whole.

Cost, called out in the code: it rewrites the entire database, so it's O(index size) in time and needs room for a second copy of the largest table. One-time at upgrade, with Open's automatic pre-migration snapshot as the rollback surface.

schema_version is rebuilt too, and the bootstrap CREATE TABLE IF NOT EXISTS in migrate() now carries STRICT for fresh databases (a no-op on anything that predates v27, which the migration converts instead).

Tests

  • TestSchemaIsAllStrict — makes the AGENTS.md convention executable: any non-STRICT table at SchemaVersion fails, so a future migration that forgets the keyword is caught.
  • TestStrictRebuildsCoverEveryV26Table — pins the conversion list against the non-STRICT tables a v26 database actually carries, so a forgotten or stale entry fails.
  • TestMigrateV26ToV27PreservesEveryRow — drives the real chain to v26 (not a hand-written fixture), seeds every table, then opens with the current binary and checks per-table row counts, ids, a blob byte-for-byte with its storage class, the files statuses, the full set of indexes and triggers, and a clean foreign_key_check.
  • TestMigrateV26ToV27KeepsSchemaGuarantees — the contents append-only triggers and the one-live-row-per-path partial unique index still bite after the rebuild, STRICT now refuses a mistyped bind, and a well-typed insert still works.

Also in the diff

  • store/schema.sql regenerated (go test ./store -update-schema). Table names come back quoted — SQLite quotes the substituted name in the stored DDL after a rename — and the rebuilt bodies pick up uniform one-tab indentation, which makes the snapshot considerably easier to read than the deep-indented original.
  • AGENTS.md amended in the same PR, per its own rule: the "~13 tables are not STRICT yet" paragraph is replaced by the v27 statement and a pointer to the enforcing test.
  • cmd/squirrel/db_test.go expectations updated for the quoted names, plus a ) STRICT assertion.

go vet ./... and go test ./... are clean. golangci-lint could not run in this environment (the installed binary is built with go1.25, the module targets 1.26.1) — CI will cover it.


Generated by Claude Code

Closes #148.

STRICT rejects any value whose storage class can't be losslessly
converted to the column's declared type instead of coercing it through
type affinity. The schema was already STRICT-compatible — only
INTEGER/TEXT/BLOB columns, ns-integer timestamps, INTEGER CHECK (x IN
(0,1)) booleans, fixed-length BLOB hashes — so no row in a healthy
database changes. What changes is the failure mode of a future mishap (a
stray string bound into size_bytes, a concatenated query, a reflection
accident): a hard error at the write instead of a wrong-storage-class row.

STRICT can't be switched on with ALTER, so migrate_v27.go rebuilds each
of the eighteen tables that predate the convention: create a STRICT copy,
INSERT…SELECT every row, drop the original, rename the copy into place,
recreate its indexes and triggers. The sweep runs in one transaction with
foreign keys off — off because the drops and renames would otherwise trip
the dense FK graph, and because a rename with FKs on would rewrite other
tables' REFERENCES clauses to point at the scratch name — and verifies
PRAGMA foreign_key_check before commit, like the v4→v5 and v24→v25
rebuilds. Per-table row counts are compared so a short copy fails the
migration instead of losing observations, and a table the chain never
materialised is skipped rather than diagnosed here.

Tests: TestSchemaIsAllStrict makes the AGENTS.md convention executable
(any non-STRICT table at SchemaVersion fails, including in future
migrations); TestStrictRebuildsCoverEveryV26Table pins the conversion
list against the non-STRICT tables a v26 database actually carries;
TestMigrateV26ToV27PreservesEveryRow drives the real chain to v26, seeds
every table, and checks every row, id, blob, index and trigger survives
with the FK graph clean; TestMigrateV26ToV27KeepsSchemaGuarantees checks
the contents append-only triggers and the one-live-row-per-path index
still bite, and that STRICT now refuses a mistyped bind.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HompuyJMt84zb3KGZENkTF
Copilot AI review requested due to automatic review settings July 25, 2026 08:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR bumps the store schema to v27 and performs a one-time bulk migration to rebuild all pre-convention tables as SQLite STRICT, aligning the on-disk schema with the documented/enforced “all tables must be STRICT” convention.

Changes:

  • Add a v26→v27 migration (migrate_v27.go) that rebuilds all legacy non-STRICT tables as STRICT in a single FK-off transaction with foreign_key_check verification.
  • Enforce the “all tables STRICT at SchemaVersion” rule via new tests, plus additional migration safety/behavior tests.
  • Regenerate store/schema.sql, update AGENTS.md guidance, and adjust CLI schema-print expectations for quoted table names / STRICT DDL.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
store/schema.sql Regenerated schema snapshot at v27 reflecting STRICT tables and updated stored DDL formatting/quoting.
store/migrations.go Bumps SchemaVersion to 27, registers the v27 migration, and makes bootstrap schema_version creation STRICT.
store/migrate_v27.go New bulk rebuild migration converting all remaining legacy tables to STRICT and recreating dropped indexes/triggers.
store/migrate_v27_test.go Adds tests enforcing STRICT-at-head and validating the v26→v27 rebuild preserves rows and schema guarantees.
cmd/squirrel/db_test.go Updates CLI DDL expectations for quoted table names and STRICT output.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread store/migrate_v27_test.go Outdated
Comment thread store/migrate_v27_test.go
Two review points from the PR feed, both about the new test file:

- assertFilesRowsIntact scanned folder_id and content_id and then
  compared only name/status, so a swapped content_id would have passed.
  The binding of a path to a *particular* content is the observation, so
  compare the full (folder_id, name, content_id, status) tuple.
- The foreign_key_check loop counted rows without checking rows.Err() —
  a swallowed iteration error would have read as "clean graph". Extracted
  countFKViolations, which defers Close and fails on an iteration error.
  Also added the missing rows.Err() check to the files read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HompuyJMt84zb3KGZENkTF
@mbertschler
mbertschler merged commit 18829cc into main Jul 25, 2026
3 checks passed
@mbertschler
mbertschler deleted the claude/issue-148-9o6mmn branch July 25, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt SQLite STRICT tables in the index schema

3 participants