diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index a10bb1476..743a564fe 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -51,9 +51,12 @@ Development contracts: | Mutating commands | `index`, `backfill-fold`, `optimize`, `vacuum` | These require writable storage and reject read-only database opens. | | Reusable index artifact | `cdidx export codeindex.cdidx.zip`; `cdidx import codeindex.cdidx.zip --db `; `cdidx import codeindex.cdidx.zip --dry-run --json` | Run export after indexing and upload the archive. Export refuses an existing destination unless `--overwrite` is explicit, publishes atomically from an owner-only temporary file, and verifies POSIX mode `0600`. Successful export JSON adds final archive byte size and SHA-256 plus the complete immutable manifest while retaining the prior result fields. Consumers import before query commands, or use `--dry-run` / `--check` to validate the archive without replacing the destination DB. Use `--prune-paths` when the archive comes from another checkout and the restored DB should advertise the import target project root; imports targeting `.../.cdidx/codeindex.db` use the sibling project directory, while other DB paths fall back to the process current directory. The archive contains only `manifest.json` plus `codeindex.db`; import validates ZIP entry names through `ZipArchiveSafetyPolicy` and rejects absolute, parent-directory, backslash, NUL, non-canonical, duplicate, and extra entries before extraction. The manifest carries bounded summary/readiness metadata including row counts, readiness bits, writer/indexed-head metadata, schema contract stamps, and unknown-extension summary when available. Import validates manifest format, manifest `user_version`, `database_sha256`, present summary counts, and the embedded SQLite file as a CodeIndex database before replacing the destination DB. Import rejects archive `codeindex.db` entries whose compressed or uncompressed metadata exceeds 8 GiB, and the extraction stream is also capped at 8 GiB. | | Maintenance checkpoint and managed rollback | `cdidx db checkpoint [--dry-run]`; `cdidx db checkpoints --list|--delete |--prune --keep [--dry-run]`; `cdidx db restore [--dry-run] [--no-backup]`; `cdidx db restore-backups --list|--prune --keep |--restore [--dry-run] [--no-backup]` | Checkpoint snapshots `codeindex.db` plus existing WAL/SHM sidecars before risky maintenance. Import and both restore forms create a consistent, verified managed SQLite rollback snapshot before replacing an existing DB unless `--no-backup` is explicit. Managed directories use `.restore-backup-/` and contain a bounded manifest plus one standalone database payload; the manifest records SHA-256, byte count, supported `user_version`, provenance, and an optional source identifier, but no local absolute source path. `restore-backups --list` exposes the ID and provenance while retaining legacy directory metadata; existing prune retention remains compatible. `restore-backups --restore ` revalidates the directory boundary, manifest, payload hash, schema, and combined staging/rollback free space, then performs an atomic replacement with transient rollback-on-failure. Its `--dry-run` reports every validation and planned backup without mutation. Checkpoint delete/prune and restore-backup prune require an explicit mutation action, and checkpoint prune skips all deletion if its bounded 1,000-directory scan is truncated. Checkpoints live under `.checkpoints//`. `backfill-fold` preflights folded rows, persisted folded values, and readiness under the index lock, creates an automatic checkpoint only when a mutation is required, accepts `--checkpoint` to force a snapshot for an already-complete DB, and accepts `--no-checkpoint` to skip mutation protection explicitly. JSON reports `checkpoint_skipped` plus `checkpoint_skipped_reason` (`already_complete`, `dry_run`, or `disabled_by_option`), and human output reports the same decision. | +| Checkpoint operation plan | `cdidx db checkpoint --dry-run [--json]` | Dry-run builds an immutable plan that separates source DB/WAL/SHM files and bytes from every output, including the versioned `manifest.txt`, its SHA-256, estimated final bytes, destination/conflict status and policy, sidecar/compression/metadata policy, and remaining uncertainty. Execution creates a fresh plan from current sources, validates source metadata and SHA-256 before and after copying, verifies every planned output, and refuses atomic publication when inputs drift. Legacy JSON `files` / `bytes` fields remain available; use `source_files` / `source_bytes`, `planned_output_files` / `estimated_output_bytes`, and `final_output_bytes` for the explicit contract. | | Binary compatibility | [COMPATIBILITY.md](COMPATIBILITY.md) | Database compatibility across `cdidx` binary upgrades and downgrades is documented there. Keep that policy updated whenever readiness bits, `codeindex_meta` contract stamps, or rebuild requirements change. | | Fold backfill preview and recovery | `backfill-fold --dry-run`; `backfill-fold --checkpoint`; MCP `backfill_fold` with `dry_run: true` or `force: true` | Dry-run previews folded-key rows without mutating the DB or stamping FoldReady. `--checkpoint` explicitly preserves a snapshot even when CLI preflight finds no mutation, while the default completed no-op leaves no checkpoint artifacts. MCP accepts the same preview and can force rewriting all folded keys when an operator needs to recover from suspicious fold metadata or row state even though the stored version/fingerprint appears current. Non-dry-run row rewrites are resumable after interruption: completed row updates remain durable, and final FoldReady metadata is stamped only after verification succeeds. MCP responses include `progress.rows_done`, `progress.rows_total`, and `progress.fraction` so clients can report and retry long backfills. | +Checkpoint plan drift detection covers DB/WAL/SHM content changes and sidecar appearance or disappearance through the final pre-publication validation. The plan's `uncertainty` value records the remaining post-validation race; copied outputs are independently hash-verified against the plan before atomic publication. DB/WAL/SHM candidates must pass native regular-file type validation before hashing so Unix FIFOs cannot block planning. `metadata_policy` reports `owner_only_files_and_directories` on POSIX and the actually inherited `inherited_windows_acls` policy on Windows. A database payload whose output name collides with `manifest.txt`, including filesystem-equivalent casing, makes the plan not ready and is rejected before mutation. + ## Filesystem Permissions | Artifact | POSIX permission and behavior | @@ -3406,9 +3409,12 @@ net9 CI lane に合わせる場合は `FRAMEWORK=net9.0 make test` を使いま | 変更系コマンド | `index`、`backfill-fold`、`optimize`、`vacuum` | 書き込み可能な storage を必要とし、read-only database open を拒否します。 | | 再利用可能な index artifact | `cdidx export codeindex.cdidx.zip`; `cdidx import codeindex.cdidx.zip --db `; `cdidx import codeindex.cdidx.zip --dry-run --json` | CI job では index 後に export して archive を upload します。export は `--overwrite` を明示しない限り既存 destination を拒否し、owner-only temporary file から atomic に publish して POSIX mode `0600` を検証します。export 成功時の JSON は従来 field を維持し、最終 archive の byte 数と SHA-256、完全で immutable な manifest を追加します。利用側は query コマンドの前に import でき、`--dry-run` / `--check` で destination DB を置き換えず archive を検証できます。別 checkout 由来の archive を import 先 project root として扱いたい場合は `--prune-paths` を使います。`.../.cdidx/codeindex.db` を import 先にした場合は sibling の project directory を使い、それ以外の DB path では process current directory に fallback します。archive は `manifest.json` と `codeindex.db` だけを含みます。import は ZIP entry 名を `ZipArchiveSafetyPolicy` で検証し、absolute path、parent-directory segment、backslash、NUL、non-canonical name、duplicate entry、extra entry を extraction 前に拒否します。manifest は row count、readiness bit、writer / indexed-head metadata、schema contract stamp、利用可能な unknown-extension summary などの bounded summary/readiness metadata を持ちます。import は manifest format、manifest `user_version`、`database_sha256`、存在する summary count、embedded SQLite file が CodeIndex database であることを検証してから destination DB を置き換えます。archive の `codeindex.db` entry は compressed / uncompressed metadata と extraction stream の双方で 8 GiB を上限に拒否されます。 | | maintenance checkpoint と managed rollback | `cdidx db checkpoint [--dry-run]`; `cdidx db checkpoints --list|--delete |--prune --keep [--dry-run]`; `cdidx db restore [--dry-run] [--no-backup]`; `cdidx db restore-backups --list|--prune --keep |--restore [--dry-run] [--no-backup]` | 危険な maintenance の前に `codeindex.db` と既存 WAL/SHM sidecar の checkpoint を作成できます。import と2種類の restore は、既存 DB を置き換える前に consistent かつ検証済みの managed SQLite rollback snapshot を既定で作成し、`--no-backup` を明示した場合だけ省略します。managed directory は `.restore-backup-/` で、bounded manifest と standalone database payload 1個を含みます。manifest は SHA-256、byte 数、対応する `user_version`、provenance、任意の source identifier を記録しますが、local absolute source path は記録しません。`restore-backups --list` は従来 directory metadata との互換性を維持しつつ ID と provenance を表示し、既存 prune retention もそのまま利用できます。`restore-backups --restore ` は directory 境界、manifest、payload hash、schema、staging と rollback を合わせた free space を再検証してから、失敗時の transient rollback を伴う atomic replacement を実行します。`--dry-run` は変更せず、すべての検証と作成予定 backup を報告します。checkpoint の delete / prune と restore-backup の prune は明示的な変更 action を必要とし、checkpoint prune の bounded scan が truncated の場合は削除をすべて skip します。checkpoint は `.checkpoints//` に置かれます。`backfill-fold` は index lock 内で folded row、永続化されたfolded値、readiness を事前確認し、mutation が必要な場合だけ automatic checkpoint を作ります。完了済みDBでもsnapshotを明示的に残すには `--checkpoint`、mutation protectionを明示的に省略するには `--no-checkpoint` を使います。JSON は `checkpoint_skipped` と `checkpoint_skipped_reason`（`already_complete`、`dry_run`、`disabled_by_option`）を返し、human output も同じ判断を表示します。 | +| checkpoint operation plan | `cdidx db checkpoint --dry-run [--json]` | dry-run は immutable な plan を作成し、source DB/WAL/SHM の file と byte 数を、version 付き `manifest.txt`、その SHA-256、最終 byte 数の見積もり、destination/conflict の状態と policy、sidecar/compression/metadata policy、残る不確実性を含む全 output から分けて報告します。実行時は現在の source から新しい plan を作成し、copy 前後の source metadata と SHA-256、および作成予定の全 output を検証して、入力が変化した場合は atomic publish を拒否します。従来の JSON `files` / `bytes` field は維持されます。明示的な contract には `source_files` / `source_bytes`、`planned_output_files` / `estimated_output_bytes`、`final_output_bytes` を使用してください。 | | binary compatibility | [COMPATIBILITY.md](COMPATIBILITY.md) | `cdidx` binary の upgrade / downgrade をまたぐ database compatibility を記載します。readiness bit、`codeindex_meta` contract stamp、rebuild requirement を変える場合は、この policy も更新してください。 | | Fold backfill の preview / recovery | `backfill-fold --dry-run`; `backfill-fold --checkpoint`; MCP `backfill_fold` の `dry_run: true` または `force: true` | dry-run は DB を変更せず FoldReady stamp も書かずに、rewrite 対象の folded-key row をプレビューします。CLI preflight でmutation不要と判断された場合でもsnapshotを明示的に保存するには `--checkpoint` を使います。既定の完了済みno-opはcheckpoint artifactを作りません。MCP も同じ preview を受け付け、stored version / fingerprint が current に見える場合でも suspicious な fold metadata や row state を復旧するため `force: true` を受け付けます。non-dry-run rewrite は中断後に resume でき、完了済み row update は durable に残り、最終 FoldReady metadata は verification 成功後にだけ stamp されます。MCP response は `progress.rows_done`、`progress.rows_total`、`progress.fraction` を含みます。 | +checkpoint plan の drift 検出は、publish 前の最終検証までに起きる DB/WAL/SHM content の変更と sidecar の出現・消失を対象にします。plan の `uncertainty` 値は最終検証後に残る race を記録し、copy 済み output は atomic publish 前に plan の hash と個別に照合されます。DB/WAL/SHM 候補は hash 読み取り前に native regular-file type validation を通すため、Unix FIFO が plan 作成を停止させることはありません。`metadata_policy` は POSIX では `owner_only_files_and_directories`、Windows では実際に継承される `inherited_windows_acls` policy を報告します。database payload の output 名が filesystem 上で同一の大小文字を含め `manifest.txt` と衝突する場合、plan は not ready となり、変更前に拒否されます。 + ## ファイルシステム権限 | artifact | POSIX permission / behavior | diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 699a6e84f..e7d37f90d 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -713,6 +713,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding - Cap issue lifecycle tests should reuse one indexed fixture across full-scan and update transitions when they assert the issue after each low/high boundary change. - Unreadable-directory full-scan coverage keeps JSON and human diagnostics, purge protection, checkpoint non-creation, and successful rescan retry in one fixture so the shared partial scan setup is not rebuilt. - Legacy-checkpoint coverage proves the save hook is never called, manually seeds the old file, and covers both deletion after a stable-snapshot file-read failure and bounded delete-failure warnings so authoritative full scans cannot resume from HEAD-only state. +- Database-checkpoint planning coverage keeps dry-run and execution in one fixed-clock fixture so source DB/WAL/SHM bytes, every planned output, manifest schema/hash, estimated and final bytes, and storage policies must agree. Separate focused fixtures cover missing sidecars in human output, destination conflicts, source drift refusal before publish, and native rejection of a Unix FIFO sidecar before hashing. - HEAD freshness coverage uses one two-commit fixture to prove a `--files` refresh remains stale before a `--commits HEAD` refresh marks the current head matched. - Hook timeout identity coverage uses a one-second callback budget so the healthy duplicate has cold worker-startup margin while the selected 30-second hook still times out well below the production-sized five-second budget. - SSE oversized-frame coverage uses the minimum practical keep-alive interval because the frame-size rejection, not elapsed idle time, is the contract; retain bounded polling for stream removal instead of a production-scale interval. @@ -1647,6 +1648,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" - cap issue lifecycle test は low/high boundary の変更ごとに issue を assertion する場合、full-scan と update の遷移で 1 つの indexed fixture を再利用してください。 - unreadable-directory の full-scan coverage は JSON/human diagnostics、purge protection、checkpoint 非作成、全体を再走査する successful retry を 1 fixture に保ち、共通の partial scan setup を再構築しないでください。 - legacy checkpoint coverage は save hook が呼ばれないことを固定し、旧fileを手動で配置してstable-snapshotのfile read failure後にも削除される経路とboundedなdelete-failure warningの両方を扱い、authoritative full scanがHEADだけの状態から再開しないことを検証してください。 +- database checkpoint plan の coverage は fixed clock の同一 fixture で dry-run と実行を扱い、source DB/WAL/SHM の byte 数、作成予定の全 output、manifest schema/hash、見積もりと最終 byte 数、storage policy が一致することを固定します。human output での sidecar 不在、destination conflict、publish 前の source drift 拒否、hash 前の Unix FIFO sidecar の native 拒否は、それぞれ焦点を絞った fixture で検証します。 - HEAD freshness coverage は 1 つの two-commit fixture で、`--files` refresh 後は stale のまま、`--commits HEAD` refresh 後は current head が matched になることを検証してください。 - hook timeoutのidentity coverageは1秒のcallback budgetを使い、正常なduplicate workerのcold startupに余裕を残しつつ、選択した30秒hookを本番相当の5秒budgetより十分早くtimeoutさせます。 - SSE oversized-frame coverage は、経過idle時間ではなくframe-size rejectionが契約なので、実用上最小のkeep-alive intervalを使います。stream除去は本番相当intervalではなく境界付きpollingで検証してください。 diff --git a/USER_GUIDE.md b/USER_GUIDE.md index b0800aadd..a896743b2 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -1259,7 +1259,9 @@ Failures from `vacuum`, `backfill-fold`, `optimize` (including `index --optimize `db schema` keeps the current full schema dump by default for support bundles. Add `--summary-only` to return only object counts, combine `--type ` and `--name ` for an exact projection, and use `--limit`, `--max-sql-chars`, and `--exclude-internal` to keep schema diagnostics bounded. -`db checkpoint --dry-run` reports the DB/WAL/SHM files and total bytes that would be copied without creating the checkpoint directory. Import, checkpoint restore, and managed-backup restore create a verified standalone rollback snapshot before replacing an existing DB. `db restore-backups --list` returns each managed ID and provenance; `--restore ` validates its bounded manifest, SHA-256, supported schema stamp, and free space before an atomic replacement that rolls back on failure. Add `--dry-run` to perform the same validations and preview the pre-restore backup without mutation. `--no-backup` is an explicit opt-out from creating rollback material and should be reserved for cases where losing the current DB is acceptable. A checkpoint name must be a non-blank single file name of at most 128 characters: it cannot be `.` or `..`, contain a directory separator, or contain characters that the operating system rejects in file names. Invalid names are input errors (`E010_USAGE_ERROR`), not database or storage failures. +`db checkpoint --dry-run` separates source DB/WAL/SHM files and bytes from every planned output without creating the checkpoint directory. The plan includes the versioned `manifest.txt`, its SHA-256, estimated final bytes, destination/conflict status and policy, sidecar/compression/metadata policy, and remaining uncertainty. The metadata policy is `owner_only_files_and_directories` on POSIX and `inherited_windows_acls` on Windows. Execution creates a fresh plan from the current sources, verifies source metadata and SHA-256 before and after copying, verifies every output, and refuses to publish if the inputs drift. Non-regular DB/WAL/SHM inputs such as Unix FIFOs are rejected before hashing. A source database named `manifest.txt` (including an equivalent case variant on a case-insensitive filesystem) conflicts with the generated manifest, so the plan reports `ready: false` and execution refuses without creating a checkpoint. Import, checkpoint restore, and managed-backup restore create a verified standalone rollback snapshot before replacing an existing DB. `db restore-backups --list` returns each managed ID and provenance; `--restore ` validates its bounded manifest, SHA-256, supported schema stamp, and free space before an atomic replacement that rolls back on failure. Add `--dry-run` to perform the same validations and preview the pre-restore backup without mutation. `--no-backup` is an explicit opt-out from creating rollback material and should be reserved for cases where losing the current DB is acceptable. A checkpoint name must be a non-blank single file name of at most 128 characters: it cannot be `.` or `..`, contain a directory separator, or contain characters that the operating system rejects in file names. Invalid names are input errors (`E010_USAGE_ERROR`), not database or storage failures. + +Here, drift refusal means drift detected through the final pre-publication validation. The `uncertainty` field explicitly notes that a source can still change after that validation; the already-copied outputs remain verified against the immutable plan. `cdidx optimize --dry-run --json` previews FTS5 maintenance without acquiring the index lock or changing the source DB/WAL/SHM files. The result includes DB/core-table/FTS sizes, page and freelist indicators, the incremental-write recommendation, current lock and readiness state, a previous-duration estimate when available, and the operations a real optimize would perform, including its repair-mode schema initialization or migration check. `object_sizes_measurement` distinguishes exact `dbstat` page bytes from the logical-payload fallback used when SQLite does not provide `dbstat`. @@ -4542,7 +4544,9 @@ DB を read-only で開いて SQLite の `PRAGMA integrity_check` を実行し `db schema` は support bundle 向けに、既定では従来どおり full schema dump を維持します。`--summary-only` を付けると object 件数だけを返し、`--type ` と `--name ` を組み合わせると exact projection を適用できます。schema diagnostics を小さく保つには `--limit`、`--max-sql-chars`、`--exclude-internal` を使います。 -`db checkpoint --dry-run` は checkpoint directory を作らずに、コピー対象になる DB/WAL/SHM file と合計 byte 数を報告します。import、checkpoint restore、managed-backup restore は既存 DB の置換前に検証済み standalone rollback snapshot を作成します。`db restore-backups --list` は managed ID と provenance を返し、`--restore ` は bounded manifest、SHA-256、対応する schema stamp、free space を検証してから、失敗時に rollback する atomic replacement を実行します。`--dry-run` を付けると、同じ検証と pre-restore backup の予定を DB 無変更で確認できます。`--no-backup` は rollback material 作成の明示的な opt-out であり、現在の DB を失ってもよい場合にだけ使用してください。checkpoint 名は空白だけではない 128 文字以下の単一 file 名でなければならず、`.`、`..`、directory separator、または OS が file 名で拒否する文字は使用できません。不正な名前は database / storage 障害ではなく入力エラー (`E010_USAGE_ERROR`) として扱われます。 +`db checkpoint --dry-run` は checkpoint directory を作らずに、source DB/WAL/SHM file と byte 数を、作成予定の全 output から分けて報告します。plan には version 付き `manifest.txt`、その SHA-256、最終 byte 数の見積もり、destination/conflict の状態と policy、sidecar/compression/metadata policy、残る不確実性が含まれます。metadata policy は POSIX では `owner_only_files_and_directories`、Windows では `inherited_windows_acls` です。実行時は現在の source から新しい plan を作成し、copy 前後の source metadata と SHA-256、および全 output を検証して、入力が変化した場合は publish を拒否します。Unix FIFO など regular file ではない DB/WAL/SHM input は hash 読み取り前に拒否します。source database 名が `manifest.txt`（case-insensitive filesystem 上の同等な大小文字違いを含む）の場合は生成 manifest と衝突するため、plan は `ready: false` を報告し、実行は checkpoint を作成せず拒否します。import、checkpoint restore、managed-backup restore は既存 DB の置換前に検証済み standalone rollback snapshot を作成します。`db restore-backups --list` は managed ID と provenance を返し、`--restore ` は bounded manifest、SHA-256、対応する schema stamp、free space を検証してから、失敗時に rollback する atomic replacement を実行します。`--dry-run` を付けると、同じ検証と pre-restore backup の予定を DB 無変更で確認できます。`--no-backup` は rollback material 作成の明示的な opt-out であり、現在の DB を失ってもよい場合にだけ使用してください。checkpoint 名は空白だけではない 128 文字以下の単一 file 名でなければならず、`.`、`..`、directory separator、または OS が file 名で拒否する文字は使用できません。不正な名前は database / storage 障害ではなく入力エラー (`E010_USAGE_ERROR`) として扱われます。 + +ここで drift の拒否とは、publish 前の最終検証までに検出した drift を指します。`uncertainty` field は、その検証後にも source が変化し得ることを明示しますが、copy 済み output は immutable plan に対して検証されたままです。 `cdidx optimize --dry-run --json` は index lock を取得せず、source DB/WAL/SHM file も変更せずに FTS5 maintenance を preview します。結果には DB/core table/FTS の size、page と freelist の指標、incremental write に基づく推奨、現在の lock/readiness 状態、利用可能な場合は前回所要時間に基づく見積もり、repair mode での schema 初期化または migration の確認を含む、実際の optimize が行う操作が含まれます。`object_sizes_measurement` は、正確な `dbstat` page byte と、SQLite が `dbstat` を提供しない場合の logical-payload fallback を区別します。 diff --git a/changelog.d/unreleased/4890.fixed.md b/changelog.d/unreleased/4890.fixed.md new file mode 100644 index 000000000..392a5cf9f --- /dev/null +++ b/changelog.d/unreleased/4890.fixed.md @@ -0,0 +1,20 @@ +--- +category: fixed +issues: + - 4890 +affected: + - src/CodeIndex/Cli/DbCommandRunner.Checkpoints.cs + - src/CodeIndex/Cli/DbCommandRunner.FileOperations.cs + - src/CodeIndex/Cli/DbCommandRunner.Maintenance.cs + - src/CodeIndex/Cli/JsonOutputContracts.cs + - src/CodeIndex/Indexer/Extensibility/ExecutableExtensionBoundary.cs + - tests/CodeIndex.Tests/DbCommandRunnerTests.cs +--- + +## English + +- **Database checkpoint previews now account for every planned artifact (#4890)** — human and JSON dry-run output separates source DB/WAL/SHM bytes from planned outputs, including the versioned manifest, its SHA-256, estimated final bytes, destination/conflict and platform-accurate storage policies, and uncertainty. Execution uses a fresh immutable plan, verifies inputs and outputs, refuses publication when it detects source drift, rejects payload names that collide with the generated manifest, and rejects non-regular inputs such as Unix FIFOs before hashing. + +## 日本語 + +- **database checkpoint preview が作成予定の全 artifact を計上するようになりました (#4890)** — human / JSON の dry-run 出力は source DB/WAL/SHM の byte 数と作成予定 output を分け、version 付き manifest、その SHA-256、最終 byte 数の見積もり、destination/conflict と platform に正確な storage policy、不確実性を報告します。実行時は新しい immutable plan を使って入力と出力を検証し、source drift を検出した場合は publish を拒否し、生成 manifest と衝突する payload 名、および Unix FIFO など regular file ではない input も hash 前に拒否します。 diff --git a/src/CodeIndex/Cli/ConsoleUi.cs b/src/CodeIndex/Cli/ConsoleUi.cs index 6bd9a6eed..c4f5e2067 100644 --- a/src/CodeIndex/Cli/ConsoleUi.cs +++ b/src/CodeIndex/Cli/ConsoleUi.cs @@ -220,7 +220,7 @@ private static readonly (string Command, string Note)[] CommandUsageNotes = ("recipes-list", "Read-only alias form for recipe discovery; filters and bounded JSON options match the parent `recipes` command."), ("recipes-list", "Example: `cdidx recipes list --names --json`."), ("db", "schema defaults to the full sqlite_master dump for support bundles; use --summary-only, --limit, --max-sql-chars, and --exclude-internal for bounded diagnostics."), - ("db", "checkpoint --dry-run reports the DB/WAL/SHM files and byte count without creating the checkpoint directory."), + ("db", "checkpoint --dry-run separates source DB/WAL/SHM bytes from every planned output, including the versioned manifest, its SHA-256, estimated final bytes, destination/conflict policy, and uncertainty."), ("db", "checkpoint creates a filesystem snapshot next to the DB; restore creates a verified managed rollback backup before replacing an existing DB unless --no-backup is explicit."), ("db", "restore --dry-run validates the checkpoint manifest, regular-file paths, rollback-backup policy, and destination free space without replacing the DB."), ("db", "checkpoints --delete and --prune remove snapshots; add --dry-run to report exact deleted/retained paths without mutation."), @@ -233,7 +233,7 @@ private static readonly (string Command, string Note)[] CommandUsageNotes = ("db-schema", "Example: `cdidx db schema --type index --summary-only --json`."), ("db-prune", "--dry-run only counts orphan rows; --apply deletes them and may run WAL checkpoint maintenance."), ("db-prune", "Example: `cdidx db prune --dry-run --json`."), - ("db-checkpoint", "--dry-run reports DB/WAL/SHM files and byte count; without --dry-run it creates a filesystem snapshot next to the DB."), + ("db-checkpoint", "--dry-run separates source DB/WAL/SHM files and bytes from every planned output, including the versioned manifest, its SHA-256, estimated final bytes, destination/conflict policy, and uncertainty; without --dry-run it creates a filesystem snapshot verified against that plan."), ("db-checkpoint", "Example: `cdidx db checkpoint before-upgrade --dry-run --json`."), ("db-checkpoints", "--list reports checkpoints; --delete and --prune remove snapshots, while --dry-run reports exact deleted/retained paths without mutation."), ("db-checkpoints", "Example: `cdidx db checkpoints --prune --keep 3 --dry-run --json`."), diff --git a/src/CodeIndex/Cli/DbCommandRunner.Checkpoints.cs b/src/CodeIndex/Cli/DbCommandRunner.Checkpoints.cs index 7d0494ef1..cc3f18a91 100644 --- a/src/CodeIndex/Cli/DbCommandRunner.Checkpoints.cs +++ b/src/CodeIndex/Cli/DbCommandRunner.Checkpoints.cs @@ -1,4 +1,6 @@ +using System.Security.Cryptography; using System.Runtime.InteropServices; +using System.Text; using System.Text.Json; using System.Text.Json.Nodes; using CodeIndex.Database; @@ -10,77 +12,148 @@ namespace CodeIndex.Cli; public static partial class DbCommandRunner { - private static DbCheckpointOperationResult CreateCheckpoint(string fullDbPath, string name) + private static DbCheckpointPlan PlanCheckpoint(string fullDbPath, string name) { ValidateCheckpointName(name); var root = GetCheckpointRoot(fullDbPath); var checkpointPath = GetCheckpointPath(fullDbPath, name); - if (Directory.Exists(checkpointPath)) - throw new InvalidOperationException($"checkpoint already exists: {FormatCheckpointNameForDiagnostic(name)}"); + var diagnostics = new List(); + var destinationStatus = FileSystemBoundary.TryGetAttributes(checkpointPath, out _); + var destinationExists = destinationStatus == FileSystemBoundaryProbeStatus.Found; + var destinationReady = destinationStatus == FileSystemBoundaryProbeStatus.Missing; + if (destinationExists) + { + diagnostics.Add(new DbDiagnosticJsonResult( + "checkpoint_already_exists", + "A checkpoint with this name already exists; execution would fail.", + ConsoleUi.FormatBoundedValue(checkpointPath))); + } + else if (!destinationReady) + { + diagnostics.Add(new DbDiagnosticJsonResult( + "checkpoint_destination_probe_failed", + "The checkpoint destination could not be inspected; execution would fail.", + ConsoleUi.FormatBoundedValue(checkpointPath))); + } + + var sourceCandidatePaths = new[] { fullDbPath, fullDbPath + "-wal", fullDbPath + "-shm" }; + var sourceFiles = ReadCheckpointSourceFiles(sourceCandidatePaths, diagnostics); + var outputNameComparer = StringComparer.FromComparison(PathCasing.ComparisonFor(checkpointPath)); + var outputNameConflict = sourceFiles.Files.Any( + source => outputNameComparer.Equals(source.OutputName, CheckpointManifestFileName)); + if (outputNameConflict) + { + diagnostics.Add(new DbDiagnosticJsonResult( + "checkpoint_output_name_conflict", + $"A checkpoint source file conflicts with the generated {CheckpointManifestFileName}; execution would overwrite a planned output and is refused.", + CheckpointManifestFileName)); + } + + var ready = destinationReady && !sourceFiles.Truncated && !outputNameConflict; + var manifestContents = $"format_version=1{Environment.NewLine}name={name}{Environment.NewLine}created_at_utc={GetUtcNow():O}{Environment.NewLine}db_file={Path.GetFileName(fullDbPath)}{Environment.NewLine}"; + var manifestBytes = Encoding.UTF8.GetBytes(manifestContents); + var plannedOutputFiles = sourceFiles.Files + .Select(source => source.OutputName) + .Append(CheckpointManifestFileName) + .Distinct(outputNameComparer) + .Order(StringComparer.Ordinal) + .ToList(); + + return new DbCheckpointPlan( + name, + root, + checkpointPath, + Array.AsReadOnly(sourceCandidatePaths), + sourceFiles.Files.AsReadOnly(), + plannedOutputFiles.AsReadOnly(), + sourceFiles.Bytes, + checked(sourceFiles.Bytes + manifestBytes.LongLength), + manifestContents, + ComputeCheckpointSha256(manifestBytes), + CheckpointDestinationPolicy, + CheckpointConflictPolicy, + CheckpointPlanUncertainty, + CheckpointManifestSchema, + CheckpointSidecarPolicy, + CheckpointCompressionPolicy, + OperatingSystem.IsWindows() + ? CheckpointWindowsMetadataPolicy + : CheckpointPosixMetadataPolicy, + ready, + destinationExists, + sourceFiles.Truncated, + diagnostics.AsReadOnly()); + } + + private static DbCheckpointOperationResult CreateCheckpoint(DbCheckpointPlan plan) + { + if (!plan.Ready) + throw new InvalidOperationException($"checkpoint plan is not ready: {FormatCheckpointNameForDiagnostic(plan.Name)}"); + + CheckpointPlanReadyForExecutionForTesting?.Invoke(); + ValidateCheckpointPlanSources(plan); - DataDirectorySecurity.CreateSensitiveDirectory(root); - var tempPath = Path.Combine(root, ".tmp-" + name + "-" + Guid.NewGuid().ToString("N")); + DataDirectorySecurity.CreateSensitiveDirectory(plan.RootPath); + var tempPath = Path.Combine(plan.RootPath, ".tmp-" + plan.Name + "-" + Guid.NewGuid().ToString("N")); DataDirectorySecurity.CreateSensitiveDirectory(tempPath); try { - CopyIfExists(fullDbPath, Path.Combine(tempPath, Path.GetFileName(fullDbPath)), privateDestination: true); - CopyIfExists(fullDbPath + "-wal", Path.Combine(tempPath, Path.GetFileName(fullDbPath) + "-wal"), privateDestination: true); - CopyIfExists(fullDbPath + "-shm", Path.Combine(tempPath, Path.GetFileName(fullDbPath) + "-shm"), privateDestination: true); - DataDirectorySecurity.WritePrivateText(Path.Combine(tempPath, "manifest.txt"), $"name={name}{Environment.NewLine}created_at_utc={GetUtcNow():O}{Environment.NewLine}db_file={Path.GetFileName(fullDbPath)}{Environment.NewLine}"); - AtomicFileWriter.PublishDirectory(tempPath, checkpointPath); + foreach (var source in plan.SourceFiles) + { + var destination = Path.Combine(tempPath, source.OutputName); + CopyIfExists(source.SourcePath, destination, privateDestination: true); + ValidateCheckpointOutput(destination, source); + } + + var manifestPath = Path.Combine(tempPath, CheckpointManifestFileName); + DataDirectorySecurity.WritePrivateText(manifestPath, plan.ManifestContents); + ValidateCheckpointOutput( + manifestPath, + new DbCheckpointSourcePlan( + manifestPath, + CheckpointManifestFileName, + Encoding.UTF8.GetByteCount(plan.ManifestContents), + LastWriteTimeUtcTicks: null, + plan.ManifestSha256)); + ValidateCheckpointPlanSources(plan); + AtomicFileWriter.PublishDirectory(tempPath, plan.CheckpointPath); } catch { TryDeleteTemporaryDirectory( tempPath, "checkpoint temporary directory", - root, + plan.RootPath, ".tmp-"); throw; } - var diagnostics = new List(); - var files = EnumerateCheckpointFileNames(checkpointPath, diagnostics); + var diagnostics = new List(plan.Diagnostics); + var files = EnumerateCheckpointFileNames(plan.CheckpointPath, diagnostics); var bytes = files.Truncated ? (Bytes: 0L, Truncated: true) - : SumCheckpointBytes(checkpointPath, diagnostics); - return new DbCheckpointOperationResult(name, checkpointPath, files.Items, files.Truncated || bytes.Truncated, diagnostics, bytes.Bytes); - } - - private static DbCheckpointOperationResult PreviewCheckpoint(string fullDbPath, string name) - { - ValidateCheckpointName(name); - var checkpointPath = GetCheckpointPath(fullDbPath, name); - var diagnostics = new List(); - if (Directory.Exists(LongPath.EnsureWindowsPrefix(checkpointPath))) - { - diagnostics.Add(new DbDiagnosticJsonResult( - "checkpoint_already_exists", - "A checkpoint with this name already exists; running without --dry-run would fail.", - ConsoleUi.FormatBoundedValue(checkpointPath))); - } - - var files = ReadCheckpointSourceFiles(fullDbPath, diagnostics); - return new DbCheckpointOperationResult(name, checkpointPath, files.Files, files.Truncated, diagnostics, files.Bytes); + : SumCheckpointBytes(plan.CheckpointPath, diagnostics); + return new DbCheckpointOperationResult(plan.Name, plan.CheckpointPath, files.Items, files.Truncated || bytes.Truncated, diagnostics, bytes.Bytes); } - private static (List Files, long Bytes, bool Truncated) ReadCheckpointSourceFiles( - string fullDbPath, + private static (List Files, long Bytes, bool Truncated) ReadCheckpointSourceFiles( + IReadOnlyList sourceCandidatePaths, List diagnostics) { - var files = new List(); + var files = new List(); long bytes = 0; - foreach (var source in new[] { fullDbPath, fullDbPath + "-wal", fullDbPath + "-shm" }) + foreach (var source in sourceCandidatePaths) { try { if (!TryGetRegularExistingFile(source, out var normalizedSource)) continue; - files.Add(Path.GetFileName(source) ?? source); - bytes += new FileInfo(normalizedSource).Length; + var sourcePlan = CaptureCheckpointSource(normalizedSource, Path.GetFileName(source) ?? source); + files.Add(sourcePlan); + bytes = checked(bytes + sourcePlan.Bytes); } - catch (Exception ex) when (IsRecoverableFilesystemException(ex)) + catch (Exception ex) when (IsRecoverableFilesystemException(ex) || ex is InvalidOperationException) { diagnostics.Add(CreateCheckpointDiagnostic( "checkpoint_source_file_stat_failed", @@ -90,10 +163,94 @@ private static (List Files, long Bytes, bool Truncated) ReadCheckpointSo } } - files.Sort(StringComparer.Ordinal); + files.Sort((left, right) => StringComparer.Ordinal.Compare(left.OutputName, right.OutputName)); return (files, bytes, Truncated: false); } + private static DbCheckpointSourcePlan CaptureCheckpointSource(string sourcePath, string outputName) + { + var fileInfo = new FileInfo(sourcePath); + fileInfo.Refresh(); + var length = fileInfo.Length; + var lastWriteTimeUtcTicks = fileInfo.LastWriteTimeUtc.Ticks; + var sha256 = ComputeCheckpointSha256(sourcePath); + fileInfo.Refresh(); + if (length != fileInfo.Length || lastWriteTimeUtcTicks != fileInfo.LastWriteTimeUtc.Ticks) + throw new DbCheckpointPlanDriftException(); + + return new DbCheckpointSourcePlan(sourcePath, outputName, length, lastWriteTimeUtcTicks, sha256); + } + + private static void ValidateCheckpointPlanSources(DbCheckpointPlan plan) + { + foreach (var sourceCandidatePath in plan.SourceCandidatePaths) + { + try + { + var outputName = Path.GetFileName(sourceCandidatePath) ?? sourceCandidatePath; + var expected = plan.SourceFiles.SingleOrDefault( + source => string.Equals(source.OutputName, outputName, StringComparison.Ordinal)); + var exists = TryGetRegularExistingFile(sourceCandidatePath, out var normalizedSource); + if (exists != (expected is not null)) + throw new DbCheckpointPlanDriftException(); + if (expected is null) + continue; + + var current = CaptureCheckpointSource(normalizedSource, expected.OutputName); + if (current.Bytes != expected.Bytes + || current.LastWriteTimeUtcTicks != expected.LastWriteTimeUtcTicks + || !string.Equals(current.Sha256, expected.Sha256, StringComparison.Ordinal)) + { + throw new DbCheckpointPlanDriftException(); + } + } + catch (DbCheckpointPlanDriftException) + { + throw; + } + catch (Exception ex) when (IsRecoverableFilesystemException(ex)) + { + throw new DbCheckpointPlanDriftException(ex); + } + } + } + + private static void ValidateCheckpointOutput(string destinationPath, DbCheckpointSourcePlan expected) + { + try + { + if (!TryGetRegularExistingFile(destinationPath, out var normalizedDestination)) + throw new DbCheckpointPlanDriftException(); + var fileInfo = new FileInfo(normalizedDestination); + if (fileInfo.Length != expected.Bytes + || !string.Equals(ComputeCheckpointSha256(normalizedDestination), expected.Sha256, StringComparison.Ordinal)) + { + throw new DbCheckpointPlanDriftException(); + } + } + catch (DbCheckpointPlanDriftException) + { + throw; + } + catch (Exception ex) when (IsRecoverableFilesystemException(ex)) + { + throw new DbCheckpointPlanDriftException(ex); + } + } + + private static string ComputeCheckpointSha256(string path) + { + using var stream = new FileStream( + LongPath.EnsureWindowsPrefix(path), + FileMode.Open, + FileAccess.Read, + FileShare.ReadWrite | FileShare.Delete); + return Convert.ToHexString(SHA256.HashData(stream)).ToLowerInvariant(); + } + + private static string ComputeCheckpointSha256(byte[] contents) + => Convert.ToHexString(SHA256.HashData(contents)).ToLowerInvariant(); + private static DbCheckpointListReadResult ListCheckpoints(string fullDbPath, int limit) { var root = GetCheckpointRoot(fullDbPath); diff --git a/src/CodeIndex/Cli/DbCommandRunner.FileOperations.cs b/src/CodeIndex/Cli/DbCommandRunner.FileOperations.cs index 65d6aca31..bf18bc6bd 100644 --- a/src/CodeIndex/Cli/DbCommandRunner.FileOperations.cs +++ b/src/CodeIndex/Cli/DbCommandRunner.FileOperations.cs @@ -4,6 +4,7 @@ using CodeIndex.Database; using CodeIndex.Diagnostics; using CodeIndex.Indexer; +using CodeIndex.Indexer.Extensibility; using Microsoft.Data.Sqlite; namespace CodeIndex.Cli; @@ -112,7 +113,8 @@ private static bool TryGetRegularExistingFile(string path, out string normalized return false; } - if ((attributes & (FileAttributes.Directory | FileAttributes.ReparsePoint | FileAttributes.Device)) != 0) + if ((attributes & (FileAttributes.Directory | FileAttributes.ReparsePoint | FileAttributes.Device)) != 0 + || !ExecutableExtensionBoundary.IsRegularFilePath(normalizedPath)) throw new InvalidOperationException($"checkpoint file is not a regular file: {ConsoleUi.FormatBoundedValue(path)}"); return true; diff --git a/src/CodeIndex/Cli/DbCommandRunner.Maintenance.cs b/src/CodeIndex/Cli/DbCommandRunner.Maintenance.cs index 3d0c4a5f5..8ab84d154 100644 --- a/src/CodeIndex/Cli/DbCommandRunner.Maintenance.cs +++ b/src/CodeIndex/Cli/DbCommandRunner.Maintenance.cs @@ -100,43 +100,55 @@ private static int RunCheckpoint(DbCommandOptions options, JsonSerializerOptions try { + var plan = PlanCheckpoint(fullDbPath, options.Name ?? MakeTimestampCheckpointName()); if (options.CheckpointDryRun) { - var preview = PreviewCheckpoint(fullDbPath, options.Name ?? MakeTimestampCheckpointName()); if (options.Json) { Console.WriteLine(JsonSerializer.Serialize( new DbCheckpointJsonResult( "dry_run", fullDbPath, - preview.Name, - preview.CheckpointPath, - preview.Files, - preview.FilesTruncated, + plan.Name, + plan.CheckpointPath, + plan.SourceFiles.Select(source => source.OutputName).ToList(), + plan.SourceFilesTruncated, CheckpointFileInspectLimit, - preview.Diagnostics, + plan.Diagnostics.ToList(), DryRun: true, - Bytes: preview.Bytes), + Bytes: plan.SourceBytes, + SourceFiles: plan.SourceFiles.Select(source => source.OutputName).ToList(), + SourceBytes: plan.SourceBytes, + PlannedOutputFiles: plan.PlannedOutputFiles.ToList(), + EstimatedOutputBytes: plan.EstimatedOutputBytes, + Ready: plan.Ready, + DestinationExists: plan.DestinationExists, + DestinationPolicy: plan.DestinationPolicy, + ConflictPolicy: plan.ConflictPolicy, + Uncertainty: plan.Uncertainty, + ManifestSchema: plan.ManifestSchema, + ManifestContents: plan.ManifestContents, + ManifestSha256: plan.ManifestSha256, + SidecarPolicy: plan.SidecarPolicy, + Compression: plan.CompressionPolicy, + MetadataPolicy: plan.MetadataPolicy), CliJsonSerializerContextFactory.Create(jsonOptions).DbCheckpointJsonResult)); } else { Console.WriteLine("Database checkpoint dry run."); Console.WriteLine($" database : {fullDbPath}"); - Console.WriteLine($" name : {preview.Name}"); - Console.WriteLine($" checkpoint: {preview.CheckpointPath}"); - Console.WriteLine($" side effect: none (run without --dry-run to copy DB/WAL/SHM files)"); - Console.WriteLine($" files : {ConsoleUi.Counted(preview.Files.Count, "file")}{(preview.FilesTruncated ? " (truncated)" : string.Empty)}"); - Console.WriteLine($" bytes : {preview.Bytes:N0}"); + WriteCheckpointPlan(plan); + Console.WriteLine(" side effect: none"); } - foreach (var diagnostic in preview.Diagnostics) + foreach (var diagnostic in plan.Diagnostics) CommandErrorWriter.WriteStderr($"Warning [{diagnostic.Code}]: {diagnostic.Message}"); return CommandExitCodes.Success; } - var result = CreateCheckpoint(fullDbPath, options.Name ?? MakeTimestampCheckpointName()); + var result = CreateCheckpoint(plan); if (options.Json) { Console.WriteLine(JsonSerializer.Serialize( @@ -149,17 +161,32 @@ private static int RunCheckpoint(DbCommandOptions options, JsonSerializerOptions result.FilesTruncated, CheckpointFileInspectLimit, result.Diagnostics, - Bytes: result.Bytes), + Bytes: result.Bytes, + SourceFiles: plan.SourceFiles.Select(source => source.OutputName).ToList(), + SourceBytes: plan.SourceBytes, + PlannedOutputFiles: plan.PlannedOutputFiles.ToList(), + EstimatedOutputBytes: plan.EstimatedOutputBytes, + FinalOutputBytes: plan.EstimatedOutputBytes, + Ready: plan.Ready, + DestinationExists: plan.DestinationExists, + DestinationPolicy: plan.DestinationPolicy, + ConflictPolicy: plan.ConflictPolicy, + Uncertainty: plan.Uncertainty, + ManifestSchema: plan.ManifestSchema, + ManifestContents: plan.ManifestContents, + ManifestSha256: plan.ManifestSha256, + SidecarPolicy: plan.SidecarPolicy, + Compression: plan.CompressionPolicy, + MetadataPolicy: plan.MetadataPolicy), CliJsonSerializerContextFactory.Create(jsonOptions).DbCheckpointJsonResult)); } else { Console.WriteLine("Created database checkpoint."); Console.WriteLine($" database : {fullDbPath}"); - Console.WriteLine($" name : {result.Name}"); - Console.WriteLine($" checkpoint: {result.CheckpointPath}"); - Console.WriteLine($" files : {ConsoleUi.Counted(result.Files.Count, "file")}{(result.FilesTruncated ? " (truncated)" : string.Empty)}"); - Console.WriteLine($" bytes : {result.Bytes:N0}"); + WriteCheckpointPlan(plan); + Console.WriteLine($" final files: {ConsoleUi.Counted(result.Files.Count, "file")}{(result.FilesTruncated ? " (truncated)" : string.Empty)}"); + Console.WriteLine($" final bytes: {plan.EstimatedOutputBytes:N0}"); } foreach (var diagnostic in result.Diagnostics) @@ -167,6 +194,17 @@ private static int RunCheckpoint(DbCommandOptions options, JsonSerializerOptions return CommandExitCodes.Success; } + catch (DbCheckpointPlanDriftException) + { + return WriteCommandError( + options.Json, + jsonOptions, + "checkpoint plan drift detected; no checkpoint was published", + CommandExitCodes.DatabaseError, + "Stop database writers and retry `cdidx db checkpoint`; execution creates a fresh plan and refuses changes before publish.", + CommandErrorCodes.DbError, + category: "checkpoint_plan_drift"); + } catch (Exception ex) { var isInputError = ex is ArgumentException; @@ -186,6 +224,36 @@ private static int RunCheckpoint(DbCommandOptions options, JsonSerializerOptions } } + private static void WriteCheckpointPlan(DbCheckpointPlan plan) + { + Console.WriteLine($" name : {plan.Name}"); + Console.WriteLine($" checkpoint: {plan.CheckpointPath}"); + Console.WriteLine($" ready : {(plan.Ready ? "yes" : "no")}"); + Console.WriteLine($" destination exists: {(plan.DestinationExists ? "yes" : "no")}"); + Console.WriteLine($" destination policy: {FormatCheckpointPolicy(plan.DestinationPolicy)}"); + Console.WriteLine($" conflict policy: {FormatCheckpointPolicy(plan.ConflictPolicy)}"); + Console.WriteLine($" source files: {ConsoleUi.Counted(plan.SourceFiles.Count, "file")}{(plan.SourceFilesTruncated ? " (truncated)" : string.Empty)}"); + foreach (var source in plan.SourceFiles) + Console.WriteLine($" {source.OutputName} ({source.Bytes:N0} bytes)"); + Console.WriteLine($" source bytes: {plan.SourceBytes:N0}"); + Console.WriteLine($" planned outputs: {ConsoleUi.Counted(plan.PlannedOutputFiles.Count, "file")}"); + foreach (var output in plan.PlannedOutputFiles) + Console.WriteLine($" {output}"); + Console.WriteLine($" estimated output bytes: {plan.EstimatedOutputBytes:N0}"); + Console.WriteLine($" manifest schema: {plan.ManifestSchema}"); + Console.WriteLine(" manifest contents:"); + foreach (var line in plan.ManifestContents.Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries)) + Console.WriteLine($" {line}"); + Console.WriteLine($" manifest sha256: {plan.ManifestSha256}"); + Console.WriteLine($" sidecar policy: {FormatCheckpointPolicy(plan.SidecarPolicy)}"); + Console.WriteLine($" compression: {FormatCheckpointPolicy(plan.CompressionPolicy)}"); + Console.WriteLine($" metadata policy: {FormatCheckpointPolicy(plan.MetadataPolicy)}"); + Console.WriteLine($" uncertainty: {FormatCheckpointPolicy(plan.Uncertainty)}"); + } + + private static string FormatCheckpointPolicy(string value) + => value.Replace('_', ' ').Replace(";", "; ", StringComparison.Ordinal); + private static int RunCheckpoints(DbCommandOptions options, JsonSerializerOptions jsonOptions) { var actionCount = (options.CheckpointsList ? 1 : 0) diff --git a/src/CodeIndex/Cli/DbCommandRunner.cs b/src/CodeIndex/Cli/DbCommandRunner.cs index ba22cd377..66d047249 100644 --- a/src/CodeIndex/Cli/DbCommandRunner.cs +++ b/src/CodeIndex/Cli/DbCommandRunner.cs @@ -22,6 +22,16 @@ public static partial class DbCommandRunner internal const int CheckpointPruneScanLimit = 1_000; internal const int CheckpointFileInspectLimit = 32; internal const int CheckpointManifestByteLimit = 16 * 1024; + internal const string CheckpointDestinationPolicy = "create_new_directory_atomically"; + internal const string CheckpointConflictPolicy = "fail_if_destination_exists"; + internal const string CheckpointPlanUncertainty = "source_files_can_change_after_final_validation;execution_replans_and_refuses_detected_drift_before_publish"; + internal const string CheckpointManifestSchema = "checkpoint_manifest_v1"; + internal const string CheckpointManifestFileName = "manifest.txt"; + internal const string CheckpointSidecarPolicy = "copy_wal_and_shm_if_present"; + internal const string CheckpointCompressionPolicy = "none"; + internal const string CheckpointMetadataPolicy = "platform_specific"; + internal const string CheckpointPosixMetadataPolicy = "owner_only_files_and_directories"; + internal const string CheckpointWindowsMetadataPolicy = "inherited_windows_acls"; internal const int RestoreBackupListEntryLimit = 100; internal const int RestoreBackupPruneScanLimit = 1_000; internal const int DefaultRestoreBackupKeepCount = 10; @@ -38,6 +48,7 @@ public static partial class DbCommandRunner internal static Func>? EnumerateCheckpointFilesForTesting { get; set; } internal static Func>? IntegrityCheckRowsForTesting { get; set; } internal static Func>? EnumerateCheckpointFileNamesForTesting { get; set; } + internal static Action? CheckpointPlanReadyForExecutionForTesting { get; set; } private static readonly AsyncLocal?> ScopedUtcNowForTesting = new(); private static readonly AsyncLocal?> ScopedAvailableFreeSpaceForTesting = new(); internal static Func? UtcNowForTesting @@ -173,7 +184,7 @@ internal static string CreateAutomaticCheckpoint(string dbPath) { var fullDbPath = Path.GetFullPath(DbPathResolver.NormalizeDbPath(dbPath)); var name = AutoCheckpointPrefix + MakeTimestampCheckpointName(); - return CreateCheckpoint(fullDbPath, name).CheckpointPath; + return CreateCheckpoint(PlanCheckpoint(fullDbPath, name)).CheckpointPath; } private static int RunIntegrityCheck(DbCommandOptions options, JsonSerializerOptions jsonOptions, string dbPath, bool isUri, CancellationToken cancellationToken) @@ -418,6 +429,49 @@ internal sealed class DbCommandOptions internal sealed record DbCheckpointOperationResult(string Name, string CheckpointPath, List Files, bool FilesTruncated, List Diagnostics, long Bytes); +internal sealed record DbCheckpointPlan( + string Name, + string RootPath, + string CheckpointPath, + IReadOnlyList SourceCandidatePaths, + IReadOnlyList SourceFiles, + IReadOnlyList PlannedOutputFiles, + long SourceBytes, + long EstimatedOutputBytes, + string ManifestContents, + string ManifestSha256, + string DestinationPolicy, + string ConflictPolicy, + string Uncertainty, + string ManifestSchema, + string SidecarPolicy, + string CompressionPolicy, + string MetadataPolicy, + bool Ready, + bool DestinationExists, + bool SourceFilesTruncated, + IReadOnlyList Diagnostics); + +internal sealed record DbCheckpointSourcePlan( + string SourcePath, + string OutputName, + long Bytes, + long? LastWriteTimeUtcTicks, + string Sha256); + +internal sealed class DbCheckpointPlanDriftException : IOException +{ + internal DbCheckpointPlanDriftException() + : base("checkpoint inputs changed after planning") + { + } + + internal DbCheckpointPlanDriftException(Exception innerException) + : base("checkpoint inputs changed after planning", innerException) + { + } +} + internal sealed record DbCheckpointListReadResult( List Entries, bool DirectoryEnumerationTruncated, diff --git a/src/CodeIndex/Cli/JsonOutputContracts.cs b/src/CodeIndex/Cli/JsonOutputContracts.cs index 14b2641a0..0dc9062bb 100644 --- a/src/CodeIndex/Cli/JsonOutputContracts.cs +++ b/src/CodeIndex/Cli/JsonOutputContracts.cs @@ -291,6 +291,22 @@ internal sealed record DbCheckpointJsonResult( [property: JsonPropertyName("diagnostics")] List? Diagnostics = null, [property: JsonPropertyName("dry_run")] bool DryRun = false, [property: JsonPropertyName("bytes")] long Bytes = 0, + [property: JsonPropertyName("source_files")] List? SourceFiles = null, + [property: JsonPropertyName("source_bytes")] long SourceBytes = 0, + [property: JsonPropertyName("planned_output_files")] List? PlannedOutputFiles = null, + [property: JsonPropertyName("estimated_output_bytes")] long EstimatedOutputBytes = 0, + [property: JsonPropertyName("final_output_bytes")] long? FinalOutputBytes = null, + [property: JsonPropertyName("ready")] bool Ready = false, + [property: JsonPropertyName("destination_exists")] bool DestinationExists = false, + [property: JsonPropertyName("destination_policy")] string DestinationPolicy = DbCommandRunner.CheckpointDestinationPolicy, + [property: JsonPropertyName("conflict_policy")] string ConflictPolicy = DbCommandRunner.CheckpointConflictPolicy, + [property: JsonPropertyName("uncertainty")] string Uncertainty = DbCommandRunner.CheckpointPlanUncertainty, + [property: JsonPropertyName("manifest_schema")] string ManifestSchema = DbCommandRunner.CheckpointManifestSchema, + [property: JsonPropertyName("manifest_contents")] string? ManifestContents = null, + [property: JsonPropertyName("manifest_sha256")] string? ManifestSha256 = null, + [property: JsonPropertyName("sidecar_policy")] string SidecarPolicy = DbCommandRunner.CheckpointSidecarPolicy, + [property: JsonPropertyName("compression")] string Compression = DbCommandRunner.CheckpointCompressionPolicy, + [property: JsonPropertyName("metadata_policy")] string MetadataPolicy = DbCommandRunner.CheckpointMetadataPolicy, [property: JsonPropertyName("api_version")] string ApiVersion = JsonOutputContract.ApiVersion) : IVersionedJsonResult; internal sealed record DbCheckpointListJsonResult( diff --git a/src/CodeIndex/Indexer/Extensibility/ExecutableExtensionBoundary.cs b/src/CodeIndex/Indexer/Extensibility/ExecutableExtensionBoundary.cs index 6fecfed03..80a661976 100644 --- a/src/CodeIndex/Indexer/Extensibility/ExecutableExtensionBoundary.cs +++ b/src/CodeIndex/Indexer/Extensibility/ExecutableExtensionBoundary.cs @@ -78,6 +78,7 @@ internal static class ExecutableExtensionBoundary private const int CopyBufferBytes = 64 * 1024; private const uint UnixFileTypeMask = 0xF000; private const uint UnixRegularFileType = 0x8000; + private const uint UnixDirectoryType = 0x4000; private const uint UnixGroupWrite = 0x0010; private const uint UnixOtherWrite = 0x0002; @@ -85,6 +86,15 @@ internal static class ExecutableExtensionBoundary internal static Action? StagedForTesting { get; set; } + internal static bool IsRegularFilePath(string path) + { + if (!OperatingSystem.IsLinux() && !OperatingSystem.IsMacOS()) + return true; + + return UnixFileIdentity.TryRead(path, out var mode, out _) + && (mode & UnixFileTypeMask) == UnixRegularFileType; + } + internal static bool TryValidateDirectory( string directory, out string fullDirectory, diff --git a/tests/CodeIndex.Tests/DbCommandRunnerTests.cs b/tests/CodeIndex.Tests/DbCommandRunnerTests.cs index 1a50c817e..80ff03e21 100644 --- a/tests/CodeIndex.Tests/DbCommandRunnerTests.cs +++ b/tests/CodeIndex.Tests/DbCommandRunnerTests.cs @@ -55,6 +55,26 @@ private static void DeleteWorkDirectory(string root) TestProjectHelper.DeleteDirectory(root); } + private static void CreateUnixFifo(string path) + { + var startInfo = new System.Diagnostics.ProcessStartInfo + { + FileName = "mkfifo", + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + startInfo.ArgumentList.Add(path); + + using var process = System.Diagnostics.Process.Start(startInfo) + ?? throw new InvalidOperationException("Failed to start mkfifo / mkfifo の起動に失敗"); + var stderr = process.StandardError.ReadToEnd(); + process.WaitForExit(); + if (process.ExitCode != 0) + throw new InvalidOperationException($"mkfifo failed: {stderr.Trim()}"); + } + [Fact] public void ParseArgs_IntegrityCheckFlagSetsFlag() { @@ -911,6 +931,235 @@ public void Run_CheckpointDryRun_JsonPreviewsFilesWithoutCreatingCheckpoint_Issu } } + [Fact] + public void Run_CheckpointPlan_JsonMatchesExecutionForDbAndSidecars_Issue4890() + { + var root = TestProjectHelper.CreateTempProject("cdidx_db_checkpoint_plan_4890"); + var dbPath = Path.Combine(root, "codeindex.db"); + var fixedTime = new DateTimeOffset(2026, 7, 30, 1, 2, 3, 456, TimeSpan.Zero); + try + { + File.WriteAllText(dbPath, "db"); + File.WriteAllText(dbPath + "-wal", "wal"); + File.WriteAllText(dbPath + "-shm", "shm!"); + DbCommandRunner.UtcNowForTesting = () => fixedTime; + + var (dryRunExit, dryRunJson) = RunAndCaptureJson(["checkpoint", "planned", "--dry-run", "--db", dbPath, "--json"]); + var (writeExit, writeJson) = RunAndCaptureJson(["checkpoint", "planned", "--db", dbPath, "--json"]); + + Assert.Equal(CommandExitCodes.Success, dryRunExit); + Assert.Equal(CommandExitCodes.Success, writeExit); + Assert.True(dryRunJson.GetProperty("ready").GetBoolean()); + Assert.False(dryRunJson.GetProperty("destination_exists").GetBoolean()); + Assert.Equal(9, dryRunJson.GetProperty("source_bytes").GetInt64()); + Assert.Equal( + ["codeindex.db", "codeindex.db-shm", "codeindex.db-wal"], + dryRunJson.GetProperty("source_files").EnumerateArray().Select(file => file.GetString()).ToArray()); + var plannedOutputs = dryRunJson.GetProperty("planned_output_files").EnumerateArray().Select(file => file.GetString()).ToArray(); + Assert.Equal(["codeindex.db", "codeindex.db-shm", "codeindex.db-wal", "manifest.txt"], plannedOutputs); + Assert.Equal( + plannedOutputs, + writeJson.GetProperty("planned_output_files").EnumerateArray().Select(file => file.GetString()).ToArray()); + Assert.Equal( + dryRunJson.GetProperty("estimated_output_bytes").GetInt64(), + writeJson.GetProperty("estimated_output_bytes").GetInt64()); + Assert.Equal( + writeJson.GetProperty("estimated_output_bytes").GetInt64(), + writeJson.GetProperty("final_output_bytes").GetInt64()); + Assert.Equal( + dryRunJson.GetProperty("manifest_sha256").GetString(), + writeJson.GetProperty("manifest_sha256").GetString()); + Assert.Equal(64, dryRunJson.GetProperty("manifest_sha256").GetString()!.Length); + Assert.Equal("checkpoint_manifest_v1", dryRunJson.GetProperty("manifest_schema").GetString()); + Assert.Contains("format_version=1", dryRunJson.GetProperty("manifest_contents").GetString(), StringComparison.Ordinal); + Assert.Contains($"created_at_utc={fixedTime:O}", dryRunJson.GetProperty("manifest_contents").GetString(), StringComparison.Ordinal); + Assert.Equal("copy_wal_and_shm_if_present", dryRunJson.GetProperty("sidecar_policy").GetString()); + Assert.Equal("none", dryRunJson.GetProperty("compression").GetString()); + Assert.Equal( + OperatingSystem.IsWindows() ? "inherited_windows_acls" : "owner_only_files_and_directories", + dryRunJson.GetProperty("metadata_policy").GetString()); + Assert.Equal("create_new_directory_atomically", dryRunJson.GetProperty("destination_policy").GetString()); + Assert.Equal("fail_if_destination_exists", dryRunJson.GetProperty("conflict_policy").GetString()); + Assert.Equal( + "source_files_can_change_after_final_validation;execution_replans_and_refuses_detected_drift_before_publish", + dryRunJson.GetProperty("uncertainty").GetString()); + var checkpointPath = writeJson.GetProperty("checkpoint_path").GetString()!; + var actualBytes = Directory.GetFiles(checkpointPath).Sum(path => new FileInfo(path).Length); + Assert.Equal(writeJson.GetProperty("final_output_bytes").GetInt64(), actualBytes); + } + finally + { + DbCommandRunner.UtcNowForTesting = null; + DeleteWorkDirectory(root); + } + } + + [Fact] + public void Run_CheckpointPlan_HumanOutputListsEveryArtifactAndMissingSidecars_Issue4890() + { + var root = TestProjectHelper.CreateTempProject("cdidx_db_checkpoint_human_plan_4890"); + var dbPath = Path.Combine(root, "codeindex.db"); + try + { + File.WriteAllText(dbPath, "db"); + + var (exitCode, stdout, _) = RunAndCaptureStreams(["checkpoint", "planned", "--dry-run", "--db", dbPath]); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Contains("source files: 1 file", stdout, StringComparison.Ordinal); + Assert.Contains("planned outputs: 2 files", stdout, StringComparison.Ordinal); + Assert.Contains("codeindex.db (2 bytes)", stdout, StringComparison.Ordinal); + Assert.Contains("manifest.txt", stdout, StringComparison.Ordinal); + Assert.DoesNotContain("codeindex.db-wal", stdout, StringComparison.Ordinal); + Assert.DoesNotContain("codeindex.db-shm", stdout, StringComparison.Ordinal); + Assert.Contains("estimated output bytes:", stdout, StringComparison.Ordinal); + Assert.Contains("manifest contents:", stdout, StringComparison.Ordinal); + Assert.Contains("format_version=1", stdout, StringComparison.Ordinal); + Assert.Contains("manifest sha256:", stdout, StringComparison.Ordinal); + Assert.Contains("conflict policy: fail if destination exists", stdout, StringComparison.Ordinal); + Assert.Contains("compression: none", stdout, StringComparison.Ordinal); + Assert.Contains( + OperatingSystem.IsWindows() + ? "metadata policy: inherited windows acls" + : "metadata policy: owner only files and directories", + stdout, + StringComparison.Ordinal); + Assert.Contains("uncertainty: source files can change after final validation; execution replans and refuses detected drift before publish", stdout, StringComparison.Ordinal); + } + finally + { + DeleteWorkDirectory(root); + } + } + + [Fact] + public void Run_CheckpointPlan_DestinationConflictIsReportedAndExecutionRefuses_Issue4890() + { + var root = TestProjectHelper.CreateTempProject("cdidx_db_checkpoint_conflict_4890"); + var dbPath = Path.Combine(root, "codeindex.db"); + var checkpointPath = Path.Combine(dbPath + ".checkpoints", "conflict"); + var sentinelPath = Path.Combine(checkpointPath, "sentinel.txt"); + try + { + File.WriteAllText(dbPath, "db"); + Directory.CreateDirectory(checkpointPath); + File.WriteAllText(sentinelPath, "keep"); + + var (dryRunExit, dryRunJson) = RunAndCaptureJson(["checkpoint", "conflict", "--dry-run", "--db", dbPath, "--json"]); + var (writeExit, writeJson) = RunAndCaptureJson(["checkpoint", "conflict", "--db", dbPath, "--json"]); + + Assert.Equal(CommandExitCodes.Success, dryRunExit); + Assert.False(dryRunJson.GetProperty("ready").GetBoolean()); + Assert.True(dryRunJson.GetProperty("destination_exists").GetBoolean()); + Assert.Contains( + dryRunJson.GetProperty("diagnostics").EnumerateArray(), + diagnostic => diagnostic.GetProperty("code").GetString() == "checkpoint_already_exists"); + Assert.Equal(CommandExitCodes.DatabaseError, writeExit); + Assert.Equal(CommandErrorCodes.DbError, writeJson.GetProperty("error_code").GetString()); + Assert.Equal("keep", File.ReadAllText(sentinelPath)); + } + finally + { + DeleteWorkDirectory(root); + } + } + + [Fact] + public void Run_CheckpointPlan_ManifestNameCollisionIsReportedAndExecutionRefuses_Issue4890() + { + var root = TestProjectHelper.CreateTempProject("cdidx_db_checkpoint_manifest_collision_4890"); + var dbPath = Path.Combine(root, "manifest.txt"); + try + { + File.WriteAllText(dbPath, "db"); + + var (dryRunExit, dryRunJson) = RunAndCaptureJson(["checkpoint", "collision", "--dry-run", "--db", dbPath, "--json"]); + var (writeExit, writeJson) = RunAndCaptureJson(["checkpoint", "collision", "--db", dbPath, "--json"]); + + Assert.Equal(CommandExitCodes.Success, dryRunExit); + Assert.False(dryRunJson.GetProperty("ready").GetBoolean()); + Assert.Equal( + ["manifest.txt"], + dryRunJson.GetProperty("planned_output_files").EnumerateArray().Select(file => file.GetString()).ToArray()); + Assert.Contains( + dryRunJson.GetProperty("diagnostics").EnumerateArray(), + diagnostic => diagnostic.GetProperty("code").GetString() == "checkpoint_output_name_conflict"); + Assert.Equal(CommandExitCodes.DatabaseError, writeExit); + Assert.Equal(CommandErrorCodes.DbError, writeJson.GetProperty("error_code").GetString()); + Assert.Equal("db", File.ReadAllText(dbPath)); + Assert.False(Directory.Exists(dbPath + ".checkpoints")); + } + finally + { + DeleteWorkDirectory(root); + } + } + + [Fact] + public void Run_CheckpointPlan_UnixFifoSidecarIsRejectedWithoutHashing_Issue4890() + { + if (OperatingSystem.IsWindows()) + return; + + var root = TestProjectHelper.CreateTempProject("cdidx_db_checkpoint_fifo_4890"); + var dbPath = Path.Combine(root, "codeindex.db"); + try + { + File.WriteAllText(dbPath, "db"); + CreateUnixFifo(dbPath + "-wal"); + + var (dryRunExit, dryRunJson) = RunAndCaptureJson(["checkpoint", "fifo", "--dry-run", "--db", dbPath, "--json"]); + var (writeExit, writeJson) = RunAndCaptureJson(["checkpoint", "fifo", "--db", dbPath, "--json"]); + + Assert.Equal(CommandExitCodes.Success, dryRunExit); + Assert.False(dryRunJson.GetProperty("ready").GetBoolean()); + Assert.Contains( + dryRunJson.GetProperty("diagnostics").EnumerateArray(), + diagnostic => diagnostic.GetProperty("code").GetString() == "checkpoint_source_file_stat_failed"); + Assert.Equal(CommandExitCodes.DatabaseError, writeExit); + Assert.Equal(CommandErrorCodes.DbError, writeJson.GetProperty("error_code").GetString()); + Assert.False(Directory.Exists(dbPath + ".checkpoints")); + } + finally + { + DeleteWorkDirectory(root); + } + } + + [Theory] + [InlineData("source_content")] + [InlineData("sidecar_appears")] + [InlineData("sidecar_disappears")] + public void Run_CheckpointPlan_SourceDriftRefusesPublish_Issue4890(string driftKind) + { + var root = TestProjectHelper.CreateTempProject("cdidx_db_checkpoint_drift_4890"); + var dbPath = Path.Combine(root, "codeindex.db"); + try + { + File.WriteAllText(dbPath, "db"); + if (driftKind == "sidecar_disappears") + File.WriteAllText(dbPath + "-wal", "wal"); + DbCommandRunner.CheckpointPlanReadyForExecutionForTesting = driftKind switch + { + "sidecar_appears" => () => File.WriteAllText(dbPath + "-wal", "wal"), + "sidecar_disappears" => () => File.Delete(dbPath + "-wal"), + _ => () => File.AppendAllText(dbPath, "-changed"), + }; + + var (exitCode, json) = RunAndCaptureJson(["checkpoint", "drift", "--db", dbPath, "--json"]); + + Assert.Equal(CommandExitCodes.DatabaseError, exitCode); + Assert.Equal(CommandErrorCodes.DbError, json.GetProperty("error_code").GetString()); + Assert.Contains("plan drift detected", json.GetProperty("message").GetString(), StringComparison.Ordinal); + Assert.False(Directory.Exists(Path.Combine(dbPath + ".checkpoints", "drift"))); + } + finally + { + DbCommandRunner.CheckpointPlanReadyForExecutionForTesting = null; + DeleteWorkDirectory(root); + } + } + [Fact] public void Run_CheckpointTraversalName_JsonUsesUsageErrorAndSyntaxHint_Issue4477() { @@ -1005,6 +1254,7 @@ public void Run_CheckpointManifestOmitsAbsoluteDbPath_Issue3833() Assert.Equal(CommandExitCodes.Success, checkpointExit); var manifest = File.ReadAllText(Path.Combine(dbPath + ".checkpoints", "manifest", "manifest.txt")); + Assert.Contains("format_version=1", manifest); Assert.Contains("db_file=codeindex.db", manifest); Assert.DoesNotContain(dbPath, manifest); Assert.DoesNotContain(root, manifest); @@ -1166,7 +1416,8 @@ public void Run_CheckpointTempCleanupFailurePreservesOriginalFailure_Issue3029() var checkpointRoot = dbPath + ".checkpoints"; Directory.CreateDirectory(checkpointRoot); - File.WriteAllText(Path.Combine(checkpointRoot, "saved"), "checkpoint path blocker"); + DbCommandRunner.CheckpointPlanReadyForExecutionForTesting = () => + File.WriteAllText(Path.Combine(checkpointRoot, "saved"), "checkpoint path blocker"); DbCommandRunner.DeleteTemporaryDirectoryForTesting = path => { cleanupPath = path; @@ -1184,6 +1435,7 @@ public void Run_CheckpointTempCleanupFailurePreservesOriginalFailure_Issue3029() } finally { + DbCommandRunner.CheckpointPlanReadyForExecutionForTesting = null; DbCommandRunner.DeleteTemporaryDirectoryForTesting = null; DeleteWorkDirectory(root); }