diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 529521b91f..22765eb02b 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -143,6 +143,8 @@ Current stable codes and triggers: Under WAL, `NORMAL` avoids per-commit fsync pressure during 500-row indexing batches while preserving database consistency after crashes. `DbWriter` runs `PRAGMA wal_checkpoint(PASSIVE)` after each outer transaction commit, and SQLite may also checkpoint automatically after the configured 1000-page threshold. Both checkpoint paths are opportunistic: active readers are not blocked, and an uncheckpointed WAL is expected state rather than corruption. If the process is killed after SQLite has committed a transaction but before checkpointing, the next normal opener rolls the WAL forward; no manual recovery step is required. If the process dies before a transaction commits, that transaction is rolled back by SQLite. +Index write batches also stamp `codeindex_meta.batch_in_progress=true` before starting a mutation transaction and clear it inside the transaction that commits the matching rows and readiness metadata. If the indexer crashes after the marker is written but before the commit clears it, the next writable DB open demotes readiness bits and warns: `Last batch did not complete; run cdidx index --rebuild to re-index from a known clean state.` Gracefully handled per-file errors clear the marker after rollback; orphaned markers are reserved for interrupted or crashed batches whose trust metadata should not be treated as clean. + Read-only fallback uses an immutable SQLite URI when the normal writable open cannot create or lock journal/WAL side files, so query commands can still read a DB from read-only or sandboxed storage. That fallback intentionally skips writable pragmas, migrations, and WAL recovery writes; if a WAL is present and must be observed, copy the `.db`, `.db-wal`, and `.db-shm` files together to a writable location or use a SQLite backup from an environment that can open the full WAL set. `status --json` exposes the resolved connection values under `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `page_count`, `freelist_count`, `page_size`) for automation and support diagnostics. `cdidx vacuum` runs `PRAGMA incremental_vacuum` against writable incremental-auto-vacuum DBs, and performs a one-time `PRAGMA auto_vacuum=INCREMENTAL` plus full `VACUUM` conversion for legacy no-autovacuum DBs. Writable opens also reject databases whose `PRAGMA user_version` contains readiness bits outside the current binary's `CurrentSchemaVersion` mask. Read-only status/query paths may still surface `index_newer_than_reader=true` as a degraded audit signal, but write-capable paths must fail with `E003_SCHEMA_TOO_NEW` so an older cdidx cannot silently rewrite a DB stamped by a newer one. @@ -2617,7 +2619,7 @@ flowchart TD - 次に OS の通常の `dlopen` 検索パス（`/lib`、`/usr/lib` など） 自己完結型 publish は `libe_sqlite3.so` を publish 出力に同梱し、リリース tarball に含め、修正後の `install.sh` がバイナリの隣に置くため、最初のプローブで成功する。これが欠けていると、`SqliteConnection` のインスタンス生成時点で `DllNotFoundException: Unable to load shared library 'e_sqlite3'` が送出され、**ユーザーコードが実行される前にプロセスが終了する**。 5. **スキーマ初期化。** `DbContext.ctor` が `PRAGMA journal_mode=WAL`、`PRAGMA busy_timeout=5000`、`CREATE TABLE IF NOT EXISTS` / `CREATE VIRTUAL TABLE IF NOT EXISTS fts_chunks USING fts5 (…)` / トリガー DDL を実行する。成功は、ネイティブライブラリがロードできるだけでなく、FTS5 がビルドに含まれた動作する SQLite であることも証明する（SQLitePCLRaw の同梱ビルドは常に FTS5 有効）。 -6. **スキャンと書き込み。** `FileIndexer` がプロジェクトツリーを走査し、ファイルを読み、言語を検出し、チャンク分割し、シンボルと参照を抽出し、`DbWriter` がトランザクションあたり500件ずつ UPSERT する。進捗は `ConsoleUi.SetProgressTheme()` でレンダリング。 +6. **スキャンと書き込み。** `FileIndexer` がプロジェクトツリーを走査し、ファイルを読み、言語を検出し、チャンク分割し、シンボルと参照を抽出し、`DbWriter` がトランザクションあたり500件ずつ UPSERT する。write batch は transaction の外で `codeindex_meta.batch_in_progress=true` を先に stamp し、同じ transaction 内で rows と readiness metadata を commit するときに marker を clear する。crash により marker が残った DB を次回 writable open すると readiness bit を落として rebuild を促す警告を出すため、stale な trust metadata を clean と誤読しない。進捗は `ConsoleUi.SetProgressTheme()` でレンダリング。 7. **FTS optimize。** 書き込みのコミット後、`INSERT INTO fts_chunks(fts_chunks) VALUES('optimize')` を実行。 8. **サマリー表示。** `Files / Chunks / Symbols / Refs / Elapsed`。 diff --git a/changelog.d/unreleased/2109.fixed.md b/changelog.d/unreleased/2109.fixed.md new file mode 100644 index 0000000000..b6423ae774 --- /dev/null +++ b/changelog.d/unreleased/2109.fixed.md @@ -0,0 +1,21 @@ +--- +category: fixed +issues: + - 2109 +affected: + - src/CodeIndex/Database/DbContext.cs + - src/CodeIndex/Database/DbWriter.cs + - src/CodeIndex/Cli/IndexCommandRunner.cs + - src/CodeIndex/Mcp/McpToolHandlers.cs + - tests/CodeIndex.Tests/DatabaseTests.cs + - tests/CodeIndex.Tests/IndexCommandRunnerTests.cs + - DEVELOPER_GUIDE.md +--- + +## English + +- **Index write batches now leave a crash-recovery marker (#2109)** — `cdidx` stamps an in-progress batch marker before mutating index rows and clears it atomically with the committing transaction, so the next writable DB open demotes readiness and warns users to rebuild when a prior batch crashed before completing. + +## 日本語 + +- **index write batch が crash recovery marker を残すようになりました (#2109)** — `cdidx` は index 行を変更する前に in-progress batch marker を stamp し、commit する transaction と同時に clear するため、前回 batch が完了前に crash した DB を次回 writable open したとき readiness を落として rebuild を促す警告を出します。 diff --git a/changelog.d/unreleased/2609.fixed.md b/changelog.d/unreleased/2609.fixed.md new file mode 100644 index 0000000000..4fb2ef2182 --- /dev/null +++ b/changelog.d/unreleased/2609.fixed.md @@ -0,0 +1,16 @@ +--- +category: fixed +issues: + - 2609 +affected: + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs + - tests/CodeIndex.Tests/SymbolExtractorTests.cs +--- + +## English + +- **Rust associated type defaults inside traits are now indexed as properties (#2609)** — The Rust symbol extractor now scans `protocol` trait containers when adding associated type default property symbols, matching the existing Rust trait kind and restoring `type Name = Value;` entries inside traits. + +## 日本語 + +- **Rust trait 内の associated type default が property として index されるようになりました (#2609)** — Rust symbol extractor は associated type default の property symbol を追加するとき、既存の Rust trait kind である `protocol` container を走査するようになり、trait 内の `type Name = Value;` entry を取りこぼさなくなりました。 diff --git a/src/CodeIndex/Cli/IndexCommandRunner.cs b/src/CodeIndex/Cli/IndexCommandRunner.cs index c0bd7bc8f9..7fff6cfc73 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.cs @@ -1766,6 +1766,7 @@ void ThrowIfUpdateCancelled() StartUpdateSpinnerIfNeeded(); currentUpdatePath = relPath; var absPath = Path.Combine(projectRoot, relPath.Replace('/', Path.DirectorySeparatorChar)); + var fileBatchMarked = false; try { if (!File.Exists(LongPath.EnsureWindowsPrefix(absPath))) @@ -2041,6 +2042,8 @@ void ThrowIfUpdateCancelled() } DemoteReadinessOnce(); + writer.MarkBatchInProgress(); + fileBatchMarked = true; using var txn = writer.BeginTransaction(); writer.PurgeStaleFilesSharingChecksum(projectRoot, record.Path, record.Checksum); if (projectRootWritten) @@ -2068,6 +2071,7 @@ void ThrowIfUpdateCancelled() // Validate content for encoding issues / エンコーディング問題を検証 var issues = FileIndexer.ValidateContent(record.Path, rawBytes, content); writer.InsertIssues(fileId, issues); + writer.ClearBatchInProgress(); txn.Commit(); updated++; @@ -2108,6 +2112,8 @@ void ThrowIfUpdateCancelled() } DemoteReadinessOnce(); + if (fileBatchMarked) + writer.ClearBatchInProgress(); GlobalToolLog.Error($"index_update_file_failed path={CollapseLineBreaks(relPath)}\n{GlobalToolLog.FormatExceptionChain(ex)}"); errors++; @@ -2167,6 +2173,8 @@ void ThrowIfUpdateCancelled() priorFoldFingerprint == currentFoldFingerprint); if (readinessDemoted && errors == 0) { + writer.MarkBatchInProgress(); + using var readinessTxn = writer.BeginTransaction(); // Restore each readiness bit independently based on what the DB carried BEFORE // ClearReadyFlags wiped them. A pre-#86 DB (user_version=3, i.e. Graph+Issues but // no Fold) must keep Graph+Issues after a successful partial update, even though @@ -2251,6 +2259,8 @@ void ThrowIfUpdateCancelled() } writer.WriteCdidxWriterVersion(ConsoleUi.LoadVersion()); writer.SetMeta(SymbolKindFilterMetaKey, options.SymbolKindFilter.Signature); + writer.ClearBatchInProgress(); + readinessTxn.Commit(); } if (errors == 0) { @@ -3085,6 +3095,7 @@ void ThrowIfFullScanCancelled(int filesProcessed, int? filesTotal) // full-scan の書き込み全体を outer transaction に入れ、中断時に readiness clear / // purge / per-file write をまとめて rollback する。 ThrowIfFullScanCancelled(0, files.Count); + writer.MarkBatchInProgress(); using var fullScanTxn = writer.BeginTransaction(); writer.ClearReadyFlags(); writer.ClearHotspotFamilyReady(); @@ -3722,6 +3733,7 @@ void StopJsonHeartbeat() // ここで stamp する。full scan / partial update を問わず最新の HEAD を保存する。 StampIndexedHeadMetadata(writer, projectRoot); } + writer.ClearBatchInProgress(); fullScanTxn.Commit(); stopwatch.Stop(); // Detect cwd drift between option-parsing and finalize. See RunUpdateMode for the diff --git a/src/CodeIndex/Database/DbContext.cs b/src/CodeIndex/Database/DbContext.cs index 0f98cbf5c2..7b663ff882 100644 --- a/src/CodeIndex/Database/DbContext.cs +++ b/src/CodeIndex/Database/DbContext.cs @@ -189,6 +189,7 @@ public DbContext(string dbPath) ApplyConnectionPerformancePragmas(); RegisterConnectionFunctionsWithRetry(_connection); _isReadOnly = true; + WarnIfBatchInProgress(); return; } catch @@ -231,6 +232,7 @@ public DbContext(string dbPath) ExecuteSynchronousPragmaWithFallback(Execute); Execute($"PRAGMA wal_autocheckpoint={DefaultWalAutocheckpointPages}"); Execute("PRAGMA optimize=0x10002"); + WarnIfBatchInProgress(); } catch (SqliteException ex) when (IsReadOnlyOpenError(ex)) { @@ -250,6 +252,7 @@ public DbContext(string dbPath) ApplyConnectionPerformancePragmas(); RegisterConnectionFunctionsWithRetry(_connection); _isReadOnly = true; + WarnIfBatchInProgress(); } catch { @@ -271,6 +274,17 @@ public DbContext(string dbPath) _suppressWriteWorkTracking = false; } + private void WarnIfBatchInProgress() + { + var raw = GetMetaString(BatchInProgressMetaKey); + if (string.Equals(raw, "true", StringComparison.OrdinalIgnoreCase)) + { + Console.Error.WriteLine("Warning: Last batch did not complete; run `cdidx index --rebuild` to re-index from a known clean state."); + if (!_isReadOnly) + Execute("PRAGMA user_version = 0"); + } + } + private void ApplyConnectionPerformancePragmas() { Execute($"PRAGMA cache_size=-{ReadPositiveIntEnvironment(CacheSizeEnvironmentVariable, DefaultCacheSizeKb)}"); @@ -1047,6 +1061,7 @@ private static void RegisterConnectionFunctionsWithRetry( // ファイル数。index 済み件数ではなく scan coverage の信号であり、現行 index が stamp // するまでは reader 側で省略する。 public const string UnknownExtensionFileCountMetaKey = "unknown_extension_file_count"; + public const string BatchInProgressMetaKey = "batch_in_progress"; // Issue #1546: case-sensitivity of the workspace filesystem the most recent successful // index ran on, persisted as the string "true" / "false". Resolved via the probe in // `PathCasing` (which honors `core.ignorecase` when the project is a git workspace and diff --git a/src/CodeIndex/Database/DbWriter.cs b/src/CodeIndex/Database/DbWriter.cs index 8d093d4e70..88fa2d6fed 100644 --- a/src/CodeIndex/Database/DbWriter.cs +++ b/src/CodeIndex/Database/DbWriter.cs @@ -1574,6 +1574,10 @@ public int RecordFtsIncrementalWrite() return value; } + public void MarkBatchInProgress() => SetMeta(DbContext.BatchInProgressMetaKey, "true"); + + public void ClearBatchInProgress() => SetMeta(DbContext.BatchInProgressMetaKey, "false"); + public bool OptimizeFtsIfIncrementalWriteThresholdReached(int threshold = DefaultFtsOptimizeIncrementalWriteThreshold) { if (threshold <= 0) diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs index 854d5957e8..75e3fd99e8 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs @@ -10070,7 +10070,7 @@ private static void ExtractRustAssociatedTypeDefaultSymbols(long fileId, string[ StartColumn = nameGroup.Index, EndLine = lineNumber, Signature = lines[lineIndex].Trim(), - ContainerKind = "interface", + ContainerKind = trait.Kind, ContainerName = trait.Name, ContainerQualifiedName = trait.ContainerQualifiedName, Visibility = match.Groups["visibility"].Success ? match.Groups["visibility"].Value : null, diff --git a/src/CodeIndex/Mcp/McpToolHandlers.cs b/src/CodeIndex/Mcp/McpToolHandlers.cs index cce9b46760..24dc2a959c 100644 --- a/src/CodeIndex/Mcp/McpToolHandlers.cs +++ b/src/CodeIndex/Mcp/McpToolHandlers.cs @@ -2419,6 +2419,7 @@ void WriteProjectRootOnce() foreach (var filePath in files) { + var fileBatchMarked = false; try { var (record, content, rawBytes, _) = indexer.BuildRecordWithRawBytes(filePath); @@ -2442,6 +2443,8 @@ void WriteProjectRootOnce() continue; } + writer.MarkBatchInProgress(); + fileBatchMarked = true; using var txn = writer.BeginTransaction(); var fileId = writer.UpsertFile(record); var chunks = ChunkSplitter.Split(fileId, content); @@ -2465,6 +2468,7 @@ void WriteProjectRootOnce() var issues = FileIndexer.ValidateContent(record.Path, rawBytes, content); writer.InsertIssues(fileId, issues); WriteProjectRootOnce(); + writer.ClearBatchInProgress(); txn.Commit(); } catch (FileIndexer.BinaryFileSkippedException) @@ -2487,6 +2491,8 @@ void WriteProjectRootOnce() } catch { + if (fileBatchMarked) + writer.ClearBatchInProgress(); errors++; } processed++; @@ -2506,6 +2512,8 @@ void WriteProjectRootOnce() _ = priorMetadataTargetCsharp; if (errors == 0) { + writer.MarkBatchInProgress(); + using var readinessTxn = writer.BeginTransaction(); writer.MarkGraphReady(); writer.MarkIssuesReady(); writer.MarkSqlGraphContractReady(); @@ -2616,6 +2624,8 @@ void WriteProjectRootOnce() { // Best-effort; never fail an otherwise-successful index run. } + writer.ClearBatchInProgress(); + readinessTxn.Commit(); } var (totalFiles, totalChunks, totalSymbols, totalReferences) = writer.GetCounts(); diff --git a/tests/CodeIndex.Tests/DatabaseTests.cs b/tests/CodeIndex.Tests/DatabaseTests.cs index 0ae07dd82e..7ff7b0133b 100644 --- a/tests/CodeIndex.Tests/DatabaseTests.cs +++ b/tests/CodeIndex.Tests/DatabaseTests.cs @@ -71,6 +71,70 @@ public void OptimizeFtsIfIncrementalWriteThresholdReached_RunsOnlyAtThreshold() Assert.Equal(0, _writer.GetFtsIncrementalWritesSinceOptimize()); } + [Fact] + public void DbContext_OpenWithBatchInProgress_Warns() + { + _writer.MarkBatchInProgress(); + + var stderr = ConsoleCapture.CaptureError(() => + { + using var reopened = new DbContext(_dbPath); + }); + + Assert.Contains("Last batch did not complete", stderr); + Assert.Contains("cdidx index --rebuild", stderr); + } + + [Fact] + public void DbContext_OpenWithBatchInProgress_DemotesReadiness() + { + _writer.MarkGraphReady(); + _writer.MarkIssuesReady(); + _writer.MarkBatchInProgress(); + + using (var reopened = new DbContext(_dbPath)) + { + Assert.Equal(0, reopened.GetUserVersion()); + } + } + + [Fact] + public void BatchInProgress_ClearInsideCommittedTransaction_PersistsCleanState() + { + _writer.MarkBatchInProgress(); + + using (var txn = _writer.BeginTransaction()) + { + _writer.ClearBatchInProgress(); + txn.Commit(); + } + + var stderr = ConsoleCapture.CaptureError(() => + { + using var reopened = new DbContext(_dbPath); + }); + + Assert.DoesNotContain("Last batch did not complete", stderr); + } + + [Fact] + public void BatchInProgress_ClearInsideRolledBackTransaction_LeavesRecoveryWarning() + { + _writer.MarkBatchInProgress(); + + using (var txn = _writer.BeginTransaction()) + { + _writer.ClearBatchInProgress(); + } + + var stderr = ConsoleCapture.CaptureError(() => + { + using var reopened = new DbContext(_dbPath); + }); + + Assert.Contains("Last batch did not complete", stderr); + } + [Fact] public void Constructor_NewDatabaseEnablesIncrementalAutoVacuum() { diff --git a/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs b/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs index 07c12b05c9..a18c59c14b 100644 --- a/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs +++ b/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs @@ -3435,8 +3435,12 @@ public void Run_FullScan_CancelledAfterReadinessDemotion_RollsBackExistingIndex( Assert.True(hookInvoked); Assert.Equal(CommandExitCodes.Interrupted, interruptedExitCode); - using (var db = new DbContext(dbPath)) - Assert.Equal(initialReadiness, db.GetUserVersion()); + var recoveryWarning = ConsoleCapture.CaptureError(() => + { + using var db = new DbContext(dbPath); + Assert.Equal(0, db.GetUserVersion()); + }); + Assert.Contains("Last batch did not complete", recoveryWarning); Assert.DoesNotContain("later.cs", ReadIndexedPaths(dbPath)); } finally diff --git a/tests/CodeIndex.Tests/SymbolExtractorTests.cs b/tests/CodeIndex.Tests/SymbolExtractorTests.cs index 40055f8f5b..4a5ea8ed4c 100644 --- a/tests/CodeIndex.Tests/SymbolExtractorTests.cs +++ b/tests/CodeIndex.Tests/SymbolExtractorTests.cs @@ -14409,13 +14409,13 @@ fn build(&self) { Assert.Contains(symbols, s => s.Kind == "property" && s.Name == "Output" - && s.ContainerKind == "interface" + && s.ContainerKind == "protocol" && s.ContainerName == "Builder" && s.ReturnType == "()"); Assert.Contains(symbols, s => s.Kind == "property" && s.Name == "Error" - && s.ContainerKind == "interface" + && s.ContainerKind == "protocol" && s.ContainerName == "Builder" && s.ReturnType == "String"); Assert.DoesNotContain(symbols, s => s.Kind == "property" && s.Name == "Pending");