Repair mixed folded backfill state

TESTING_GUIDE.md

@@ -379,6 +379,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding
   share seeded databases to verify that optimize dry-run reports sizes, readiness, lock state, recommendations, and planned work while preserving the source bytes and creating no lock artifacts; the same fixtures retain the writable optimize and lock/read-only URI mutation guards.
   `ConsoleUiTests.PrintUsage_WithoutBanner_HidesAsciiArtAndEasterEggFlags` keeps the optimize dry-run flag visible in full CLI usage.
 - Issue #4889 backfill-checkpoint coverage keeps the mutation preflight, folded-value validation, index lock, and artifact decision in one contract. Reuse the completed fixture for default no-op, forced `--checkpoint`, `--no-checkpoint` mutation, human output, a non-NULL drifted fold that requires a protected full repair, and a zero-row pending graph refresh; keep WAL/SHM open during the forced snapshot so Windows sharing semantics and the complete DB/WAL/SHM/manifest file set are exercised. Separate dry-run, invalid-DB, and held-lock assertions must prove that no checkpoint directory is created, while a real pending rewrite or pending graph refresh must retain the automatic checkpoint.
+- Issue #4946 fold-backfill coverage keeps CLI and MCP preflight aligned for a database that simultaneously contains NULL folded values and non-NULL drifted values. Both surfaces must promote that mixed state to one full rewrite and verify successfully in the first invocation. If that promoted rewrite is interrupted, its checkpoint must keep the retry in full-rewrite mode until completion and then be cleared; even an end-of-table checkpoint must be consumed and exactly verified before a later scoped NULL update can use targeted repair. Complete databases without a checkpoint must not pay for an additional present-value scan before their existing exact verification.
 - Issue #4856 maintenance-error coverage spans `CommandErrorWriterTests`, `QueryCommandRunnerTests`, `IndexCommandRunnerTests`, and `DbCommandRunnerTests`.
   Keep the SQLite primary-code matrix (`5`, `6`, `8`, `11`, `26`), caller-spelled relative paths, whole-value Unix/Windows/file-URI redaction, missing and invalid-header files, directory/inaccessible preflight classification, integrity corruption rows, lock-holder metadata, the `index --optimize` human preamble, valid JSON, and category-specific recovery hints aligned with classifier version `1`.
 - `IndexCommandRunnerTests.Run_DryRunWithRebuildAndMemoryTrace_SkipsConfirmationAndPreservesWorkspace_Issue4580`
@@ -1338,6 +1339,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests"
   seed 済み database を共有し、optimize dry-run が size、readiness、lock state、推奨、planned work を報告しつつ source byte を保持し、lock artifact を作成しないことを検証する。同じ fixture で、書き込み版 optimize と lock/read-only URI の mutation guard も維持する。
   `ConsoleUiTests.PrintUsage_WithoutBanner_HidesAsciiArtAndEasterEggFlags` は、full CLI usage に optimize dry-run flag が表示され続けることを固定します。
 - Issue #4889 の backfill-checkpoint coverage は、mutation preflight、folded値の検証、index lock、artifact 作成判断を1つの契約として固定します。完了済み fixture を既定no-op、明示 `--checkpoint`、`--no-checkpoint` mutation、human output、保護された全行修復を必要とする非NULLのfold drift、zero-rowのpending graph refreshで再利用し、明示snapshot中は WAL/SHM をopenしたままにしてWindowsの共有 semanticsとDB/WAL/SHM/manifest一式を検証してください。dry-run、無効DB、lock保持中のassertionではcheckpoint directoryが作られないことを個別に証明し、実際にpending rewriteまたはpending graph refreshがある場合はautomatic checkpointを維持してください。
+- Issue #4946 の fold-backfill coverage は、NULL の folded 値と非 NULL だが drift した値が同時に存在する database で CLI と MCP の preflight を同期させます。両 surface はこの混在状態を1回の全行 rewrite に昇格し、最初の invocation で検証成功しなければなりません。昇格した rewrite が中断された場合は、完了まで checkpoint に従って全行 rewrite mode を維持してから checkpoint を消去します。table 末尾に到達した checkpoint も、後続の scoped NULL update が対象修復を使う前に消費して厳密検証してください。checkpoint のない complete database では、既存の厳密検証より前に追加の present-value scan を実行しないでください。
 - Issue #4856 の maintenance-error coverage は `CommandErrorWriterTests`、`QueryCommandRunnerTests`、`IndexCommandRunnerTests`、`DbCommandRunnerTests` にまたがります。
   SQLite primary-code matrix (`5`、`6`、`8`、`11`、`26`)、呼び出し側表記を維持する relative path、Unix / Windows / file URI の値全体 redaction、missing / invalid-header file、directory / inaccessible preflight classification、integrity corruption row、lock-holder metadata、`index --optimize` の human preamble、有効な JSON、category 別 recovery hint を classifier version `1` と同期させてください。
 - `IndexCommandRunnerTests.Run_DryRunWithRebuildAndMemoryTrace_SkipsConfirmationAndPreservesWorkspace_Issue4580`

changelog.d/unreleased/4946.fixed.md

@@ -0,0 +1,20 @@
+---
+category: fixed
+issues:
+  - 4946
+affected:
+  - src/CodeIndex/Database/DbWriter.FoldBackfill.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Maintenance.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.Backfill.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
+  - tests/CodeIndex.Tests/McpServerToolsCallTests.cs
+  - TESTING_GUIDE.md
+---
+
+## English
+
+- **Fold backfill now repairs mixed missing and stale folded keys in one pass (#4946)** — `backfill-fold` and MCP `backfill_fold` distinguish targetable NULL values from non-NULL drift during preflight and promote mixed state to a full rewrite, so verification succeeds without requiring a rebuild or retry. An interrupted promoted rewrite remains in full-rewrite mode until its checkpointed work completes, and the CLI consumes and verifies even a completed persisted checkpoint before later scoped updates can use targeted repair.
+
+## 日本語
+
+- **fold backfill が missing / stale folded key の混在を1回で修復するようになりました (#4946)** — `backfill-fold` と MCP `backfill_fold` は preflight で対象修復可能な NULL 値と非 NULL の drift を区別し、混在時は全行 rewrite に昇格するため、rebuild や再試行なしで検証が成功します。昇格後に中断された rewrite は checkpoint の処理が完了するまで全行 rewrite mode を維持し、CLI は完了済みの永続 checkpoint も消費して検証してから、後続の scoped update に対する対象修復へ戻ります。

src/CodeIndex/Cli/IndexCommandRunner.Maintenance.cs

@@ -638,13 +638,27 @@ internal static int RunBackfillFold(
             // Missing or mismatched fold metadata means persisted keys may have been generated
             // by a different fold algorithm/runtime, so refresh every row from source names.
             // fold metadata 未記録 / 不一致時は全行再計算して version/runtime skew を解消する。
-            var rewriteAll = writer.ResolveFoldBackfillRewriteAll(!foldMetadataCurrentBefore);
+            var hasRewriteCheckpoint = writer.HasFoldBackfillRewriteCheckpoint();
+            var rewriteAll = writer.ResolveFoldBackfillRewriteAll(
+                !foldMetadataCurrentBefore || hasRewriteCheckpoint);
 
             var symbols = 0;
             var symbolReferences = 0;
             var verified = false;
             var userVersionAfter = userVersionBefore;
             var pendingRows = writer.CountBackfillFoldedColumns(rewriteAll);
+            if (!rewriteAll
+                && (pendingRows.Symbols > 0 || pendingRows.SymbolReferences > 0)
+                && !writer.AllPresentFoldedColumnValuesMatchCurrentFold())
+            {
+                // Missing folded values can use the targeted backfill, but any non-NULL
+                // drift must promote the same invocation to a full refresh. This avoids a
+                // partial pass failing verification when both states coexist.
+                // NULL の folded 値は対象行だけ修復できるが、非 NULL の drift も混在する場合は
+                // 同じ invocation を全行 refresh に昇格し、partial pass 後の検証失敗を防ぐ。
+                rewriteAll = true;
+                pendingRows = writer.CountBackfillFoldedColumns(rewriteAll);
+            }
             if (!rewriteAll
                 && pendingRows.Symbols == 0
                 && pendingRows.SymbolReferences == 0
@@ -662,6 +676,11 @@ internal static int RunBackfillFold(
             var symbolExtractorVersionsCurrent = writer.SymbolExtractorVersionsMatchCurrent();
             var mutationRequired = pendingRows.Symbols > 0
                 || pendingRows.SymbolReferences > 0
+                // Even when its cursors have reached the end, a persisted rewrite checkpoint
+                // must run through BackfillFoldedColumns so it is cleared before exact verification.
+                // cursor が末尾でも、永続化済み rewrite checkpoint は BackfillFoldedColumns を
+                // 通して消去してから厳密検証する必要がある。
+                || hasRewriteCheckpoint
                 || !foldReadyBefore
                 || csharpSymbolNameContractUpgradeRequired
                 || graphRefreshPending

src/CodeIndex/Database/DbWriter.FoldBackfill.cs

@@ -127,6 +127,12 @@ private bool AllFoldedColumnsBackfilledCore(
     }
 
     public bool AllFoldedColumnValuesMatchCurrentFold()
+        => AllFoldedColumnValuesMatchCurrentFoldCore(allowMissingValues: false);
+
+    internal bool AllPresentFoldedColumnValuesMatchCurrentFold()
+        => AllFoldedColumnValuesMatchCurrentFoldCore(allowMissingValues: true);
+
+    private bool AllFoldedColumnValuesMatchCurrentFoldCore(bool allowMissingValues)
     {
         var markdownSymbolIdentityFolds = BuildMarkdownSymbolIdentityFoldMap();
         var hasDisplayNameFolded =
@@ -156,7 +162,7 @@ WHERE s.name IS NOT NULL
                     reader.IsDBNull(6) ? null : reader.GetString(6),
                     markdownSymbolIdentityFolds);
                 var actual = reader.IsDBNull(2) ? null : reader.GetString(2);
-                if (!string.Equals(actual, expected, StringComparison.Ordinal))
+                if (!FoldedValueMatches(actual, expected, allowMissingValues))
                     return false;
                 var foldedDisplay = DbReader.FoldNameForLanguage(
                     reader.GetString(1),
@@ -170,7 +176,7 @@ WHERE s.name IS NOT NULL
                         ? foldedDisplay
                         : null;
                 var actualDisplay = reader.IsDBNull(3) ? null : reader.GetString(3);
-                if (!string.Equals(actualDisplay, expectedDisplay, StringComparison.Ordinal))
+                if (!FoldedValueMatches(actualDisplay, expectedDisplay, allowMissingValues))
                     return false;
             }
         }
@@ -200,7 +206,7 @@ FROM symbol_references r
                         reader.IsDBNull(4) ? null : reader.GetString(4),
                         reader.GetString(5));
                     var actual = reader.IsDBNull(1) ? null : reader.GetString(1);
-                    if (!string.Equals(actual, expected, StringComparison.Ordinal))
+                    if (!FoldedValueMatches(actual, expected, allowMissingValues))
                         return false;
                 }
 
@@ -210,7 +216,7 @@ FROM symbol_references r
                         reader.GetString(2),
                         reader.IsDBNull(4) ? null : reader.GetString(4));
                     var actual = reader.IsDBNull(3) ? null : reader.GetString(3);
-                    if (!string.Equals(actual, expected, StringComparison.Ordinal))
+                    if (!FoldedValueMatches(actual, expected, allowMissingValues))
                         return false;
                 }
             }
@@ -223,6 +229,10 @@ FROM symbol_references r
         return true;
     }
 
+    private static bool FoldedValueMatches(string? actual, string? expected, bool allowMissingValues)
+        => (allowMissingValues && actual == null)
+            || string.Equals(actual, expected, StringComparison.Ordinal);
+
     public bool AllFoldedColumnsBackfilled(IReadOnlyCollection<string> requireCurrentSymbolExtractorLanguages)
     {
         if (IsInTransaction())
@@ -441,6 +451,11 @@ internal bool IsFoldBackfillGraphRefreshPending()
             "1",
             StringComparison.Ordinal);
 
+    internal bool HasFoldBackfillRewriteCheckpoint()
+        => GetMetaString(FoldBackfillPhaseMetaKey) != null
+            || GetMetaString(FoldBackfillLastSymbolIdMetaKey) != null
+            || GetMetaString(FoldBackfillLastReferenceIdMetaKey) != null;
+
     private static int ToInt32Count(object? value)
     {
         var count = value is long l ? l : (value is int i ? i : 0);

src/CodeIndex/Mcp/McpToolHandlers.Backfill.cs

@@ -68,15 +68,27 @@ private async Task<JsonNode> ExecuteBackfillFoldAsync(JsonNode? id, JsonNode? ar
             foldReadyBefore = foldReadyBefore && foldMetadataCurrentBefore;
             var force = args?["force"]?.GetValue<bool>() ?? false;
             var rewriteAll = writer.ResolveFoldBackfillRewriteAll(
-                force || !foldMetadataCurrentBefore);
+                force
+                || !foldMetadataCurrentBefore
+                || writer.HasFoldBackfillRewriteCheckpoint());
+            (var totalSymbols, var totalSymbolReferences) =
+                writer.CountBackfillFoldedColumns(rewriteAll);
+            if (!rewriteAll
+                && (totalSymbols > 0 || totalSymbolReferences > 0)
+                && !writer.AllPresentFoldedColumnValuesMatchCurrentFold())
+            {
+                // Keep MCP aligned with the CLI: mixed missing and non-current folded
+                // values require one full repair rather than a partial pass that cannot verify.
+                // CLI と同様に、missing と non-current の folded 値が混在する場合は
+                // 検証不能な partial pass ではなく1回の全行修復へ昇格する。
+                rewriteAll = true;
+                (totalSymbols, totalSymbolReferences) =
+                    writer.CountBackfillFoldedColumns(rewriteAll);
+            }
             var symbols = 0;
             var symbolReferences = 0;
-            var totalSymbols = 0;
-            var totalSymbolReferences = 0;
             var verified = false;
             var userVersionAfter = userVersionBefore;
-
-            (totalSymbols, totalSymbolReferences) = writer.CountBackfillFoldedColumns(rewriteAll);
             if (dryRun)
             {
                 symbols = totalSymbols;