[codex] Fix index watch failure handling (#2956)

* Fix watch pending batch cap (#2903)

* Propagate watch sub-run failures (#2902)

* Document watch failure events (#2902)

* Document watch pending cap fallback (#2903)

Author: Widthdom

USER_GUIDE.md

@@ -724,7 +724,7 @@ small repositories, and minutes or longer on very large monorepos with around
 
 By default, `cdidx index` stores the database in `<projectPath>/.cdidx/codeindex.db`, even if you run the command from another directory.
 
-`--watch` keeps the process alive after the initial scan and rebuilds the index incrementally as files are created, edited, renamed, or deleted. It uses `FileSystemWatcher` (FSEvents on macOS, inotify on Linux, ReadDirectoryChangesW on Windows), debounces bursts of events (`--debounce <ms>`, default 500 ms) into a single `--files` update, releases the per-DB index lock between batches so other `cdidx` commands can still query, and falls back to a full incremental rescan if the watcher buffer overflows. With `--json` it streams `status: "watching" / "updated" / "rescanned" / "overflow" / "stopped"` lifecycle events to stdout; otherwise it writes `[watch] …` summaries to stderr. Stop the loop with Ctrl+C (or SIGTERM); the final exit code is `0` for a clean stop. `--watch` cannot be combined with `--commits`, `--files`, or `--dry-run` — the loop already drives continuous incremental updates.
+`--watch` keeps the process alive after the initial scan and rebuilds the index incrementally as files are created, edited, renamed, or deleted. It uses `FileSystemWatcher` (FSEvents on macOS, inotify on Linux, ReadDirectoryChangesW on Windows), debounces bursts of events (`--debounce <ms>`, default 500 ms) into a single `--files` update, releases the per-DB index lock between batches so other `cdidx` commands can still query, and falls back to a full incremental rescan if the watcher buffer overflows or a pending path batch reaches its safety cap. With `--json` it streams `status: "watching" / "updated" / "rescanned" / "overflow" / "failed" / "stopped"` lifecycle events to stdout, and update/rescan events include `exit_code`; otherwise it writes `[watch] …` summaries with exit codes to stderr. Stop the loop with Ctrl+C (or SIGTERM); the final exit code is `0` when every batch succeeds, or the most recent non-zero sub-run exit code if a watch update/rescan failed before stop. `--watch` cannot be combined with `--commits`, `--files`, or `--dry-run` — the loop already drives continuous incremental updates.
 
 Indexing keeps the built-in skip lists (`node_modules`, `bin`, `obj`, lockfiles, etc.) and also honors user `.gitignore` plus optional `.cdidxignore` rules across full scans, `--files`, and `--commits` updates. A project-root `.codeindex/.cdidxignore` is also loaded as a workspace-scoped ignore file, which lets multi-workspace manifests keep local cdidx-only ignore rules out of the repository root. Ignore files are read as UTF-8, so non-ASCII patterns behave the same across platforms. On Windows, paths marked with the Hidden or System attribute are skipped before language detection so broad scans do not enter OS-owned caches such as `System Volume Information` or `$Recycle.Bin`; clear those attributes before indexing project-owned source files because ignore rules only exclude additional paths. When the project is inside Git, ignore matching follows the repository's `core.ignorecase` setting, even when the indexed project path is a subdirectory inside that repo; repo-root and other ancestor `.gitignore` files above that subdirectory still apply, and `--commits` resolves changed paths from the repository root before narrowing them back to the indexed project root. Nested directories that contain their own `.git` directory or gitfile are treated as repository boundaries and skipped by default. Indexed file paths are stored in Unicode NFC form so composed and decomposed path spellings match across platforms. `**` only gets Git-style special handling in the documented path forms rather than as an unrestricted cross-directory wildcard. If an update refresh includes ignore-file changes, cdidx automatically falls back to a full scan so newly ignored files are purged safely. Invalid ignore lines are skipped with a warning instead of aborting the whole run, while unreadable ignore files fail closed for that directory scope so cdidx does not index with incomplete rules.
 
@@ -2863,7 +2863,7 @@ interactive terminal では spinner と progress bar が動き続けます。待
 
 `cdidx index` は、別ディレクトリから実行しても、デフォルトでは `<projectPath>/.cdidx/codeindex.db` にDBを保存します。
 
-`--watch` を付けると初回スキャン後もプロセスが残り、`FileSystemWatcher`（macOS は FSEvents、Linux は inotify、Windows は ReadDirectoryChangesW）でファイルの作成・編集・リネーム・削除を検知して差分更新を繰り返します。`--debounce <ms>`（既定 500 ms）の窓内で発生したイベントは 1 つの `--files` 更新にまとめられ、バッチ間ではデータベースごとの index lock を解放するため別の `cdidx` コマンドからの問い合わせも可能です。Watcher バッファがオーバーフローした場合は変更を黙って捨てる代わりにフル差分再走査へフォールバックします。`--json` 時は `status: "watching" / "updated" / "rescanned" / "overflow" / "stopped"` のライフサイクルイベントを stdout に流し、そうでなければ `[watch] …` の要約を stderr に出力します。Ctrl+C（または SIGTERM）で正常に停止し、終了コードは正常停止で `0` です。`--watch` は連続的な差分更新を内蔵しているため `--commits` / `--files` / `--dry-run` と併用できません。
+`--watch` を付けると初回スキャン後もプロセスが残り、`FileSystemWatcher`（macOS は FSEvents、Linux は inotify、Windows は ReadDirectoryChangesW）でファイルの作成・編集・リネーム・削除を検知して差分更新を繰り返します。`--debounce <ms>`（既定 500 ms）の窓内で発生したイベントは 1 つの `--files` 更新にまとめられ、バッチ間ではデータベースごとの index lock を解放するため別の `cdidx` コマンドからの問い合わせも可能です。Watcher バッファがオーバーフローした場合や pending path batch が安全上限に達した場合は、変更を黙って捨てる代わりにフル差分再走査へフォールバックします。`--json` 時は `status: "watching" / "updated" / "rescanned" / "overflow" / "failed" / "stopped"` のライフサイクルイベントを stdout に流し、update/rescan event には `exit_code` を含めます。そうでなければ exit code 付きの `[watch] …` 要約を stderr に出力します。Ctrl+C（または SIGTERM）で停止し、すべての batch が成功していれば終了コードは `0`、停止前に watch update/rescan が失敗していれば直近の non-zero sub-run exit code です。`--watch` は連続的な差分更新を内蔵しているため `--commits` / `--files` / `--dry-run` と併用できません。
 
 デフォルト出力:
 

changelog.d/unreleased/2902.fixed.md

@@ -0,0 +1,18 @@
+---
+category: fixed
+issues:
+  - 2902
+affected:
+  - USER_GUIDE.md
+  - src/CodeIndex/Cli/IndexWatchRunner.cs
+  - src/CodeIndex/Cli/JsonOutputContracts.cs
+  - tests/CodeIndex.Tests/IndexWatchRunnerTests.cs
+---
+
+## English
+
+- **`index --watch` now reports failed sub-runs (#2902)** — watch batch events and human summaries include the sub-run exit code, and failed sub-runs are surfaced as failed watch batches instead of normal updates.
+
+## 日本語
+
+- **`index --watch` が sub-run failure を報告するようになりました (#2902)** — watch batch event と human summary に sub-run exit code を含め、失敗した sub-run を通常更新ではなく failed batch として表示します。

changelog.d/unreleased/2903.fixed.md

@@ -0,0 +1,17 @@
+---
+category: fixed
+issues:
+  - 2903
+affected:
+  - USER_GUIDE.md
+  - src/CodeIndex/Cli/IndexWatchRunner.cs
+  - tests/CodeIndex.Tests/IndexWatchRunnerTests.cs
+---
+
+## English
+
+- **`index --watch` now caps pending path batches (#2903)** — watch mode collapses very large pending file-event sets into a full rescan request instead of retaining an unbounded list of changed paths.
+
+## 日本語
+
+- **`index --watch` の pending path batch に上限を設けました (#2903)** — watch mode は大量のファイルイベントを個別 path の無制限リストとして保持せず、full rescan 要求へ畳み込むようになりました。

src/CodeIndex/Cli/IndexWatchRunner.cs

@@ -54,6 +54,7 @@ internal static int RunCore(
 
         var ignoreRuleRoot = GitHelper.TryGetRepositoryRoot(projectRoot) ?? Path.GetFullPath(projectRoot);
         var fileIndexer = new FileIndexer(projectRoot, ignoreCase, ignoreRuleRoot);
+        var watchExitCode = CommandExitCodes.Success;
 
         FileSystemWatcher? watcher = null;
         try
@@ -133,14 +134,14 @@ void Enqueue(string fullPath)
                 if (fullRescan)
                 {
                     EmitWatchOverflow(baseOptions, overflowReason);
-                    RunFullRescan(baseOptions, jsonOptions);
+                    RecordSubRunExitCode(ref watchExitCode, RunFullRescan(baseOptions, jsonOptions));
                     continue;
                 }
 
                 if (batch.Count == 0)
                     continue;
 
-                RunPartialUpdate(baseOptions, jsonOptions, batch);
+                RecordSubRunExitCode(ref watchExitCode, RunPartialUpdate(baseOptions, jsonOptions, batch));
             }
         }
         finally
@@ -153,10 +154,10 @@ void Enqueue(string fullPath)
         }
 
         EmitWatchStopped(baseOptions);
-        return CommandExitCodes.Success;
+        return watchExitCode;
     }
 
-    private static void RunPartialUpdate(
+    private static int RunPartialUpdate(
         IndexCommandOptions baseOptions,
         JsonSerializerOptions jsonOptions,
         IReadOnlyList<string> changedPaths)
@@ -167,18 +168,24 @@ private static void RunPartialUpdate(
         foreach (var path in changedPaths)
             args.Add(path);
 
-        InvokeSubRunAndEmit(baseOptions, jsonOptions, args, stopwatch, "updated", changedPaths.Count);
+        return InvokeSubRunAndEmit(baseOptions, jsonOptions, args, stopwatch, "updated", changedPaths.Count);
     }
 
-    private static void RunFullRescan(
+    private static int RunFullRescan(
         IndexCommandOptions baseOptions,
         JsonSerializerOptions jsonOptions)
     {
         var stopwatch = Stopwatch.StartNew();
         var args = BuildSubRunArgs(baseOptions);
         // No --files: this is a default incremental full scan.
         // --files を付けない: 通常のインクリメンタル全件スキャン。
-        InvokeSubRunAndEmit(baseOptions, jsonOptions, args, stopwatch, "rescanned", batchSize: null);
+        return InvokeSubRunAndEmit(baseOptions, jsonOptions, args, stopwatch, "rescanned", batchSize: null);
+    }
+
+    private static void RecordSubRunExitCode(ref int watchExitCode, int subRunExitCode)
+    {
+        if (subRunExitCode != CommandExitCodes.Success)
+            watchExitCode = subRunExitCode;
     }
 
     private static List<string> BuildSubRunArgs(IndexCommandOptions baseOptions)
@@ -214,7 +221,7 @@ private static List<string> BuildSubRunArgs(IndexCommandOptions baseOptions)
         return args;
     }
 
-    private static void InvokeSubRunAndEmit(
+    private static int InvokeSubRunAndEmit(
         IndexCommandOptions baseOptions,
         JsonSerializerOptions jsonOptions,
         List<string> args,
@@ -223,19 +230,24 @@ private static void InvokeSubRunAndEmit(
         int? batchSize)
     {
         string capturedJson;
+        int subRunExitCode;
         var previousOut = Console.Out;
         using var captureWriter = new StringWriter();
         Console.SetOut(captureWriter);
         try
         {
-            IndexCommandRunner.Run(args.ToArray(), jsonOptions);
+            subRunExitCode = IndexCommandRunner.Run(args.ToArray(), jsonOptions);
         }
         finally
         {
             Console.SetOut(previousOut);
         }
         stopwatch.Stop();
         capturedJson = captureWriter.ToString();
+        var eventStatus = subRunExitCode == CommandExitCodes.Success ? status : "failed";
+        var failureReason = subRunExitCode == CommandExitCodes.Success
+            ? null
+            : $"{status} sub-run exited with code {subRunExitCode.ToString(CultureInfo.InvariantCulture)}";
 
         if (baseOptions.Json)
         {
@@ -244,9 +256,11 @@ private static void InvokeSubRunAndEmit(
             // watch バッチであることを示すヘッダ行を先頭に流し、その後にサブ実行 JSON を出す。
             Console.Out.WriteLine(JsonSerializer.Serialize(new IndexWatchEventJsonResult
             {
-                Status = status,
+                Status = eventStatus,
                 BatchSize = batchSize,
                 ElapsedMs = stopwatch.ElapsedMilliseconds,
+                ExitCode = subRunExitCode,
+                Reason = failureReason,
             }, CliJsonSerializerContextFactory.Create(jsonOptions).IndexWatchEventJsonResult));
 
             var trimmed = capturedJson.TrimEnd('\r', '\n');
@@ -255,22 +269,32 @@ private static void InvokeSubRunAndEmit(
         }
         else
         {
-            var human = FormatHumanSummary(status, batchSize, stopwatch.ElapsedMilliseconds, capturedJson);
+            var human = FormatHumanSummary(eventStatus, batchSize, stopwatch.ElapsedMilliseconds, capturedJson, subRunExitCode);
             Console.Error.WriteLine(human);
         }
+
+        return subRunExitCode;
     }
 
-    private static string FormatHumanSummary(string status, int? batchSize, long elapsedMs, string subRunJson)
+    private static string FormatHumanSummary(string status, int? batchSize, long elapsedMs, string subRunJson, int exitCode)
     {
-        var prefix = status == "rescanned" ? "[watch] rescanned" : "[watch] updated";
+        var prefix = status switch
+        {
+            "rescanned" => "[watch] rescanned",
+            "failed" => "[watch] failed",
+            _ => "[watch] updated",
+        };
         var batchLabel = batchSize is int n
             ? $" {ConsoleUi.Counted(n, "path", format: "N0")}"
             : string.Empty;
 
         // Best-effort parse of the sub-run JSON to surface updated/removed/errors counts.
         // The summary is informational; a parse failure must not break the watch loop.
         // サブ実行 JSON から件数を best-effort で抽出。失敗してもループは続行する。
-        string detail = string.Empty;
+        var details = new List<string>
+        {
+            $"exit code {exitCode.ToString(CultureInfo.InvariantCulture)}",
+        };
         try
         {
             var trimmed = subRunJson.TrimEnd('\r', '\n');
@@ -284,15 +308,17 @@ private static string FormatHumanSummary(string status, int? batchSize, long ela
                     int updated = summary.TryGetProperty("updated", out var u) && u.TryGetInt32(out var uv) ? uv : 0;
                     int removed = summary.TryGetProperty("removed", out var r) && r.TryGetInt32(out var rv) ? rv : 0;
                     int errors = summary.TryGetProperty("errors", out var er) && er.TryGetInt32(out var erv) ? erv : 0;
-                    detail = $" (updated {updated}, removed {removed}, errors {errors})";
+                    details.Add($"updated {updated}");
+                    details.Add($"removed {removed}");
+                    details.Add($"errors {errors}");
                 }
             }
         }
         catch (JsonException)
         {
-            detail = string.Empty;
         }
 
+        var detail = details.Count > 0 ? $" ({string.Join(", ", details)})" : string.Empty;
         return $"{prefix}{batchLabel}{detail} in {elapsedMs.ToString("N0", System.Globalization.CultureInfo.InvariantCulture)} ms";
     }
 
@@ -373,18 +399,29 @@ private static void EmitWatchStopped(IndexCommandOptions baseOptions)
 /// </summary>
 internal sealed class FileChangeBatcher
 {
+    internal const int DefaultMaxPendingPaths = 4096;
+
     private readonly object _gate = new();
     private readonly HashSet<string> _pending;
     private DateTime _lastEventUtc = DateTime.MinValue;
     private bool _overflowRequested;
     private string? _overflowReason;
     private readonly TimeSpan _debounce;
     private readonly Func<DateTime> _clock;
+    private readonly int _maxPendingPaths;
 
-    public FileChangeBatcher(TimeSpan debounce, Func<DateTime>? clock = null, bool ignoreCase = true)
+    public FileChangeBatcher(
+        TimeSpan debounce,
+        Func<DateTime>? clock = null,
+        bool ignoreCase = true,
+        int maxPendingPaths = DefaultMaxPendingPaths)
     {
+        if (maxPendingPaths <= 0)
+            throw new ArgumentOutOfRangeException(nameof(maxPendingPaths), "Maximum pending path count must be positive.");
+
         _debounce = debounce;
         _clock = clock ?? (() => DateTime.UtcNow);
+        _maxPendingPaths = maxPendingPaths;
         // On case-sensitive filesystems (Linux ext4), `foo.py` and `Foo.py` are distinct files,
         // so coalescing them via OrdinalIgnoreCase would drop one rename leg and leave the
         // renamed-to file unindexed. The watch loop passes the filesystem's case sensitivity in.
@@ -397,7 +434,24 @@ public void Add(string path)
     {
         lock (_gate)
         {
-            _pending.Add(path);
+            if (_overflowRequested)
+            {
+                _lastEventUtc = _clock();
+                return;
+            }
+
+            if (!_pending.Contains(path))
+            {
+                if (_pending.Count >= _maxPendingPaths)
+                {
+                    RequestFullRescanLocked(
+                        $"pending path limit exceeded ({_maxPendingPaths.ToString("N0", CultureInfo.InvariantCulture)} paths)");
+                    return;
+                }
+
+                _pending.Add(path);
+            }
+
             _lastEventUtc = _clock();
         }
     }
@@ -406,10 +460,7 @@ public void RequestFullRescan(string? reason = null)
     {
         lock (_gate)
         {
-            _overflowRequested = true;
-            if (!string.IsNullOrEmpty(reason))
-                _overflowReason = reason;
-            _lastEventUtc = _clock();
+            RequestFullRescanLocked(reason);
         }
     }
 
@@ -445,4 +496,13 @@ public bool TryDrain(out IReadOnlyList<string> batch, out bool fullRescan, out s
             return true;
         }
     }
+
+    private void RequestFullRescanLocked(string? reason)
+    {
+        _pending.Clear();
+        _overflowRequested = true;
+        if (!string.IsNullOrEmpty(reason))
+            _overflowReason = reason;
+        _lastEventUtc = _clock();
+    }
 }

src/CodeIndex/Cli/JsonOutputContracts.cs

@@ -181,6 +181,7 @@ internal sealed class IndexWatchEventJsonResult
     public int? DebounceMs { get; init; }
     public int? BatchSize { get; init; }
     public long? ElapsedMs { get; init; }
+    public int? ExitCode { get; init; }
     public string? Reason { get; init; }
 }