Fix ignore rule parsing and loading edge cases

DEVELOPER_GUIDE.md

@@ -132,7 +132,9 @@ Hook failures are isolated to that hook invocation: assembly load, construction,
 
 ### Ignore file parsing
 
-`.gitignore` and `.cdidxignore` parsing follows Git's whitespace rules for pattern lines: leading spaces and tabs are literal pattern characters, `#` starts a comment only when it is the first unescaped character, and unescaped trailing spaces or tabs are trimmed. Escape a trailing space or tab with `\` when the whitespace is part of the filename pattern.
+`.gitignore` and `.cdidxignore` parsing follows Git's whitespace rules for pattern lines: leading unescaped spaces and tabs are ignored before comment/pattern parsing, `#` starts a comment only when it is the first unescaped character after that trim, and unescaped trailing spaces or tabs are trimmed. Escape a leading, trailing, or `#` character with `\` when it is part of the filename pattern.
+
+Ignore-file reads avoid `File.Exists` / `File.ReadLines` time-of-check/time-of-use races: the scanner attempts the UTF-8 read directly, treats missing files as absent rules, treats permission-denied ignore files as warnings while preserving inherited ancestor rules, and treats other I/O failures as unavailable rules so callers can avoid indexing with stale or unknown local ignore state. Ignore patterns are capped at 512 tokens and compiled with the non-backtracking regex engine plus a match timeout so malformed or untrusted ignore files cannot stall a scan with excessive regex work.
 
 Bracket expressions follow Git-compatible glob behavior: both `[!a]` and `[^a]` are treated as negated character classes when `!` or `^` appears immediately after `[`. A caret elsewhere in the class is literal (`[a^b]`), and a literal leading caret must be escaped (`[\^a]`).
 

USER_GUIDE.md

@@ -1360,6 +1360,10 @@ Successful narrow update JSON reports `mode: "update"` with `summary.updated`, `
 
 `cdidx index` automatically adds `.cdidx/` to `.git/info/exclude`. You don't need to edit `.gitignore` just to hide the local index, and user-authored `.gitignore` rules are honored during scanning and scoped updates. If you want cdidx-only exclusions without changing Git behavior, add a `.cdidxignore` file.
 
+### Project-local ignore
+
+Place `.cdidxignore` in the project root or any subdirectory to add cdidx-only ignore rules without changing Git behavior. It uses the same Git-style pattern syntax as `.gitignore`: leading unescaped spaces and tabs are ignored, `#` comments are recognized after that leading whitespace, trailing unescaped whitespace is trimmed, and `\#` or `\ ` keep literal characters in the pattern. In each directory, `.gitignore` is loaded first and `.cdidxignore` second, so `.cdidxignore` extends the active rules and later `!` patterns can re-include paths ignored earlier in the same directory scope. A project-root `.codeindex/.cdidxignore` is also loaded as a workspace-scoped ignore file for cdidx-only rules that should not sit at the repository root.
+
 `.git/info/exclude` is a standard Git mechanism that works just like `.gitignore`. Many tools use `.git/info/exclude` or store data inside `.git/` to avoid polluting `.gitignore` — git-lfs, git-secret, git-crypt, git-annex, Husky, pre-commit, JetBrains IDEs, VS Code (GitLens), Eclipse, etc.
 
 ## Git branch switching

changelog.d/unreleased/1620.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 1620
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+---
+
+## English
+
+- **Ignore parsing now follows leading-whitespace comment and pattern rules (#1620)** — `.gitignore` and `.cdidxignore` lines now trim leading unescaped whitespace before detecting comments or patterns, while escaped leading spaces and escaped `#` remain literal pattern characters.
+
+## 日本語
+
+- **ignore parsing が先頭空白つきのコメントとパターン規則に従うようになりました (#1620)** — `.gitignore` / `.cdidxignore` の行はコメントやパターン判定の前に未エスケープの先頭空白を取り除き、エスケープされた先頭空白と `#` は引き続きリテラルのパターン文字として扱います。

changelog.d/unreleased/1621.docs.md

@@ -0,0 +1,16 @@
+---
+category: docs
+issues:
+  - 1621
+affected:
+  - USER_GUIDE.md
+  - src/CodeIndex/Cli/ConsoleUi.cs
+---
+
+## English
+
+- **Documented project-local `.cdidxignore` semantics (#1621)** — the user guide and CLI help now describe where `.cdidxignore` is loaded, its Git-style syntax, and how it extends `.gitignore` rules.
+
+## 日本語
+
+- **project-local `.cdidxignore` の意味論を文書化しました (#1621)** — user guide と CLI help で、`.cdidxignore` の読み込み場所、Git 形式の構文、`.gitignore` ルールを拡張する扱いを説明しました。

changelog.d/unreleased/1653.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 1653
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Ignore-file reads now avoid exists/read TOCTOU gaps (#1653)** — the scanner reads ignore files directly, treats files deleted during the read window as absent rules, and keeps other I/O failures from silently indexing with unknown local ignore state.
+
+## 日本語
+
+- **ignore file 読み込みが exists/read の TOCTOU 差分を避けるようになりました (#1653)** — scanner は ignore file を直接読み、読み込み窓で削除されたファイルはルールなしとして扱い、その他の I/O 失敗では未知のローカル ignore 状態で黙って index しないようにしました。

changelog.d/unreleased/1657.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 1657
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+---
+
+## English
+
+- **Permission-denied ignore files now preserve inherited rules (#1657)** — unreadable `.gitignore` / `.cdidxignore` files are reported as warnings while previously loaded ancestor rules remain active for the scan.
+
+## 日本語
+
+- **権限不足の ignore file でも継承済みルールを維持するようになりました (#1657)** — 読み取れない `.gitignore` / `.cdidxignore` は warning として報告しつつ、既に読み込まれた ancestor rules は scan 中に有効なまま維持します。

changelog.d/unreleased/1658.fixed.md

@@ -0,0 +1,17 @@
+---
+category: fixed
+issues:
+  - 1658
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Ignore-pattern regexes now have complexity bounds (#1658)** — ignore rules are capped in length and compiled with the non-backtracking regex engine plus a match timeout to avoid excessive regex work from malformed patterns.
+
+## 日本語
+
+- **ignore pattern regex に複雑性上限を追加しました (#1658)** — malformed pattern による過剰な regex 処理を避けるため、ignore rule の長さを制限し、non-backtracking regex engine と match timeout でコンパイルします。

src/CodeIndex/Cli/ConsoleUi.cs

@@ -767,6 +767,7 @@ private static void PrintFlagReference(Action<string> WriteHelpLine)
         Console.WriteLine();
         Console.WriteLine("Index and update options:");
         Console.WriteLine("  --db <path>                Database file path (default for index: <projectPath>/.cdidx/codeindex.db)");
+        WriteHelpLine("  .cdidxignore               Optional project-local ignore file; loaded after .gitignore in each directory");
         Console.WriteLine("  --rebuild                  Delete existing DB and rebuild from scratch");
         Console.WriteLine("  --verbose                  Show per-file status ([OK  ]/[SKIP]/[DEL ]/[ERR ])");
         Console.WriteLine("  --dry-run                  Scan files without writing to the database");

src/CodeIndex/Indexer/Scanning/FileIndexer.cs

@@ -79,6 +79,8 @@ PathFilterKind.ExcludedByDefaultFile or
     private static readonly string[] HotspotFamilyMarkerLanguages = ["csharp", "vb", "fsharp", "msbuild"];
     private const int ConflictMarkerScanLimitBytes = 50 * 1024;
     private static readonly string[] IgnoreFileNames = [".gitignore", ".cdidxignore"];
+    private const int MaxIgnorePatternLength = 512;
+    private static readonly TimeSpan IgnoreRegexMatchTimeout = TimeSpan.FromMilliseconds(100);
     // Extension-to-language mapping / 拡張子→言語名マッピング
     private static readonly Dictionary<string, string> LangMap = new(StringComparer.OrdinalIgnoreCase)
     {
@@ -496,6 +498,12 @@ internal static bool TryParse(string sourceDirectory, string rawLine, bool ignor
             if (!TryTokenize(rawLine, out var tokens))
                 return false;
 
+            if (tokens.Count > MaxIgnorePatternLength)
+            {
+                errorMessage = $"Invalid ignore rule skipped: pattern exceeds {MaxIgnorePatternLength} characters";
+                return false;
+            }
+
             if (tokens[0] is { Value: '#', Escaped: false })
                 return false;
 
@@ -596,6 +604,8 @@ private static bool TryTokenize(string rawLine, out List<PatternToken> tokens)
 
             while (tokens.Count > 0 && tokens[^1] is { Value: ' ' or '\t', Escaped: false })
                 tokens.RemoveAt(tokens.Count - 1);
+            while (tokens.Count > 0 && tokens[0] is { Value: ' ' or '\t', Escaped: false })
+                tokens.RemoveAt(0);
 
             return tokens.Count > 0;
         }
@@ -663,7 +673,10 @@ private static Regex BuildMatcher(IReadOnlyList<PatternToken> pattern, bool igno
             }
 
             builder.Append('$');
-            return new Regex(builder.ToString(), RegexOptions.CultureInvariant | RegexOptions.Compiled);
+            return new Regex(
+                builder.ToString(),
+                RegexOptions.CultureInvariant | RegexOptions.Compiled | RegexOptions.NonBacktracking,
+                IgnoreRegexMatchTimeout);
         }
 
         private static bool TryBuildCharacterClass(IReadOnlyList<PatternToken> pattern, ref int index, StringBuilder builder, bool ignoreCase)
@@ -2230,8 +2243,6 @@ private IgnoreRuleLoadResult LoadIgnoreRulesForDirectory(
         {
             var ignorePath = Path.Combine(dir, ignoreFileName);
             var prefixedIgnorePath = LongPath.EnsureWindowsPrefix(ignorePath);
-            if (!File.Exists(prefixedIgnorePath))
-                continue;
 
             try
             {
@@ -2247,9 +2258,16 @@ private IgnoreRuleLoadResult LoadIgnoreRulesForDirectory(
             }
             catch (UnauthorizedAccessException)
             {
-                errors.Add(new ScanError(ToRelativePath(ignorePath), $"Could not read {ignoreFileName}."));
-                fullyScanned = false;
-                ignoreRulesAvailable = false;
+                if (!File.Exists(prefixedIgnorePath))
+                    throw;
+
+                errors.Add(new ScanError(ToRelativePath(ignorePath), $"Could not read {ignoreFileName} due to permissions.", ScanIssueSeverity.Warning));
+            }
+            catch (FileNotFoundException)
+            {
+            }
+            catch (DirectoryNotFoundException)
+            {
             }
             catch (IOException)
             {
@@ -2270,9 +2288,6 @@ private IgnoreRuleLoadResult LoadWorkspaceConfigIgnoreRules(
         ref bool fullyScanned)
     {
         var configIgnorePath = Path.Combine(_projectRoot, ".codeindex", ".cdidxignore");
-        if (!File.Exists(LongPath.EnsureWindowsPrefix(configIgnorePath)))
-            return new IgnoreRuleLoadResult(inheritedIgnoreRules, IgnoreRulesAvailable: true);
-
         return LoadIgnoreRulesFile(
             sourceDirectory: _projectRoot,
             ignorePath: configIgnorePath,
@@ -2328,9 +2343,19 @@ private IgnoreRuleLoadResult LoadIgnoreRulesFile(
         }
         catch (UnauthorizedAccessException)
         {
-            errors.Add(new ScanError(ToRelativePath(ignorePath), $"Could not read {ignoreFileName}."));
-            fullyScanned = false;
-            return new IgnoreRuleLoadResult(inheritedIgnoreRules, IgnoreRulesAvailable: false);
+            if (!File.Exists(prefixedIgnorePath))
+                throw;
+
+            errors.Add(new ScanError(ToRelativePath(ignorePath), $"Could not read {ignoreFileName} due to permissions.", ScanIssueSeverity.Warning));
+            return new IgnoreRuleLoadResult(inheritedIgnoreRules, IgnoreRulesAvailable: true);
+        }
+        catch (FileNotFoundException)
+        {
+            return new IgnoreRuleLoadResult(inheritedIgnoreRules, IgnoreRulesAvailable: true);
+        }
+        catch (DirectoryNotFoundException)
+        {
+            return new IgnoreRuleLoadResult(inheritedIgnoreRules, IgnoreRulesAvailable: true);
         }
         catch (IOException)
         {

tests/CodeIndex.Tests/FileIndexerTests.cs

@@ -1476,6 +1476,62 @@ public void ScanFiles_RespectsGitignorePatternsAndNegation()
         }
     }
 
+    [Fact]
+    public void ScanFiles_TrimsLeadingWhitespaceBeforeParsingIgnoreLines()
+    {
+        var tempDir = Path.Combine(Path.GetTempPath(), $"codeindex_test_{Guid.NewGuid():N}");
+        try
+        {
+            Directory.CreateDirectory(tempDir);
+            File.WriteAllText(Path.Combine(tempDir, ".gitignore"), "  # comment\n  *.tmp\n\\ leading.py\n\\#literal.py\n", Encoding.UTF8);
+            File.WriteAllText(Path.Combine(tempDir, "keep.py"), "print('keep')");
+            File.WriteAllText(Path.Combine(tempDir, "ignored.tmp"), "ignored");
+            File.WriteAllText(Path.Combine(tempDir, " leading.py"), "print('literal leading space')");
+            File.WriteAllText(Path.Combine(tempDir, "#literal.py"), "print('literal hash')");
+
+            var indexer = new FileIndexer(tempDir);
+            var files = indexer.ScanFiles()
+                .Select(path => Path.GetRelativePath(tempDir, path).Replace('\\', '/'))
+                .OrderBy(path => path, StringComparer.Ordinal)
+                .ToList();
+
+            Assert.Equal([".gitignore", "keep.py"], files);
+        }
+        finally
+        {
+            Directory.Delete(tempDir, true);
+        }
+    }
+
+    [Fact]
+    public void ScanFiles_ReportsOverlongIgnorePatternAndContinues()
+    {
+        var tempDir = Path.Combine(Path.GetTempPath(), $"codeindex_test_{Guid.NewGuid():N}");
+        try
+        {
+            Directory.CreateDirectory(tempDir);
+            File.WriteAllText(Path.Combine(tempDir, ".gitignore"), $"{new string('a', 513)}\n*.tmp\n", Encoding.UTF8);
+            File.WriteAllText(Path.Combine(tempDir, "keep.py"), "print('keep')");
+            File.WriteAllText(Path.Combine(tempDir, "ignored.tmp"), "ignored");
+
+            var result = new FileIndexer(tempDir).ScanFilesDetailed();
+            var files = result.Files
+                .Select(path => Path.GetRelativePath(tempDir, path).Replace('\\', '/'))
+                .OrderBy(path => path, StringComparer.Ordinal)
+                .ToList();
+
+            Assert.Equal([".gitignore", "keep.py"], files);
+            var warning = Assert.Single(result.Errors);
+            Assert.Equal(FileIndexer.ScanIssueSeverity.Warning, warning.Severity);
+            Assert.Contains("pattern exceeds 512 characters", warning.Message);
+            Assert.False(result.HadErrors);
+        }
+        finally
+        {
+            Directory.Delete(tempDir, true);
+        }
+    }
+
     [Fact]
     public void ScanFiles_RespectsCdidxignoreAndNestedGitignore()
     {
@@ -1639,7 +1695,7 @@ public void BuildRecord_NormalizesRelativePathToNfc()
     }
 
     [Fact]
-    public void ScanFiles_FailsClosedWhenRootIgnoreFileIsUnreadable()
+    public void ScanFiles_PreservesInheritedRulesWhenRootIgnoreFileIsUnreadable()
     {
         if (OperatingSystem.IsWindows())
             return;
@@ -1659,8 +1715,17 @@ public void ScanFiles_FailsClosedWhenRootIgnoreFileIsUnreadable()
             var indexer = new FileIndexer(tempDir);
             var result = indexer.ScanFilesDetailed();
 
-            Assert.Empty(result.Files);
-            Assert.Contains(result.Errors, error => error.Path == ".gitignore" && error.Message == "Could not read .gitignore.");
+            var files = result.Files
+                .Select(path => Path.GetRelativePath(tempDir, path).Replace('\\', '/'))
+                .OrderBy(path => path, StringComparer.Ordinal)
+                .ToList();
+
+            Assert.Equal([".gitignore", "keep.py", "secret.py"], files);
+            Assert.Contains(result.Errors, error =>
+                error.Path == ".gitignore" &&
+                error.Message == "Could not read .gitignore due to permissions." &&
+                error.Severity == FileIndexer.ScanIssueSeverity.Warning);
+            Assert.False(result.HadErrors);
         }
         finally
         {
@@ -1672,7 +1737,7 @@ public void ScanFiles_FailsClosedWhenRootIgnoreFileIsUnreadable()
     }
 
     [Fact]
-    public void ScanFiles_FailsClosedWhenNestedIgnoreFileIsUnreadable()
+    public void ScanFiles_PreservesInheritedRulesWhenNestedIgnoreFileIsUnreadable()
     {
         if (OperatingSystem.IsWindows())
             return;
@@ -1698,8 +1763,12 @@ public void ScanFiles_FailsClosedWhenNestedIgnoreFileIsUnreadable()
                 .OrderBy(path => path, StringComparer.Ordinal)
                 .ToList();
 
-            Assert.Equal(["keep.py"], files);
-            Assert.Contains(result.Errors, error => error.Path == "src/.gitignore" && error.Message == "Could not read .gitignore.");
+            Assert.Equal(["keep.py", "src/.gitignore", "src/keep_nested.py", "src/secret.py"], files);
+            Assert.Contains(result.Errors, error =>
+                error.Path == "src/.gitignore" &&
+                error.Message == "Could not read .gitignore due to permissions." &&
+                error.Severity == FileIndexer.ScanIssueSeverity.Warning);
+            Assert.False(result.HadErrors);
         }
         finally
         {
@@ -1921,19 +1990,19 @@ public void ScanFiles_HandlesGitIgnoreWhitespaceLikeGit()
             File.WriteAllText(
                 Path.Combine(tempDir, ".gitignore"),
                 "  #*.py\n  *.py\n*.cs\t\n");
-            File.WriteAllText(Path.Combine(tempDir, "  #x.py"), "print('ignored because leading-space # is literal')");
-            File.WriteAllText(Path.Combine(tempDir, "a.py"), "print('kept because leading spaces are literal')");
-            File.WriteAllText(Path.Combine(tempDir, "  a.py"), "print('ignored by leading-space pattern')");
+            File.WriteAllText(Path.Combine(tempDir, "  #x.py"), "print('kept because leading-space # is a comment')");
+            File.WriteAllText(Path.Combine(tempDir, "a.py"), "print('ignored after leading-space trim')");
+            File.WriteAllText(Path.Combine(tempDir, "  a.py"), "print('ignored by trimmed basename pattern')");
             File.WriteAllText(Path.Combine(tempDir, "a.cs"), "public class IgnoredAfterTrailingTabTrim { }");
-            File.WriteAllText(Path.Combine(tempDir, "keep.py"), "print('kept')");
+            File.WriteAllText(Path.Combine(tempDir, "keep.js"), "export const kept = true;");
 
             var indexer = new FileIndexer(tempDir);
             var files = indexer.ScanFiles()
                 .Select(path => Path.GetRelativePath(tempDir, path).Replace('\\', '/'))
                 .OrderBy(path => path, StringComparer.Ordinal)
                 .ToList();
 
-            Assert.Equal([".gitignore", "a.py", "keep.py"], files);
+            Assert.Equal([".gitignore", "keep.js"], files);
         }
         finally
         {