Fix indexing diagnostics, caps, and scan edge cases

DEVELOPER_GUIDE.md

@@ -153,6 +153,8 @@ git status --short -- '**/packages.lock.json'
 | DTOs | `Models/FileRecord.cs`, `Models/ChunkRecord.cs`, `Models/SymbolRecord.cs`, `Models/ReferenceRecord.cs` | Records shared by indexing, storage, query, and MCP layers. |
 | Tests | `tests/CodeIndex.Tests/*Tests.cs`, `TestProjectHelper.cs`, `TestConsoleLock.cs` | Focused unit/integration coverage for chunking, extraction, DB reads/writes, CLI behavior, MCP behavior, git helpers, and shared test harness utilities. |
 
+Full-scan resume checkpoints live at `.cdidx/scan-checkpoint.json`. The internal JSON schema is versioned (`Version` is currently `1`) and binds `GitHead` to a bounded `Directories` array; malformed, stale, future-version, oversized, or over-depth payloads are ignored with a bounded warning and a full scan.
+
 Large command and extractor files have a tracked decomposition plan in
 [docs/large-file-decomposition-plan.md](docs/large-file-decomposition-plan.md).
 Use that plan when splitting `QueryCommandRunner`, `SymbolExtractor`,

README.md

@@ -90,6 +90,10 @@ For a faster first pass when you only need text search, `definition`, `symbols`,
 or `map`, run `cdidx . --symbols-only`. Reference graph commands remain degraded
 until you rerun `cdidx .` without that flag.
 
+For generated or dense source that emits excessive reference rows, use
+`cdidx . --max-references-per-file <n>` to keep text search and symbols indexed
+while skipping references for only the over-limit file.
+
 ## Highlights
 
 | Area | What to use |
@@ -265,6 +269,10 @@ cdidx lsp --db .cdidx/codeindex.db
 `cdidx . --symbols-only` を使えます。reference graph 系コマンドは、このフラグなしで
 `cdidx .` を再実行するまで degraded のままです。
 
+生成コードや高密度なソースが過剰な reference 行を生成する場合は
+`cdidx . --max-references-per-file <n>` を使うと、text search と symbols は保持しつつ
+上限を超えたファイルだけ references をスキップできます。
+
 ## 特長
 
 | 分野 | 使うもの |

changelog.d/unreleased/3712.fixed.md

@@ -0,0 +1,18 @@
+---
+category: fixed
+issues:
+  - 3712
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Scan checkpoint recovery now reports schema/version diagnostics (#3712)** — malformed, stale, future-version, oversized, over-depth, or invalid-directory checkpoint payloads fall back to a full scan with a bounded `<scan_checkpoint>` warning instead of being ignored silently.
+
+## 日本語
+
+- **scan checkpoint の復旧で schema/version 診断を出すようになりました (#3712)** — malformed、stale、future-version、oversized、over-depth、invalid-directory な checkpoint payload は、無言で無視される代わりに bounded な `<scan_checkpoint>` warning を出して full scan へフォールバックします。

changelog.d/unreleased/3719.fixed.md

@@ -0,0 +1,24 @@
+---
+category: fixed
+issues:
+  - 3719
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.cs
+  - src/CodeIndex/Indexer/References/ReferenceExtractionContext.cs
+  - src/CodeIndex/Indexer/References/ReferenceExtractor.Core.cs
+  - src/CodeIndex/Indexer/References/ReferenceExtractor.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.cs
+  - src/CodeIndex/Mcp/McpToolDefinitions.cs
+  - src/CodeIndex/Mcp/McpToolArgumentContracts.cs
+  - tests/CodeIndex.Tests/McpServerTests.cs
+---
+
+## English
+
+- **Reference extraction now has a per-file cap (#3719)** — `cdidx index` and MCP indexing accept `--max-references-per-file` / `maxReferencesPerFile`, stop extraction after the diagnostic sentinel count, preserve text search and symbols, and emit `reference_count_exceeded` file issues when a file's references are skipped.
+
+## 日本語
+
+- **reference 抽出にファイル単位の上限を追加しました (#3719)** — `cdidx index` と MCP index が `--max-references-per-file` / `maxReferencesPerFile` を受け付け、診断用 sentinel 件数で抽出を止め、text search と symbols は保持しつつ、references をスキップしたファイルには `reference_count_exceeded` file issue を出します。

changelog.d/unreleased/3720.fixed.md

@@ -0,0 +1,24 @@
+---
+category: fixed
+issues:
+  - 3720
+affected:
+  - src/CodeIndex/Cli/CdidxConfigFile.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.cs
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - src/CodeIndex/Indexer/Scanning/GeneratedCodePatternMatcher.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.cs
+  - tests/CodeIndex.Tests/CdidxConfigFileTests.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
+  - tests/CodeIndex.Tests/McpServerTests.cs
+  - USER_GUIDE.md
+---
+
+## English
+
+- **Generated-code extraction suppression is now configurable (#3720)** — project config can suppress symbol/reference extraction for filename or path patterns while preserving normal text search visibility and recording `generated_code_extraction_skipped` in `file_issues`.
+
+## 日本語
+
+- **generated-code extraction suppression を設定可能にしました (#3720)** — project config で filename/path pattern の symbol/reference 抽出を抑制でき、通常の text search での可視性は維持しつつ、`file_issues` に `generated_code_extraction_skipped` を記録します。

changelog.d/unreleased/3762.fixed.md

@@ -0,0 +1,18 @@
+---
+category: fixed
+issues:
+  - 3762
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.cs
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
+---
+
+## English
+
+- **Project discovery budget diagnostics now name the budget that fired (#3762)** — project marker fingerprint truncation records directory/marker-file budget reasons in warnings and fingerprint salt, forwards truncation warnings through CLI JSON output, and solution fallback discovery has explicit directory-budget coverage.
+
+## 日本語
+
+- **project discovery budget 診断が発火した budget 名を示すようになりました (#3762)** — project marker fingerprint の truncation は directory / marker-file budget の理由を warning と fingerprint salt に記録し、truncation warning を CLI JSON 出力にも転送し、solution fallback discovery には directory budget の明示テストを追加しました。

changelog.d/unreleased/3763.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 3763
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+---
+
+## English
+
+- **Dangling filesystem entry probes are now capped (#3763)** — per-directory dangling symlink checks stop after a bounded candidate count and emit a warning when additional diagnostics may be omitted, preserving normal scan correctness while avoiding unbounded probe work.
+
+## 日本語
+
+- **dangling filesystem entry probe に上限を追加しました (#3763)** — ディレクトリ単位の dangling symlink 確認は bounded な candidate 数で停止し、追加診断が省略される可能性を warning として出すため、通常の scan correctness を保ちながら無制限な probe 作業を避けます。

changelog.d/unreleased/3785.fixed.md

@@ -0,0 +1,17 @@
+---
+category: fixed
+issues:
+  - 3785
+affected:
+  - src/CodeIndex/Indexer/References/ReferenceExtractor.Preparation.cs
+  - src/CodeIndex/Indexer/Scanning/ChunkSplitter.cs
+  - tests/CodeIndex.Tests/ChunkSplitterTests.cs
+---
+
+## English
+
+- **Chunk and reference preparation now avoid more full line-string materialization (#3785)** — chunk splitting tracks line start offsets and slices only the persisted chunk bodies, and reference preparation reuses structural line arrays when sanitizer output is unchanged, reducing duplicate allocations for large but valid files while preserving line ranges and trailing-newline behavior.
+
+## 日本語
+
+- **chunk/reference preparation がより多くの全行 string 配列 materialization を避けるようになりました (#3785)** — chunk splitting は行開始 offset を追跡して永続化する chunk 本文だけを切り出し、reference preparation は sanitizer output が変わらない場合に structural line 配列を再利用するため、大きくても有効なファイルでの重複 allocation を減らしつつ、行範囲と末尾改行の挙動を維持します。

changelog.d/unreleased/3800.fixed.md

@@ -0,0 +1,19 @@
+---
+category: fixed
+issues:
+  - 3800
+affected:
+  - src/CodeIndex/Indexer/BoundedRegex.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractionWorker.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.cs
+---
+
+## English
+
+- **Regex timeout fallbacks are now visible in file issues (#3800)** — symbol and reference extraction record bounded `regex_timeout` diagnostics with hashed pattern metadata when a guarded regex falls back to no-match behavior, so skipped matches no longer disappear silently.
+
+## 日本語
+
+- **regex timeout fallback が file issue に表示されるようになりました (#3800)** — symbol 抽出と reference 抽出中に保護された regex が no-match fallback へ切り替わった場合、ハッシュ化した pattern メタデータ付きの bounded な `regex_timeout` 診断を記録し、match の欠落が無言で起きないようにしました。

changelog.d/unreleased/3835.fixed.md

@@ -0,0 +1,22 @@
+---
+category: fixed
+issues:
+  - 3835
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.cs
+  - src/CodeIndex/Indexer/Scanning/FileContentLoader.cs
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+  - tests/CodeIndex.Tests/McpServerTests.cs
+  - USER_GUIDE.md
+---
+
+## English
+
+- **File scanning diagnostics now cover more edge cases (#3835)** — Git LFS pointer files keep pointer-identity checksums, BOM-less UTF-16 heuristic decoding reports the chosen encoding, NUL-byte binary skips include the byte offset and persist a `null_byte` file issue during indexing, out-of-policy symlink warnings no longer expose absolute external targets, and extensionless `env -S` shebangs resolve their interpreter.
+
+## 日本語
+
+- **ファイル走査診断がより多くの端ケースを扱うようになりました (#3835)** — Git LFS pointer ファイルは pointer identity に基づく checksum を保持し、BOM なし UTF-16 heuristic decode は選択した encoding を報告し、NUL バイトによる binary skip は byte offset を含めて index 時にも `null_byte` file issue として永続化し、ポリシー外 symlink の warning は外部絶対 target を露出せず、拡張子なし `env -S` shebang は interpreter を解決するようになりました。

src/CodeIndex/Cli/CdidxConfigFile.cs

@@ -56,7 +56,7 @@ internal static class CdidxConfigFile
         "mcp",
     };
 
-    private static readonly IReadOnlyList<string> KnownIndexingKeys = new[] { "includeKinds", "excludeKinds" };
+    private static readonly IReadOnlyList<string> KnownIndexingKeys = new[] { "includeKinds", "excludeKinds", "generatedCodePatterns" };
     private static readonly IReadOnlyList<string> KnownSearchKeys = new[] { "limit", "snippet_lines", "max_line_width" };
     private static readonly IReadOnlyList<string> KnownOutputKeys = new[] { "format", "locale" };
     private static readonly IReadOnlyList<string> KnownGraphKeys = new[] { "max_hops" };
@@ -277,6 +277,14 @@ private static void AddIndexingEnvironmentSettings(JsonElement root, string path
             else if (value!.Length > 0)
                 pending.Add((IndexCommandRunner.ExcludeSymbolKindsEnvironmentVariable, string.Join(",", value)));
         }
+
+        if (indexing.TryGetProperty("generatedCodePatterns", out var generatedCodePatterns))
+        {
+            if (!TryReadStringArray(generatedCodePatterns, "indexing.generatedCodePatterns", path, out var value, out var err))
+                errors.Add(err!);
+            else if (value!.Length > 0)
+                pending.Add((IndexCommandRunner.GeneratedCodePatternsEnvironmentVariable, string.Join(",", value)));
+        }
     }
 
     private static void AddSearchEnvironmentSettings(JsonElement root, string path, List<(string EnvName, string Value)> pending, List<string> errors)

src/CodeIndex/Cli/CliFlagSchema.cs

@@ -367,6 +367,7 @@ private static IReadOnlyList<CliFlag> BuildAll()
             new() { Name = "--duration-format", ValuePlaceholder = "<auto|seconds|hms>", Description = "Index elapsed time display format", Commands = Set("index") },
             new() { Name = "--max-file-bytes", ValuePlaceholder = "<bytes>", Description = "Override the per-file indexing size limit", Commands = Set("index") },
             new() { Name = "--max-symbols-per-file", ValuePlaceholder = "<n>", Description = "Skip file content, symbols, and references when one file emits too many symbols (max 50000)", Commands = Set("index") },
+            new() { Name = "--max-references-per-file", ValuePlaceholder = "<n>", Description = "Skip references when one file emits too many references (max 1000000)", Commands = Set("index") },
             new() { Name = "--parallelism", ValuePlaceholder = "<n>", Description = "Full-scan extraction worker count (default: CPU count capped at 16; also honors CDIDX_INDEX_PARALLELISM)", Commands = Set("index") },
             new() { Name = "--memory-trace", Description = "Include phase memory samples in index JSON output", Commands = Set("index") },
             new() { Name = "--commits", ValuePlaceholder = "<commit-ref>", Description = "Update files changed in given git commits", Commands = Set("index") },

src/CodeIndex/Cli/ConsoleUi.cs

@@ -80,7 +80,7 @@ public static class ConsoleUi
 
     private static readonly (string Command, string Usage)[] CommandUsageLines =
     [
-        ("index", "cdidx index <projectPath> [--db <path>] [--rebuild] [--optimize] [--symbols-only] [--verbose] [--dry-run] [--force] [--quiet] [--json] [--memory-trace] [--duration-format <auto|seconds|hms>] [--notify <auto|bell|osc9|desktop|none>] [--max-file-bytes <bytes>] [--max-symbols-per-file <n>] [--follow-symlinks <none|internal|all>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]] [--watch [--debounce <ms>]]"),
+        ("index", "cdidx index <projectPath> [--db <path>] [--rebuild] [--optimize] [--symbols-only] [--verbose] [--dry-run] [--force] [--quiet] [--json] [--memory-trace] [--duration-format <auto|seconds|hms>] [--notify <auto|bell|osc9|desktop|none>] [--max-file-bytes <bytes>] [--max-symbols-per-file <n>] [--max-references-per-file <n>] [--follow-symlinks <none|internal|all>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]] [--watch [--debounce <ms>]]"),
         ("hooks", "cdidx hooks <install|uninstall|status> [--project <path>] [--force] [--json]"),
         ("backfill-fold", "cdidx backfill-fold [--db <path>] [--dry-run] [--no-checkpoint] [--json]"),
         ("optimize", "cdidx optimize [--db <path>] [--json]"),
@@ -990,6 +990,7 @@ private static void PrintFlagReference(Action<string> WriteHelpLine)
         WriteHelpLine("  --notify <mode>           Long index completion signal: auto, bell, osc9, desktop, or none (also honors CDIDX_NOTIFY; quiet/json suppress it)");
         WriteHelpLine("  --max-file-bytes <bytes>  Index only files up to this size (default: 4MiB; also honors CDIDX_MAX_FILE_BYTES; accepts K/M/G suffixes)");
         WriteHelpLine("  --max-symbols-per-file <n> Skip file content, symbols, and references when one file emits too many symbols (default: 5000; max: 50000)");
+        WriteHelpLine("  --max-references-per-file <n> Skip references when one file emits too many references (default: 100000; max: 1000000)");
         WriteHelpLine("  --parallelism <n>         Full-scan extraction workers (default: CPU count capped at 16; also honors CDIDX_INDEX_PARALLELISM)");
         WriteHelpLine("  --follow-symlinks <mode>  Symlink policy for directories and files: none (default), internal, or all");
         WriteHelpLine("  --include-symbol-kind <kind>[,<kind>]  Keep only matching symbol kinds during indexing");

src/CodeIndex/Cli/IndexCommandRunner.DryRun.cs

@@ -20,7 +20,14 @@ private static int RunDryRun(
         CancellationToken cancellationToken)
     {
         var projectPath = options.ProjectPath!;
-        var dryIndexer = new FileIndexer(projectPath, ignoreCase, ignoreRuleRoot, options.MaxFileSizeBytes, directoryIgnoreCaseProbe: null, symlinkPolicy: options.SymlinkPolicy);
+        var dryIndexer = new FileIndexer(
+            projectPath,
+            ignoreCase,
+            ignoreRuleRoot,
+            options.MaxFileSizeBytes,
+            directoryIgnoreCaseProbe: null,
+            symlinkPolicy: options.SymlinkPolicy,
+            generatedCodePatterns: options.GeneratedCodePatterns);
         IReadOnlyList<string> dryCandidates;
         IReadOnlyList<string> dryDeleteCandidates;
         bool authoritativeFullScan;