Skip to content
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ After the first command, use these cues and follow-up commands:
| Edits or branch switches | Refresh incrementally with `--files`, `--commits`, or `--changed-between <old-ref> <new-ref>` instead of rebuilding. See [Quick Start](USER_GUIDE.md#quick-start) and [Incremental update reliability](USER_GUIDE.md#incremental-update-reliability). |
| Intentional rebuilds | Interactive terminals ask before deleting the DB. Scripts and CI must pass `--yes` or `--force`. |
| Long-lived DB compaction | Run `cdidx optimize` or `cdidx index <projectPath> --optimize` to compact FTS5 segments immediately. Incremental refreshes also optimize opportunistically. |
| Pathological generated files | `--max-symbols-per-file <n>` skips indexing file content, symbols, and references when one file emits too many symbols, leaving a `symbol_count_exceeded` issue for audit. |
| Maintenance rollback | Run `cdidx db checkpoint <name>` before risky DB maintenance and `cdidx db restore <name>` to roll back. `backfill-fold` creates an automatic checkpoint unless `--no-checkpoint` is passed. |
| Permission or I/O scan errors | `cdidx` records the scan error, continues other directories, and writes `.cdidx/scan-checkpoint.json` so same-HEAD retries can skip completed directories. |

Expand Down Expand Up @@ -337,6 +338,7 @@ extractor fixture を確認できます。詳細は
| 編集後やブランチ切り替え後 | 再構築ではなく `--files`、`--commits`、`--changed-between <old-ref> <new-ref>` で差分更新します。詳細は [クイックスタート](USER_GUIDE.md#クイックスタート) と [インクリメンタル更新の信頼性](USER_GUIDE.md#インクリメンタル更新の信頼性) を参照してください。 |
| 意図的な再構築 | interactive terminal では既存 DB 削除前に確認を求めます。script / CI では `--yes` または `--force` が必要です。 |
| 長期間使っている DB の compact | `cdidx optimize` または `cdidx index <projectPath> --optimize` で FTS5 segment をすぐに compact できます。差分更新中も必要に応じて自動 optimize します。 |
| 病的な generated file | 1 ファイルが過剰な symbol を出す場合、`--max-symbols-per-file <n>` は file content / symbols / references を保存せず、監査用の `symbol_count_exceeded` issue を残します。 |
| 保守作業の rollback | risky な DB 保守の前に `cdidx db checkpoint <name>`、戻す場合は `cdidx db restore <name>` を使います。`backfill-fold` は `--no-checkpoint` を渡さない限り自動 checkpoint を作成します。 |
| 権限や I/O の scan error | `cdidx` は scan error を記録し、他のディレクトリの走査を続けます。同じ HEAD の再実行では `.cdidx/scan-checkpoint.json` により成功済みディレクトリを読み飛ばせます。 |

Expand Down
24 changes: 24 additions & 0 deletions changelog.d/unreleased/1604.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
category: fixed
issues:
- 1604
affected:
- src/CodeIndex/Cli/IndexCommandRunner.cs
- src/CodeIndex/Cli/IndexCommandRunner.Parse.cs
- src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
- src/CodeIndex/Cli/IndexCommandRunner.Update.cs
- src/CodeIndex/Cli/ConsoleUi.cs
- src/CodeIndex/Cli/IndexWatchRunner.cs
- src/CodeIndex/Cli/CliFlagSchema.cs
- src/CodeIndex/Database/DbWriter.cs
- README.md
- tests/CodeIndex.Tests/IndexCommandRunnerTests.cs
---

## English

- **Indexing now caps per-file symbol output (#1604)** - `cdidx index --max-symbols-per-file N` skips file content, symbols, and references when one file emits too many symbols, leaving a `symbol_count_exceeded` issue instead of ballooning the database.

## 日本語

- **index が 1 ファイルあたりの symbol 出力数を制限できるようになりました (#1604)** - `cdidx index --max-symbols-per-file N` は 1 ファイルが過剰な symbol を出す場合に file content / symbols / references を保存せず、DB を肥大化させる代わりに `symbol_count_exceeded` issue を残します。
15 changes: 15 additions & 0 deletions changelog.d/unreleased/2744.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
category: fixed
issues:
- 2744
affected:
- src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
---

## English

- **Extraction-stall diagnostics now point at bounded recovery for pathological symbol output (#2744)** - `E013_INDEX_EXTRACTION_STALLED` keeps the active file/phase and now suggests using `--max-symbols-per-file` or excluding the reported file when full refresh stalls on symbol extraction.

## 日本語

- **extraction stall 診断が病的な symbol 出力への bounded recovery を示すようになりました (#2744)** - `E013_INDEX_EXTRACTION_STALLED` は active file / phase を保持したまま、full refresh が symbol extraction で止まる場合に `--max-symbols-per-file` または該当ファイルの除外を案内します。
1 change: 1 addition & 0 deletions src/CodeIndex/Cli/CliFlagSchema.cs
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@ private static IReadOnlyList<CliFlag> BuildAll()
new() { Name = "--force", Description = "Bypass the per-database index lock", Commands = Set("index") },
new() { Name = "--duration-format", ValuePlaceholder = "<auto|seconds|hms>", Description = "Index elapsed time display format", Commands = Set("index") },
new() { Name = "--max-file-bytes", ValuePlaceholder = "<bytes>", Description = "Override the per-file indexing size limit", Commands = Set("index") },
new() { Name = "--max-symbols-per-file", ValuePlaceholder = "<n>", Description = "Skip file content, symbols, and references when one file emits too many symbols", Commands = Set("index") },
new() { Name = "--parallelism", ValuePlaceholder = "<n>", Description = "Full-scan extraction worker count (default: CPU count capped at 16; also honors CDIDX_INDEX_PARALLELISM)", Commands = Set("index") },
new() { Name = "--memory-trace", Description = "Include phase memory samples in index JSON output", Commands = Set("index") },
new() { Name = "--commits", ValuePlaceholder = "<id>", Description = "Update files changed in given git commits", Commands = Set("index") },
Expand Down
3 changes: 2 additions & 1 deletion src/CodeIndex/Cli/ConsoleUi.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ public static class ConsoleUi

private static readonly (string Command, string Usage)[] CommandUsageLines =
[
("index", "cdidx index <projectPath> [--db <path>] [--rebuild] [--optimize] [--verbose] [--dry-run] [--force] [--quiet] [--json] [--memory-trace] [--duration-format <auto|seconds|hms>] [--notify <auto|bell|osc9|desktop|none>] [--max-file-bytes <bytes>] [--follow-symlinks <none|internal|all>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]] [--watch [--debounce <ms>]]"),
("index", "cdidx index <projectPath> [--db <path>] [--rebuild] [--optimize] [--verbose] [--dry-run] [--force] [--quiet] [--json] [--memory-trace] [--duration-format <auto|seconds|hms>] [--notify <auto|bell|osc9|desktop|none>] [--max-file-bytes <bytes>] [--max-symbols-per-file <n>] [--follow-symlinks <none|internal|all>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]] [--watch [--debounce <ms>]]"),
("hooks", "cdidx hooks <install|uninstall|status> [--project <path>] [--force] [--json]"),
("backfill-fold", "cdidx backfill-fold [--db <path>] [--dry-run] [--no-checkpoint] [--json]"),
("optimize", "cdidx optimize [--db <path>] [--json]"),
Expand Down Expand Up @@ -875,6 +875,7 @@ private static void PrintFlagReference(Action<string> WriteHelpLine)
Console.WriteLine(" --duration-format <format> Index elapsed time format: `auto` (default), `seconds`, or `hms`; JSON keeps raw elapsed_ms");
WriteHelpLine(" --notify <mode> Long index completion signal: auto, bell, osc9, desktop, or none (also honors CDIDX_NOTIFY; quiet/json suppress it)");
WriteHelpLine(" --max-file-bytes <bytes> Index only files up to this size (default: 4MiB; also honors CDIDX_MAX_FILE_BYTES; accepts K/M/G suffixes)");
WriteHelpLine(" --max-symbols-per-file <n> Skip file content, symbols, and references when one file emits too many symbols (default: 5000)");
WriteHelpLine(" --parallelism <n> Full-scan extraction workers (default: CPU count capped at 16; also honors CDIDX_INDEX_PARALLELISM)");
WriteHelpLine(" --follow-symlinks <mode> Directory symlink policy: none (default), internal, or all");
WriteHelpLine(" --include-symbol-kind <kind>[,<kind>] Keep only matching symbol kinds during indexing");
Expand Down
69 changes: 67 additions & 2 deletions src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,15 @@ private static string FormatExtractionStalledMessage(IndexExtractionStalledExcep
return $"Index extraction made no progress for {ConsoleUi.FormatDuration(ex.Timeout)}.{pathSuffix}";
}

private static FileIssue BuildSymbolCountExceededIssue(string path, int symbolCount, int maxSymbolsPerFile) =>
new()
{
Path = path,
Kind = "symbol_count_exceeded",
Line = 0,
Message = $"Symbol extraction produced {symbolCount:N0} symbols, exceeding the --max-symbols-per-file limit of {maxSymbolsPerFile:N0}; file content, symbols, and references were not indexed. Exclude the generated/pathological file or raise --max-symbols-per-file if this is expected.",
};

internal static string FormatIndexPhasePath(string path, string phase) =>
$"{path} ({phase})";

Expand Down Expand Up @@ -191,7 +200,7 @@ private static int WriteExtractionStalledResult(bool json, JsonSerializerOptions
jsonOptions,
$"Index extraction made no progress for {ConsoleUi.FormatDuration(ex.Timeout)} ({ex.FilesProcessed:N0}{totalSuffix} files processed).{pathSuffix}",
CommandExitCodes.CancelledBySignal,
"Rerun with `--verbose` to inspect progress, lower `--parallelism`, or file a bug with the reported active phase.",
"Rerun with `--verbose` to inspect progress, lower `--parallelism`, exclude the reported file, or lower `--max-symbols-per-file` to skip pathological symbol output.",
CommandErrorCodes.IndexExtractionStalled);
}

Expand Down Expand Up @@ -865,6 +874,14 @@ void StopJsonHeartbeat()
Path.GetFullPath(options.ProjectPath!),
activeJsonExtractionPhases[workerIndex],
extractionCancellationToken);
if (symbols.Count > options.MaxSymbolsPerFile)
{
var issue = BuildSymbolCountExceededIssue(record.Path, symbols.Count, options.MaxSymbolsPerFile);
extractionResults.Add(
FullScanFileWorkItem.Success(filePath, record, string.Empty, rawBytes, issue.Message, [], [], [], [issue]),
extractionCancellationToken);
continue;
}
SymbolExtractor.ApplyFamilyScope(symbols, indexer.GetFamilyScopeKey(filePath, record.Lang));
activeJsonExtractionPhases[workerIndex] = FormatIndexPhasePath(record.Path, "references");
references = ReferenceExtractor.Extract(
Expand Down Expand Up @@ -1025,6 +1042,14 @@ void StopJsonHeartbeat()
&& AllowReuseWithCurrentHotspotFamilyTrust(record.Lang, hotspotFamilyTrustMatchesCurrent));
}
if (existingId != null)
{
if (writer.CountSymbolsForFile(existingId.Value) > options.MaxSymbolsPerFile
|| writer.HasIssueForFile(existingId.Value, "symbol_count_exceeded"))
{
existingId = null;
}
}
if (existingId != null)
{
writer.PurgeStaleFilesSharingChecksum(projectRoot, record.Path, record.Checksum);
skipped++;
Expand Down Expand Up @@ -1058,7 +1083,6 @@ void StopJsonHeartbeat()
var chunks = item.Chunks == null
? ChunkSplitter.Split(fileId, item.Content!)
: ReassignChunkFileIds(item.Chunks, fileId);
writer.InsertChunks(chunks);
currentJsonIndexFile = FormatIndexPhasePath(record.Path, "symbols");
var symbols = item.Symbols == null
? ExtractSymbolsWithStallTimeout(
Expand All @@ -1070,13 +1094,54 @@ void StopJsonHeartbeat()
currentJsonIndexFile,
cancellationToken)
: ReassignSymbolFileIds(item.Symbols, fileId);
if (symbols.Count > options.MaxSymbolsPerFile)
{
var issue = BuildSymbolCountExceededIssue(record.Path, symbols.Count, options.MaxSymbolsPerFile);
writer.InsertSymbols([]);
writer.InsertReferences([]);
writer.InsertIssues(fileId, [issue]);
if (options.Verbose)
WriteIndexVerboseStatus($" [SKIP] {record.Path} ({issue.Message})");
txn.Commit();
processed++;
if (!options.Json && !options.Quiet)
{
PauseIndexSpinnerForConsoleWrite();
ConsoleUi.PrintProgress(processed, files.Count);
ResumeIndexSpinnerAfterConsoleWrite();
}
ReportJsonIndexProgressIfNeeded();
currentJsonIndexFile = null;
continue;
}
if (item.Symbols == null)
SymbolExtractor.ApplyFamilyScope(symbols, indexer.GetFamilyScopeKey(item.FilePath, record.Lang));
var fileContext = new FileContext(projectRoot, record.Path, item.FilePath, record.Lang);
var mutableSymbols = symbols as IList<SymbolRecord> ?? symbols.ToList();
postExtractionHooks.OnSymbolsExtracted(fileContext, mutableSymbols);
symbolsDroppedByKindFilter += options.SymbolKindFilter.Apply(mutableSymbols);
symbols = (IReadOnlyList<SymbolRecord>)mutableSymbols;
if (symbols.Count > options.MaxSymbolsPerFile)
{
var issue = BuildSymbolCountExceededIssue(record.Path, symbols.Count, options.MaxSymbolsPerFile);
writer.InsertSymbols([]);
writer.InsertReferences([]);
writer.InsertIssues(fileId, [issue]);
if (options.Verbose)
WriteIndexVerboseStatus($" [SKIP] {record.Path} ({issue.Message})");
txn.Commit();
processed++;
if (!options.Json && !options.Quiet)
{
PauseIndexSpinnerForConsoleWrite();
ConsoleUi.PrintProgress(processed, files.Count);
ResumeIndexSpinnerAfterConsoleWrite();
}
ReportJsonIndexProgressIfNeeded();
currentJsonIndexFile = null;
continue;
}
writer.InsertChunks(chunks);
FileIndexer.ValidateSymbolLineRanges(record, symbols);
writer.InsertSymbols(symbols);
currentJsonIndexFile = FormatIndexPhasePath(record.Path, "references");
Expand Down
19 changes: 18 additions & 1 deletion src/CodeIndex/Cli/IndexCommandRunner.Parse.cs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ public static partial class IndexCommandRunner
private static readonly string[] AcceptedIndexFlags =
[
"--db", "--data-dir", "--rebuild", "--verbose", "--json", "--quiet", "--dry-run", "--force",
"--yes", "--watch", "--debounce", "--duration-format", "--max-file-bytes",
"--yes", "--watch", "--debounce", "--duration-format", "--max-file-bytes", "--max-symbols-per-file",
"--notify",
"--parallelism", "--memory-trace", "--follow-symlinks",
"--commits", "--changed-between", "--files", "--solution", "--project",
Expand Down Expand Up @@ -43,6 +43,7 @@ public static IndexCommandOptions ParseArgs(string[] args)
var durationFormat = DurationOutputFormat.Auto;
var notifyMode = ReadCompletionNotificationModeFromEnvironment();
long? maxFileSizeBytes = ReadMaxFileSizeBytesFromEnvironment();
var maxSymbolsPerFile = DefaultMaxSymbolsPerFile;
var parallelism = ReadIndexParallelismFromEnvironment();
var symlinkPolicy = FileIndexer.SymlinkPolicy.None;
string? easterEgg = null;
Expand Down Expand Up @@ -151,6 +152,12 @@ public static IndexCommandOptions ParseArgs(string[] args)
case var option when option.StartsWith("--max-file-bytes=", StringComparison.Ordinal):
maxFileSizeBytes = ParseMaxFileBytes(option["--max-file-bytes=".Length..], maxFileSizeBytes);
break;
case "--max-symbols-per-file" when i + 1 < args.Length:
maxSymbolsPerFile = ParseMaxSymbolsPerFile(args[++i], maxSymbolsPerFile, "--max-symbols-per-file");
break;
case var option when option.StartsWith("--max-symbols-per-file=", StringComparison.Ordinal):
maxSymbolsPerFile = ParseMaxSymbolsPerFile(option["--max-symbols-per-file=".Length..], maxSymbolsPerFile, "--max-symbols-per-file");
break;
case "--parallelism" when i + 1 < args.Length:
parallelism = ParseIndexParallelism(args[++i], parallelism, "--parallelism");
break;
Expand Down Expand Up @@ -309,6 +316,7 @@ public static IndexCommandOptions ParseArgs(string[] args)
DurationFormat = durationFormat,
NotifyMode = notifyMode,
MaxFileSizeBytes = maxFileSizeBytes,
MaxSymbolsPerFile = maxSymbolsPerFile,
Parallelism = parallelism,
SymlinkPolicy = symlinkPolicy,
SymbolKindFilter = SymbolKindFilter.Create(includeSymbolKinds, excludeSymbolKinds, symbolKindFilterError),
Expand Down Expand Up @@ -407,6 +415,15 @@ private static int ParseIndexParallelism(string value, int fallback, string sour
return fallback;
}

private static int ParseMaxSymbolsPerFile(string value, int fallback, string source)
{
if (int.TryParse(value, System.Globalization.NumberStyles.Integer, System.Globalization.CultureInfo.InvariantCulture, out var parsed) && parsed > 0)
return parsed;

Console.Error.WriteLine($"Warning: invalid {source} value '{value}' (ignored; use a positive integer) / 不正な {source} 値 '{value}'(無視。正の整数を指定)");
return fallback;
}

private static DurationOutputFormat ParseDurationFormat(string value, DurationOutputFormat fallback)
{
return value.Trim().ToLowerInvariant() switch
Expand Down
Loading
Loading