Add custom extractor configuration workflows

DEVELOPER_GUIDE.md

@@ -3047,6 +3047,54 @@ Cloud セッションは開発ループの中で `dotnet build` にフォール
 - ドキュメント（README, CHANGELOG）は前半英語、後半日本語の構成。
 - 不要な本番パッケージは入れない。test-only package は、テストハーネスの改善に明確に寄与し、`tests/CodeIndex.Tests/` に閉じる限り許容されるが、本番依存ルールを緩めるものではない。
 
+## Custom Language Extraction
+
+Downstream users can add lightweight language support without rebuilding
+`cdidx`:
+
+- extension aliases are read from `~/.config/cdidx/langmap.yaml` and the first
+  workspace ancestor `.cdidx-langmap.yaml`; workspace entries override user
+  entries;
+- regex-backed symbol patterns are read from `.cdidx/patterns/*.yaml` and
+  `~/.config/cdidx/patterns/*.yaml`;
+- `cdidx test-extractor --language <lang> --file <path> --json` runs symbol
+  extraction without building an index, and `--expect-symbols <json>` compares
+  the extracted JSON to a fixture.
+
+Minimal examples:
+
+```yaml
+# .cdidx-langmap.yaml
+entries:
+  - extension: ".kts.in"
+    language: "kotlin"
+```
+
+```yaml
+# .cdidx/patterns/toydsl.yaml
+language: "toydsl"
+extensions:
+  - extension: ".toy"
+patterns:
+  - kind: "class"
+    regex: "^entity (?<name>\\w+)"
+```
+
+Each configured regex should expose a named `name` capture. If it does not,
+`cdidx` uses the full match text as the symbol name. Invalid sidecar files are
+ignored so a broken local experiment does not prevent indexing.
+
+## カスタム言語抽出
+
+下流ユーザーは `cdidx` を再ビルドせずに軽量な言語対応を追加できます。
+
+- 拡張子 alias は `~/.config/cdidx/langmap.yaml` と、最初に見つかった workspace
+  祖先の `.cdidx-langmap.yaml` から読み込まれ、workspace 側が user 側を上書きします。
+- regex ベースのシンボルパターンは `.cdidx/patterns/*.yaml` と
+  `~/.config/cdidx/patterns/*.yaml` から読み込まれます。
+- `cdidx test-extractor --language <lang> --file <path> --json` は index を作らずに
+  symbol extraction だけを実行し、`--expect-symbols <json>` で fixture JSON と比較できます。
+
 ## SQLite reader のデバッグ
 
 `Database/DbDebug.cs` は `ExecuteTrackedReader` / `TrackedRead` の最後に流れた SQL、パラメーター、行ごとの状態を記録し、ループ途中で `SqliteException` が発生した場合に再現に十分な文脈を stderr へダンプする。インデックス済みのソースバイトが想定外の経路に漏れないよう、ダンプ経路はゲート制御されている:

README.md

@@ -58,6 +58,12 @@ cdidx validate
 cdidx mcp
 ```
 
+Custom language loops can stay out of tree: put extension aliases in
+`.cdidx-langmap.yaml`, put regex symbol patterns in `.cdidx/patterns/*.yaml`,
+and run `cdidx test-extractor --language <lang> --file <path> --json` to test
+an extractor fixture without building a full index. See
+[Custom Language Extraction](DEVELOPER_GUIDE.md#custom-language-extraction).
+
 After the first command, use these cues and follow-up commands:
 
 | Situation | What to expect or run |
@@ -299,6 +305,12 @@ cdidx validate
 cdidx mcp
 ```
 
+カスタム言語の開発ループは out-of-tree で回せます。拡張子 alias は
+`.cdidx-langmap.yaml`、regex シンボルパターンは `.cdidx/patterns/*.yaml` に置き、
+`cdidx test-extractor --language <lang> --file <path> --json` で full index を作らずに
+extractor fixture を確認できます。詳細は
+[Custom Language Extraction](DEVELOPER_GUIDE.md#custom-language-extraction) を参照してください。
+
 初回実行後は、次の見方と追加コマンドをよく使います。
 
 | 状況 | 見るもの / 使うもの |

changelog.d/unreleased/1756.added.md

@@ -0,0 +1,16 @@
+---
+category: added
+issues:
+  - 1756
+affected:
+  - src/CodeIndex/Indexer/Extensibility/ConfiguredSymbolExtractor.cs
+  - src/CodeIndex/Indexer/Extensibility/ExtractorPluginRegistry.cs
+---
+
+## English
+
+- **Added YAML-backed custom symbol patterns (#1756)** — `cdidx` now loads regex symbol extractors from `.cdidx/patterns/*.yaml` and `~/.config/cdidx/patterns/*.yaml` so lightweight language patterns can live outside the binary.
+
+## 日本語
+
+- **YAML ベースのカスタムシンボルパターンを追加しました (#1756)** — `.cdidx/patterns/*.yaml` と `~/.config/cdidx/patterns/*.yaml` から regex symbol extractor を読み込めるようになり、軽量な言語パターンをバイナリ外に置けます。

changelog.d/unreleased/1757.added.md

@@ -0,0 +1,16 @@
+---
+category: added
+issues:
+  - 1757
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - src/CodeIndex/Indexer/Scanning/LanguageMapOverrides.cs
+---
+
+## English
+
+- **Added per-user and per-workspace language-map overrides (#1757)** — extension aliases from `~/.config/cdidx/langmap.yaml` and `.cdidx-langmap.yaml` now override built-in language detection, including compound suffixes such as `.kts.in`.
+
+## 日本語
+
+- **ユーザー単位・workspace 単位の language-map override を追加しました (#1757)** — `~/.config/cdidx/langmap.yaml` と `.cdidx-langmap.yaml` の拡張子 alias が組み込み言語判定を上書きし、`.kts.in` のような複合サフィックスにも対応します。

changelog.d/unreleased/1759.added.md

@@ -0,0 +1,15 @@
+---
+category: added
+issues:
+  - 1759
+affected:
+  - src/CodeIndex/Cli/ProgramRunner.cs
+---
+
+## English
+
+- **Added `cdidx test-extractor` (#1759)** — contributors can run symbol extraction for one file with `--language` and `--file`, emit JSON, and compare against an expected symbols fixture without building a full index.
+
+## 日本語
+
+- **`cdidx test-extractor` を追加しました (#1759)** — contributor は full index を作らず、`--language` と `--file` で単一ファイルの symbol extraction を実行し、JSON 出力や expected symbols fixture との比較ができます。

src/CodeIndex/Cli/ProgramRunner.cs

@@ -254,6 +254,7 @@ internal static int Run(
                     "validate-config" => CdidxConfigFile.RunValidate(subArgs, jsonOptions),
                     "db" => DbCommandRunner.RunIntegrityCheck(subArgs, jsonOptions),
                     "report" => ReportCommandRunner.Run(subArgs, jsonOptions, appVersion),
+                    "test-extractor" => RunTestExtractor(subArgs, jsonOptions),
                     _ when IsProjectPathArg(commandName)
                         => IndexCommandRunner.Run(args, jsonOptions),
                     _ => ShowError(args, $"Unknown command: {commandName}")
@@ -301,6 +302,74 @@ _ when IsProjectPathArg(commandName)
     internal static bool IsProjectPathArg(string arg) =>
         !arg.StartsWith('-') && (Directory.Exists(arg) || arg.Contains('/') || arg.Contains('\\') || arg == ".");
 
+    private static int RunTestExtractor(string[] args, JsonSerializerOptions jsonOptions)
+    {
+        string? language = null;
+        string? file = null;
+        string? expect = null;
+        var json = false;
+        for (var i = 0; i < args.Length; i++)
+        {
+            var arg = args[i];
+            if (TryConsumeInlineOrNext(args, ref i, arg, "--language", out var value))
+                language = value;
+            else if (TryConsumeInlineOrNext(args, ref i, arg, "--file", out value))
+                file = value;
+            else if (TryConsumeInlineOrNext(args, ref i, arg, "--expect-symbols", out value) || TryConsumeInlineOrNext(args, ref i, arg, "--expect", out value))
+                expect = value;
+            else if (arg == "--json")
+                json = true;
+            else
+                return CommandErrorWriter.Write($"Unknown test-extractor argument: {arg}", CommandExitCodes.InvalidArgument, "use --language <lang> --file <path> [--expect-symbols <json>] [--json].");
+        }
+
+        if (string.IsNullOrWhiteSpace(language) || string.IsNullOrWhiteSpace(file))
+            return CommandErrorWriter.Write("test-extractor requires --language and --file.", CommandExitCodes.InvalidArgument, "use --language <lang> --file <path> [--expect-symbols <json>] [--json].");
+        if (!File.Exists(file))
+            return CommandErrorWriter.Write($"File not found: {file}", CommandExitCodes.NotFound);
+
+        var source = File.ReadAllText(file);
+        var symbols = Indexer.SymbolExtractor.Extract(1, language, source, file);
+        if (expect != null)
+        {
+            var expected = File.ReadAllText(expect);
+            var actual = JsonSerializer.Serialize(symbols);
+            if (!JsonEquivalent(expected, actual))
+            {
+                Console.Error.WriteLine("Expected symbols did not match extracted symbols.");
+                Console.Error.WriteLine(actual);
+                return CommandExitCodes.InvalidArgument;
+            }
+        }
+
+        if (json || expect == null)
+            Console.WriteLine(JsonSerializer.Serialize(symbols));
+        return CommandExitCodes.Success;
+    }
+
+    private static bool TryConsumeInlineOrNext(string[] args, ref int index, string arg, string flag, out string value)
+    {
+        value = string.Empty;
+        if (arg.StartsWith(flag + "=", StringComparison.Ordinal))
+        {
+            value = arg[(flag.Length + 1)..];
+            return true;
+        }
+
+        if (arg != flag || index + 1 >= args.Length)
+            return false;
+
+        value = args[++index];
+        return true;
+    }
+
+    private static bool JsonEquivalent(string expected, string actual)
+    {
+        using var expectedDoc = JsonDocument.Parse(expected);
+        using var actualDoc = JsonDocument.Parse(actual);
+        return JsonSerializer.Serialize(expectedDoc.RootElement) == JsonSerializer.Serialize(actualDoc.RootElement);
+    }
+
     internal static void EnsureRedirectedStdoutUsesUtf8()
     {
         if (!Console.IsOutputRedirected || Console.Out is StringWriter || Console.Out.GetType().Assembly != typeof(Console).Assembly)

src/CodeIndex/Indexer/Extensibility/ConfiguredSymbolExtractor.cs

@@ -0,0 +1,50 @@
+using System.Text.RegularExpressions;
+using CodeIndex.Models;
+
+namespace CodeIndex.Indexer.Extensibility;
+
+internal sealed class ConfiguredSymbolExtractor(
+    string language,
+    IReadOnlyCollection<string> fileExtensions,
+    IReadOnlyList<ConfiguredSymbolExtractor.PatternRule> patterns) : ISymbolExtractor
+{
+    internal sealed record PatternRule(string Kind, Regex Regex);
+
+    public string Language { get; } = language;
+
+    public IReadOnlyCollection<string> FileExtensions { get; } = fileExtensions;
+
+    public IReadOnlyList<SymbolRecord> Extract(long fileId, string source, ExtractionContext context)
+    {
+        var symbols = new List<SymbolRecord>();
+        var lineNumber = 0;
+        foreach (var line in source.Replace("\r\n", "\n", StringComparison.Ordinal).Replace('\r', '\n').Split('\n'))
+        {
+            lineNumber++;
+            foreach (var pattern in patterns)
+            {
+                var match = pattern.Regex.Match(line);
+                if (!match.Success)
+                    continue;
+
+                var name = match.Groups["name"].Success ? match.Groups["name"].Value : match.Value.Trim();
+                if (string.IsNullOrWhiteSpace(name))
+                    continue;
+
+                symbols.Add(new SymbolRecord
+                {
+                    FileId = fileId,
+                    Kind = pattern.Kind,
+                    Name = name,
+                    Line = lineNumber,
+                    StartLine = lineNumber,
+                    EndLine = lineNumber,
+                    Signature = line.Trim(),
+                });
+                break;
+            }
+        }
+
+        return symbols;
+    }
+}