diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index b19fe1d19..89005b24c 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -2009,7 +2009,7 @@ Process exit codes are coarse (`0` success including valid zero-row queries, `1` ## Design decisions - **Language capability patterns remain typed at the integration boundary** — CLI/MCP `languages` rows expose suffix-only `extensions`, literal `exact_filenames`, and ``-rendered `filename_prefix_patterns`. `legacy_patterns` preserves the former combined list during deprecation, and `pattern_provenance` identifies built-in, plugin/pattern, and language-map override ownership. Round-trip tests feed every advertised typed pattern back through `FileIndexer.DetectLanguage` (#4617). -- **Ambiguous source extensions stay explicit** — `.m` and `.pl` are not assigned to Objective-C and Perl by default. `FileIndexer` checks an authoritative recognized shebang, then a 64 KiB bounded prefix for strong mutually exclusive Objective-C/MATLAB or Perl/Prolog markers, then at most 256 entries per ancestor directory for conservative project markers. Conflicting or weak evidence is indexed as `ambiguous_m` / `ambiguous_pl`; unresolved `.m` files run the bounded MATLAB and Objective-C symbol/reference paths after a shared position-preserving comment mask, while Prolog and `ambiguous_pl` advertise conservative reference/graph support and the ambiguous `.pl` bucket uses union symbol/reference rules without changing content-based classification (#4612, #4738, #4746). +- **Ambiguous source extensions stay explicit** — `.m` and `.pl` are not assigned to Objective-C and Perl by default. After language-map overrides and built-in exact/prefix filename rules, `FileIndexer` checks an authoritative recognized shebang whose first physical line is bounded to 256 bytes, then a 64 KiB bounded prefix for strong mutually exclusive Objective-C/MATLAB or Perl/Prolog markers, then at most 256 entries per ancestor directory for conservative project markers. A first line that reaches the shebang boundary without a terminator falls through instead of selecting an interpreter. Conflicting or weak evidence is indexed as `ambiguous_m` / `ambiguous_pl`; unresolved `.m` files run the bounded MATLAB and Objective-C symbol/reference paths after a shared position-preserving comment mask, while Prolog and `ambiguous_pl` advertise conservative reference/graph support and the ambiguous `.pl` bucket uses union symbol/reference rules without changing content-based classification. The detector owns the ordered candidate descriptors, filename patterns, exact content patterns, project markers, bounded shebang rules, and reason/confidence vocabulary; CLI/MCP `extension_lookup` diagnostics and dry-run `language_detections` consume that same source so catalog guidance cannot drift from indexing decisions (#4612, #4738, #4746, #4901). - **Dynamic reference-graph readiness follows extractor contracts** — when indexed Crystal, Groovy, Tcl, Prolog, or `ambiguous_pl` rows have a missing or stale symbol-extractor version stamp, status reports `dynamic_reference_graph_contract_stale` and keeps `reference_graph_complete` / `graph_data_current` false until a normal index refresh rewrites those rows (#4746). - **Hotspot marker fingerprints share one bounded tree traversal** — full/update CLI and MCP indexing compute C#, VB, F#, and MSBuild marker fingerprints together instead of walking the directory tree once per language. Each distinct marker glob retains the platform filesystem's matching behavior and is enumerated once per visited directory, while child directories are enumerated once; marker sets, budgets, truncation sentinels, and warning order remain isolated per language. The single-language API delegates to the same engine, preserving ignore rules, nested-repository/submodule boundaries, and MCP authorized-read failures. - **Lock-file dependency graphs model package relationships** — `packages.lock.json`, `package-lock.json`, and `npm-shrinkwrap.json` keep package declarations as symbols, but emit `dependency` references only for explicit parent-package to child-package entries. NuGet lock symbols and references preserve the current file, target/RID, parent package, and exact JSON property span; candidate resolution stays file-local, while file-level `deps` suppresses cross-file package-name inference. Normal index updates invalidate the prior dependency-lock extraction and reference-identity contracts, so `callers` identifies the requiring package without connecting unrelated lock files or collapsing repeated declarations to the first matching line (#4409, #4845). @@ -5454,7 +5454,7 @@ USER_GUIDEの[終了コード](USER_GUIDE.md#終了コード)セクションを ## 設計判断 - **integration boundary では language capability pattern の型を維持** — CLI/MCP の `languages` 行は suffix のみの `extensions`、literal な `exact_filenames`、`` 表記の `filename_prefix_patterns` を公開します。`legacy_patterns` は deprecation 中に従来の combined list を保持し、`pattern_provenance` は built-in、plugin/pattern、language-map override の所有元を示します。round-trip test は広告した全 typed pattern を `FileIndexer.DetectLanguage` に戻して検証します(#4617)。 -- **曖昧な source extension は曖昧なまま明示** — `.m` と `.pl` を既定で Objective-C / Perl に割り当てません。`FileIndexer` は authoritative な認識済み shebang、64 KiB 上限 prefix 内の相互排他的で強い Objective-C/MATLAB または Perl/Prolog marker、各 ancestor directory 最大 256 entry の保守的な project marker の順に確認します。競合または弱い証拠は `ambiguous_m` / `ambiguous_pl` として index し、未確定の `.m` は位置を保つ共通コメントマスクの後で上限付きの MATLAB / Objective-C symbol・reference 経路を実行します。一方、Prolog と `ambiguous_pl` は保守的な reference / graph 対応を広告し、曖昧な `.pl` bucket は content-based classification を変えずに symbol / reference rule の和集合を使います(#4612、#4738、#4746)。 +- **曖昧な source extension は曖昧なまま明示** — `.m` と `.pl` を既定で Objective-C / Perl に割り当てません。language-map override と built-in の完全一致/prefix filename rule の後で、`FileIndexer` は先頭物理行を 256 byte に制限した authoritative な認識済み shebang、64 KiB 上限 prefix 内の相互排他的で強い Objective-C/MATLAB または Perl/Prolog marker、各 ancestor directory 最大 256 entry の保守的な project marker の順に確認します。行終端なしで shebang 境界に達した先頭行は interpreter を選択せず、後続判定へ進みます。競合または弱い証拠は `ambiguous_m` / `ambiguous_pl` として index し、未確定の `.m` は位置を保つ共通コメントマスクの後で上限付きの MATLAB / Objective-C symbol・reference 経路を実行します。一方、Prolog と `ambiguous_pl` は保守的な reference / graph 対応を広告し、曖昧な `.pl` bucket は content-based classification を変えずに symbol / reference rule の和集合を使います。順序付き candidate descriptor、filename pattern、正確な content pattern、project marker、上限付き shebang rule、reason/confidence 語彙は detector 自身が所有し、CLI/MCP の `extension_lookup` diagnostic と dry-run の `language_detections` は同じ source を使うため、catalog guidance と indexing 判定が乖離しません(#4612、#4738、#4746、#4901)。 - **動的言語の reference-graph readiness は extractor contract に従う** — index 済みの Crystal、Groovy、Tcl、Prolog、`ambiguous_pl` row で symbol-extractor version stamp が欠落または古い場合、status は `dynamic_reference_graph_contract_stale` を報告し、通常の index refresh が対象 row を更新するまで `reference_graph_complete` / `graph_data_current` を false に保ちます(#4746)。 - **hotspot marker fingerprint は上限付きtree traversalを1回共有** — full/update CLIとMCP indexingは、directory treeを言語ごとに歩かず、C#、VB、F#、MSBuildのmarker fingerprintをまとめて計算します。各directoryでは固有marker globごとにplatform filesystemのmatching挙動を保って1回ずつ列挙し、child directoryも1回だけ列挙する一方、marker集合、budget、truncation sentinel、warning順は言語別に分離します。single-language APIも同じengineへ委譲し、ignore rule、nested repository/submodule境界、MCP authorized read failureを維持します。 - **lock file の依存グラフは package 間の関係をモデル化** — `packages.lock.json`、`package-lock.json`、`npm-shrinkwrap.json` は package 宣言を symbol として保持しますが、`dependency` reference は明示された親 package → 子 package の項目だけに出力します。NuGet lock の symbol / reference は現在の file、target/RID、親 package、正確な JSON property span を保持し、candidate 解決を file 内に限定します。file 単位の `deps` は package 名による file 間推論を抑止し、通常の index update は以前の dependency-lock 抽出 contract と reference-identity contract を無効化します。そのため、`callers` は無関係な lock file を接続したり、反復宣言を最初の一致行へ畳み込んだりせず、要求元 package を特定できます(#4409、#4845)。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 490afdc78..adc0dca70 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -157,6 +157,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding TypeScript same-line sibling-class coverage keeps distinct and identical method-name cases in one source, using four class containers to preserve attribution diagnostics. - `FileIndexerTests.cs`, `FileIndexerContentLoadingTests.cs`, `FileIndexerTestSupport.cs` File scanning, language detection, scan-result language reuse, content-sensitive header safeguards, content loading/canonicalization, checksum, Git LFS pointer detection, and record-building behavior, including authoritative language-map suffix overrides across exact/prefix/extension rules, fail-visible child-map probe/read diagnostics that block parent inheritance, typed `languages` pattern schemas with provenance and detection round trips, extensionless/unknown/ambiguous-extension shebang detection's override precedence, `detection_source`, 256-byte first-line cap, binary/NUL-byte rejection, conservative `.m` / `.pl` content/project classification with explicit ambiguity buckets, and Windows-only >=260-character path walker/purge coverage. Keep `.h` lexical-detection coverage consolidated across comment-only markers, spliced strings/comments and delimiters, long macro/raw-string/block-comment state, UTF-8 byte-budget sampling, partial-token sample boundaries, long physical lines, genuine mixed C/C++ code, and source/confidence metadata. Shared `FileIndexerTests` helpers live in `FileIndexerTestSupport.cs`. + Ambiguous-extension diagnostic coverage keeps `.M` case normalization, ordered Objective-C/MATLAB candidates, aliases, exact/prefix filename precedence, the 256-byte unterminated-shebang boundary, shebang/content/project evidence, override examples, empty/binary outcomes, and detector reason/confidence parity together across `FileIndexerTests`, CLI `languages`, MCP `languages`, and index dry-run fixtures (#4901). Directory-entry snapshot coverage requires one ordered, scan-local filesystem enumeration for each normally visited root/child directory and shares it between the default case probe and entry processing. Keep custom case-probe invocation counts, the legacy injected `enumerateFiles` path, dangling-entry diagnostics, and single-error enumeration failures explicit when changing traversal. Project-marker fingerprint coverage requires the C#/VB/F#/MSBuild batch to share one directory-tree traversal while retaining one enumeration per distinct marker glob and one child-directory enumeration per visited directory. Pin known hashes and single-language delegation, independent budgets/truncation/warning order, ignore/nested-repository/submodule and platform-casing boundaries, traversal failure and cancellation propagation, and MCP fingerprint/authorization parity. - `PathCompatibilityMatrixTests.cs` @@ -1107,6 +1108,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" TypeScript same-line sibling-class coverage は、distinct/identical method-name case を1 source にまとめ、4つの class container で attribution の診断性を維持します。 - `FileIndexerTests.cs`、`FileIndexerContentLoadingTests.cs`、`FileIndexerTestSupport.cs` ファイル走査、言語判定、scan result 言語の再利用、content loading / canonicalization、checksum、レコード構築のテスト。完全一致/prefix/extension rule を横断する authoritative な language-map suffix override、親継承を遮断して失敗を可視化する child map の probe/read diagnostic、provenance と detection round trip を持つ typed `languages` pattern schema、拡張子なし/未知拡張子/曖昧拡張子の shebang 判定における override precedence、`detection_source`、「先頭物理行 256 byte 上限」、binary/NUL byte 除外、明示 ambiguity bucket を持つ保守的な `.m` / `.pl` content/project 分類、Windows 専用の 260 文字以上 path walker/purge カバレッジも含みます。`.h` の字句判定カバレッジは、コメントだけのマーカー、splice された文字列・コメント・delimiter、長いマクロ・raw string・block comment の状態、UTF-8 byte budget sampling、token 途中の sample 境界、長い物理行、実際の C/C++ 混在コード、判定元・信頼度 metadata を一まとまりで検証します。共有 `FileIndexerTests` helper は `FileIndexerTestSupport.cs` に置きます。 + 曖昧拡張子 diagnostic の coverage は、`.M` の大小文字正規化、順序付き Objective-C/MATLAB candidate、alias、完全一致/prefix filename の優先順位、行終端なしで到達する 256 byte shebang 境界、shebang/content/project evidence、override 例、空・binary 入力の結果、detector の reason/confidence parity を `FileIndexerTests`、CLI `languages`、MCP `languages`、index dry-run fixture の間でまとめて維持します(#4901)。 directory entry snapshot の coverage では、通常scanで訪問するroot/child directoryごとに順序付き・scan-localなfilesystem列挙を1回だけ行い、既定case probeとentry処理で共有することを必須とします。traversal変更時はcustom case probeの呼出回数、旧injected `enumerateFiles`経路、dangling entry診断、列挙失敗が1 errorだけになる契約を明示的に維持してください。 project marker fingerprint の coverage では、C#/VB/F#/MSBuild batch が1回の directory-tree traversal を共有しつつ、各directoryで固有marker globごとに1回、child directoryは1回だけ列挙することを必須とします。既知hashとsingle-language delegation、言語別budget/truncation/warning順、ignore/nested repository/submoduleとplatform casingの境界、traversal failure/cancellationの伝播、MCP fingerprint/authorization parityを固定してください。 - `PathCompatibilityMatrixTests.cs` diff --git a/USER_GUIDE.md b/USER_GUIDE.md index f5f9b2c57..da2e17a96 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -2517,6 +2517,8 @@ CLI JSON and MCP `languages` responses share one catalog snapshot and expose `la All indexed languages are searchable through FTS5. Rows with **Symbols = yes** also support structured queries by function, class, import, or language-specific symbol name. Use `cdidx languages --indexed-only --json` to list only languages present in the current DB; JSON rows expose `symbol_extraction`, `reference_extraction`, `graph_queries`, `capability_gaps`, `unsupported_guidance`, and `indexed_file_count`. Pattern capabilities are split into suffix-only `extensions`, literal `exact_filenames`, and `filename_prefix_patterns` whose `` placeholder requires one or more trailing characters. `legacy_patterns` retains the previous combined list for a deprecation window, while `pattern_provenance` reports each pattern's kind and `built_in`, `plugin_or_pattern`, or `language_map_override` source. The top-level `detection_policy` reports that extensions remain case-insensitive while exact filenames and filename-prefix patterns follow the indexed filesystem's `path_case_sensitive` policy; its `precedence` array records that a trusted language-map suffix override wins before built-in exact-filename, prefix-pattern, and extension rules. `language_map_diagnostics` reports stable `code`, sanitized `config`, `reason`, and `blocks_parent_fallback` fields. If the closest workspace `.cdidx-langmap.yaml` cannot be probed or read, cdidx reports the failure and does not silently inherit a parent workspace map for that subtree. When references or graph queries are unsupported, `unsupported_guidance` explains why empty reference/graph results are not authoritative and lists fallback commands. Add `--language `, `--extension `, or `--alias ` to retrieve one language row by canonical name, recognized suffix extension, or display alias. Add `--capability graph|references|symbols|missing-graph|missing-references|missing-symbols|search-only` to narrow the table to languages that support a structured capability or still have a capability gap. YAML reference extraction recognizes GitHub Actions `uses` targets, `needs` job edges, and unambiguous repository-local script/project paths in `run` steps. JSON reference extraction recognizes conservative repository-local file paths in string values, including paths embedded in command strings, and emits them as `project_reference` edges; URLs, parent-directory paths, and ambiguous bare filenames are ignored. +For one `--extension` lookup, CLI JSON and the MCP `languages` tool return the same `extension_lookup` structure. Ambiguous `.m` / `.pl` lookups preserve the case-insensitive normalized extension and ambiguity bucket, then list ordered candidate IDs, display names, aliases, recognized shebang interpreters, exact content regexes, project markers, reason/confidence rules, bounded probe limits, empty/binary handling, and `.cdidx-langmap.yaml` override entries. The ordered rules retain the detector's language-map, exact-filename, and non-empty filename-prefix precedence before shebang inspection. The shebang rule lists every globally recognized exact or prefix interpreter mapping, its case-insensitive matching policy, and the 256-byte first-physical-line boundary; a line that reaches the boundary without a terminator is not accepted as a shebang. These fields describe the same detector used by indexing; `index --dry-run --json` reports a selected ambiguous-extension file in `language_detections` with the corresponding `source` and `confidence`. + | Language | Extensions | Symbols | |---|---|:---:| | Python | `.py`, `.pyi`, `.pyw`, `BUILD`, `BUILD.bazel`, `WORKSPACE`, `WORKSPACE.bazel` (Bazel Starlark) | yes | @@ -5789,6 +5791,8 @@ CLI JSON と MCP の `languages` response は同じ catalog snapshot を共有 全言語が FTS5 全文検索に対応しています。**シンボル = yes** の行は、関数・クラス・import 名などの構造化検索にも対応します。現在の DB に存在する言語だけを一覧するには `cdidx languages --indexed-only --json` を使います。JSON 行には `symbol_extraction`、`reference_extraction`、`graph_queries`、`capability_gaps`、`unsupported_guidance`、`indexed_file_count` が含まれます。pattern capability は suffix だけを含む `extensions`、literal な `exact_filenames`、1 文字以上の末尾文字列を要求する `` placeholder 付き `filename_prefix_patterns` に分離されます。`legacy_patterns` は deprecation 期間のため従来の combined list を保持し、`pattern_provenance` は各 pattern の kind と `built_in`、`plugin_or_pattern`、`language_map_override` source を示します。top-level の `detection_policy` は、拡張子が引き続き case-insensitive である一方、完全一致ファイル名と filename-prefix pattern が index 対象 filesystem の `path_case_sensitive` ポリシーに従うことを示します。`precedence` 配列は、信頼済み language-map suffix override が built-in の完全一致 filename、prefix pattern、extension rule より先に適用されることを示します。`language_map_diagnostics` は安定した `code`、sanitization 済み `config`、`reason`、`blocks_parent_fallback` を返します。最も近い workspace の `.cdidx-langmap.yaml` を probe または read できない場合、cdidx は失敗を報告し、その subtree で親 workspace map を暗黙に継承しません。参照抽出やグラフクエリが未対応の場合、`unsupported_guidance` は空の参照/グラフ結果を根拠として扱えない理由と代替コマンドを示します。言語名・認識済み suffix extension・表示 alias から 1 行を取得するには `--language `、`--extension `、`--alias ` を追加してください。YAML の参照抽出は GitHub Actions の `uses` target、`needs` job edge、`run` step 内の明確なリポジトリローカル script / project path を認識します。JSON の参照抽出は command 文字列内を含む string value から保守的にリポジトリローカル file path を認識し、`project_reference` edge として記録します。URL、親ディレクトリ path、曖昧な basename だけのファイル名は無視します。 +`--extension` を 1 件指定した lookup では、CLI JSON と MCP の `languages` tool が同じ `extension_lookup` 構造を返します。曖昧な `.m` / `.pl` lookup は大小文字を区別しない正規化済み拡張子と ambiguity bucket を保持し、順序付きの候補 ID、表示名、alias、認識済み shebang interpreter、正確な content regex、project marker、reason/confidence rule、上限付き probe 制限、空・binary 入力の扱い、`.cdidx-langmap.yaml` の override entry を公開します。順序付き rule は、shebang を調べる前に detector が適用する language-map、完全一致 filename、空でない suffix を持つ filename-prefix の優先順位も保持します。authoritative な shebang は拡張子の候補外言語を選択する場合があるため、shebang rule は globally recognized な exact / prefix interpreter mapping、大小文字を区別しない matching policy、先頭物理行 256 byte の境界も列挙します。行終端なしでこの境界へ達した行は shebang として受理されません。これらの field は indexing と同じ detector を表し、`index --dry-run --json` は選択された曖昧拡張子ファイルを、対応する `source` と `confidence` とともに `language_detections` へ出力します。 + | 言語 | 拡張子 | シンボル | |---|---|:---:| | Python | `.py`, `.pyi`, `.pyw`, `BUILD`, `BUILD.bazel`, `WORKSPACE`, `WORKSPACE.bazel`(Bazel Starlark) | yes | diff --git a/changelog.d/unreleased/4901.added.md b/changelog.d/unreleased/4901.added.md new file mode 100644 index 000000000..a50c0a94c --- /dev/null +++ b/changelog.d/unreleased/4901.added.md @@ -0,0 +1,21 @@ +--- +category: added +issues: + - 4901 +affected: + - src/CodeIndex/Indexer/Scanning/FileIndexer.AmbiguousLanguageDetection.cs + - src/CodeIndex/Indexer/Scanning/FileIndexer.FileNameLanguages.cs + - src/CodeIndex/Indexer/Scanning/FileIndexer.Shebang.cs + - src/CodeIndex/Models/LanguageCapabilityCatalog.cs + - src/CodeIndex/Cli/QueryCommandRunner.Languages.cs + - src/CodeIndex/Mcp/McpToolHandlers.Languages.cs + - USER_GUIDE.md +--- + +## English + +- **Ambiguous extension lookups now explain their candidates (#4901)** — CLI and MCP `languages` lookups for `.m` and `.pl` return ordered candidate languages, aliases, exact/prefix filename precedence, all authoritative shebang outcomes and the 256-byte first-line boundary, detector evidence and confidence rules, case behavior, bounded input handling, and explicit language-map override entries from the same rules used during indexing. Index dry-run output also reports the selected language with the matching reason and confidence. + +## 日本語 + +- **曖昧な拡張子の lookup が候補を説明するようになりました(#4901)** — CLI と MCP の `languages` による `.m` / `.pl` lookup は、indexing と同じ rule から順序付き候補言語、alias、完全一致/prefix filename の優先順位、authoritative な全 shebang outcome と先頭行 256 byte の境界、detector の evidence と confidence rule、大小文字の扱い、上限付き入力処理、明示的な language-map override entry を返します。index dry-run 出力も、選択された言語を同じ reason と confidence とともに報告します。 diff --git a/src/CodeIndex/Cli/IndexCommandRunner.DryRun.cs b/src/CodeIndex/Cli/IndexCommandRunner.DryRun.cs index 1dc5facab..ea8bf9a8a 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.DryRun.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.DryRun.cs @@ -578,7 +578,7 @@ void RecordDryRunScanErrors(IEnumerable scanErrors) foreach (var detection in languageDetectionSamples) { CommandOutputWriter.WriteLine( - $" header detection {detection.Path}: {detection.Language} ({detection.Source}, confidence {detection.Confidence})"); + $" language detection {detection.Path}: {detection.Language} ({detection.Source}, confidence {detection.Confidence})"); } } return CommandExitCodes.Success; @@ -935,19 +935,29 @@ private static DryRunFileProbe ProbeDryRunFile( return DryRunFileProbe.FromUnsupported(); string? reusableLanguage = knownLanguage; - if (reusableLanguage == null) + FileIndexer.LanguageDetectionResult? preLoadDetection = null; + var isAmbiguousExtension = FileIndexer.TryGetAmbiguousLanguageDescriptor( + Path.GetExtension(absolutePath), + out _); + if (reusableLanguage == null || isAmbiguousExtension) { var detection = indexer.TryDetectLanguageForIndexing(absolutePath); - if (detection.Status == FileIndexer.FileProbeStatus.ProbeFailed) - return DryRunFileProbe.FromError("Could not probe file for indexability/language."); - if (detection.Status != FileIndexer.FileProbeStatus.Supported) - return string.IsNullOrEmpty(Path.GetExtension(absolutePath)) - ? DryRunFileProbe.FromUnsupported() - : DryRunFileProbe.FromUnknownExtension(); - - reusableLanguage = FileIndexer.CanReuseDetectedLanguageWithoutContent(absolutePath, detection.Language) - ? detection.Language - : null; + if (reusableLanguage == null) + { + if (detection.Status == FileIndexer.FileProbeStatus.ProbeFailed) + return DryRunFileProbe.FromError("Could not probe file for indexability/language."); + if (detection.Status != FileIndexer.FileProbeStatus.Supported) + return string.IsNullOrEmpty(Path.GetExtension(absolutePath)) + ? DryRunFileProbe.FromUnsupported() + : DryRunFileProbe.FromUnknownExtension(); + + reusableLanguage = FileIndexer.CanReuseDetectedLanguageWithoutContent(absolutePath, detection.Language) + ? detection.Language + : null; + } + + if (detection.Status == FileIndexer.FileProbeStatus.Supported) + preLoadDetection = detection; } try @@ -957,6 +967,13 @@ private static DryRunFileProbe ProbeDryRunFile( relativePath, reusableLanguage); var record = loaded.Record; + var reportDetection = loaded.LanguageDetection; + if (reportDetection.DetectionSource is null + && preLoadDetection is { DetectionSource: not null } detectedBeforeLoad + && string.Equals(detectedBeforeLoad.Language, record.Lang, StringComparison.Ordinal)) + { + reportDetection = detectedBeforeLoad; + } return new DryRunFileProbe( true, record.Lang ?? "unknown", @@ -964,8 +981,8 @@ private static DryRunFileProbe ProbeDryRunFile( loaded.Warning, Unsupported: false, UnknownExtension: false, - DetectionSource: loaded.LanguageDetection.DetectionSource, - DetectionConfidence: loaded.LanguageDetection.Confidence, + DetectionSource: reportDetection.DetectionSource, + DetectionConfidence: reportDetection.Confidence, Loaded: loaded, PolicySkipped: false, DryRunPolicySkipKind.None, diff --git a/src/CodeIndex/Cli/JsonOutputContracts.cs b/src/CodeIndex/Cli/JsonOutputContracts.cs index 44a48fdfa..afd785d7e 100644 --- a/src/CodeIndex/Cli/JsonOutputContracts.cs +++ b/src/CodeIndex/Cli/JsonOutputContracts.cs @@ -832,6 +832,7 @@ internal sealed record LanguagesJsonResult( [property: JsonPropertyName("detection_policy")] LanguageDetectionPolicyJsonResult DetectionPolicy, [property: JsonPropertyName("language_map_diagnostics")] List LanguageMapDiagnostics, [property: JsonPropertyName("reference_extraction_limits")] ReferenceExtractionSafetyLimits ReferenceExtractionLimits, + [property: JsonPropertyName("extension_lookup"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] JsonObject? ExtensionLookup = null, [property: JsonPropertyName("api_version")] string ApiVersion = JsonOutputContract.ApiVersion) : IVersionedJsonResult; internal sealed record IndexLanguageDetectionJsonResult( diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Languages.cs b/src/CodeIndex/Cli/QueryCommandRunner.Languages.cs index 0606ab300..53a2ed509 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Languages.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Languages.cs @@ -79,6 +79,7 @@ int WriteLanguages( catalog.Languages, filtered, indexedLanguageCounts); + var unpaginatedFiltered = filtered; if (json && (options.SummaryOnly || options.CountOnly || options.OutputFormat == OutputFormatCount)) { var payload = BuildLanguageSummaryPayload( @@ -143,7 +144,8 @@ int WriteLanguages( scopedCounts.ToJson(), BuildLanguageDetectionPolicy(), BuildLanguageMapDiagnostics(languageMapDiagnostics), - ReferenceExtractor.GetSafetyLimits()), + ReferenceExtractor.GetSafetyLimits(), + BuildSingleExtensionLookup(options, unpaginatedFiltered, catalog.Languages)), CliJsonSerializerContextFactory.Create(jsonOptions).LanguagesJsonResult, jsonOptions)); } @@ -310,6 +312,8 @@ private static JsonObject BuildLanguageSummaryPayload( payload["language_filters"] = BuildStringArray(options.LanguageLookups); if (options.LanguageExtensionLookups.Count > 0) payload["extension_filters"] = BuildStringArray(options.LanguageExtensionLookups); + if (BuildSingleExtensionLookup(options, languages, catalogLanguages) is { } extensionLookup) + payload["extension_lookup"] = extensionLookup; if (options.LanguageAliasLookups.Count > 0) payload["alias_filters"] = BuildStringArray(options.LanguageAliasLookups); @@ -330,6 +334,24 @@ private static JsonObject BuildLanguageSummaryPayload( return payload; } + private static JsonObject? BuildSingleExtensionLookup( + QueryCommandOptions options, + IReadOnlyList> matches, + IReadOnlyList> catalog) + { + if (options.LanguageExtensionLookups.Count != 1) + return null; + + var extension = options.LanguageExtensionLookups[0]; + var extensionMatches = matches + .Where(pair => LanguageCapabilityCatalog.MatchesExtension(pair.Value, extension)) + .ToList(); + return LanguageCapabilityCatalog.BuildExtensionLookup( + extension, + extensionMatches, + catalog); + } + private static JsonObject BuildLanguageCapabilityCounts(IReadOnlyList> languages) { static bool HasAll(LanguageCatalogSupportInfo language) diff --git a/src/CodeIndex/Indexer/Scanning/FileIndexer.AmbiguousLanguageDetection.cs b/src/CodeIndex/Indexer/Scanning/FileIndexer.AmbiguousLanguageDetection.cs index 9725f2ff2..612aca025 100644 --- a/src/CodeIndex/Indexer/Scanning/FileIndexer.AmbiguousLanguageDetection.cs +++ b/src/CodeIndex/Indexer/Scanning/FileIndexer.AmbiguousLanguageDetection.cs @@ -6,21 +6,74 @@ namespace CodeIndex.Indexer; public partial class FileIndexer { - private const int AmbiguousLanguageProbeByteLimit = 64 * 1024; - private const int AmbiguousProjectMarkerEntryLimit = 256; + internal const int AmbiguousLanguageProbeByteLimit = 64 * 1024; + internal const int AmbiguousProjectMarkerEntryLimit = 256; + internal const int AmbiguousProjectMarkerAncestorLimit = 32; - private static readonly Regex ObjectiveCContentMarker = new( - @"^\s*(?:#\s*import\b|@(?:interface|implementation|protocol)\b)", - RegexOptions.Compiled | RegexOptions.Multiline | RegexOptions.CultureInvariant); - private static readonly Regex MatlabContentMarker = new( - @"^\s*(?:function\b|classdef\b)", - RegexOptions.Compiled | RegexOptions.Multiline | RegexOptions.CultureInvariant); - private static readonly Regex PerlContentMarker = new( - @"^\s*(?:use\s+(?:strict|warnings)\s*;|package\s+[A-Za-z_]\w*(?:::\w+)*\s*;|(?:my|our|state)\s+[$@%][A-Za-z_]\w*|sub\s+[A-Za-z_]\w*)", - RegexOptions.Compiled | RegexOptions.Multiline | RegexOptions.CultureInvariant); - private static readonly Regex PrologContentMarker = new( - @"^\s*(?::-\s*(?:module|use_module|dynamic|multifile|discontiguous|initialization)\b|\?-\s*|[a-z][A-Za-z0-9_]*(?:\s*\([^\r\n.]*\))?\s*(?::-|-->))", - RegexOptions.Compiled | RegexOptions.Multiline | RegexOptions.CultureInvariant); + internal sealed record AmbiguousLanguageCandidate( + string Language, + string DisplayName, + string ContentPattern, + IReadOnlyList ProjectMarkerPatterns) + { + internal Regex ContentMarker { get; } = new( + ContentPattern, + RegexOptions.Compiled | RegexOptions.Multiline | RegexOptions.CultureInvariant); + + internal bool MatchesProjectMarker(string name) + => ProjectMarkerPatterns.Any(pattern => + pattern.StartsWith('*') + ? name.EndsWith(pattern[1..], StringComparison.OrdinalIgnoreCase) + : string.Equals(name, pattern, StringComparison.OrdinalIgnoreCase)); + } + + internal sealed record AmbiguousLanguageDescriptor( + string Extension, + string BucketLanguage, + IReadOnlyList Candidates); + + private static readonly AmbiguousLanguageDescriptor[] AmbiguousLanguageDescriptors = + [ + new( + ".m", + "ambiguous_m", + [ + new( + "objc", + "Objective-C", + @"^\s*(?:#\s*import\b|@(?:interface|implementation|protocol)\b)", + ["*.xcodeproj", "*.xcworkspace", "project.pbxproj"]), + new( + "matlab", + "MATLAB", + @"^\s*(?:function\b|classdef\b)", + ["*.prj", "*.slx", "*.mlx"]), + ]), + new( + ".pl", + "ambiguous_pl", + [ + new( + "perl", + "Perl", + @"^\s*(?:use\s+(?:strict|warnings)\s*;|package\s+[A-Za-z_]\w*(?:::\w+)*\s*;|(?:my|our|state)\s+[$@%][A-Za-z_]\w*|sub\s+[A-Za-z_]\w*)", + ["Makefile.PL", "Build.PL", "cpanfile"]), + new( + "prolog", + "Prolog", + @"^\s*(?::-\s*(?:module|use_module|dynamic|multifile|discontiguous|initialization)\b|\?-\s*|[a-z][A-Za-z0-9_]*(?:\s*\([^\r\n.]*\))?\s*(?::-|-->))", + ["*.pro", "*.prolog", "pack.pl"]), + ]), + ]; + + internal static bool TryGetAmbiguousLanguageDescriptor( + string extension, + out AmbiguousLanguageDescriptor descriptor) + { + descriptor = AmbiguousLanguageDescriptors.FirstOrDefault(candidate => + string.Equals(candidate.Extension, extension, StringComparison.OrdinalIgnoreCase))!; + return descriptor is not null; + } private static LanguageDetectionResult TryDetectAmbiguousExtensionLanguage( string filePath, @@ -31,9 +84,8 @@ private static LanguageDetectionResult TryDetectAmbiguousExtensionLanguage( Func? openReadForIndexContent, Func>? enumerateFileSystemEntries) { - var ambiguityBucket = string.Equals(extension, ".m", StringComparison.OrdinalIgnoreCase) - ? "ambiguous_m" - : "ambiguous_pl"; + if (!TryGetAmbiguousLanguageDescriptor(extension, out var descriptor)) + throw new ArgumentOutOfRangeException(nameof(extension), extension, "Expected an ambiguous language extension."); if (content == null) { @@ -45,49 +97,59 @@ private static LanguageDetectionResult TryDetectAmbiguousExtensionLanguage( if (readResult != FileProbeStatus.Supported) { if (!knownIndexability.HasValue && readResult == FileProbeStatus.Missing) - return new LanguageDetectionResult(FileProbeStatus.Supported, ambiguityBucket, "ambiguous"); + { + return new LanguageDetectionResult( + FileProbeStatus.Supported, + descriptor.BucketLanguage, + AmbiguousFallbackDetectionSource, + LanguageDetectionConfidence.Low); + } return new LanguageDetectionResult(readResult, null); } } - var firstFamily = string.Equals(extension, ".m", StringComparison.OrdinalIgnoreCase) - ? "objc" - : "perl"; - var secondFamily = string.Equals(extension, ".m", StringComparison.OrdinalIgnoreCase) - ? "matlab" - : "prolog"; - var firstContentMarker = string.Equals(extension, ".m", StringComparison.OrdinalIgnoreCase) - ? ObjectiveCContentMarker.IsMatch(content) - : PerlContentMarker.IsMatch(content); - var secondContentMarker = string.Equals(extension, ".m", StringComparison.OrdinalIgnoreCase) - ? MatlabContentMarker.IsMatch(content) - : PrologContentMarker.IsMatch(content); + var firstCandidate = descriptor.Candidates[0]; + var secondCandidate = descriptor.Candidates[1]; + var firstContentMarker = firstCandidate.ContentMarker.IsMatch(content); + var secondContentMarker = secondCandidate.ContentMarker.IsMatch(content); if (firstContentMarker != secondContentMarker) { return new LanguageDetectionResult( FileProbeStatus.Supported, - firstContentMarker ? firstFamily : secondFamily, - "content"); + firstContentMarker ? firstCandidate.Language : secondCandidate.Language, + AmbiguousContentDetectionSource, + LanguageDetectionConfidence.High); } if (firstContentMarker) - return new LanguageDetectionResult(FileProbeStatus.Supported, ambiguityBucket, "ambiguous"); + { + return new LanguageDetectionResult( + FileProbeStatus.Supported, + descriptor.BucketLanguage, + AmbiguousFallbackDetectionSource, + LanguageDetectionConfidence.Low); + } var (firstProjectMarker, secondProjectMarker) = ProbeAmbiguousLanguageProjectMarkers( filePath, projectRoot, - string.Equals(extension, ".m", StringComparison.OrdinalIgnoreCase), + descriptor, enumerateFileSystemEntries); if (firstProjectMarker != secondProjectMarker) { return new LanguageDetectionResult( FileProbeStatus.Supported, - firstProjectMarker ? firstFamily : secondFamily, - "project"); + firstProjectMarker ? firstCandidate.Language : secondCandidate.Language, + AmbiguousProjectDetectionSource, + LanguageDetectionConfidence.Medium); } - return new LanguageDetectionResult(FileProbeStatus.Supported, ambiguityBucket, "ambiguous"); + return new LanguageDetectionResult( + FileProbeStatus.Supported, + descriptor.BucketLanguage, + AmbiguousFallbackDetectionSource, + LanguageDetectionConfidence.Low); } private static FileProbeStatus TryReadAmbiguousLanguagePrefix( @@ -163,7 +225,7 @@ private static string DecodeBoundedPrefix(Encoding encoding, ReadOnlySpan private static (bool FirstFamily, bool SecondFamily) ProbeAmbiguousLanguageProjectMarkers( string filePath, string? projectRoot, - bool isMExtension, + AmbiguousLanguageDescriptor descriptor, Func>? enumerateFileSystemEntries) { string? directory; @@ -185,7 +247,7 @@ private static (bool FirstFamily, bool SecondFamily) ProbeAmbiguousLanguageProje var firstFamily = false; var secondFamily = false; - for (var depth = 0; depth < 32 && !string.IsNullOrEmpty(directory); depth++) + for (var depth = 0; depth < AmbiguousProjectMarkerAncestorLimit && !string.IsNullOrEmpty(directory); depth++) { try { @@ -198,9 +260,8 @@ private static (bool FirstFamily, bool SecondFamily) ProbeAmbiguousLanguageProje break; var name = Path.GetFileName(entry); - var markerFamilies = GetAmbiguousLanguageProjectMarkerFamilies(name, isMExtension); - firstFamily |= markerFamilies.FirstFamily; - secondFamily |= markerFamilies.SecondFamily; + firstFamily |= descriptor.Candidates[0].MatchesProjectMarker(name); + secondFamily |= descriptor.Candidates[1].MatchesProjectMarker(name); } } catch (Exception ex) when (ex is ArgumentException or IOException or UnauthorizedAccessException or NotSupportedException or System.Security.SecurityException) @@ -226,33 +287,7 @@ private static (bool FirstFamily, bool SecondFamily) ProbeAmbiguousLanguageProje internal static bool IsAmbiguousLanguageProjectMarkerPath(string path) { var name = Path.GetFileName(Path.TrimEndingDirectorySeparator(path)); - var mFamilies = GetAmbiguousLanguageProjectMarkerFamilies(name, isMExtension: true); - var plFamilies = GetAmbiguousLanguageProjectMarkerFamilies(name, isMExtension: false); - return mFamilies.FirstFamily || mFamilies.SecondFamily - || plFamilies.FirstFamily || plFamilies.SecondFamily; - } - - private static (bool FirstFamily, bool SecondFamily) GetAmbiguousLanguageProjectMarkerFamilies( - string name, - bool isMExtension) - { - if (isMExtension) - { - return ( - name.EndsWith(".xcodeproj", StringComparison.OrdinalIgnoreCase) - || name.EndsWith(".xcworkspace", StringComparison.OrdinalIgnoreCase) - || string.Equals(name, "project.pbxproj", StringComparison.OrdinalIgnoreCase), - name.EndsWith(".prj", StringComparison.OrdinalIgnoreCase) - || name.EndsWith(".slx", StringComparison.OrdinalIgnoreCase) - || name.EndsWith(".mlx", StringComparison.OrdinalIgnoreCase)); - } - - return ( - string.Equals(name, "Makefile.PL", StringComparison.OrdinalIgnoreCase) - || string.Equals(name, "Build.PL", StringComparison.OrdinalIgnoreCase) - || string.Equals(name, "cpanfile", StringComparison.OrdinalIgnoreCase), - name.EndsWith(".pro", StringComparison.OrdinalIgnoreCase) - || name.EndsWith(".prolog", StringComparison.OrdinalIgnoreCase) - || string.Equals(name, "pack.pl", StringComparison.OrdinalIgnoreCase)); + return AmbiguousLanguageDescriptors.Any(descriptor => + descriptor.Candidates.Any(candidate => candidate.MatchesProjectMarker(name))); } } diff --git a/src/CodeIndex/Indexer/Scanning/FileIndexer.FileNameLanguages.cs b/src/CodeIndex/Indexer/Scanning/FileIndexer.FileNameLanguages.cs index b8c90412d..088fcd3b0 100644 --- a/src/CodeIndex/Indexer/Scanning/FileIndexer.FileNameLanguages.cs +++ b/src/CodeIndex/Indexer/Scanning/FileIndexer.FileNameLanguages.cs @@ -78,4 +78,27 @@ private static readonly (string Prefix, string Language)[] FileNamePrefixMap = ("Makefile.", "makefile"), ("GNUmakefile.", "makefile"), ]; + + internal sealed record FilenameLanguageRule( + string Pattern, + string Language); + + internal static IReadOnlyList GetExactFilenameLanguageRulesForExtension( + string extension) + => FileNameMap + .Where(pair => string.Equals( + Path.GetExtension(pair.Key), + extension, + StringComparison.OrdinalIgnoreCase)) + .OrderBy(pair => pair.Key, StringComparer.Ordinal) + .Select(pair => new FilenameLanguageRule(pair.Key, pair.Value)) + .ToArray(); + + internal static IReadOnlyList GetFilenamePrefixLanguageRules() + => FileNamePrefixMap + .OrderBy(rule => rule.Prefix, StringComparer.Ordinal) + .Select(rule => new FilenameLanguageRule( + rule.Prefix + "", + rule.Language)) + .ToArray(); } diff --git a/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs b/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs index 3ab117574..425560422 100644 --- a/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs +++ b/src/CodeIndex/Indexer/Scanning/FileIndexer.LanguageDetection.cs @@ -526,7 +526,15 @@ internal static LanguageDetectionResult TryDetectLanguage( // 信頼済みの明示 suffix override は拡張子、suffix を含む完全一致 special filename、 // suffix 付き prefix variant のいずれでも最優先する。 if (TryDetectLanguageOverride(filePath, fileName, languageMapOverrideResolver, out var overrideLang)) - return new LanguageDetectionResult(FileProbeStatus.Supported, overrideLang); + { + return TryGetAmbiguousLanguageDescriptor(ext, out _) + ? new LanguageDetectionResult( + FileProbeStatus.Supported, + overrideLang, + LanguageMapOverrideDetectionSource, + LanguageDetectionConfidence.High) + : new LanguageDetectionResult(FileProbeStatus.Supported, overrideLang); + } // Exact filename matching beats built-in extension lookup so manifest-style filenames // like `pyproject.toml` map to a dependency category instead of the generic file type. @@ -551,7 +559,14 @@ internal static LanguageDetectionResult TryDetectLanguage( knownIndexability, openReadForIndexContent); if (shebangLanguage.Status == FileProbeStatus.Supported) - return shebangLanguage; + { + return TryGetAmbiguousLanguageDescriptor(ext, out _) + ? shebangLanguage with + { + Confidence = LanguageDetectionConfidence.High, + } + : shebangLanguage; + } if (knownIndexability.HasValue && shebangLanguage.Status is FileProbeStatus.Missing or FileProbeStatus.ProbeFailed) { diff --git a/src/CodeIndex/Indexer/Scanning/FileIndexer.Shebang.cs b/src/CodeIndex/Indexer/Scanning/FileIndexer.Shebang.cs index e5a1274b5..1b13993cf 100644 --- a/src/CodeIndex/Indexer/Scanning/FileIndexer.Shebang.cs +++ b/src/CodeIndex/Indexer/Scanning/FileIndexer.Shebang.cs @@ -7,10 +7,44 @@ public partial class FileIndexer // Shebang detection reads at most the first physical line within this // byte cap. NUL bytes or a line that reaches the cap without LF/CR are treated as // unsupported so binary executables and minified data are not parsed as scripts. - private const int ShebangProbeByteLimit = 256; + internal const int ShebangProbeByteLimit = 256; + private const string PythonShebangInterpreterPrefix = "python"; private static readonly UTF8Encoding StrictShebangUtf8Encoding = new(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true); private static readonly UnicodeEncoding StrictShebangUtf16LittleEndianEncoding = new(bigEndian: false, byteOrderMark: false, throwOnInvalidBytes: true); private static readonly UnicodeEncoding StrictShebangUtf16BigEndianEncoding = new(bigEndian: true, byteOrderMark: false, throwOnInvalidBytes: true); + private static readonly IReadOnlyDictionary ExactShebangInterpreterLanguages = + new Dictionary(StringComparer.Ordinal) + { + ["bash"] = "shell", + ["sh"] = "shell", + ["zsh"] = "shell", + ["fish"] = "shell", + ["dash"] = "shell", + ["ksh"] = "shell", + ["ash"] = "shell", + ["node"] = "javascript", + ["nodejs"] = "javascript", + ["ruby"] = "ruby", + ["perl"] = "perl", + ["tclsh"] = "tcl", + ["wish"] = "tcl", + ["matlab"] = "matlab", + ["octave"] = "matlab", + ["octave-cli"] = "matlab", + ["prolog"] = "prolog", + ["swipl"] = "prolog", + ["sicstus"] = "prolog", + ["gprolog"] = "prolog", + ["php"] = "php", + ["lua"] = "lua", + ["pwsh"] = "powershell", + ["powershell"] = "powershell", + }; + + internal sealed record ShebangInterpreterRule( + string MatchKind, + string Pattern, + string Language); /// /// Try to infer a language from a script shebang. @@ -91,7 +125,7 @@ private static LanguageDetectionResult TryDetectLanguageFromShebang( var language = MapShebangInterpreterToLanguage(interpreter); return language != null - ? new LanguageDetectionResult(FileProbeStatus.Supported, language, DetectionSource: "shebang") + ? new LanguageDetectionResult(FileProbeStatus.Supported, language, DetectionSource: ShebangDetectionSource) : new LanguageDetectionResult(FileProbeStatus.Unsupported, null); } catch (FileNotFoundException) @@ -321,19 +355,32 @@ private static IReadOnlyList TokenizeShebangCommandLine(string commandLi return Path.GetFileName(candidate).ToLowerInvariant(); } - private static string? MapShebangInterpreterToLanguage(string interpreter) => interpreter switch + internal static IReadOnlyList GetShebangInterpretersForLanguage(string language) + => ExactShebangInterpreterLanguages + .Where(pair => string.Equals(pair.Value, language, StringComparison.Ordinal)) + .Select(pair => pair.Key) + .OrderBy(value => value, StringComparer.Ordinal) + .ToArray(); + + internal static IReadOnlyList GetShebangInterpreterRules() { - "bash" or "sh" or "zsh" or "fish" or "dash" or "ksh" or "ash" => "shell", - "node" or "nodejs" => "javascript", - "ruby" => "ruby", - "perl" => "perl", - "tclsh" or "wish" => "tcl", - "matlab" or "octave" or "octave-cli" => "matlab", - "prolog" or "swipl" or "sicstus" or "gprolog" => "prolog", - "php" => "php", - "lua" => "lua", - "pwsh" or "powershell" => "powershell", - _ when interpreter.StartsWith("python", StringComparison.Ordinal) => "python", - _ => null, - }; + var rules = ExactShebangInterpreterLanguages + .OrderBy(pair => pair.Key, StringComparer.Ordinal) + .Select(pair => new ShebangInterpreterRule("exact", pair.Key, pair.Value)) + .ToList(); + rules.Add(new ShebangInterpreterRule( + "prefix", + PythonShebangInterpreterPrefix, + "python")); + return rules; + } + + private static string? MapShebangInterpreterToLanguage(string interpreter) + { + if (ExactShebangInterpreterLanguages.TryGetValue(interpreter, out var language)) + return language; + return interpreter.StartsWith(PythonShebangInterpreterPrefix, StringComparison.Ordinal) + ? "python" + : null; + } } diff --git a/src/CodeIndex/Indexer/Scanning/FileIndexer.Types.cs b/src/CodeIndex/Indexer/Scanning/FileIndexer.Types.cs index a247d0f3d..484b64cab 100644 --- a/src/CodeIndex/Indexer/Scanning/FileIndexer.Types.cs +++ b/src/CodeIndex/Indexer/Scanning/FileIndexer.Types.cs @@ -29,6 +29,13 @@ internal enum LanguageDetectionConfidence internal const string HeaderLexicalFallbackDetectionSource = "header_lexical_fallback"; internal const string HeaderSampledLexicalFallbackDetectionSource = "header_sampled_lexical_fallback"; internal const string HeaderExtensionFallbackDetectionSource = "header_extension_fallback"; + internal const string LanguageMapOverrideDetectionSource = "language_map_override"; + internal const string ExactFilenameDetectionSource = "exact_filename"; + internal const string FilenamePrefixPatternDetectionSource = "filename_prefix_pattern"; + internal const string ShebangDetectionSource = "shebang"; + internal const string AmbiguousContentDetectionSource = "content"; + internal const string AmbiguousProjectDetectionSource = "project"; + internal const string AmbiguousFallbackDetectionSource = "ambiguous"; internal readonly record struct LanguageDetectionResult( FileProbeStatus Status, diff --git a/src/CodeIndex/Mcp/McpToolCatalog.cs b/src/CodeIndex/Mcp/McpToolCatalog.cs index 03bc5c69d..9549b5976 100644 --- a/src/CodeIndex/Mcp/McpToolCatalog.cs +++ b/src/CodeIndex/Mcp/McpToolCatalog.cs @@ -479,7 +479,7 @@ private static JsonArray CreateToolCatalog() ["indexedOnly"] = new JsonObject { ["type"] = "boolean", ["description"] = "Return only languages currently present in the index. Requires the configured database.", ["default"] = false }, ["capability"] = new JsonObject { ["oneOf"] = new JsonArray { new JsonObject { ["type"] = "string", ["enum"] = CreateLanguageCapabilityEnum() }, new JsonObject { ["type"] = "array", ["items"] = new JsonObject { ["type"] = "string", ["enum"] = CreateLanguageCapabilityEnum() } } }, ["description"] = "Filter by the same capability or capability-gap values as CLI `languages --capability`. Accepts a single value or an array; all requested capabilities must match." }, ["language"] = new JsonObject { ["type"] = "string", ["description"] = "Look up one canonical language using the same exact normalization as CLI `languages --language`, e.g. `csharp` or `cs`." }, - ["extension"] = new JsonObject { ["type"] = "string", ["description"] = "Look up languages by file extension. Accepts `cs` or `.cs` style values." }, + ["extension"] = new JsonObject { ["type"] = "string", ["description"] = "Look up languages by file extension. Accepts `cs` or `.cs` style values; ambiguous extensions also return ordered candidates, detector evidence/rules, and override guidance in extension_lookup." }, ["alias"] = new JsonObject { ["type"] = "string", ["description"] = "Look up languages by exact CLI language alias; canonical language names remain accepted for backward compatibility." }, ["limit"] = new JsonObject { ["type"] = "integer", ["minimum"] = 1, ["maximum"] = MaxLimit, ["description"] = "Maximum catalog entries to return per page.", ["default"] = QueryCommandRunner.DefaultQueryLimit }, ["cursor"] = new JsonObject { ["type"] = "string", ["maxLength"] = MaxMcpQueryCursorCharacters, ["description"] = "Opaque next_cursor from the previous languages page. Keep every filter, limit, and maxBytes unchanged." }, diff --git a/src/CodeIndex/Mcp/McpToolHandlers.Languages.cs b/src/CodeIndex/Mcp/McpToolHandlers.Languages.cs index b32f9037f..88f3d8294 100644 --- a/src/CodeIndex/Mcp/McpToolHandlers.Languages.cs +++ b/src/CodeIndex/Mcp/McpToolHandlers.Languages.cs @@ -105,7 +105,7 @@ JsonNode BuildResponse( var queryFingerprint = BuildMcpQueryFingerprint( "languages", limit, - "catalog-v1", + "catalog-v2", [ new("alias", normalizedAlias), new("extension", normalizedExtension is null @@ -284,10 +284,10 @@ private JsonObject BuildLanguageCatalogPayload( } if (extensionFilter is not null) { - payload["extension_lookup"] = BuildLanguageLookup( - "extension", + payload["extension_lookup"] = LanguageCapabilityCatalog.BuildExtensionLookup( extensionFilter, - filtered); + filtered, + catalog.Languages); } if (aliasFilter is not null) { @@ -396,7 +396,7 @@ private static string BuildLanguageCatalogGenerationFingerprint( { var components = new List { - "mcp-language-catalog-generation:v1", + "mcp-language-catalog-generation:v2", }; foreach (var (language, entry) in catalog.Languages) { diff --git a/src/CodeIndex/Models/LanguageCapabilityCatalog.cs b/src/CodeIndex/Models/LanguageCapabilityCatalog.cs index 39da511b4..a3fd44310 100644 --- a/src/CodeIndex/Models/LanguageCapabilityCatalog.cs +++ b/src/CodeIndex/Models/LanguageCapabilityCatalog.cs @@ -222,6 +222,126 @@ internal static bool MatchesAlias(LanguageCatalogSupportInfo language, string lo string.Equals(NormalizeLookupKey(alias), normalized, StringComparison.Ordinal)); } + internal static JsonObject BuildExtensionLookup( + string value, + IReadOnlyList> matches, + IReadOnlyList> catalog) + { + var normalizedExtension = NormalizeExtension(value); + var result = new JsonObject + { + ["extension"] = value, + ["normalized_extension"] = normalizedExtension, + ["matched"] = matches.Count, + ["languages"] = BuildStringArray(matches.Select(pair => pair.Key)), + ["extension_case_policy"] = "case_insensitive", + ["ambiguous"] = false, + }; + + if (!FileIndexer.TryGetAmbiguousLanguageDescriptor(normalizedExtension, out var descriptor)) + return result; + + result["ambiguous"] = true; + result["bucket_language"] = descriptor.BucketLanguage; + result["candidates"] = new JsonArray(descriptor.Candidates.Select(candidate => + { + var support = catalog.FirstOrDefault(pair => + string.Equals(pair.Key, candidate.Language, StringComparison.Ordinal)).Value; + return (JsonNode)new JsonObject + { + ["lang"] = candidate.Language, + ["display_name"] = candidate.DisplayName, + ["aliases"] = BuildStringArray(support?.Aliases ?? []), + ["evidence"] = new JsonObject + { + ["shebang_interpreters"] = BuildStringArray( + FileIndexer.GetShebangInterpretersForLanguage(candidate.Language)), + ["content_pattern"] = candidate.ContentPattern, + ["content_case_policy"] = "case_sensitive", + ["project_markers"] = BuildStringArray(candidate.ProjectMarkerPatterns), + ["project_marker_case_policy"] = "case_insensitive", + }, + }; + }).ToArray()); + result["detection_rules"] = new JsonArray( + BuildDetectionRule( + FileIndexer.LanguageMapOverrideDetectionSource, + FileIndexer.LanguageDetectionConfidence.High, + precedence: 1, + selection: "authoritative"), + BuildDetectionRule( + FileIndexer.ExactFilenameDetectionSource, + confidence: null, + precedence: 2, + selection: "authoritative_when_exact_filename_matches", + ("case_policy", JsonValue.Create("filesystem")), + ("case_policy_source", JsonValue.Create("path_case_sensitive")), + ("applicable_patterns", BuildFilenameRuleArray( + FileIndexer.GetExactFilenameLanguageRulesForExtension(normalizedExtension)))), + BuildDetectionRule( + FileIndexer.FilenamePrefixPatternDetectionSource, + confidence: null, + precedence: 3, + selection: "authoritative_when_non_empty_suffix_matches", + ("case_policy", JsonValue.Create("filesystem")), + ("case_policy_source", JsonValue.Create("path_case_sensitive")), + ("patterns", BuildFilenameRuleArray( + FileIndexer.GetFilenamePrefixLanguageRules()))), + BuildDetectionRule( + FileIndexer.ShebangDetectionSource, + FileIndexer.LanguageDetectionConfidence.High, + precedence: 4, + selection: "recognized_interpreter_authoritative", + ("candidate_restricted", JsonValue.Create(false)), + ("probe_scope", JsonValue.Create("first_physical_line")), + ("probe_byte_limit", JsonValue.Create(FileIndexer.ShebangProbeByteLimit)), + ("line_termination_policy", JsonValue.Create("required_before_limit_unless_eof")), + ("interpreter_case_policy", JsonValue.Create("case_insensitive")), + ("interpreter_rules", new JsonArray( + FileIndexer.GetShebangInterpreterRules() + .Select(rule => (JsonNode)new JsonObject + { + ["match"] = rule.MatchKind, + ["pattern"] = rule.Pattern, + ["language"] = rule.Language, + }) + .ToArray()))), + BuildDetectionRule( + FileIndexer.AmbiguousContentDetectionSource, + FileIndexer.LanguageDetectionConfidence.High, + precedence: 5, + selection: "exactly_one_candidate_matches", + ("probe_byte_limit", JsonValue.Create(FileIndexer.AmbiguousLanguageProbeByteLimit))), + BuildDetectionRule( + FileIndexer.AmbiguousProjectDetectionSource, + FileIndexer.LanguageDetectionConfidence.Medium, + precedence: 6, + selection: "exactly_one_candidate_matches", + ("ancestor_directory_limit", JsonValue.Create(FileIndexer.AmbiguousProjectMarkerAncestorLimit)), + ("entry_limit_per_directory", JsonValue.Create(FileIndexer.AmbiguousProjectMarkerEntryLimit))), + BuildDetectionRule( + FileIndexer.AmbiguousFallbackDetectionSource, + FileIndexer.LanguageDetectionConfidence.Low, + precedence: 7, + selection: "zero_or_multiple_candidates_match")); + result["input_handling"] = new JsonObject + { + ["empty_text"] = "project_then_ambiguous", + ["binary_or_invalid_text"] = "unsupported", + ["missing_without_indexability_context"] = "ambiguous", + }; + result["override_guidance"] = new JsonObject + { + ["config_file"] = LanguageMapOverrides.WorkspaceFileName, + ["entries"] = new JsonArray(descriptor.Candidates.Select(candidate => (JsonNode)new JsonObject + { + ["extension"] = descriptor.Extension, + ["language"] = candidate.Language, + }).ToArray()), + }; + return result; + } + internal static string NormalizeLookupKey(string value) { var builder = new StringBuilder(value.Length); @@ -234,6 +354,48 @@ internal static string NormalizeLookupKey(string value) return builder.ToString(); } + private static string NormalizeExtension(string value) + => "." + NormalizeLookupKey(value); + + private static JsonArray BuildStringArray(IEnumerable values) + => new(values + .OrderBy(value => value, StringComparer.Ordinal) + .Select(value => JsonValue.Create(value)) + .ToArray()); + + private static JsonArray BuildFilenameRuleArray( + IEnumerable rules) + => new(rules + .Select(rule => (JsonNode)new JsonObject + { + ["pattern"] = rule.Pattern, + ["language"] = rule.Language, + }) + .ToArray()); + + private static JsonObject BuildDetectionRule( + string source, + FileIndexer.LanguageDetectionConfidence? confidence, + int precedence, + string selection, + params (string Name, JsonNode? Value)[] details) + { + var rule = new JsonObject + { + ["source"] = source, + ["precedence"] = precedence, + ["selection"] = selection, + }; + if (confidence is { } reportedConfidence) + { + rule["confidence"] = + FileIndexer.GetLanguageDetectionConfidenceCode(reportedConfidence); + } + foreach (var (name, value) in details) + rule[name] = value; + return rule; + } + private static LanguageCapabilityCountScope CountScope( string scope, IReadOnlyList> languages) diff --git a/tests/CodeIndex.Tests/FileIndexerTests.cs b/tests/CodeIndex.Tests/FileIndexerTests.cs index bc384dcc1..5b5372c05 100644 --- a/tests/CodeIndex.Tests/FileIndexerTests.cs +++ b/tests/CodeIndex.Tests/FileIndexerTests.cs @@ -2437,7 +2437,7 @@ public void DetectLanguage_StrongExtensionWinsConflictingShebang_Issue4611() } [Theory] - [InlineData("Widget.m", "#import \n@interface Widget : NSObject\n@end\n", "objc")] + [InlineData("Widget.M", "#import \n@interface Widget : NSObject\n@end\n", "objc")] [InlineData("add.m", "function result = add(left, right)\nresult = left + right;\nend\n", "matlab")] [InlineData("Worker.pl", "use strict;\nuse warnings;\nsub run { return 1; }\n", "perl")] [InlineData("family.pl", "ancestor(X, Y) :- parent(X, Y).\n", "prolog")] @@ -2455,6 +2455,7 @@ public void DetectLanguage_AmbiguousExtensionsUseStrongContentMarkers_Issue4612( Assert.Equal(FileIndexer.FileProbeStatus.Supported, detection.Status); Assert.Equal(expectedLanguage, detection.Language); Assert.Equal("content", detection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.High, detection.Confidence); Assert.Equal(expectedLanguage, scan.FileLanguages[path]); } @@ -2471,6 +2472,7 @@ public void DetectLanguage_ZeroArityPrologRulesAreStrongContentMarkers_Issue4612 Assert.Equal(FileIndexer.FileProbeStatus.Supported, detection.Status); Assert.Equal("prolog", detection.Language); Assert.Equal("content", detection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.High, detection.Confidence); } [Fact] @@ -2512,6 +2514,7 @@ public void DetectLanguage_WeakOrConflictingMarkersRemainExplicitlyAmbiguous_Iss Assert.Equal(expectedLanguage, detection.Language); Assert.Equal("ambiguous", detection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.Low, detection.Confidence); } [Theory] @@ -2532,6 +2535,7 @@ public void DetectLanguage_AmbiguousExtensionsUseConservativeProjectMarkers_Issu Assert.Equal(expectedLanguage, detection.Language); Assert.Equal("project", detection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.Medium, detection.Confidence); } [Fact] @@ -2547,6 +2551,89 @@ public void DetectLanguage_PrologShebangOverridesAmbiguousPl_Issue4612() Assert.Equal("prolog", detection.Language); Assert.Equal("shebang", detection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.High, detection.Confidence); + } + + [Fact] + public void DetectLanguage_FilenamePrefixPrecedesAmbiguousExtensionShebang_Issue4901() + { + using var project = TestProjectHelper.CreateTempProjectScope("cdidx_ambiguous_filename_precedence"); + var path = TestProjectHelper.WriteTextFile( + project.Root, + "Makefile.pl", + "#!/usr/bin/env ruby\nputs 1\n"); + + var detection = FileIndexer.TryDetectLanguage(path); + + Assert.Equal("makefile", detection.Language); + Assert.Null(detection.DetectionSource); + Assert.Null(detection.Confidence); + } + + [Fact] + public void DetectLanguage_AmbiguousShebangRequiresTerminatorBeforeByteLimit_Issue4901() + { + using var project = TestProjectHelper.CreateTempProjectScope("cdidx_ambiguous_shebang_boundary"); + const string shebang = "#!/usr/bin/env ruby"; + var path = TestProjectHelper.WriteTextFile( + project.Root, + "tool.pl", + shebang + + new string(' ', FileIndexer.ShebangProbeByteLimit - shebang.Length) + + "\nputs 1\n"); + + var detection = FileIndexer.TryDetectLanguage(path); + + Assert.Equal("ambiguous_pl", detection.Language); + Assert.Equal("ambiguous", detection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.Low, detection.Confidence); + } + + [Fact] + public void DetectLanguage_AmbiguousExtensionOverrideReportsAuthoritativeReason_Issue4901() + { + using var project = TestProjectHelper.CreateTempProjectScope("cdidx_ambiguous_override"); + LanguageMapOverrides.ClearEffectiveMapCacheForTesting(); + try + { + TestProjectHelper.WriteTextFile( + project.Root, + LanguageMapOverrides.WorkspaceFileName, + "entries:\n - extension: \".m\"\n language: \"matlab\"\n"); + var path = TestProjectHelper.WriteTextFile( + project.Root, + "Widget.M", + "#import \n"); + + var detection = FileIndexer.TryDetectLanguage(path); + + Assert.Equal("matlab", detection.Language); + Assert.Equal("language_map_override", detection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.High, detection.Confidence); + } + finally + { + LanguageMapOverrides.ClearEffectiveMapCacheForTesting(); + } + } + + [Fact] + public void DetectLanguage_EmptyAndBinaryAmbiguousFilesHaveExplicitOutcomes_Issue4901() + { + using var project = TestProjectHelper.CreateTempProjectScope("cdidx_ambiguous_inputs"); + var emptyPath = TestProjectHelper.WriteTextFile(project.Root, "empty.M", string.Empty); + var binaryPath = TestProjectHelper.ProjectPath(project.Root, "binary.m"); + File.WriteAllBytes(binaryPath, [(byte)'x', 0, (byte)'y']); + + var emptyDetection = FileIndexer.TryDetectLanguage(emptyPath); + var binaryDetection = FileIndexer.TryDetectLanguage(binaryPath); + + Assert.Equal(FileIndexer.FileProbeStatus.Supported, emptyDetection.Status); + Assert.Equal("ambiguous_m", emptyDetection.Language); + Assert.Equal("ambiguous", emptyDetection.DetectionSource); + Assert.Equal(FileIndexer.LanguageDetectionConfidence.Low, emptyDetection.Confidence); + Assert.Equal(FileIndexer.FileProbeStatus.Unsupported, binaryDetection.Status); + Assert.Null(binaryDetection.Language); } [Fact] diff --git a/tests/CodeIndex.Tests/IndexCommandRunnerDryRunTests.cs b/tests/CodeIndex.Tests/IndexCommandRunnerDryRunTests.cs index e6217053b..20331313f 100644 --- a/tests/CodeIndex.Tests/IndexCommandRunnerDryRunTests.cs +++ b/tests/CodeIndex.Tests/IndexCommandRunnerDryRunTests.cs @@ -1040,6 +1040,135 @@ public void Run_DryRun_HeaderLanguageDetectionExposesSourceAndConfidence_Issue46 } } + [Theory] + [InlineData( + "Widget.M", + "#import \n@interface Widget : NSObject\n@end\n", + "objc", + "content")] + [InlineData( + "tool.M", + "#!/usr/bin/env ruby\nputs 'hi'\n", + "ruby", + "shebang")] + public void Run_DryRun_AmbiguousExtensionReportsSelectedLanguageAndSharedReason_Issue4901( + string fileName, + string content, + string expectedLanguage, + string expectedSource) + { + var projectRoot = CreateTempProject(); + try + { + File.WriteAllText( + Path.Combine(projectRoot, fileName), + content); + + var (exitCode, json) = RunAndCaptureJson([ + projectRoot, + "--files", + fileName, + "--dry-run", + "--json", + ]); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(1, json.GetProperty("language_detections_total").GetInt32()); + var detection = Assert.Single(json.GetProperty("language_detections").EnumerateArray().ToArray()); + Assert.Equal(fileName, detection.GetProperty("path").GetString()); + Assert.Equal(expectedLanguage, detection.GetProperty("language").GetString()); + Assert.Equal(expectedSource, detection.GetProperty("source").GetString()); + Assert.Equal("high", detection.GetProperty("confidence").GetString()); + + var (humanExitCode, stdout, stderr) = RunAndCaptureStreams([ + projectRoot, + "--files", + fileName, + "--dry-run", + ]); + Assert.Equal(CommandExitCodes.Success, humanExitCode); + Assert.Equal(string.Empty, stderr); + Assert.Contains( + $"language detection {fileName}: {expectedLanguage} ({expectedSource}, confidence high)", + stdout, + StringComparison.Ordinal); + Assert.DoesNotContain("header detection", stdout, StringComparison.Ordinal); + } + finally + { + DeleteDirectory(projectRoot); + } + } + + [Fact] + public void Run_DryRun_FullScanPreservesBoundedAmbiguousProjectReason_Issue4901() + { + var projectRoot = CreateTempProject(); + try + { + File.WriteAllText(Path.Combine(projectRoot, "App.xcodeproj"), string.Empty); + var contentBeyondProbe = new string(' ', FileIndexer.AmbiguousLanguageProbeByteLimit + 1024); + File.WriteAllText( + Path.Combine(projectRoot, "Widget.M"), + contentBeyondProbe + "\n#import \n"); + File.WriteAllText( + Path.Combine(projectRoot, "late.M"), + contentBeyondProbe + "\nfunction result = late()\nend\n"); + + var (exitCode, json) = RunAndCaptureJson([ + projectRoot, + "--dry-run", + "--json", + ]); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(2, json.GetProperty("language_detections_total").GetInt32()); + var detections = json.GetProperty("language_detections").EnumerateArray() + .OrderBy(detection => detection.GetProperty("path").GetString(), StringComparer.Ordinal) + .ToArray(); + Assert.Equal(["Widget.M", "late.M"], detections.Select(detection => detection.GetProperty("path").GetString())); + Assert.All(detections, detection => + { + Assert.Equal("objc", detection.GetProperty("language").GetString()); + Assert.Equal("project", detection.GetProperty("source").GetString()); + Assert.Equal("medium", detection.GetProperty("confidence").GetString()); + }); + } + finally + { + DeleteDirectory(projectRoot); + } + } + + [Fact] + public void Run_DryRun_TShebangDoesNotGainAmbiguousDescriptorConfidence_Issue4901() + { + var projectRoot = CreateTempProject(); + try + { + File.WriteAllText( + Path.Combine(projectRoot, "tool.t"), + "#!/usr/bin/env ruby\nputs 1\n"); + + var results = new[] + { + RunAndCaptureJson([projectRoot, "--dry-run", "--json"]), + RunAndCaptureJson([projectRoot, "--files", "tool.t", "--dry-run", "--json"]), + }; + + Assert.All(results, result => + { + Assert.Equal(CommandExitCodes.Success, result.ExitCode); + Assert.Equal(1, result.Json.GetProperty("languages").GetProperty("ruby").GetInt32()); + Assert.Equal(0, result.Json.GetProperty("language_detections_total").GetInt32()); + }); + } + finally + { + DeleteDirectory(projectRoot); + } + } + [Fact] public void Run_DryRun_WithFiles_NormalizesUnicodeDbPathForEstimates() { diff --git a/tests/CodeIndex.Tests/McpServerIssue4896Tests.cs b/tests/CodeIndex.Tests/McpServerIssue4896Tests.cs index 907d8eb87..32ba3a4ce 100644 --- a/tests/CodeIndex.Tests/McpServerIssue4896Tests.cs +++ b/tests/CodeIndex.Tests/McpServerIssue4896Tests.cs @@ -1,4 +1,5 @@ using System.Text.Json.Nodes; +using CodeIndex.Indexer; using CodeIndex.Indexer.Extensibility; using CodeIndex.Mcp; using CodeIndex.Models; @@ -39,12 +40,46 @@ public void Languages_ExactFiltersReportSeparateCatalogAndMatchCounts_Issue4896( var ambiguousExtension = CallIssue4896Languages(new JsonObject { - ["extension"] = "m", + ["extension"] = ".M", }, id: 4); Assert.Equal( "ambiguous_m", Assert.Single(ambiguousExtension["languages"]!.AsArray())!["lang"]!.GetValue()); - Assert.Equal(1, ambiguousExtension["extension_lookup"]!["matched"]!.GetValue()); + var ambiguousLookup = ambiguousExtension["extension_lookup"]!; + Assert.Equal(1, ambiguousLookup["matched"]!.GetValue()); + Assert.Equal(".m", ambiguousLookup["normalized_extension"]!.GetValue()); + Assert.True(ambiguousLookup["ambiguous"]!.GetValue()); + Assert.Equal( + ["objc", "matlab"], + ambiguousLookup["candidates"]!.AsArray() + .Select(candidate => candidate!["lang"]!.GetValue())); + Assert.Contains( + "octave", + ambiguousLookup["candidates"]![1]!["aliases"]!.AsArray() + .Select(aliasValue => aliasValue!.GetValue())); + var shebangRule = ambiguousLookup["detection_rules"]!.AsArray() + .Single(rule => rule!["source"]!.GetValue() == "shebang")!; + Assert.Equal(4, shebangRule["precedence"]!.GetValue()); + Assert.Equal(FileIndexer.ShebangProbeByteLimit, shebangRule["probe_byte_limit"]!.GetValue()); + Assert.Equal( + "required_before_limit_unless_eof", + shebangRule["line_termination_policy"]!.GetValue()); + Assert.Equal("case_insensitive", shebangRule["interpreter_case_policy"]!.GetValue()); + var shebangRules = shebangRule["interpreter_rules"]!.AsArray(); + Assert.Contains( + shebangRules, + rule => rule!["pattern"]!.GetValue() == "ruby" + && rule["language"]!.GetValue() == "ruby"); + var filenamePrefixRule = ambiguousLookup["detection_rules"]!.AsArray() + .Single(rule => rule!["source"]!.GetValue() == "filename_prefix_pattern")!; + Assert.Equal(3, filenamePrefixRule["precedence"]!.GetValue()); + Assert.Contains( + filenamePrefixRule["patterns"]!.AsArray(), + rule => rule!["pattern"]!.GetValue() == "Makefile." + && rule["language"]!.GetValue() == "makefile"); + Assert.Equal( + LanguageMapOverrides.WorkspaceFileName, + ambiguousLookup["override_guidance"]!["config_file"]!.GetValue()); var emptyUnicodeLookup = CallIssue4896Languages(new JsonObject { @@ -57,6 +92,32 @@ public void Languages_ExactFiltersReportSeparateCatalogAndMatchCounts_Issue4896( Assert.Equal("complete", emptyUnicodeLookup["continuation_reason"]!.GetValue()); } + [Theory] + [InlineData("m-", ".m", "ambiguous_m", "objc", "matlab")] + [InlineData(".m_", ".m", "ambiguous_m", "objc", "matlab")] + [InlineData(".p.l", ".pl", "ambiguous_pl", "perl", "prolog")] + public void Languages_SeparatorNormalizedAmbiguousExtensionKeepsDiagnostics_Issue4901( + string extension, + string normalizedExtension, + string bucketLanguage, + string firstCandidate, + string secondCandidate) + { + var response = CallIssue4896Languages(new JsonObject + { + ["extension"] = extension, + }, id: 4901); + + var lookup = response["extension_lookup"]!; + Assert.Equal(normalizedExtension, lookup["normalized_extension"]!.GetValue()); + Assert.True(lookup["ambiguous"]!.GetValue()); + Assert.Equal(bucketLanguage, lookup["bucket_language"]!.GetValue()); + Assert.Equal( + [firstCandidate, secondCandidate], + lookup["candidates"]!.AsArray() + .Select(candidate => candidate!["lang"]!.GetValue())); + } + [Fact] public void Languages_AcceptsEveryCliCapabilityFilter_Issue4896() { diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs index 8735037ca..296a4ae4a 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs @@ -3521,6 +3521,156 @@ public void RunLanguages_JsonListsModernNodeModuleExtensions() Assert.Contains(".m", ambiguousM.GetProperty("extensions").EnumerateArray().Select(ext => ext.GetString())); } + [Fact] + public void RunLanguages_AmbiguousUppercaseExtensionExplainsCandidatesAndOverrides_Issue4901() + { + var (exitCode, stdout, stderr) = CaptureConsole(() => + QueryCommandRunner.RunLanguages(["--extension", ".M", "--json"], _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + Assert.Equal( + "ambiguous_m", + Assert.Single(document.RootElement.GetProperty("languages").EnumerateArray().ToArray()) + .GetProperty("lang") + .GetString()); + + var lookup = document.RootElement.GetProperty("extension_lookup"); + Assert.Equal(".M", lookup.GetProperty("extension").GetString()); + Assert.Equal(".m", lookup.GetProperty("normalized_extension").GetString()); + Assert.Equal("case_insensitive", lookup.GetProperty("extension_case_policy").GetString()); + Assert.True(lookup.GetProperty("ambiguous").GetBoolean()); + Assert.Equal("ambiguous_m", lookup.GetProperty("bucket_language").GetString()); + + var candidates = lookup.GetProperty("candidates").EnumerateArray().ToArray(); + Assert.Equal(["objc", "matlab"], candidates.Select(candidate => candidate.GetProperty("lang").GetString())); + Assert.Equal(["Objective-C", "MATLAB"], candidates.Select(candidate => candidate.GetProperty("display_name").GetString())); + Assert.Empty(candidates[0].GetProperty("aliases").EnumerateArray()); + Assert.Contains( + "octave", + candidates[1].GetProperty("aliases").EnumerateArray().Select(alias => alias.GetString())); + Assert.Contains( + "octave-cli", + candidates[1].GetProperty("evidence").GetProperty("shebang_interpreters") + .EnumerateArray() + .Select(interpreter => interpreter.GetString())); + Assert.Contains( + "*.xcodeproj", + candidates[0].GetProperty("evidence").GetProperty("project_markers") + .EnumerateArray() + .Select(marker => marker.GetString())); + Assert.Contains( + "classdef", + candidates[1].GetProperty("evidence").GetProperty("content_pattern").GetString()); + + var rules = lookup.GetProperty("detection_rules").EnumerateArray() + .ToDictionary(rule => rule.GetProperty("source").GetString()!, rule => rule); + Assert.Equal(2, rules["exact_filename"].GetProperty("precedence").GetInt32()); + Assert.Empty(rules["exact_filename"].GetProperty("applicable_patterns").EnumerateArray()); + Assert.Equal(3, rules["filename_prefix_pattern"].GetProperty("precedence").GetInt32()); + Assert.Contains( + rules["filename_prefix_pattern"].GetProperty("patterns").EnumerateArray(), + rule => rule.GetProperty("pattern").GetString() == "Makefile." + && rule.GetProperty("language").GetString() == "makefile"); + Assert.Equal(4, rules["shebang"].GetProperty("precedence").GetInt32()); + Assert.False(rules["shebang"].GetProperty("candidate_restricted").GetBoolean()); + Assert.Equal(256, rules["shebang"].GetProperty("probe_byte_limit").GetInt32()); + Assert.Equal( + "required_before_limit_unless_eof", + rules["shebang"].GetProperty("line_termination_policy").GetString()); + Assert.Equal( + "case_insensitive", + rules["shebang"].GetProperty("interpreter_case_policy").GetString()); + var shebangRules = rules["shebang"].GetProperty("interpreter_rules").EnumerateArray().ToArray(); + Assert.Contains( + shebangRules, + rule => rule.GetProperty("match").GetString() == "exact" + && rule.GetProperty("pattern").GetString() == "ruby" + && rule.GetProperty("language").GetString() == "ruby"); + Assert.Contains( + shebangRules, + rule => rule.GetProperty("match").GetString() == "prefix" + && rule.GetProperty("pattern").GetString() == "python" + && rule.GetProperty("language").GetString() == "python"); + Assert.Equal("high", rules["content"].GetProperty("confidence").GetString()); + Assert.Equal(64 * 1024, rules["content"].GetProperty("probe_byte_limit").GetInt32()); + Assert.Equal("medium", rules["project"].GetProperty("confidence").GetString()); + Assert.Equal("low", rules["ambiguous"].GetProperty("confidence").GetString()); + Assert.Equal( + "unsupported", + lookup.GetProperty("input_handling").GetProperty("binary_or_invalid_text").GetString()); + + var guidance = lookup.GetProperty("override_guidance"); + Assert.Equal(LanguageMapOverrides.WorkspaceFileName, guidance.GetProperty("config_file").GetString()); + Assert.Equal( + ["objc", "matlab"], + guidance.GetProperty("entries").EnumerateArray() + .Select(entry => entry.GetProperty("language").GetString())); + + var (combinedExitCode, combinedStdout, combinedStderr) = CaptureConsole(() => + QueryCommandRunner.RunLanguages( + ["--extension", ".m", "--language", "csharp", "--json"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, combinedExitCode); + Assert.Equal(string.Empty, combinedStderr); + using var combinedDocument = ParseJsonOutput(combinedStdout); + var combinedLookup = combinedDocument.RootElement.GetProperty("extension_lookup"); + Assert.Equal(1, combinedLookup.GetProperty("matched").GetInt32()); + Assert.Equal( + ["ambiguous_m"], + combinedLookup.GetProperty("languages").EnumerateArray() + .Select(language => language.GetString())); + + var (pagedExitCode, pagedStdout, pagedStderr) = CaptureConsole(() => + QueryCommandRunner.RunLanguages( + ["--extension", ".pl", "--language", "ada", "--limit", "1", "--json"], + _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, pagedExitCode); + Assert.Equal(string.Empty, pagedStderr); + using var pagedDocument = ParseJsonOutput(pagedStdout); + Assert.Equal( + "ada", + Assert.Single(pagedDocument.RootElement.GetProperty("languages").EnumerateArray().ToArray()) + .GetProperty("lang") + .GetString()); + var pagedLookup = pagedDocument.RootElement.GetProperty("extension_lookup"); + Assert.Equal(1, pagedLookup.GetProperty("matched").GetInt32()); + Assert.Equal( + ["ambiguous_pl"], + pagedLookup.GetProperty("languages").EnumerateArray() + .Select(language => language.GetString())); + } + + [Theory] + [InlineData("m-", ".m", "ambiguous_m", "objc", "matlab")] + [InlineData(".m_", ".m", "ambiguous_m", "objc", "matlab")] + [InlineData(".p.l", ".pl", "ambiguous_pl", "perl", "prolog")] + public void RunLanguages_SeparatorNormalizedAmbiguousExtensionKeepsDiagnostics_Issue4901( + string extension, + string normalizedExtension, + string bucketLanguage, + string firstCandidate, + string secondCandidate) + { + var (exitCode, stdout, stderr) = CaptureConsole(() => + QueryCommandRunner.RunLanguages(["--extension", extension, "--json"], _jsonOptions)); + + Assert.Equal(CommandExitCodes.Success, exitCode); + Assert.Equal(string.Empty, stderr); + using var document = ParseJsonOutput(stdout); + var lookup = document.RootElement.GetProperty("extension_lookup"); + Assert.Equal(normalizedExtension, lookup.GetProperty("normalized_extension").GetString()); + Assert.True(lookup.GetProperty("ambiguous").GetBoolean()); + Assert.Equal(bucketLanguage, lookup.GetProperty("bucket_language").GetString()); + Assert.Equal( + [firstCandidate, secondCandidate], + lookup.GetProperty("candidates").EnumerateArray() + .Select(candidate => candidate.GetProperty("lang").GetString())); + } + [Fact] public void RunLanguages_JsonReportsCythonAndCudaReferences_Issues4737And4738() {