diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index eb7e44263..c448e13b8 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -1639,7 +1639,7 @@ Process exit codes are coarse (`0` success including valid zero-row queries, `1` ## Design decisions - **Language capability patterns remain typed at the integration boundary** — CLI/MCP `languages` rows expose suffix-only `extensions`, literal `exact_filenames`, and ``-rendered `filename_prefix_patterns`. `legacy_patterns` preserves the former combined list during deprecation, and `pattern_provenance` identifies built-in, plugin/pattern, and language-map override ownership. Round-trip tests feed every advertised typed pattern back through `FileIndexer.DetectLanguage` (#4617). -- **Ambiguous source extensions stay explicit** — `.m` and `.pl` are not assigned to Objective-C and Perl by default. `FileIndexer` checks an authoritative recognized shebang, then a 64 KiB bounded prefix for strong mutually exclusive Objective-C/MATLAB or Perl/Prolog markers, then at most 256 entries per ancestor directory for conservative project markers. Conflicting or weak evidence is indexed as `ambiguous_m` / `ambiguous_pl`; MATLAB and Prolog have conservative symbol patterns but deliberately advertise no reference/graph support (#4612). +- **Ambiguous source extensions stay explicit** — `.m` and `.pl` are not assigned to Objective-C and Perl by default. `FileIndexer` checks an authoritative recognized shebang, then a 64 KiB bounded prefix for strong mutually exclusive Objective-C/MATLAB or Perl/Prolog markers, then at most 256 entries per ancestor directory for conservative project markers. Conflicting or weak evidence is indexed as `ambiguous_m` / `ambiguous_pl`; unresolved `.m` files run the bounded MATLAB and Objective-C symbol/reference paths after a shared position-preserving comment mask, while Prolog deliberately remains without reference/graph support (#4612, #4738). - **Hotspot marker fingerprints share one bounded tree traversal** — full/update CLI and MCP indexing compute C#, VB, F#, and MSBuild marker fingerprints together instead of walking the directory tree once per language. Each distinct marker glob retains the platform filesystem's matching behavior and is enumerated once per visited directory, while child directories are enumerated once; marker sets, budgets, truncation sentinels, and warning order remain isolated per language. The single-language API delegates to the same engine, preserving ignore rules, nested-repository/submodule boundaries, and MCP authorized-read failures. - **Lock-file dependency graphs model package relationships** — `packages.lock.json`, `package-lock.json`, and `npm-shrinkwrap.json` keep package declarations as symbols, but emit `dependency` references only for explicit parent-package to child-package entries. The parent package is stored as the reference container, so `callers` can identify which package requires a child and `deps` does not infer lock-file-to-lock-file similarity merely because two files contain the same resolved package set (#4409). - **Dependency-cycle audits separate analysis from display** — CLI `deps --cycles` and MCP `deps` with `cycles=true` analyze a deterministic, path-ordered edge set up to the independent `--graph-budget` / `graphBudget` before computing and stably ranking strongly connected components. `--limit` / `limit` only paginates that ranked SCC set, and opaque cursors are bound to the filters, graph budget, and indexed graph that produced them. Machine-readable responses expose `analysis_complete`, graph edge count/budget, stable ranking mode, authoritative total-cycle status, and continuation metadata; exhausting the graph budget is reported as an explicitly incomplete analysis rather than a complete cycle audit (#4731). @@ -4666,7 +4666,7 @@ USER_GUIDEの[終了コード](USER_GUIDE.md#終了コード)セクションを ## 設計判断 - **integration boundary では language capability pattern の型を維持** — CLI/MCP の `languages` 行は suffix のみの `extensions`、literal な `exact_filenames`、`` 表記の `filename_prefix_patterns` を公開します。`legacy_patterns` は deprecation 中に従来の combined list を保持し、`pattern_provenance` は built-in、plugin/pattern、language-map override の所有元を示します。round-trip test は広告した全 typed pattern を `FileIndexer.DetectLanguage` に戻して検証します（#4617）。 -- **曖昧な source extension は曖昧なまま明示** — `.m` と `.pl` を既定で Objective-C / Perl に割り当てません。`FileIndexer` は authoritative な認識済み shebang、64 KiB 上限 prefix 内の相互排他的で強い Objective-C/MATLAB または Perl/Prolog marker、各 ancestor directory 最大 256 entry の保守的な project marker の順に確認します。競合または弱い証拠は `ambiguous_m` / `ambiguous_pl` として index し、MATLAB / Prolog は保守的な symbol pattern を持つ一方、reference / graph 対応を意図的に広告しません（#4612）。 +- **曖昧な source extension は曖昧なまま明示** — `.m` と `.pl` を既定で Objective-C / Perl に割り当てません。`FileIndexer` は authoritative な認識済み shebang、64 KiB 上限 prefix 内の相互排他的で強い Objective-C/MATLAB または Perl/Prolog marker、各 ancestor directory 最大 256 entry の保守的な project marker の順に確認します。競合または弱い証拠は `ambiguous_m` / `ambiguous_pl` として index し、未確定の `.m` は位置を保つ共通コメントマスクの後で上限付きの MATLAB / Objective-C symbol・reference 経路を実行します。Prolog は意図的に reference / graph 未対応のままです（#4612、#4738）。 - **hotspot marker fingerprint は上限付きtree traversalを1回共有** — full/update CLIとMCP indexingは、directory treeを言語ごとに歩かず、C#、VB、F#、MSBuildのmarker fingerprintをまとめて計算します。各directoryでは固有marker globごとにplatform filesystemのmatching挙動を保って1回ずつ列挙し、child directoryも1回だけ列挙する一方、marker集合、budget、truncation sentinel、warning順は言語別に分離します。single-language APIも同じengineへ委譲し、ignore rule、nested repository/submodule境界、MCP authorized read failureを維持します。 - **lock file の依存グラフは package 間の関係をモデル化** — `packages.lock.json`、`package-lock.json`、`npm-shrinkwrap.json` は package 宣言を symbol として保持しますが、`dependency` reference は明示された親 package → 子 package の項目だけに出力します。親 package を reference container に保存するため、`callers` はどの package が子 package を必要としているかを特定でき、`deps` は同じ resolved package 集合を持つだけの lock file 同士を類似関係として推論しません（#4409）。 - **依存サイクル監査では解析と表示を分離** — CLI の `deps --cycles` と MCP `deps` の `cycles=true` は、独立した `--graph-budget` / `graphBudget` まで path 順で決定的な edge 集合を解析してから、強連結成分を安定順位付けします。`--limit` / `limit` はその SCC 順位集合をページ分割するだけで、不透明 cursor は生成時の filter、graph budget、indexed graph に結び付けます。machine-readable 応答は `analysis_complete`、graph edge 件数/予算、安定 ranking mode、authoritative な総 cycle 件数かどうか、continuation metadata を公開し、graph budget 枯渇時は完全な cycle 監査を装わず明示的な未完了解析として報告します（#4731）。 diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 7e8033b00..768d0316f 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -2381,14 +2381,14 @@ All indexed languages are searchable through FTS5. Rows with **Symbols = yes** a - C/C++ headers: `.h` stays on the C path unless lexical code (after comments, strings, and macro payloads are masked) has clear C++ markers such as `namespace`, `template`, `using`, `class`, or `std::`; those headers are promoted to `cpp` at index time. Detection scores the full header up to 48 KiB, then uses head/middle/tail ranges for larger files while retaining lexical state across skipped bytes, so long license blocks do not impose a fixed line cutoff. `index --dry-run --json` reports ambiguous-header decisions in `language_detections` with stable `source` and `confidence` values. - C++ callables: balanced declarators preserve constructors, destructors, conversion operators, ordinary functions, and trailing-return functions as navigable function symbols. Trailing return types populate `return_type` metadata. -- Cython and CUDA: Cython `cdef` / `cpdef` declarations, `cimport` entries, and extern declarations are indexed as symbols. CUDA files reuse C++ symbols and classify `__global__`, `__device__`, and `__host__` functions with CUDA-specific sub-kinds. +- Cython and CUDA: Cython `cdef` / `cpdef` declarations, `cimport` entries, and extern declarations are indexed as symbols; bounded cimport, base-type, and call references feed graph queries. CUDA files reuse C++ symbols and classify `__global__`, `__device__`, and `__host__` functions with CUDA-specific sub-kinds. - Shaders: GLSL, HLSL, Metal, and WGSL entry points, structs, type aliases, resource bindings, constant buffers, samplers, textures, and uniform/input/output declarations are indexed as symbols. - HDL: Verilog, SystemVerilog, and VHDL module/package/type/function/resource declarations are indexed as symbols, with bounded syntax-visible reference edges available to graph queries. - SQL: query-time `--lang tsql` is accepted as a SQL alias, and T-SQL aggregate, assembly, and XML schema collection declarations are searchable. - R: function assignments, S4/R6 class declarations, validity/generic/method declarations, inherit vectors, public/private/active methods, and `library` / `require` imports are indexed. - Functional graph languages: Clojure, Erlang, OCaml, and Raku expose conservative declarations plus bounded imports, aliases, calls, and type/protocol/behaviour relationships. References and graph queries are advertised for these languages. -- Dynamic symbol-only languages: Crystal, Groovy, Julia, and Tcl expose conservative declarations as symbols. References and graph queries are not advertised for these languages yet. -- Systems symbol-only languages: Ada, D, and Nim expose conservative declarations as symbols. References and graph queries are not advertised for these languages yet. +- Dynamic symbol-only languages: Crystal, Groovy, and Tcl expose conservative declarations as symbols. References and graph queries are not advertised for these languages yet. +- Scientific and native-extension graphs: Julia, MATLAB, Nim, D, Cython, and Ada emit bounded language-aware import/module, base/type, and call references. Julia macro invocations and Ada procedure-style calls without parentheses are also represented. - Markdown, JSON/YAML, and CSS: Markdown heading and local-anchor symbols are indexed; JSON/YAML configuration keys are indexed as structural key paths; CSS variables, placeholders, and `@extend` references are indexed. - Dockerfile, Assembly, Common Lisp, and Racket: `ARG` build args, labels/PROC/MACRO blocks, package/module forms, definitions, classes/structs, requires, and provides are surfaced as symbols where applicable. - Shell, PowerShell, and Batch: command-style function calls, functions/filters, classes/enums, imports, labels, `goto` / `call` targets, and inline control-flow forms are indexed where the language supports them. @@ -2401,7 +2401,7 @@ All indexed languages are searchable through FTS5. Rows with **Symbols = yes** a - Dependency manifests and lockfiles: use `--lang dependency_manifest` or `--lang dependency_lock` for dependency/security audits. `Directory.Packages.props`, `packages.config`, `requirements.txt`, `pyproject.toml`, `packages.lock.json`, and npm `package-lock.json` / `npm-shrinkwrap.json` expose package symbols and `dependency` references with version, scope, and direct/transitive metadata where the format provides it. - Solution and application manifests: `.sln` files expose project entries as symbols and project path references; `.manifest` files expose assembly identity, requested execution level, supported OS, and long-path settings as symbols. - Shebang scripts: recognized first-line shebangs index extensionless and unknown-extension files for shell (`sh`, `bash`, `zsh`, `fish`, `dash`, `ksh`, `ash`), Python, Ruby, Perl, Tcl (`tclsh`, `wish`), Node.js, PHP, Lua, and PowerShell. Explicit language-map overrides remain authoritative; for ambiguous `.t` files, a recognized shebang overrides the Perl default, while strong known extensions continue to win conflicts. -- Ambiguous `.m` / `.pl`: recognized shebangs win first, then bounded content checks use only strong Objective-C/MATLAB or Perl/Prolog markers, followed by conservative project markers. Scoped updates that add, change, or remove one of those markers automatically rescan the workspace so unchanged ambiguous files do not retain stale classifications. Weak or conflicting evidence remains searchable under `ambiguous_m` or `ambiguous_pl` instead of being assigned unconditionally. MATLAB and Prolog expose declaration symbols but no reference or graph support yet. +- Ambiguous `.m` / `.pl`: recognized shebangs win first, then bounded content checks use only strong Objective-C/MATLAB or Perl/Prolog markers, followed by conservative project markers. Scoped updates that add, change, or remove one of those markers automatically rescan the workspace so unchanged ambiguous files do not retain stale classifications. Weak or conflicting evidence remains searchable under `ambiguous_m` or `ambiguous_pl` instead of being assigned unconditionally. Unresolved `.m` content exposes the conservative union of MATLAB and Objective-C symbols/references after both comment syntaxes are position-preservingly masked; Prolog remains symbol-only. ### Language extraction matrix @@ -2428,8 +2428,8 @@ entries with the unsupported capability, an explanatory message, and | Java / Kotlin / Scala | packages/imports, classes/interfaces, methods, properties | calls, constructors, annotations, type references | Kotlin inline lambda body modeling is limited; verify with `references` before relying on deep call chains. | | JavaScript / TypeScript / Vue / Svelte | functions, classes, exports, imports, variables | calls, constructors, static/dynamic imports, workers, service workers | Dynamic property calls and computed module specifiers are best-effort. `cdidx references render --lang typescript` | | Python / Ruby / PHP / Perl / R | functions, classes/modules, imports where supported | calls, constructors, decorators/annotations where supported | Dynamic dispatch and metaprogramming may require `search`. PHPDoc/static import patterns are indexed when statically visible. | -| MATLAB / Prolog | classes/modules, functions/predicates, imports | none yet | `.m` / `.pl` are classified conservatively; declaration symbols are searchable, but use `search` for reference and graph questions. | -| Cython | `cdef` / `cpdef` declarations, cimports, extern declarations | none yet | Cython native-extension declarations are searchable as symbols; use `search` for call/reference questions. | +| MATLAB / Julia / Nim / D / Ada / Cython | classes/modules/types, functions/procedures, imports | bounded calls, imports/modules, base/type references; Julia macros and Ada procedure-style calls | Static syntax is indexed conservatively; dynamic dispatch, generated code, and macro expansion may still require `search`. | +| Prolog | modules, predicates, imports | none yet | `.pl` is classified conservatively; declaration symbols are searchable, but use `search` for reference and graph questions. | | C / C++ / Objective-C / Swift / Rust / Go / Zig | functions, types, methods, imports/modules | calls, constructors, macro invocations where supported, type references | C++ templates/macros and Rust macro expansion are not evaluated; Rust macro invocations are still reference edges. | | CUDA | C++-style functions/types plus CUDA kernel/device/host sub-kinds | calls and kernel launches, includes, workspace-backed user-defined type references, constant bindings, scoped kernel-parameter resource uses | CUDA references are bounded syntactic edges. Macro-generated launches, function pointers, and semantic data flow still require `search`. | | GLSL / HLSL / Metal / WGSL | entry points, structs, type aliases, resource bindings, constant buffers, samplers, textures, uniforms/inputs/outputs | entry-point/helper calls, includes where supported, workspace-backed user-defined type references, block/direct resource uses, binding metadata | Shader references are bounded syntactic edges. They do not validate binding compatibility or model semantic data flow; use `search` for those questions. | @@ -5549,14 +5549,14 @@ indexing はファイル単位の SQLite transaction を commit します。長 - C/C++ ヘッダー: `.h` は既定では C として扱います。コメント、文字列、マクロのペイロードをマスクした後の字句コードに `namespace`、`template`、`using`、`class`、`std::` などの明確な C++ マーカーがある場合だけ、index 時に `cpp` へ昇格します。48 KiB まではヘッダー全体、それを超える場合は評価対象外の byte をまたいで字句状態を保持しながら先頭・中央・末尾 range を評価するため、長いライセンスブロックが固定行数の打ち切りを引き起こしません。`index --dry-run --json` は曖昧なヘッダー判定を、安定した `source` と `confidence` を持つ `language_detections` として報告します。 - C++ callable: 括弧の対応を考慮した declarator 解析により、constructor、destructor、conversion operator、通常関数、後置戻り値関数を移動可能な function シンボルとして保持します。後置戻り値は `return_type` メタデータへ格納します。 -- Cython と CUDA: Cython の `cdef` / `cpdef` 宣言、`cimport`、extern 宣言をシンボルとして索引します。CUDA ファイルは C++ のシンボル抽出を再利用し、`__global__`、`__device__`、`__host__` 関数に CUDA 固有の sub-kind を付けます。 +- Cython と CUDA: Cython の `cdef` / `cpdef` 宣言、`cimport`、extern 宣言をシンボルとして索引し、上限付きの cimport・基底型・call 参照を graph query に提供します。CUDA ファイルは C++ のシンボル抽出を再利用し、`__global__`、`__device__`、`__host__` 関数に CUDA 固有の sub-kind を付けます。 - Shaders: GLSL、HLSL、Metal、WGSL の entry point、struct、type alias、resource binding、constant buffer、sampler、texture、uniform/input/output 宣言をシンボルとして索引します。 - HDL: Verilog、SystemVerilog、VHDL の module / package / type / function / resource 宣言をシンボルとして索引し、構文上確認できる上限付き reference edge を graph query で利用できます。 - SQL: クエリ時の `--lang tsql` は SQL の別名です。T-SQL の aggregate、assembly、XML schema collection 宣言も検索対象です。 - R: 関数代入、S4/R6 class 宣言、validity/generic/method 宣言、inherit vector、public/private/active method、`library` / `require` import を索引します。 - 関数型言語のグラフ対応: Clojure、Erlang、OCaml、Raku は保守的な宣言に加え、上限付きの import、alias、call、type / protocol / behaviour 関係を公開します。これらの言語では references と graph queries を対応済みとして広告します。 -- 動的言語のシンボル専用対応: Crystal、Groovy、Julia、Tcl は保守的な宣言をシンボルとして公開します。これらの言語では references と graph queries はまだ対応として広告しません。 -- システム系言語のシンボル専用対応: Ada、D、Nim は保守的な宣言をシンボルとして公開します。これらの言語では references と graph queries はまだ対応として広告しません。 +- 動的言語のシンボル専用対応: Crystal、Groovy、Tcl は保守的な宣言をシンボルとして公開します。これらの言語では references と graph queries はまだ対応として広告しません。 +- 科学技術・ネイティブ拡張言語のグラフ: Julia、MATLAB、Nim、D、Cython、Ada は、言語構文に応じた import/module、基底型/type、call 参照を上限付きで出力します。Julia の macro invocation と、括弧を伴わない Ada の procedure call も記録します。 - Markdown、JSON/YAML、CSS: Markdown の heading / local anchor、JSON/YAML の configuration key path、CSS の variable、placeholder、`@extend` をシンボルとして扱います。 - Dockerfile、Assembly、Common Lisp、Racket: `ARG` build arg、label、PROC/MACRO、package/module form、definition、class/struct、require/provide を必要に応じて表面化します。 - Shell、PowerShell、Batch: command-style function call、function/filter、class/enum、import、label、`goto` / `call` target、inline control-flow を言語仕様に合わせて索引します。 @@ -5569,7 +5569,7 @@ indexing はファイル単位の SQLite transaction を commit します。長 - Dependency manifest / lockfile: dependency / security audit では `--lang dependency_manifest` または `--lang dependency_lock` を使います。`Directory.Packages.props`、`packages.config`、`requirements.txt`、`pyproject.toml`、`packages.lock.json`、npm の `package-lock.json` / `npm-shrinkwrap.json` は、format が提供する範囲で version、scope、direct/transitive metadata を持つ package symbol と `dependency` reference を公開します。 - ソリューションとアプリケーションマニフェスト: `.sln` は project entry をシンボルとして公開し、project path を参照として記録します。`.manifest` は assembly identity、requested execution level、supported OS、long-path 設定をシンボルとして公開します。 - shebang script: 先頭行の shebang を認識できる拡張子なし/未知拡張子ファイルは、shell (`sh`, `bash`, `zsh`, `fish`, `dash`, `ksh`, `ash`)、Python、Ruby、Perl、Tcl (`tclsh`, `wish`)、Node.js、PHP、Lua、PowerShell として index 対象です。明示的な language-map override は常に優先し、曖昧な `.t` では認識済み shebang が Perl の既定値を上書きします。一方、曖昧でない既知拡張子は競合する shebang より優先されます。 -- 曖昧な `.m` / `.pl`: 認識済み shebang を最優先し、その後は bounded content check で Objective-C/MATLAB または Perl/Prolog の強い marker だけを使い、最後に保守的な project marker を確認します。これらの marker を追加・変更・削除する scoped update は workspace を自動的に再 scan し、未変更の曖昧ファイルに古い分類を残しません。弱い証拠や競合する証拠は無条件に言語を割り当てず、`ambiguous_m` / `ambiguous_pl` として全文検索可能なまま残します。MATLAB と Prolog は宣言 symbol を公開しますが、reference / graph はまだ未対応です。 +- 曖昧な `.m` / `.pl`: 認識済み shebang を最優先し、その後は bounded content check で Objective-C/MATLAB または Perl/Prolog の強い marker だけを使い、最後に保守的な project marker を確認します。これらの marker を追加・変更・削除する scoped update は workspace を自動的に再 scan し、未変更の曖昧ファイルに古い分類を残しません。弱い証拠や競合する証拠は無条件に言語を割り当てず、`ambiguous_m` / `ambiguous_pl` として全文検索可能なまま残します。未確定の `.m` は両方のコメント構文を位置を保ってマスクした後、MATLAB と Objective-C の symbol/reference を保守的に統合します。Prolog は引き続き symbol-only です。 ### 言語別 extraction matrix @@ -5590,8 +5590,8 @@ indexing はファイル単位の SQLite transaction を commit します。長 | Java / Kotlin / Scala | package/import、class/interface、method、property | call、constructor、annotation、type reference | Kotlin inline lambda body の modeling は限定的です。深い call chain を信頼する前に `references` で確認してください。 | | JavaScript / TypeScript / Vue / Svelte | function、class、export、import、variable | call、constructor、static/dynamic import、worker、service worker | dynamic property call と computed module specifier は best-effort です。`cdidx references render --lang typescript` | | Python / Ruby / PHP / Perl / R | function、class/module、対応言語の import | call、constructor、対応言語の decorator/annotation | dynamic dispatch と metaprogramming は `search` が必要な場合があります。PHPDoc/static import pattern は静的に見える範囲で索引されます。 | -| MATLAB / Prolog | class/module、function/predicate、import | まだなし | `.m` / `.pl` は保守的に分類され、宣言 symbol は検索できます。reference / graph の調査には `search` を使ってください。 | -| Cython | `cdef` / `cpdef` 宣言、cimport、extern 宣言 | まだなし | Cython の native extension 宣言はシンボルとして検索できます。call/reference の調査には `search` を使ってください。 | +| MATLAB / Julia / Nim / D / Ada / Cython | class/module/type、function/procedure、import | 上限付きの call、import/module、基底型/type reference。Julia macro と Ada の procedure-style call | 静的な構文を保守的に索引します。dynamic dispatch、generated code、macro expansion には `search` が必要な場合があります。 | +| Prolog | module、predicate、import | まだなし | `.pl` は保守的に分類され、宣言 symbol は検索できます。reference / graph の調査には `search` を使ってください。 | | C / C++ / Objective-C / Swift / Rust / Go / Zig | function、type、method、import/module | call、constructor、対応言語の macro invocation、type reference | C++ template/macro と Rust macro expansion は評価しません。Rust macro invocation 自体は reference edge です。 | | CUDA | C++ 風の function/type と CUDA kernel/device/host sub-kind | call と kernel launch、include、workspace に基づくユーザー定義型参照、constant binding、scope 付き kernel parameter の resource 利用 | CUDA の参照は上限付きの構文エッジです。macro 生成 launch、function pointer、意味的 data flow には引き続き `search` を使ってください。 | | GLSL / HLSL / Metal / WGSL | entry point、struct、type alias、resource binding、constant buffer、sampler、texture、uniform/input/output | entry point/helper の call、対応言語の include、workspace に基づくユーザー定義型参照、block / direct resource 利用、binding metadata | Shader の参照は上限付きの構文エッジです。binding compatibility の検証や意味的 data flow の modeling は行わないため、それらには `search` を使ってください。 | diff --git a/changelog.d/unreleased/4738.added.md b/changelog.d/unreleased/4738.added.md new file mode 100644 index 000000000..87822929d --- /dev/null +++ b/changelog.d/unreleased/4738.added.md @@ -0,0 +1,37 @@ +--- +category: added +issues: + - 4738 +affected: + - src/CodeIndex/Database/DbContext.cs + - src/CodeIndex/Database/DbReader.GraphQueries.cs + - src/CodeIndex/Database/DbReader.References.cs + - src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs + - src/CodeIndex/Database/DbWriter.References.cs + - src/CodeIndex/Indexer/AmbiguousMContentMasker.cs + - src/CodeIndex/Indexer/References/Languages/ScientificNativeReferenceExtractor.cs + - src/CodeIndex/Indexer/References/Languages/ScientificNativeCommentMasker.cs + - src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs + - src/CodeIndex/Indexer/References/ReferenceExtractor.Preparation.cs + - src/CodeIndex/Indexer/References/ReferenceExtractor.State.cs + - src/CodeIndex/Indexer/References/ReferenceExtractor.TypeReferences.cs + - src/CodeIndex/Indexer/References/ReferenceExtractor.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.ScientificNative.cs + - src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs + - tests/CodeIndex.Tests/QueryCommandRunnerTests.cs + - tests/CodeIndex.Tests/DatabaseTests.cs + - tests/CodeIndex.Tests/ReferenceExtractorTests.ScientificNative.cs + - tests/CodeIndex.Tests/SymbolExtractorTests.cs + - tests/CodeIndex.Tests/golden/status.json + - USER_GUIDE.md + - DEVELOPER_GUIDE.md +--- + +## English + +- **Scientific and native-extension languages now expose reference graphs (#4738)** — Nim, MATLAB, Julia, D, Cython, and Ada emit bounded language-aware call, import/module, and type/base references. Ambiguous `.m` files preserve content-based classification, expose a conservative MATLAB/Objective-C symbol and reference union when evidence remains unresolved, and resolve those references against definitively classified MATLAB or Objective-C files. + +## 日本語 + +- **科学技術・ネイティブ拡張言語が reference graph に対応しました (#4738)** — Nim、MATLAB、Julia、D、Cython、Ada は、言語構文に応じた call、import/module、type/base reference を上限付きで出力します。曖昧な `.m` は content-based classification を維持し、証拠が未確定のときは MATLAB / Objective-C の symbol・reference を保守的に統合して公開し、確定済みの MATLAB / Objective-C ファイルにある参照先へ解決します。 diff --git a/src/CodeIndex/Database/DbContext.cs b/src/CodeIndex/Database/DbContext.cs index c771f275f..51fc2a6e4 100644 --- a/src/CodeIndex/Database/DbContext.cs +++ b/src/CodeIndex/Database/DbContext.cs @@ -2056,7 +2056,7 @@ public static string BuildIncompleteHotspotFamilyMarkerFingerprint(string? finge public const string SqlGraphContractVersionMetaKey = "sql_graph_contract_version"; public const int HdlGraphContractVersion = 1; public const string HdlGraphContractVersionMetaKey = "hdl_graph_contract_version"; - public const int ReferenceIdentityContractVersion = 1; + public const int ReferenceIdentityContractVersion = 2; public const string ReferenceIdentityContractVersionMetaKey = "reference_identity_contract_version"; public const string SymbolsOnlyGraphOmittedMetaKey = "symbols_only_graph_omitted"; public const string IndexedProjectRootMetaKey = "indexed_project_root"; diff --git a/src/CodeIndex/Database/DbReader.GraphQueries.cs b/src/CodeIndex/Database/DbReader.GraphQueries.cs index 2a80eb176..3d382ccaf 100644 --- a/src/CodeIndex/Database/DbReader.GraphQueries.cs +++ b/src/CodeIndex/Database/DbReader.GraphQueries.cs @@ -126,7 +126,7 @@ FROM symbol_reference_candidates AS identity_candidate var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false); var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - sql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name_folded = @query) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; + sql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; } else if (useSqlQualifiedContextMatch && exact) { @@ -152,9 +152,9 @@ FROM symbol_reference_candidates AS identity_candidate else if (exact && _foldReady) sql += allowSqlLeafFallback ? cssScssVariableAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : " AND (r.symbol_name_folded = @query OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : " AND r.symbol_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" + : $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; else if (exact) sql += allowSqlLeafFallback ? cssScssVariableAlias != null @@ -166,7 +166,11 @@ FROM symbol_reference_candidates AS identity_candidate ? $" AND (r.symbol_name LIKE @query ESCAPE '\\' OR (r.symbol_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" : " AND (r.symbol_name LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))"; if (lang != null) - sql += " AND f.lang = @lang"; + { + sql += IncludeAmbiguousMSourceForIdentityTarget(lang, targetSymbolId) + ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" + : " AND f.lang = @lang"; + } sql += BuildCSharpBareMemberGraphReferenceFilter(query, lang, exact, contextSql, "f", "r"); AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); sql += @" @@ -191,10 +195,13 @@ FROM logical_references r if (!exact) callersQueryParam = $"%{EscapeLikeQuery(query)}%"; else if (_foldReady) - callersQueryParam = NameFold.Fold(query) ?? query; + callersQueryParam = FoldNameForLanguage(query, lang); else callersQueryParam = query; - SqliteCommandPolicy.Add(cmd, "@query", callersQueryParam); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", callersQueryParam); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); @@ -285,7 +292,7 @@ FROM symbol_references r var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false); var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name_folded = @query) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; + groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; } else if (useSqlQualifiedContextMatch && exact) { @@ -311,9 +318,9 @@ FROM symbol_references r else if (exact && _foldReady) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : " AND (r.symbol_name_folded = @query OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : " AND r.symbol_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" + : $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; else if (exact) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null @@ -335,9 +342,12 @@ FROM symbol_references r var value = !exact ? $"%{EscapeLikeQuery(query)}%" : _foldReady - ? NameFold.Fold(query) ?? query + ? FoldNameForLanguage(query, lang) : query; - SqliteCommandPolicy.Add(cmd, "@query", value); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", value); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); @@ -392,7 +402,7 @@ FROM symbol_references r var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false); var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name_folded = @query) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; + groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; } else if (useSqlQualifiedContextMatch && exact) { @@ -418,9 +428,9 @@ FROM symbol_references r else if (exact && _foldReady) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : " AND (r.symbol_name_folded = @query OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : " AND r.symbol_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" + : $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; else if (exact) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null @@ -442,9 +452,12 @@ FROM symbol_references r var value = !exact ? $"%{EscapeLikeQuery(query)}%" : _foldReady - ? NameFold.Fold(query) ?? query + ? FoldNameForLanguage(query, lang) : query; - SqliteCommandPolicy.Add(cmd, "@query", value); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", value); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); @@ -515,7 +528,7 @@ WHERE r.container_name IS NOT NULL if (exact && useSqlQualifiedContainerMatch && _foldReady) { var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - sql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND r.container_name_folded = @query) OR {qualifiedLeafFallbackSql})"; + sql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}) OR {qualifiedLeafFallbackSql})"; } else if (exact && useSqlQualifiedContainerMatch) { @@ -535,9 +548,9 @@ WHERE r.container_name IS NOT NULL else if (exact && _foldReady) sql += allowSqlLeafFallback ? cssScssVariableAlias != null - ? $" AND (r.container_name_folded = @query OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : " AND (r.container_name_folded = @query OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : " AND r.container_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" + : $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}"; else if (exact) sql += allowSqlLeafFallback ? cssScssVariableAlias != null @@ -568,10 +581,13 @@ FROM logical_references r if (!exact) calleesQueryParam = $"%{EscapeLikeQuery(query)}%"; else if (_foldReady) - calleesQueryParam = NameFold.Fold(query) ?? query; + calleesQueryParam = FoldNameForLanguage(query, lang); else calleesQueryParam = query; - SqliteCommandPolicy.Add(cmd, "@query", calleesQueryParam); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", calleesQueryParam); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalized", SqlNameResolver.NormalizeQualifiedName(query)); @@ -657,7 +673,7 @@ FROM symbol_references r if (exact && useSqlQualifiedContainerMatch && _foldReady) { var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND r.container_name_folded = @query) OR {qualifiedLeafFallbackSql})"; + groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}) OR {qualifiedLeafFallbackSql})"; } else if (exact && useSqlQualifiedContainerMatch) { @@ -677,9 +693,9 @@ FROM symbol_references r else if (exact && _foldReady) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null - ? $" AND (r.container_name_folded = @query OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : " AND (r.container_name_folded = @query OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : " AND r.container_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" + : $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}"; else if (exact) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null @@ -700,9 +716,12 @@ FROM symbol_references r var value = !exact ? $"%{EscapeLikeQuery(query)}%" : _foldReady - ? NameFold.Fold(query) ?? query + ? FoldNameForLanguage(query, lang) : query; - SqliteCommandPolicy.Add(cmd, "@query", value); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", value); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalized", SqlNameResolver.NormalizeQualifiedName(query)); @@ -756,7 +775,7 @@ FROM symbol_references r if (exact && useSqlQualifiedContainerMatch && _foldReady) { var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND r.container_name_folded = @query) OR {qualifiedLeafFallbackSql})"; + groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}) OR {qualifiedLeafFallbackSql})"; } else if (exact && useSqlQualifiedContainerMatch) { @@ -776,9 +795,9 @@ FROM symbol_references r else if (exact && _foldReady) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null - ? $" AND (r.container_name_folded = @query OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : " AND (r.container_name_folded = @query OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : " AND r.container_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" + : $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}"; else if (exact) groupedSql += allowSqlLeafFallback ? cssScssVariableAlias != null @@ -799,9 +818,12 @@ FROM symbol_references r var value = !exact ? $"%{EscapeLikeQuery(query)}%" : _foldReady - ? NameFold.Fold(query) ?? query + ? FoldNameForLanguage(query, lang) : query; - SqliteCommandPolicy.Add(cmd, "@query", value); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", value); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalized", SqlNameResolver.NormalizeQualifiedName(query)); @@ -932,13 +954,13 @@ ELSE 5 /// SQL 側で要求された LIMIT/OFFSET を適用し、呼び出し側が要求以上の中間ページを /// materialize しないようにする。 /// - private List GetCallersExact(string symbolName, int limit, int offset = 0, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false) - => GetCallersExactCore(symbolName, limit, offset, lang, pathPatterns, excludePathPatterns, excludeTests, targetSymbolId: null); + private List GetCallersExact(string symbolName, int limit, int offset = 0, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool includeAmbiguousMSource = false) + => GetCallersExactCore(symbolName, limit, offset, lang, pathPatterns, excludePathPatterns, excludeTests, targetSymbolId: null, includeAmbiguousMSource); - private List GetCallersExactForTarget(string symbolName, long targetSymbolId, int limit, int offset, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests) - => GetCallersExactCore(symbolName, limit, offset, lang, pathPatterns, excludePathPatterns, excludeTests, targetSymbolId); + private List GetCallersExactForTarget(string symbolName, long targetSymbolId, int limit, int offset, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, bool includeAmbiguousMSource = false) + => GetCallersExactCore(symbolName, limit, offset, lang, pathPatterns, excludePathPatterns, excludeTests, targetSymbolId, includeAmbiguousMSource); - private List GetCallersExactCore(string symbolName, int limit, int offset, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, long? targetSymbolId) + private List GetCallersExactCore(string symbolName, int limit, int offset, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, long? targetSymbolId, bool includeAmbiguousMSource) { if (!_hasReferencesTable) return new List(); using var cmd = _conn.CreateCommand(); @@ -970,9 +992,9 @@ private List GetCallersExactCore(string symbolName, int limit, int var nameCondition = _foldReady ? allowSqlLeafFallback ? @" - AND (r.symbol_name_folded = @symbolNameFolded OR (f.lang = 'sql' AND r.symbol_name_folded = @symbolNameLeafFolded)" + polymorphicNameCondition + " OR (f.lang = 'solution' AND r.reference_kind = 'project_reference' AND r.container_name = @symbolName COLLATE NOCASE))" + AND (" + BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@symbolNameFolded") + " OR (f.lang = 'sql' AND r.symbol_name_folded = @symbolNameLeafFolded)" + polymorphicNameCondition + " OR (f.lang = 'solution' AND r.reference_kind = 'project_reference' AND r.container_name = @symbolName COLLATE NOCASE))" : @" - AND (((f.lang = 'sql') AND sql_context_has_name_folded_at(" + contextSql + @", @symbolName, r.column_number) = 1) OR ((f.lang != 'sql') AND r.symbol_name_folded = @symbolNameFolded) OR " + BuildCSharpQualifiedContextFallbackSql(BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false)) + " OR " + BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true) + polymorphicNameCondition + " OR (f.lang = 'solution' AND r.reference_kind = 'project_reference' AND r.container_name = @symbolName COLLATE NOCASE))" + AND (((f.lang = 'sql') AND sql_context_has_name_folded_at(" + contextSql + @", @symbolName, r.column_number) = 1) OR ((f.lang != 'sql') AND " + BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@symbolNameFolded") + ") OR " + BuildCSharpQualifiedContextFallbackSql(BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false)) + " OR " + BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true) + polymorphicNameCondition + " OR (f.lang = 'solution' AND r.reference_kind = 'project_reference' AND r.container_name = @symbolName COLLATE NOCASE))" : allowSqlLeafFallback ? @" AND (r.symbol_name = @symbolName COLLATE NOCASE OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@symbolName) COLLATE NOCASE)" + polymorphicNameCondition + " OR (f.lang = 'solution' AND r.reference_kind = 'project_reference' AND r.container_name = @symbolName COLLATE NOCASE))" @@ -1010,7 +1032,11 @@ AND r.reference_kind IN {CallGraphReferenceKindsSql} AND {supportedLangFilter} {targetCondition}"; if (lang != null) - sql += " AND f.lang = @lang"; + { + sql += includeAmbiguousMSource + ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" + : " AND f.lang = @lang"; + } sql += BuildCSharpBareMemberGraphReferenceFilter(symbolName, lang, exact: true, contextSql, "f", "r"); AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); sql += @" @@ -1034,7 +1060,7 @@ FROM logical_references r SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(symbolName)) ?? SqlNameResolver.GetLeafName(symbolName)); SqliteCommandPolicy.Add(cmd, "@symbolNameLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(symbolName)) ?? SqlNameResolver.GetLeafName(symbolName)); if (_foldReady) - SqliteCommandPolicy.Add(cmd, "@symbolNameFolded", NameFold.Fold(symbolName) ?? symbolName); + AddPersistedFoldedNameQueryParameters(cmd, "@symbolNameFolded", symbolName, lang); for (var i = 0; i < polymorphicCSharpSymbolNames.Count; i++) { if (_foldReady) @@ -1138,6 +1164,14 @@ private static string BuildImpactVisitedKey(CallerResult caller, string callerNa && rootDefinitions[0].Lang == "csharp" ? rootDefinitions[0].SymbolId : null; + var ambiguousMRootSymbolId = hasResolvedIdentityGraph + && rootDefinitions.Count == 1 + && lang is "matlab" or "objc" + && string.Equals(rootDefinitions[0].Lang, lang, StringComparison.Ordinal) + ? rootDefinitions[0].SymbolId + : null; + var identityRootSymbolId = qualifiedRootSymbolId ?? ambiguousMRootSymbolId; + var includeAmbiguousMSource = ambiguousMRootSymbolId != null; var results = new List(); resultOffset = Math.Max(0, resultOffset); @@ -1200,9 +1234,9 @@ private static string BuildImpactVisitedKey(CallerResult caller, string callerNa while (discoveredResultCount < resultWindowEnd && fetchIterations < maxFetchIterations && !graphStateBudgetHit && !boundaryProbeBudgetHit) { fetchIterations++; - var page = depth == 0 && qualifiedRootSymbolId is long targetSymbolId - ? GetCallersExactForTarget(currentSymbol, targetSymbolId, pageSize, pageOffset, lang, pathPatterns, excludePathPatterns, excludeTests) - : GetCallersExact(currentSymbol, pageSize, pageOffset, lang, pathPatterns, excludePathPatterns, excludeTests); + var page = depth == 0 && identityRootSymbolId is long targetSymbolId + ? GetCallersExactForTarget(currentSymbol, targetSymbolId, pageSize, pageOffset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource) + : GetCallersExact(currentSymbol, pageSize, pageOffset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource); if (page.Count == 0) break; // No more callers for this symbol / このシンボルの caller は尽きた @@ -1349,7 +1383,8 @@ private static string BuildImpactVisitedKey(CallerResult caller, string callerNa lang, pathPatterns, excludePathPatterns, - excludeTests); + excludeTests, + includeAmbiguousMSource); maxDepthReached |= boundaryInspection.HasUnvisitedCaller; if (boundaryInspection.ProbeBudgetHit) { @@ -1448,7 +1483,8 @@ private ImpactBoundaryInspection InspectBoundaryCallers( string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, - bool excludeTests) + bool excludeTests, + bool includeAmbiguousMSource) { var offset = 0; var probes = 0; @@ -1458,7 +1494,7 @@ private ImpactBoundaryInspection InspectBoundaryCallers( return new ImpactBoundaryInspection(HasUnvisitedCaller: true, ProbeBudgetHit: true); var pageSize = Math.Min(ImpactBoundaryCallerProbePageSize, ImpactBoundaryCallerProbeBudget - probes); - var page = GetCallersExact(symbolName, pageSize, offset, lang, pathPatterns, excludePathPatterns, excludeTests); + var page = GetCallersExact(symbolName, pageSize, offset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource); if (page.Count == 0) return new ImpactBoundaryInspection(HasUnvisitedCaller: false, ProbeBudgetHit: false); probes += page.Count; @@ -2033,8 +2069,8 @@ private ImpactDefinitionResolution ResolveImpactDefinitions( GetSymbolColumnSql("family_key")); var nameCondition = _foldReady ? allowLeafFallback - ? "(s.name_folded = @resolvedNameFolded OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded) OR sql_leaf_name_folded(s.name) = @resolvedNameLeafFolded)))" - : "(s.name_folded = @resolvedNameFolded OR (f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded))" + ? $"({BuildPersistedFoldedNameMatchSql("s.name_folded", "@resolvedNameFolded")} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded) OR sql_leaf_name_folded(s.name) = @resolvedNameLeafFolded)))" + : $"({BuildPersistedFoldedNameMatchSql("s.name_folded", "@resolvedNameFolded")} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded))" : allowLeafFallback ? "(s.name = @resolvedName COLLATE NOCASE OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @resolvedNameLeaf COLLATE NOCASE)))" : "(s.name = @resolvedName COLLATE NOCASE OR (f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized COLLATE NOCASE))"; @@ -2159,9 +2195,9 @@ CROSS JOIN definition_stats stats cmd.CommandText = sql; SqliteCommandPolicy.Add(cmd, "@resolvedName", resolvedName); SqliteCommandPolicy.Add(cmd, "@resolvedNameNormalized", normalizedName); - SqliteCommandPolicy.Add(cmd, "@resolvedNameNormalizedFolded", NameFold.Fold(normalizedName) ?? normalizedName); + SqliteCommandPolicy.Add(cmd, "@resolvedNameNormalizedFolded", FoldNameForLanguage(normalizedName, lang)); SqliteCommandPolicy.Add(cmd, "@resolvedNameLeaf", leafName); - SqliteCommandPolicy.Add(cmd, "@resolvedNameLeafFolded", NameFold.Fold(leafName) ?? leafName); + SqliteCommandPolicy.Add(cmd, "@resolvedNameLeafFolded", FoldNameForLanguage(leafName, lang)); SqliteCommandPolicy.Add(cmd, "@resolvedNameSegmentCount", segmentCount); SqliteCommandPolicy.Add(cmd, "@allowLeafFallback", allowLeafFallback ? 1 : 0); if (SqlNameResolver.HasQualifier(resolvedName)) @@ -2171,7 +2207,7 @@ CROSS JOIN definition_stats stats SqliteCommandPolicy.Add(cmd, "@resolvedNameContainerSuffix", $"%.{EscapeLikeQuery(container)}"); } if (_foldReady) - SqliteCommandPolicy.Add(cmd, "@resolvedNameFolded", NameFold.Fold(resolvedName) ?? resolvedName); + AddPersistedFoldedNameQueryParameters(cmd, "@resolvedNameFolded", resolvedName, lang); if (lang != null) SqliteCommandPolicy.Add(cmd, "@lang", lang); SqliteCommandPolicy.Add(cmd, "@definitionLimit", Math.Max(1, representativeLimit)); diff --git a/src/CodeIndex/Database/DbReader.References.cs b/src/CodeIndex/Database/DbReader.References.cs index b03786505..58f06294d 100644 --- a/src/CodeIndex/Database/DbReader.References.cs +++ b/src/CodeIndex/Database/DbReader.References.cs @@ -8,6 +8,9 @@ public partial class DbReader private const int CSharpUsingStaticReferenceFilterChunkSize = 64; private const int CSharpUsingStaticReferenceFilterMaxRawLimit = 65536; private sealed record SearchReferenceRawRow(string Path, string? Lang, string SymbolName, string ReferenceKind, int Line, int Column, string Context, string? ContainerKind, string? ContainerName, bool IsSelfReference, bool IsMutualRecursion, long? TargetSymbolId, string? TargetSymbolKey, string? ResolutionState, int ResolutionCandidateCount); + private static bool IncludeAmbiguousMSourceForIdentityTarget(string? language, long? targetSymbolId) => + targetSymbolId != null && language is "matlab" or "objc"; + internal sealed record ReferencePositionCandidate(SymbolResult Definition, bool Authoritative); internal sealed record ReferencePositionResolution( bool IdentityAvailable, @@ -311,8 +314,8 @@ FROM symbol_references r var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); sql += referencesSuffixAlias != null - ? $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryAttributeAlias{referencesAliasScope}))) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})" - : $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name_folded = @query) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; + ? $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryAttributeAlias{referencesAliasScope}))) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})" + : $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; } else if (useSqlQualifiedContextMatch && exact) { @@ -343,12 +346,12 @@ FROM symbol_references r } else if (exact && _foldReady) sql += referencesSuffixAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryAttributeAlias{referencesAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafReferenceScope})" : string.Empty)})" + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryAttributeAlias{referencesAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafReferenceScope})" : string.Empty)})" : referencesCssScssVariableAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryCssScssVariableAlias{referencesCssScssVariableAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafReferenceScope})" : string.Empty)})" + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{referencesCssScssVariableAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafReferenceScope})" : string.Empty)})" : allowSqlLeafFallback - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafReferenceScope}))" - : " AND r.symbol_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafReferenceScope}))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; else if (exact) sql += referencesSuffixAlias != null ? $" AND (r.symbol_name = @query COLLATE NOCASE OR (r.symbol_name = @queryAttributeAlias COLLATE NOCASE{referencesAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE{sqlLeafReferenceScope})" : string.Empty)})" @@ -379,7 +382,11 @@ FROM symbol_reference_candidates AS identity_candidate )"; } if (lang != null) - sql += " AND f.lang = @lang"; + { + sql += IncludeAmbiguousMSourceForIdentityTarget(lang, targetSymbolId) + ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" + : " AND f.lang = @lang"; + } AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); if (referenceKind == null) { @@ -401,10 +408,13 @@ FROM symbol_reference_candidates AS identity_candidate if (!exact) queryParam = $"%{EscapeLikeQuery(query)}%"; else if (_foldReady) - queryParam = NameFold.Fold(query) ?? query; + queryParam = FoldNameForLanguage(query, lang); else queryParam = query; - SqliteCommandPolicy.Add(cmd, "@query", queryParam); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", queryParam); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); @@ -715,8 +725,8 @@ FROM symbol_references r var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); innerSql += countSuffixAlias != null - ? $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryAttributeAlias{countAliasScope}))) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})" - : $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name_folded = @query) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; + ? $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryAttributeAlias{countAliasScope}))) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})" + : $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; } else if (useSqlQualifiedContextMatch && exact) { @@ -747,10 +757,10 @@ FROM symbol_references r } else if (exact && _foldReady) innerSql += countSuffixAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryAttributeAlias{countAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafCountScope})" : string.Empty)})" + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryAttributeAlias{countAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafCountScope})" : string.Empty)})" : allowSqlLeafFallback - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafCountScope}))" - : " AND r.symbol_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafCountScope}))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; else if (exact) innerSql += countSuffixAlias != null ? $" AND (r.symbol_name = @query COLLATE NOCASE OR (r.symbol_name = @queryAttributeAlias COLLATE NOCASE{countAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE{sqlLeafCountScope})" : string.Empty)})" @@ -777,9 +787,12 @@ FROM symbol_references r var value = !exact ? $"%{EscapeLikeQuery(query)}%" : _foldReady - ? NameFold.Fold(query) ?? query + ? FoldNameForLanguage(query, lang) : query; - SqliteCommandPolicy.Add(cmd, "@query", value); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", value); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); @@ -848,8 +861,8 @@ FROM symbol_references r var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); innerSql += totalSuffixAlias != null - ? $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryAttributeAlias{totalAliasScope}))) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})" - : $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name_folded = @query) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; + ? $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryAttributeAlias{totalAliasScope}))) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})" + : $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; } else if (useSqlQualifiedContextMatch && exact) { @@ -880,12 +893,12 @@ FROM symbol_references r } else if (exact && _foldReady) innerSql += totalSuffixAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryAttributeAlias{totalAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafTotalScope})" : string.Empty)})" + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryAttributeAlias{totalAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafTotalScope})" : string.Empty)})" : totalCssScssVariableAlias != null - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @queryCssScssVariableAlias{totalCssScssVariableAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafTotalScope})" : string.Empty)})" + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{totalCssScssVariableAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafTotalScope})" : string.Empty)})" : allowSqlLeafFallback - ? $" AND (r.symbol_name_folded = @query OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafTotalScope}))" - : " AND r.symbol_name_folded = @query"; + ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @aliasQueryLeafFolded{sqlLeafTotalScope}))" + : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; else if (exact) innerSql += totalSuffixAlias != null ? $" AND (r.symbol_name = @query COLLATE NOCASE OR (r.symbol_name = @queryAttributeAlias COLLATE NOCASE{totalAliasScope}){(allowSqlLeafFallback ? $" OR (r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE{sqlLeafTotalScope})" : string.Empty)})" @@ -916,9 +929,12 @@ FROM symbol_references r var value = !exact ? $"%{EscapeLikeQuery(query)}%" : _foldReady - ? NameFold.Fold(query) ?? query + ? FoldNameForLanguage(query, lang) : query; - SqliteCommandPolicy.Add(cmd, "@query", value); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", value); SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); diff --git a/src/CodeIndex/Database/DbReader.cs b/src/CodeIndex/Database/DbReader.cs index 7d8286df2..709786bfd 100644 --- a/src/CodeIndex/Database/DbReader.cs +++ b/src/CodeIndex/Database/DbReader.cs @@ -1659,6 +1659,28 @@ internal static bool IsSqlLanguage(string? lang) : normalized; } + internal static string FoldNameForLanguage(string value, string? lang) => + string.Equals(NormalizeQueryLanguage(lang), "nim", StringComparison.Ordinal) + ? NimIdentifierIdentity.Fold(value) ?? value + : NameFold.Fold(value) ?? value; + + private static string BuildPersistedFoldedNameMatchSql( + string foldedNameSql, + string parameterSql, + string fileLanguageSql = "f.lang") + => $"(({fileLanguageSql} <> 'nim' AND {foldedNameSql} = {parameterSql})" + + $" OR ({fileLanguageSql} = 'nim' AND {foldedNameSql} = {parameterSql}Nim))"; + + private static void AddPersistedFoldedNameQueryParameters( + SqliteCommand command, + string parameterName, + string value, + string? lang) + { + SqliteCommandPolicy.Add(command, parameterName, FoldNameForLanguage(value, lang)); + SqliteCommandPolicy.Add(command, $"{parameterName}Nim", FoldNameForLanguage(value, "nim")); + } + internal static bool ContainsSqlLanguage(IEnumerable langs) => langs.Any(IsSqlLanguage); diff --git a/src/CodeIndex/Database/DbSymbolReader.Analysis.cs b/src/CodeIndex/Database/DbSymbolReader.Analysis.cs index 3f2f5e601..a8050fa45 100644 --- a/src/CodeIndex/Database/DbSymbolReader.Analysis.cs +++ b/src/CodeIndex/Database/DbSymbolReader.Analysis.cs @@ -612,8 +612,8 @@ public bool HasExactGraphSupportedDefinition( var allowLeafFallback = !SqlNameResolver.HasQualifier(normalizedQuery); var nameCondition = _foldReady ? allowLeafFallback - ? "(s.name_folded = @queryFolded OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded) OR sql_leaf_name_folded(s.name) = @queryLeafFolded)))" - : "(s.name_folded = @queryFolded OR (f.lang = 'sql' AND sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded))" + ? $"({BuildPersistedFoldedNameMatchSql("s.name_folded", "@queryFolded")} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded) OR sql_leaf_name_folded(s.name) = @queryLeafFolded)))" + : $"({BuildPersistedFoldedNameMatchSql("s.name_folded", "@queryFolded")} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded))" : allowLeafFallback ? "(s.name = @queryRaw COLLATE NOCASE OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name(s.name) = @queryNormalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @queryLeaf COLLATE NOCASE)))" : "(s.name = @queryRaw COLLATE NOCASE OR (f.lang = 'sql' AND sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name(s.name) = @queryNormalized COLLATE NOCASE))"; @@ -635,7 +635,7 @@ FROM symbols s cmd.CommandText = sql; SqliteCommandPolicy.Add(cmd, "@queryRaw", query); - SqliteCommandPolicy.Add(cmd, "@queryFolded", NameFold.Fold(query) ?? query); + AddPersistedFoldedNameQueryParameters(cmd, "@queryFolded", query, lang); SqliteCommandPolicy.Add(cmd, "@queryNormalized", SqlNameResolver.NormalizeQualifiedName(query)); SqliteCommandPolicy.Add(cmd, "@queryNormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(query)) ?? SqlNameResolver.NormalizeQualifiedName(query)); SqliteCommandPolicy.Add(cmd, "@queryLeaf", SqlNameResolver.GetLeafName(query)); diff --git a/src/CodeIndex/Database/DbSymbolReader.Definitions.cs b/src/CodeIndex/Database/DbSymbolReader.Definitions.cs index 024a9a0aa..26ba01a4f 100644 --- a/src/CodeIndex/Database/DbSymbolReader.Definitions.cs +++ b/src/CodeIndex/Database/DbSymbolReader.Definitions.cs @@ -335,8 +335,8 @@ FROM symbols s : " AND ((s.container_qualified_name = @queryRustContainer COLLATE NOCASE OR s.container_name = @queryRustContainer COLLATE NOCASE) AND s.name = @queryRustLeaf COLLATE NOCASE)" : _foldReady ? allowLeafFallback - ? " AND (s.name_folded = @query OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded) OR sql_leaf_name_folded(s.name) = @queryLeafFolded)))" - : $" AND (s.name_folded = @query OR (f.lang = 'sql' AND sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" + ? $" AND ({BuildPersistedFoldedNameMatchSql("s.name_folded", "@query")} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded) OR sql_leaf_name_folded(s.name) = @queryLeafFolded)))" + : $" AND ({BuildPersistedFoldedNameMatchSql("s.name_folded", "@query")} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name_folded(s.name) = @queryNormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" : allowLeafFallback ? " AND (s.name = @query COLLATE NOCASE OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name(s.name) = @queryNormalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @queryLeaf COLLATE NOCASE)))" : $" AND (s.name = @query COLLATE NOCASE OR (f.lang = 'sql' AND sql_segment_count(s.name) = @querySegmentCount AND sql_normalize_name(s.name) = @queryNormalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" @@ -367,9 +367,12 @@ FROM chunks c var paramValue = !exact ? $"%{EscapeLikeQuery(normalizedQuery)}%" : _foldReady - ? NameFold.Fold(normalizedQuery) ?? normalizedQuery + ? FoldNameForLanguage(normalizedQuery, lang) : normalizedQuery; - SqliteCommandPolicy.Add(cmd, "@query", paramValue); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query", normalizedQuery, lang); + else + SqliteCommandPolicy.Add(cmd, "@query", paramValue); SqliteCommandPolicy.Add(cmd, "@queryNormalized", SqlNameResolver.NormalizeQualifiedName(normalizedQuery)); SqliteCommandPolicy.Add(cmd, "@queryNormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(normalizedQuery)) ?? SqlNameResolver.NormalizeQualifiedName(normalizedQuery)); SqliteCommandPolicy.Add(cmd, "@queryLeaf", SqlNameResolver.GetLeafName(normalizedQuery)); diff --git a/src/CodeIndex/Database/DbSymbolReader.Search.cs b/src/CodeIndex/Database/DbSymbolReader.Search.cs index a32f7989f..707420db4 100644 --- a/src/CodeIndex/Database/DbSymbolReader.Search.cs +++ b/src/CodeIndex/Database/DbSymbolReader.Search.cs @@ -220,8 +220,8 @@ FROM symbols s : " AND ((s.container_qualified_name = @query0RustContainer COLLATE NOCASE OR s.container_name = @query0RustContainer COLLATE NOCASE) AND s.name = @query0RustLeaf COLLATE NOCASE)" : _foldReady ? allowLeafFallback - ? " AND (s.name_folded = @query0 OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name_folded(s.name) = @query0NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query0LeafFolded)))" - : $" AND (s.name_folded = @query0 OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name_folded(s.name) = @query0NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" + ? $" AND ({BuildPersistedFoldedNameMatchSql("s.name_folded", "@query0")} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name_folded(s.name) = @query0NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query0LeafFolded)))" + : $" AND ({BuildPersistedFoldedNameMatchSql("s.name_folded", "@query0")} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name_folded(s.name) = @query0NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" : allowLeafFallback ? " AND (s.name = @query0 COLLATE NOCASE OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name(s.name) = @query0Normalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @query0Leaf COLLATE NOCASE)))" : $" AND (s.name = @query0 COLLATE NOCASE OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name(s.name) = @query0Normalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" @@ -246,9 +246,12 @@ FROM symbols s var paramValue = !exact ? $"%{EscapeLikeQuery(value)}%" : _foldReady - ? NameFold.Fold(value) ?? value + ? FoldNameForLanguage(value, lang) : value; - SqliteCommandPolicy.Add(cmd, "@query0", paramValue); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@query0", value, lang); + else + SqliteCommandPolicy.Add(cmd, "@query0", paramValue); SqliteCommandPolicy.Add(cmd, "@query0Normalized", SqlNameResolver.NormalizeQualifiedName(value)); SqliteCommandPolicy.Add(cmd, "@query0NormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(value)) ?? SqlNameResolver.NormalizeQualifiedName(value)); SqliteCommandPolicy.Add(cmd, "@query0Leaf", SqlNameResolver.GetLeafName(value)); @@ -330,8 +333,8 @@ FROM symbols s : $"((s.container_qualified_name = @query{idx}RustContainer COLLATE NOCASE OR s.container_name = @query{idx}RustContainer COLLATE NOCASE) AND s.name = @query{idx}RustLeaf COLLATE NOCASE)"; return _foldReady ? allowLeafFallback - ? $"(s.name_folded = @query{idx}{swiftBacktickClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query{idx}LeafFolded)))" - : $"(s.name_folded = @query{idx}{swiftBacktickClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" + ? $"({BuildPersistedFoldedNameMatchSql("s.name_folded", $"@query{idx}")}{swiftBacktickClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query{idx}LeafFolded)))" + : $"({BuildPersistedFoldedNameMatchSql("s.name_folded", $"@query{idx}")}{swiftBacktickClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" : allowLeafFallback ? $"(s.name = @query{idx} COLLATE NOCASE{swiftBacktickClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @query{idx}Leaf COLLATE NOCASE)))" : $"(s.name = @query{idx} COLLATE NOCASE{swiftBacktickClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})"; @@ -363,9 +366,12 @@ FROM symbols s var paramValue = !exact ? $"%{EscapeLikeQuery(value)}%" : _foldReady - ? NameFold.Fold(value) ?? value + ? FoldNameForLanguage(value, lang) : value; - SqliteCommandPolicy.Add(cmd, $"@query{i}", paramValue); + if (exact && _foldReady) + AddPersistedFoldedNameQueryParameters(cmd, $"@query{i}", value, lang); + else + SqliteCommandPolicy.Add(cmd, $"@query{i}", paramValue); SqliteCommandPolicy.Add(cmd, $"@query{i}Normalized", SqlNameResolver.NormalizeQualifiedName(value)); SqliteCommandPolicy.Add(cmd, $"@query{i}NormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(value)) ?? SqlNameResolver.NormalizeQualifiedName(value)); SqliteCommandPolicy.Add(cmd, $"@query{i}Leaf", SqlNameResolver.GetLeafName(value)); @@ -608,8 +614,8 @@ FROM symbols s : $"((s.container_qualified_name = @query{idx}RustContainer COLLATE NOCASE OR s.container_name = @query{idx}RustContainer COLLATE NOCASE) AND s.name = @query{idx}RustLeaf COLLATE NOCASE)"; return _foldReady ? allowLeafFallback - ? $"(s.name_folded = @query{idx}{swiftBacktickClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query{idx}LeafFolded)))" - : $"(s.name_folded = @query{idx}{swiftBacktickClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" + ? $"({BuildPersistedFoldedNameMatchSql("s.name_folded", $"@query{idx}")}{swiftBacktickClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query{idx}LeafFolded)))" + : $"({BuildPersistedFoldedNameMatchSql("s.name_folded", $"@query{idx}")}{swiftBacktickClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" : allowLeafFallback ? $"(s.name = @query{idx} COLLATE NOCASE{swiftBacktickClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @query{idx}Leaf COLLATE NOCASE)))" : $"(s.name = @query{idx} COLLATE NOCASE{swiftBacktickClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})"; @@ -652,10 +658,21 @@ FROM symbols s if (!exact) paramValue = $"%{EscapeLikeQuery(effectiveQueries[idx])}%"; else if (_foldReady) - paramValue = NameFold.Fold(effectiveQueries[idx]) ?? effectiveQueries[idx]; + paramValue = FoldNameForLanguage(effectiveQueries[idx], lang); else paramValue = effectiveQueries[idx]; - SqliteCommandPolicy.Add(cmd, $"@query{idx}", paramValue); + if (exact && _foldReady) + { + AddPersistedFoldedNameQueryParameters( + cmd, + $"@query{idx}", + effectiveQueries[idx], + lang); + } + else + { + SqliteCommandPolicy.Add(cmd, $"@query{idx}", paramValue); + } SqliteCommandPolicy.Add(cmd, $"@query{idx}Normalized", SqlNameResolver.NormalizeQualifiedName(effectiveQueries[idx])); SqliteCommandPolicy.Add(cmd, $"@query{idx}NormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(effectiveQueries[idx])) ?? SqlNameResolver.NormalizeQualifiedName(effectiveQueries[idx])); SqliteCommandPolicy.Add(cmd, $"@query{idx}Leaf", SqlNameResolver.GetLeafName(effectiveQueries[idx])); @@ -1052,7 +1069,7 @@ private static bool ShouldPreserveRustQualifiedExactQuery(string? query, string? if (ShouldPreserveRustQualifiedExactQuery(query, lang, exact)) return query?.Trim(); - return NormalizeSymbolSearchQuery(query, lang) ?? query; + return NormalizeSymbolSearchQuery(query, lang, exact) ?? query; } private static (string? QualifiedPath, string? ContainerPath, string? LeafName) NormalizeRustQualifiedExactQueryParts(string query) diff --git a/src/CodeIndex/Database/DbWriter.BatchSql.cs b/src/CodeIndex/Database/DbWriter.BatchSql.cs index fa6849b70..cb1f3eb0d 100644 --- a/src/CodeIndex/Database/DbWriter.BatchSql.cs +++ b/src/CodeIndex/Database/DbWriter.BatchSql.cs @@ -30,6 +30,14 @@ private static object FoldedNameDbValue(string? name, Dictionary cache) => + identityNameFolded != null + ? identityNameFolded + : FoldedNameDbValue(name, cache); + private static Dictionary CreateFoldedNameCache(int rowCount, int namesPerRow) { if (rowCount <= 0 || namesPerRow <= 0) diff --git a/src/CodeIndex/Database/DbWriter.ChunkSymbolBatches.cs b/src/CodeIndex/Database/DbWriter.ChunkSymbolBatches.cs index 469dc6699..b0ff53bec 100644 --- a/src/CodeIndex/Database/DbWriter.ChunkSymbolBatches.cs +++ b/src/CodeIndex/Database/DbWriter.ChunkSymbolBatches.cs @@ -244,7 +244,10 @@ private void InsertSymbolBatch(IReadOnlyList symbols, int start, i ? (symbol.IsMetadataTarget.Value ? 1 : 0) : (object)DBNull.Value; cmd.Parameters[parameterIndex++].Value = (object?)symbol.MetadataTargetSource ?? DBNull.Value; - cmd.Parameters[parameterIndex++].Value = FoldedNameDbValue(symbol.Name, foldedNameCache); + cmd.Parameters[parameterIndex++].Value = FoldedNameDbValue( + symbol.Name, + symbol.IdentityNameFolded, + foldedNameCache); } cmd.ExecuteNonQuery(); diff --git a/src/CodeIndex/Database/DbWriter.FoldBackfill.cs b/src/CodeIndex/Database/DbWriter.FoldBackfill.cs index f25c617e4..cd1659175 100644 --- a/src/CodeIndex/Database/DbWriter.FoldBackfill.cs +++ b/src/CodeIndex/Database/DbWriter.FoldBackfill.cs @@ -8,6 +8,7 @@ public partial class DbWriter private const string FoldBackfillPhaseMetaKey = "fold_backfill_phase"; private const string FoldBackfillLastSymbolIdMetaKey = "fold_backfill_last_symbol_id"; private const string FoldBackfillLastReferenceIdMetaKey = "fold_backfill_last_reference_id"; + private const string FoldBackfillGraphRefreshPendingMetaKey = "fold_backfill_graph_refresh_pending"; private static readonly AsyncLocal ScopedFoldBackfillRowUpdatedForTesting = new(); private static readonly AsyncLocal ScopedFoldBackfillVerificationForTesting = new(); @@ -94,14 +95,21 @@ private bool AllFoldedColumnsBackfilledCore( public bool AllFoldedColumnValuesMatchCurrentFold() { var symbols = RentCommand( - "SELECT name, name_folded FROM symbols WHERE name IS NOT NULL", + """ + SELECT s.name, s.name_folded, f.lang + FROM symbols s + JOIN files f ON f.id = s.file_id + WHERE s.name IS NOT NULL + """, static _ => { }); try { using var reader = symbols.ExecuteTrackedReader(); while (reader.TrackedRead()) { - var expected = NameFold.Fold(reader.GetString(0)); + var expected = DbReader.FoldNameForLanguage( + reader.GetString(0), + reader.IsDBNull(2) ? null : reader.GetString(2)); var actual = reader.IsDBNull(1) ? null : reader.GetString(1); if (!string.Equals(actual, expected, StringComparison.Ordinal)) return false; @@ -114,9 +122,12 @@ public bool AllFoldedColumnValuesMatchCurrentFold() var references = RentCommand( @" - SELECT symbol_name, symbol_name_folded, container_name, container_name_folded - FROM symbol_references - WHERE symbol_name IS NOT NULL OR container_name IS NOT NULL", + SELECT r.symbol_name, r.symbol_name_folded, + r.container_name, r.container_name_folded, + f.lang + FROM symbol_references r + JOIN files f ON f.id = r.file_id + WHERE r.symbol_name IS NOT NULL OR r.container_name IS NOT NULL", static _ => { }); try { @@ -125,7 +136,9 @@ FROM symbol_references { if (!reader.IsDBNull(0)) { - var expected = NameFold.Fold(reader.GetString(0)); + var expected = DbReader.FoldNameForLanguage( + reader.GetString(0), + reader.IsDBNull(4) ? null : reader.GetString(4)); var actual = reader.IsDBNull(1) ? null : reader.GetString(1); if (!string.Equals(actual, expected, StringComparison.Ordinal)) return false; @@ -133,7 +146,9 @@ FROM symbol_references if (!reader.IsDBNull(2)) { - var expected = NameFold.Fold(reader.GetString(2)); + var expected = DbReader.FoldNameForLanguage( + reader.GetString(2), + reader.IsDBNull(4) ? null : reader.GetString(4)); var actual = reader.IsDBNull(3) ? null : reader.GetString(3); if (!string.Equals(actual, expected, StringComparison.Ordinal)) return false; @@ -240,6 +255,21 @@ private bool SymbolExtractorVersionMatchesCurrent(string? lang) CancellationToken cancellationToken = default) { cancellationToken.ThrowIfCancellationRequested(); + var graphRefreshPending = string.Equals( + GetMetaString(FoldBackfillGraphRefreshPendingMetaKey), + "1", + StringComparison.Ordinal); + var pendingRows = CountBackfillFoldedColumns(rewriteAll); + if (!graphRefreshPending && (pendingRows.Symbols > 0 || pendingRows.SymbolReferences > 0)) + { + // Persist this before the first row mutation so cancellation after the rewrite but + // before graph refresh cannot make a retry mistake the operation for a no-op. + // 最初の行を書き換える前に pending を永続化し、書換え後から graph refresh + // までの中断を retry が no-op と誤認しないようにする。 + SetMeta(FoldBackfillGraphRefreshPendingMetaKey, "1"); + graphRefreshPending = true; + } + var foldBackfillPhase = rewriteAll ? GetMetaString(FoldBackfillPhaseMetaKey) : null; var symbols = BackfillSymbolFoldedRows(rewriteAll, cancellationToken); if (rewriteAll && foldBackfillPhase != "references") @@ -250,6 +280,15 @@ private bool SymbolExtractorVersionMatchesCurrent(string? lang) var symbolReferences = BackfillReferenceFoldedRows(rewriteAll, cancellationToken); cancellationToken.ThrowIfCancellationRequested(); + if (graphRefreshPending) + { + // Candidate membership and resolved identities depend on the persisted folded keys. + // Refresh them before advertising the rewritten rows as current. + // candidate と解決済み identity は永続化 folded key に依存するため、 + // 書換え後の key を current と公開する前に graph を再解決する。 + RefreshMutualRecursionFlags(cancellationToken); + SetMeta(FoldBackfillGraphRefreshPendingMetaKey, null); + } if (rewriteAll) ClearFoldBackfillCheckpoint(); @@ -318,10 +357,21 @@ private int BackfillSymbolFoldedRows(bool rewriteAll, CancellationToken cancella return 0; var lastSymbolId = rewriteAll ? GetFoldBackfillCheckpoint(FoldBackfillLastSymbolIdMetaKey) : 0; - var rows = new List<(long Id, string Name)>(); + var rows = new List<(long Id, string Name, string? Lang)>(); var selectSql = rewriteAll - ? "SELECT id, name FROM symbols WHERE name IS NOT NULL AND id > @lastSymbolId ORDER BY id" - : "SELECT id, name FROM symbols WHERE name IS NOT NULL AND name_folded IS NULL"; + ? """ + SELECT s.id, s.name, f.lang + FROM symbols s + JOIN files f ON f.id = s.file_id + WHERE s.name IS NOT NULL AND s.id > @lastSymbolId + ORDER BY s.id + """ + : """ + SELECT s.id, s.name, f.lang + FROM symbols s + JOIN files f ON f.id = s.file_id + WHERE s.name IS NOT NULL AND s.name_folded IS NULL + """; var select = RentCommand( selectSql, rewriteAll @@ -335,7 +385,10 @@ private int BackfillSymbolFoldedRows(bool rewriteAll, CancellationToken cancella while (reader.TrackedRead()) { cancellationToken.ThrowIfCancellationRequested(); - rows.Add((reader.GetInt64(0), reader.GetString(1))); + rows.Add(( + reader.GetInt64(0), + reader.GetString(1), + reader.IsDBNull(2) ? null : reader.GetString(2))); } } finally @@ -360,7 +413,7 @@ private int BackfillSymbolFoldedRows(bool rewriteAll, CancellationToken cancella foreach (var row in rows) { cancellationToken.ThrowIfCancellationRequested(); - pFolded.Value = (object?)NameFold.Fold(row.Name) ?? DBNull.Value; + pFolded.Value = DbReader.FoldNameForLanguage(row.Name, row.Lang); pId.Value = row.Id; update.ExecuteNonQuery(); if (rewriteAll) @@ -379,17 +432,23 @@ private int BackfillSymbolFoldedRows(bool rewriteAll, CancellationToken cancella private int BackfillReferenceFoldedRows(bool rewriteAll, CancellationToken cancellationToken) { var lastReferenceId = rewriteAll ? GetFoldBackfillCheckpoint(FoldBackfillLastReferenceIdMetaKey) : 0; - var rows = new List<(long Id, string? SymbolName, string? ContainerName)>(); + var rows = new List<(long Id, string? SymbolName, string? ContainerName, string? Lang)>(); var selectSql = rewriteAll - ? @"SELECT id, symbol_name, container_name - FROM symbol_references - WHERE id > @lastReferenceId - AND (symbol_name IS NOT NULL OR container_name IS NOT NULL) - ORDER BY id" - : @"SELECT id, symbol_name, container_name - FROM symbol_references - WHERE (symbol_name IS NOT NULL AND symbol_name_folded IS NULL) - OR (container_name IS NOT NULL AND container_name_folded IS NULL)"; + ? """ + SELECT r.id, r.symbol_name, r.container_name, f.lang + FROM symbol_references r + JOIN files f ON f.id = r.file_id + WHERE r.id > @lastReferenceId + AND (r.symbol_name IS NOT NULL OR r.container_name IS NOT NULL) + ORDER BY r.id + """ + : """ + SELECT r.id, r.symbol_name, r.container_name, f.lang + FROM symbol_references r + JOIN files f ON f.id = r.file_id + WHERE (r.symbol_name IS NOT NULL AND r.symbol_name_folded IS NULL) + OR (r.container_name IS NOT NULL AND r.container_name_folded IS NULL) + """; var select = RentCommand( selectSql, rewriteAll @@ -406,7 +465,8 @@ FROM symbol_references rows.Add(( reader.GetInt64(0), reader.IsDBNull(1) ? null : reader.GetString(1), - reader.IsDBNull(2) ? null : reader.GetString(2))); + reader.IsDBNull(2) ? null : reader.GetString(2), + reader.IsDBNull(3) ? null : reader.GetString(3))); } } finally @@ -436,8 +496,12 @@ FROM symbol_references foreach (var row in rows) { cancellationToken.ThrowIfCancellationRequested(); - pSymbolNameFolded.Value = (object?)NameFold.Fold(row.SymbolName) ?? DBNull.Value; - pContainerNameFolded.Value = (object?)NameFold.Fold(row.ContainerName) ?? DBNull.Value; + pSymbolNameFolded.Value = row.SymbolName == null + ? DBNull.Value + : DbReader.FoldNameForLanguage(row.SymbolName, row.Lang); + pContainerNameFolded.Value = row.ContainerName == null + ? DBNull.Value + : DbReader.FoldNameForLanguage(row.ContainerName, row.Lang); pId.Value = row.Id; update.ExecuteNonQuery(); if (rewriteAll) diff --git a/src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs b/src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs index 949333775..bbc65bc9d 100644 --- a/src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs +++ b/src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs @@ -61,7 +61,29 @@ CROSS JOIN symbols AS s INDEXED BY idx_symbols_name_folded JOIN files AS target_file ON target_file.id = s.file_id WHERE s.name_folded = dirty_name.name_folded AND target_file.lang = dirty_name.lang + AND target_file.lang <> 'ambiguous_m' GROUP BY target_file.lang, s.name_folded + HAVING COUNT(DISTINCT target_file.path || char(31) || + COALESCE(s.container_qualified_name, s.container_name, '') || char(31) || + COALESCE(s.name, '')) = 1; + + -- Keep the scoped projection aligned with the full-refresh union-wide + -- uniqueness contract for callers whose .m dialect is unresolved. + -- .m 方言が未確定な呼出し元について、差分更新でも全件更新と同じ + -- 言語横断の一意性契約を維持する。 + INSERT INTO temp.reference_unique_symbol_families(lang, name_folded, family_key) + SELECT 'ambiguous_m', + s.name_folded, + MIN(target_file.path || char(31) || + COALESCE(s.container_qualified_name, s.container_name, '') || char(31) || + COALESCE(s.name, '')) AS family_key + FROM temp.{ReferenceGraphLookupNamesTable} AS dirty_name + CROSS JOIN symbols AS s INDEXED BY idx_symbols_name_folded + JOIN files AS target_file ON target_file.id = s.file_id + WHERE dirty_name.lang = 'ambiguous_m' + AND s.name_folded = dirty_name.name_folded + AND target_file.lang IN ('matlab', 'objc') + GROUP BY s.name_folded HAVING COUNT(DISTINCT target_file.path || char(31) || COALESCE(s.container_qualified_name, s.container_name, '') || char(31) || COALESCE(s.name, '')) = 1; @@ -197,7 +219,10 @@ FROM temp.{ReferenceGraphDirtyNamesTable} AS dirty_name CROSS JOIN symbol_references AS r INDEXED BY idx_symbol_refs_symbol_name_folded JOIN files AS source_file ON source_file.id = r.file_id WHERE r.symbol_name_folded = dirty_name.name_folded - AND source_file.lang = dirty_name.lang; + AND ( + source_file.lang = dirty_name.lang + OR (source_file.lang = 'ambiguous_m' AND dirty_name.lang IN ('matlab', 'objc')) + ); """; private const string MaterializeReferenceGraphLookupNamesSql = $""" @@ -209,6 +234,19 @@ FROM temp.{ReferenceGraphDirtyReferencesTable} AS dirty WHERE source_file.lang IS NOT NULL AND r.symbol_name_folded IS NOT NULL; + INSERT OR IGNORE INTO temp.{ReferenceGraphLookupNamesTable}(lang, name_folded) + SELECT target_lang.lang, r.symbol_name_folded + FROM temp.{ReferenceGraphDirtyReferencesTable} AS dirty + JOIN symbol_references AS r ON r.id = dirty.reference_id + JOIN files AS source_file ON source_file.id = r.file_id + CROSS JOIN ( + SELECT 'matlab' AS lang + UNION ALL + SELECT 'objc' + ) AS target_lang + WHERE source_file.lang = 'ambiguous_m' + AND r.symbol_name_folded IS NOT NULL; + INSERT OR IGNORE INTO temp.{ReferenceGraphLookupNamesTable}(lang, name_folded) SELECT 'csharp', r.symbol_name_folded || 'attribute' FROM temp.{ReferenceGraphDirtyReferencesTable} AS dirty diff --git a/src/CodeIndex/Database/DbWriter.References.cs b/src/CodeIndex/Database/DbWriter.References.cs index 40e3e3047..2a0cecdfa 100644 --- a/src/CodeIndex/Database/DbWriter.References.cs +++ b/src/CodeIndex/Database/DbWriter.References.cs @@ -122,7 +122,27 @@ INSERT INTO temp.reference_unique_symbol_families(lang, name_folded, family_key) FROM symbols AS s JOIN files AS target_file ON target_file.id = s.file_id WHERE s.name_folded IS NOT NULL + AND target_file.lang <> 'ambiguous_m' GROUP BY target_file.lang, s.name_folded + HAVING COUNT(DISTINCT target_file.path || char(31) || + COALESCE(s.container_qualified_name, s.container_name, '') || char(31) || + COALESCE(s.name, '')) = 1; + + -- An ambiguous .m caller can bind to either dialect, so uniqueness must hold + -- across the MATLAB/Objective-C union rather than within either language alone. + -- ambiguous .m の呼出し先は両方の方言になり得るため、一意性は各言語内ではなく + -- MATLAB/Objective-C の和集合全体で成立させる。 + INSERT INTO temp.reference_unique_symbol_families(lang, name_folded, family_key) + SELECT 'ambiguous_m', + s.name_folded, + MIN(target_file.path || char(31) || + COALESCE(s.container_qualified_name, s.container_name, '') || char(31) || + COALESCE(s.name, '')) AS family_key + FROM symbols AS s + JOIN files AS target_file ON target_file.id = s.file_id + WHERE s.name_folded IS NOT NULL + AND target_file.lang IN ('matlab', 'objc') + GROUP BY s.name_folded HAVING COUNT(DISTINCT target_file.path || char(31) || COALESCE(s.container_qualified_name, s.container_name, '') || char(31) || COALESCE(s.name, '')) = 1; @@ -142,13 +162,46 @@ ON s.name_folded IN ( THEN r.symbol_name_folded || 'attribute' END ) JOIN files AS target_file ON target_file.id = s.file_id - WHERE source_file.lang = target_file.lang + WHERE ( + (source_file.lang = target_file.lang + AND (source_file.lang <> 'ambiguous_m' OR source_file.id = target_file.id)) + OR (source_file.lang = 'ambiguous_m' AND target_file.lang IN ('matlab', 'objc')) + ) AND r.target_qualifier IS NOT NULL AND r.target_qualifier NOT LIKE char(31) || 'receiver:%' AND ( s.container_name = r.target_qualifier COLLATE NOCASE OR s.container_qualified_name = r.target_qualifier COLLATE NOCASE OR s.container_qualified_name LIKE '%.' || r.target_qualifier COLLATE NOCASE + OR ( + source_file.lang IN ( + 'ada', + 'ambiguous_m', + 'cython', + 'd', + 'julia', + 'matlab', + 'nim', + 'objc' + ) + AND COALESCE(s.container_name, '') = '' + AND COALESCE(s.container_qualified_name, '') = '' + AND EXISTS ( + SELECT 1 + FROM symbols AS target_scope + WHERE target_scope.file_id = s.file_id + AND target_scope.kind IN ('namespace', 'module', 'package') + AND ( + target_scope.name = r.target_qualifier COLLATE NOCASE + OR target_scope.container_qualified_name = r.target_qualifier COLLATE NOCASE + OR substr( + target_scope.name, + 1, + length(r.target_qualifier) + 1 + ) = (r.target_qualifier || '.') COLLATE NOCASE + ) + ) + ) ); INSERT INTO symbol_reference_candidates(reference_id, symbol_id, scope_rank) @@ -195,7 +248,11 @@ THEN r.symbol_name_folded || 'attribute' END ) JOIN files AS target_file ON target_file.id = s.file_id JOIN symbols AS source ON source.id = r.source_symbol_id - WHERE source_file.lang = target_file.lang + WHERE ( + (source_file.lang = target_file.lang + AND (source_file.lang <> 'ambiguous_m' OR source_file.id = target_file.id)) + OR (source_file.lang = 'ambiguous_m' AND target_file.lang IN ('matlab', 'objc')) + ) AND r.target_qualifier IS NULL AND s.file_id = r.file_id AND source.container_name IS NOT NULL @@ -221,7 +278,11 @@ THEN r.symbol_name_folded || 'attribute' END ) JOIN files AS target_file ON target_file.id = s.file_id JOIN symbols AS source ON source.id = r.source_symbol_id - WHERE source_file.lang = target_file.lang + WHERE ( + (source_file.lang = target_file.lang + AND (source_file.lang <> 'ambiguous_m' OR source_file.id = target_file.id)) + OR (source_file.lang = 'ambiguous_m' AND target_file.lang IN ('matlab', 'objc')) + ) AND r.target_qualifier IS NULL AND source.container_qualified_name IS NOT NULL AND source.container_qualified_name <> '' @@ -242,7 +303,11 @@ ON s.name_folded IN ( THEN r.symbol_name_folded || 'attribute' END ) JOIN files AS target_file ON target_file.id = s.file_id - WHERE source_file.lang = target_file.lang + WHERE ( + (source_file.lang = target_file.lang + AND (source_file.lang <> 'ambiguous_m' OR source_file.id = target_file.id)) + OR (source_file.lang = 'ambiguous_m' AND target_file.lang IN ('matlab', 'objc')) + ) AND r.target_qualifier IS NULL AND s.file_id = r.file_id AND NOT EXISTS ( @@ -262,7 +327,11 @@ THEN r.symbol_name_folded || 'attribute' END ) JOIN files AS target_file ON target_file.id = s.file_id JOIN symbols AS source ON source.id = r.source_symbol_id - WHERE source_file.lang = target_file.lang + WHERE ( + (source_file.lang = target_file.lang + AND (source_file.lang <> 'ambiguous_m' OR source_file.id = target_file.id)) + OR (source_file.lang = 'ambiguous_m' AND target_file.lang IN ('matlab', 'objc')) + ) AND r.target_qualifier IS NULL AND source.container_name IS NOT NULL AND source.container_name <> '' @@ -281,13 +350,34 @@ JOIN temp.reference_unique_symbol_families AS unique_family AND unique_family.name_folded = r.symbol_name_folded JOIN symbols AS target ON target.name_folded = unique_family.name_folded JOIN files AS target_file - ON target_file.id = target.file_id - AND target_file.lang = unique_family.lang + ON target_file.id = target.file_id + AND ( + ( + unique_family.lang <> 'ambiguous_m' + AND target_file.lang = unique_family.lang + ) + OR ( + unique_family.lang = 'ambiguous_m' + AND target_file.lang IN ('matlab', 'objc') + ) + ) AND target_file.path || char(31) || COALESCE(target.container_qualified_name, target.container_name, '') || char(31) || COALESCE(target.name, '') = unique_family.family_key WHERE source_file.lang <> 'csharp' - AND r.target_qualifier IS NULL + AND ( + r.target_qualifier IS NULL + OR source_file.lang IN ( + 'ada', + 'ambiguous_m', + 'cython', + 'd', + 'julia', + 'matlab', + 'nim', + 'objc' + ) + ) AND NOT EXISTS ( SELECT 1 FROM symbol_reference_candidates AS existing WHERE existing.reference_id = r.id @@ -746,8 +836,14 @@ private void InsertReferenceBatch( cmd.Parameters[parameterIndex++].Value = previousReferenceLineId; cmd.Parameters[parameterIndex++].Value = (object?)reference.ContainerKind ?? DBNull.Value; cmd.Parameters[parameterIndex++].Value = (object?)reference.ContainerName ?? DBNull.Value; - cmd.Parameters[parameterIndex++].Value = FoldedNameDbValue(reference.SymbolName, foldedNameCache); - cmd.Parameters[parameterIndex++].Value = FoldedNameDbValue(reference.ContainerName, foldedNameCache); + cmd.Parameters[parameterIndex++].Value = FoldedNameDbValue( + reference.SymbolName, + reference.IdentitySymbolNameFolded, + foldedNameCache); + cmd.Parameters[parameterIndex++].Value = FoldedNameDbValue( + reference.ContainerName, + reference.IdentityContainerNameFolded, + foldedNameCache); cmd.Parameters[parameterIndex++].Value = reference.IsSelfReference ? 1 : 0; cmd.Parameters[parameterIndex++].Value = reference.IsMutualRecursion ? 1 : 0; cmd.Parameters[parameterIndex++].Value = (object?)ExtractTargetQualifier(reference) ?? DBNull.Value; @@ -1120,6 +1216,9 @@ internal void RefreshMutualRecursionFlags(CancellationToken cancellationToken = private static string? ExtractTargetQualifier(ReferenceRecord reference) { + if (reference.SuppressInferredTargetQualifier) + return null; + if (!string.IsNullOrWhiteSpace(reference.TargetQualifier)) { var explicitQualifier = reference.TargetQualifier.Trim(); diff --git a/src/CodeIndex/Database/NameFold.cs b/src/CodeIndex/Database/NameFold.cs index fa522dae8..f84b8b05b 100644 --- a/src/CodeIndex/Database/NameFold.cs +++ b/src/CodeIndex/Database/NameFold.cs @@ -51,16 +51,16 @@ public static class NameFold ]; /// - /// Algorithm version for the persisted `name_folded` / `*_folded` keys. Bumped whenever - /// changes semantics so an upgraded reader can detect that a previously - /// stamped DB's folded keys were generated by a different fold function and fall back to - /// the `COLLATE NOCASE` path until a rebuild regenerates the keys. Without this, a - /// Unicode CaseFold table refresh (or any future fold tweak) would silently mismatch - /// against legacy keys and return wrong zero results on `--exact` queries. - /// fold アルゴリズムのバージョン。Fold の意味が変わるたびに bump し、旧 key 版の DB を - /// NOCASE fallback に降格させる（fold table 更新などに備えるメタデータ）。 + /// Persisted `name_folded` / `*_folded` key contract version. Bumped whenever + /// or language-specific key generation changes semantics so an upgraded + /// reader can detect previously stamped incompatible keys and fall back to the + /// `COLLATE NOCASE` path until a rebuild/backfill regenerates them. Without this, a + /// Unicode CaseFold table refresh, language-specific identity rule, or future fold tweak + /// would silently mismatch legacy keys and return wrong zero results on `--exact` queries. + /// 永続化 folded key 契約のバージョン。Fold 自体または言語固有 key 生成の意味が + /// 変わるたびに bump し、旧 key 版の DB を NOCASE fallback に降格させる。 /// - public const int Version = 2; + public const int Version = 3; /// /// Runtime-sensitive canary fingerprint for the current observable fold output. diff --git a/src/CodeIndex/Indexer/AmbiguousMContentMasker.cs b/src/CodeIndex/Indexer/AmbiguousMContentMasker.cs new file mode 100644 index 000000000..2bb217fe9 --- /dev/null +++ b/src/CodeIndex/Indexer/AmbiguousMContentMasker.cs @@ -0,0 +1,379 @@ +using System.Text; + +namespace CodeIndex.Indexer; + +/// +/// Masks MATLAB and Objective-C comments in unresolved .m content while preserving positions. +/// 未確定の .m 内容にある MATLAB / Objective-C コメントを、位置を保ったままマスクする。 +/// +internal static class AmbiguousMContentMasker +{ + internal static string MaskComments( + string content, + bool maskMatlabComments, + bool maskObjectiveCComments, + bool preserveObjectiveCModuloExpressions = false) + { + if (preserveObjectiveCModuloExpressions) + { + preserveObjectiveCModuloExpressions = + HasStrongObjectiveCModuloEvidence(content); + } + + return MaskContent( + content, + maskMatlabComments, + maskObjectiveCComments, + preserveObjectiveCModuloExpressions, + maskStrings: false); + } + + private static string MaskContent( + string content, + bool maskMatlabComments, + bool maskObjectiveCComments, + bool preserveObjectiveCModuloExpressions, + bool maskStrings) + { + if ((!maskMatlabComments || content.IndexOf('%') < 0) + && (!maskObjectiveCComments || content.IndexOf('/') < 0) + && (!maskStrings + || (content.IndexOf('"') < 0 && content.IndexOf('\'') < 0))) + { + return content; + } + + StringBuilder? masked = null; + var inBlockComment = false; + var inMatlabBlockComment = false; + var inLineComment = false; + var quote = '\0'; + + for (var index = 0; index < content.Length; index++) + { + var current = content[index]; + if (current is '\r' or '\n') + { + inLineComment = false; + if (!maskStrings + || quote == '\0' + || !IsEscapedLineBreak(content, index)) + { + quote = '\0'; + } + continue; + } + + if (inLineComment) + { + MaskAt(index); + continue; + } + + if (inMatlabBlockComment) + { + MaskAt(index); + if (current == '%' + && index + 1 < content.Length + && content[index + 1] == '}' + && IsStandaloneMatlabBlockDelimiter(content, index)) + { + MaskAt(++index); + inMatlabBlockComment = false; + } + continue; + } + + if (inBlockComment) + { + MaskAt(index); + if (current == '*' && index + 1 < content.Length && content[index + 1] == '/') + { + MaskAt(++index); + inBlockComment = false; + } + continue; + } + + if (quote != '\0') + { + if (maskStrings) + MaskAt(index); + + if ((preserveObjectiveCModuloExpressions || maskStrings) + && current == '\\' + && index + 1 < content.Length) + { + if (maskStrings && content[index + 1] is not ('\r' or '\n')) + MaskAt(index + 1); + index++; + continue; + } + + if (current == quote) + { + if (quote == '\'' && index + 1 < content.Length && content[index + 1] == '\'') + { + if (maskStrings) + MaskAt(index + 1); + index++; + continue; + } + + quote = '\0'; + } + continue; + } + + if (current == '"' + || (current == '\'' + && (preserveObjectiveCModuloExpressions + || IsMatlabSingleQuoteStart(content, index)))) + { + quote = current; + if (maskStrings) + MaskAt(index); + continue; + } + + if (maskMatlabComments && current == '%') + { + if (preserveObjectiveCModuloExpressions + && LooksLikeObjectiveCModuloOperator(content, index)) + { + continue; + } + + MaskAt(index); + if (index + 1 < content.Length + && content[index + 1] == '{' + && IsStandaloneMatlabBlockDelimiter(content, index)) + { + inMatlabBlockComment = true; + } + else + { + inLineComment = true; + } + continue; + } + + if (!maskObjectiveCComments || current != '/' || index + 1 >= content.Length) + continue; + + var next = content[index + 1]; + if (next == '/') + { + MaskAt(index); + MaskAt(++index); + inLineComment = true; + } + else if (next == '*') + { + MaskAt(index); + MaskAt(++index); + inBlockComment = true; + } + } + + return masked?.ToString() ?? content; + + void MaskAt(int index) + { + masked ??= new StringBuilder(content); + masked[index] = ' '; + } + } + + private static bool IsMatlabSingleQuoteStart(string content, int quoteIndex) + { + var previousIndex = quoteIndex - 1; + return previousIndex < 0 + || content[previousIndex] is '\r' or '\n' + || (!IsTransposeOperandEnd(content[previousIndex]) + && !(content[previousIndex] == '.' + && previousIndex > 0 + && IsTransposeOperandEnd(content[previousIndex - 1]))); + } + + private static bool IsStandaloneMatlabBlockDelimiter(string content, int percentIndex) + { + for (var index = percentIndex - 1; index >= 0 && content[index] is not '\r' and not '\n'; index--) + { + if (!char.IsWhiteSpace(content[index])) + return false; + } + + for (var index = percentIndex + 2; index < content.Length && content[index] is not '\r' and not '\n'; index++) + { + if (!char.IsWhiteSpace(content[index])) + return false; + } + + return true; + } + + private static bool LooksLikeObjectiveCModuloOperator(string content, int percentIndex) + { + var previousIndex = percentIndex - 1; + while (previousIndex >= 0 + && content[previousIndex] is not '\r' and not '\n' + && char.IsWhiteSpace(content[previousIndex])) + { + previousIndex--; + } + + if (previousIndex > 0 + && content[previousIndex] is '+' or '-' + && content[previousIndex - 1] == content[previousIndex]) + { + previousIndex -= 2; + while (previousIndex >= 0 + && content[previousIndex] is not '\r' and not '\n' + && char.IsWhiteSpace(content[previousIndex])) + { + previousIndex--; + } + } + + if (previousIndex < 0 + || content[previousIndex] is '\r' or '\n' + || !IsObjectiveCModuloLeftOperandEnd(content[previousIndex])) + { + return false; + } + + var nextIndex = percentIndex + 1; + if (nextIndex < content.Length && content[nextIndex] == '=') + nextIndex++; + + while (nextIndex < content.Length + && content[nextIndex] is not '\r' and not '\n' + && char.IsWhiteSpace(content[nextIndex])) + { + nextIndex++; + } + + while (nextIndex < content.Length + && content[nextIndex] is not '\r' and not '\n' + && content[nextIndex] is '+' or '-' or '!' or '~' or '*' or '&') + { + nextIndex++; + while (nextIndex < content.Length + && content[nextIndex] is not '\r' and not '\n' + && char.IsWhiteSpace(content[nextIndex])) + { + nextIndex++; + } + } + + return nextIndex < content.Length + && content[nextIndex] is not '\r' and not '\n' + && (char.IsLetterOrDigit(content[nextIndex]) + || content[nextIndex] is '_' or '(' or '[' or '@' or '\'' or '"'); + } + + private static bool IsObjectiveCModuloLeftOperandEnd(char value) => + IsTransposeOperandEnd(value) || value is '\'' or '"'; + + private static bool HasStrongObjectiveCModuloEvidence(string content) + { + var evidenceContent = MaskContent( + content, + maskMatlabComments: true, + maskObjectiveCComments: true, + preserveObjectiveCModuloExpressions: false, + maskStrings: true); + var lineStart = 0; + while (lineStart < evidenceContent.Length) + { + var lineEnd = lineStart; + while (lineEnd < evidenceContent.Length + && evidenceContent[lineEnd] is not ('\r' or '\n')) + { + lineEnd++; + } + + var line = evidenceContent.AsSpan(lineStart, lineEnd - lineStart).TrimStart(); + if (StartsWithObjectiveCPreprocessorDirective(line) + || StartsWithObjectiveCAtKeyword(line) + || StartsWithObjectiveCMethodDeclaration(line)) + { + return true; + } + + lineStart = lineEnd + 1; + if (lineEnd < evidenceContent.Length + && evidenceContent[lineEnd] == '\r' + && lineStart < evidenceContent.Length + && evidenceContent[lineStart] == '\n') + { + lineStart++; + } + } + + return false; + } + + private static bool IsEscapedLineBreak(string content, int lineBreakIndex) + { + var previousIndex = lineBreakIndex - 1; + if (content[lineBreakIndex] == '\n' + && previousIndex >= 0 + && content[previousIndex] == '\r') + { + previousIndex--; + } + + var backslashCount = 0; + while (previousIndex >= 0 && content[previousIndex] == '\\') + { + backslashCount++; + previousIndex--; + } + + return backslashCount % 2 != 0; + } + + private static bool StartsWithObjectiveCPreprocessorDirective(ReadOnlySpan line) + { + if (line.IsEmpty || line[0] != '#') + return false; + + var keywordStart = 1; + while (keywordStart < line.Length && char.IsWhiteSpace(line[keywordStart])) + keywordStart++; + + return StartsWithToken(line[keywordStart..], "import") + || StartsWithToken(line[keywordStart..], "include"); + } + + private static bool StartsWithObjectiveCAtKeyword(ReadOnlySpan line) => + StartsWithToken(line, "@interface") + || StartsWithToken(line, "@implementation") + || StartsWithToken(line, "@protocol") + || StartsWithToken(line, "@class") + || StartsWithToken(line, "@property") + || StartsWithToken(line, "@synthesize") + || StartsWithToken(line, "@dynamic") + || StartsWithToken(line, "@autoreleasepool"); + + private static bool StartsWithObjectiveCMethodDeclaration(ReadOnlySpan line) + { + if (line.IsEmpty || line[0] is not ('-' or '+')) + return false; + + var openingParenthesis = 1; + while (openingParenthesis < line.Length && char.IsWhiteSpace(line[openingParenthesis])) + openingParenthesis++; + + return openingParenthesis < line.Length && line[openingParenthesis] == '('; + } + + private static bool StartsWithToken(ReadOnlySpan line, string token) => + line.StartsWith(token, StringComparison.Ordinal) + && (line.Length == token.Length || !IsTransposeOperandEnd(line[token.Length])); + + private static bool IsTransposeOperandEnd(char value) => + char.IsLetterOrDigit(value) || value is '_' or ')' or ']' or '}'; +} diff --git a/src/CodeIndex/Indexer/Hooks/PostExtractionHookMutationMaterializer.cs b/src/CodeIndex/Indexer/Hooks/PostExtractionHookMutationMaterializer.cs index 42b5ddc35..273edd63c 100644 --- a/src/CodeIndex/Indexer/Hooks/PostExtractionHookMutationMaterializer.cs +++ b/src/CodeIndex/Indexer/Hooks/PostExtractionHookMutationMaterializer.cs @@ -59,6 +59,27 @@ internal static bool TrimToLimit(List? items, int? maxCount) return true; } + internal static void RefreshLanguageIdentity(string? language, IEnumerable symbols) + { + if (!string.Equals(language, "nim", StringComparison.Ordinal)) + return; + + foreach (var symbol in symbols) + symbol.IdentityNameFolded = NimIdentifierIdentity.Fold(symbol.Name); + } + + internal static void RefreshLanguageIdentity(string? language, IEnumerable references) + { + if (!string.Equals(language, "nim", StringComparison.Ordinal)) + return; + + foreach (var reference in references) + { + reference.IdentitySymbolNameFolded = NimIdentifierIdentity.Fold(reference.SymbolName); + reference.IdentityContainerNameFolded = NimIdentifierIdentity.Fold(reference.ContainerName); + } + } + private static SymbolRecord CloneSymbol(SymbolRecord symbol) => new() { @@ -67,6 +88,7 @@ private static SymbolRecord CloneSymbol(SymbolRecord symbol) Kind = symbol.Kind, SubKind = symbol.SubKind, Name = symbol.Name, + IdentityNameFolded = symbol.IdentityNameFolded, Line = symbol.Line, StartLine = symbol.StartLine, StartColumn = symbol.StartColumn, @@ -91,12 +113,16 @@ private static ReferenceRecord CloneReference(ReferenceRecord reference) Id = reference.Id, FileId = reference.FileId, SymbolName = reference.SymbolName, + IdentitySymbolNameFolded = reference.IdentitySymbolNameFolded, ReferenceKind = reference.ReferenceKind, Line = reference.Line, Column = reference.Column, Context = reference.Context, ContainerKind = reference.ContainerKind, ContainerName = reference.ContainerName, + IdentityContainerNameFolded = reference.IdentityContainerNameFolded, + TargetQualifier = reference.TargetQualifier, + SuppressInferredTargetQualifier = reference.SuppressInferredTargetQualifier, IsSelfReference = reference.IsSelfReference, IsMutualRecursion = reference.IsMutualRecursion, }; diff --git a/src/CodeIndex/Indexer/Hooks/PostExtractionHooks.cs b/src/CodeIndex/Indexer/Hooks/PostExtractionHooks.cs index b957ac002..4eda16fe5 100644 --- a/src/CodeIndex/Indexer/Hooks/PostExtractionHooks.cs +++ b/src/CodeIndex/Indexer/Hooks/PostExtractionHooks.cs @@ -400,6 +400,12 @@ private void OnSymbolsExtractedCore( PostExtractionHookMutationMaterializer.ReplaceList(symbols, workingSymbols); } } + + // Hooks can rename or add records but cannot set the internal persisted identity key. + // Re-derive it from the accepted public name after all mutations. + // hook は record の rename/add はできるが内部の永続化 identity key は設定できないため、 + // 全 mutation 受理後の公開名から再導出する。 + PostExtractionHookMutationMaterializer.RefreshLanguageIdentity(context.Language, symbols); } public void OnReferencesExtracted(FileContext context, IList references, CancellationToken cancellationToken = default) @@ -434,6 +440,8 @@ public void OnReferencesExtracted(FileContext context, IList re PostExtractionHookMutationMaterializer.ReplaceList(references, workingReferences); } } + + PostExtractionHookMutationMaterializer.RefreshLanguageIdentity(context.Language, references); } private bool InvokeHookWithBudget( diff --git a/src/CodeIndex/Indexer/NimIdentifierIdentity.cs b/src/CodeIndex/Indexer/NimIdentifierIdentity.cs new file mode 100644 index 000000000..a1eda664a --- /dev/null +++ b/src/CodeIndex/Indexer/NimIdentifierIdentity.cs @@ -0,0 +1,32 @@ +using System.Text; + +namespace CodeIndex.Indexer; + +internal static class NimIdentifierIdentity +{ + internal static string? Fold(string? value) + { + if (value == null) + return null; + + var folded = new StringBuilder(value.Length); + var atIdentifierStart = true; + foreach (var character in value) + { + if (character == '_') + continue; + + if (char.IsLetterOrDigit(character)) + { + folded.Append(atIdentifierStart ? character : char.ToLowerInvariant(character)); + atIdentifierStart = false; + continue; + } + + folded.Append(character); + atIdentifierStart = true; + } + + return folded.ToString(); + } +} diff --git a/src/CodeIndex/Indexer/References/Languages/ScientificNativeCommentMasker.cs b/src/CodeIndex/Indexer/References/Languages/ScientificNativeCommentMasker.cs new file mode 100644 index 000000000..ca5ee7421 --- /dev/null +++ b/src/CodeIndex/Indexer/References/Languages/ScientificNativeCommentMasker.cs @@ -0,0 +1,891 @@ +namespace CodeIndex.Indexer; + +internal static class ScientificNativeCommentMasker +{ + internal static string[] MaskBlockComments(string language, string[] lines) => + language switch + { + "d" => MaskDNonCodeRegions(lines), + "julia" => MaskNestedBlockComments( + MaskTripleQuotedStrings( + lines, + "#=", + "=#", + "#", + singleQuoteCanBePostfix: true, + tripleQuoteUsesBackslashEscapes: true, + maskMultilineBacktickStrings: true), + "#=", + "=#", + "#", + singleQuoteCanBePostfix: true), + "nim" => MaskNimNonCodeRegions(lines), + "matlab" => MaskMatlabBlockComments(lines), + _ => lines, + }; + + internal static string MaskLineStringLiteralsPreservingPostfixSingleQuotes( + string line, + bool useMatlabStringRules, + bool useBackslashEscapes, + bool singleQuoteCanBePostfix = true) + { + char[]? chars = null; + + void MaskAt(int index) => + (chars ??= line.ToCharArray())[index] = ' '; + + for (var cursor = 0; cursor < line.Length; cursor++) + { + var quote = line[cursor]; + if (quote is not ('"' or '\'' or '`')) + continue; + + if (quote == '\'' + && singleQuoteCanBePostfix + && IsPostfixSingleQuote( + line, + cursor, + skipWhitespace: !useMatlabStringRules)) + continue; + + MaskAt(cursor); + cursor++; + while (cursor < line.Length) + { + var current = line[cursor]; + MaskAt(cursor); + if (useBackslashEscapes + && current == '\\' + && cursor + 1 < line.Length) + { + MaskAt(++cursor); + cursor++; + continue; + } + + if (current != quote) + { + cursor++; + continue; + } + + if (cursor + 1 < line.Length && line[cursor + 1] == quote) + { + MaskAt(++cursor); + cursor++; + continue; + } + + break; + } + } + + return chars is null ? line : new string(chars); + } + + private static string[] MaskTripleQuotedStrings( + string[] lines, + string blockOpening, + string blockClosing, + string lineComment, + bool singleQuoteCanBePostfix = false, + bool tripleQuoteUsesBackslashEscapes = false, + bool maskMultilineBacktickStrings = false) + { + if (!MayContain(lines, "\"\"\"") + && (!maskMultilineBacktickStrings || !MayContain(lines, "`"))) + { + return lines; + } + + var result = new string[lines.Length]; + var inTripleQuotedString = false; + var inBacktickString = false; + var blockCommentDepth = 0; + for (var lineIndex = 0; lineIndex < lines.Length; lineIndex++) + { + var line = lines[lineIndex]; + char[]? chars = null; + var quote = '\0'; + + void MaskAt(int index) => + (chars ??= line.ToCharArray())[index] = ' '; + + var cursor = 0; + while (cursor < line.Length) + { + if (inBacktickString) + { + var current = line[cursor]; + MaskAt(cursor++); + if (current == '`' && !HasOddBackslashPrefix(line, cursor - 1)) + inBacktickString = false; + continue; + } + + if (inTripleQuotedString) + { + if (StartsWith(line, cursor, "\"\"\"") + && (!tripleQuoteUsesBackslashEscapes + || !HasOddBackslashPrefix(line, cursor))) + { + MaskToken("\"\"\""); + inTripleQuotedString = false; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (blockCommentDepth > 0) + { + if (StartsWith(line, cursor, blockOpening)) + { + blockCommentDepth++; + cursor += blockOpening.Length; + continue; + } + + if (StartsWith(line, cursor, blockClosing)) + { + blockCommentDepth--; + cursor += blockClosing.Length; + continue; + } + + cursor++; + continue; + } + + if (quote != '\0') + { + if (line[cursor] == '\\' && cursor + 1 < line.Length) + { + cursor += 2; + continue; + } + + if (line[cursor] == quote) + quote = '\0'; + cursor++; + continue; + } + + if (StartsWith(line, cursor, blockOpening)) + { + blockCommentDepth++; + cursor += blockOpening.Length; + continue; + } + + if (StartsWith(line, cursor, lineComment)) + break; + + if (StartsWith(line, cursor, "\"\"\"")) + { + MaskToken("\"\"\""); + inTripleQuotedString = true; + continue; + } + + if (maskMultilineBacktickStrings && line[cursor] == '`') + { + MaskAt(cursor++); + inBacktickString = true; + continue; + } + + if (line[cursor] is '"' or '\'' or '`') + { + if (line[cursor] == '\'' + && singleQuoteCanBePostfix + && IsPostfixSingleQuote(line, cursor)) + { + cursor++; + continue; + } + + quote = line[cursor++]; + continue; + } + + cursor++; + } + + result[lineIndex] = chars is null ? line : new string(chars); + + void MaskToken(string token) + { + for (var tokenIndex = 0; tokenIndex < token.Length; tokenIndex++) + MaskAt(cursor++); + } + } + + return result; + } + + private static bool HasOddBackslashPrefix(string line, int index) + { + var backslashCount = 0; + for (index--; index >= 0 && line[index] == '\\'; index--) + backslashCount++; + + return (backslashCount & 1) != 0; + } + + internal static string MaskNimRawStringLiterals(string line) + { + if (!line.Contains('"')) + return line; + + char[]? chars = null; + for (var cursor = 0; cursor < line.Length;) + { + if (line[cursor] == '"') + { + if (cursor + 2 < line.Length + && line[cursor + 1] == '"' + && line[cursor + 2] == '"') + { + var closing = line.IndexOf("\"\"\"", cursor + 3, StringComparison.Ordinal); + cursor = closing < 0 ? line.Length : closing + 3; + continue; + } + + cursor++; + while (cursor < line.Length) + { + if (line[cursor] == '\\' && cursor + 1 < line.Length) + { + cursor += 2; + continue; + } + + if (line[cursor++] == '"') + break; + } + continue; + } + + if (!(char.IsLetter(line[cursor]) || line[cursor] == '_') + || (cursor > 0 && IsIdentifierChar(line[cursor - 1]))) + { + cursor++; + continue; + } + + var prefixStart = cursor; + while (cursor < line.Length && IsIdentifierChar(line[cursor])) + cursor++; + if (cursor >= line.Length || line[cursor] != '"') + continue; + + chars ??= line.ToCharArray(); + for (var prefixIndex = prefixStart; prefixIndex < cursor; prefixIndex++) + chars[prefixIndex] = ' '; + + if (cursor + 2 < line.Length + && line[cursor + 1] == '"' + && line[cursor + 2] == '"') + { + // Leave the triple-quote delimiter for the stateful multiline masker. + var closing = line.IndexOf("\"\"\"", cursor + 3, StringComparison.Ordinal); + cursor = closing < 0 ? line.Length : closing + 3; + continue; + } + + chars[cursor++] = ' '; + while (cursor < line.Length) + { + var current = line[cursor]; + chars[cursor++] = ' '; + if (current != '"') + continue; + + if (cursor < line.Length && line[cursor] == '"') + { + chars[cursor++] = ' '; + continue; + } + + break; + } + } + + return chars is null ? line : new string(chars); + } + + private static string[] MaskNimNonCodeRegions(string[] lines) + { + string[]? rawStringMaskedLines = null; + for (var lineIndex = 0; lineIndex < lines.Length; lineIndex++) + { + var maskedLine = MaskNimRawStringLiterals(lines[lineIndex]); + if (ReferenceEquals(maskedLine, lines[lineIndex])) + continue; + + rawStringMaskedLines ??= (string[])lines.Clone(); + rawStringMaskedLines[lineIndex] = maskedLine; + } + + var preparedLines = rawStringMaskedLines ?? lines; + return MaskNestedBlockComments( + MaskTripleQuotedStrings(preparedLines, "#[", "]#", "#"), + "#[", + "]#", + "#"); + } + + private static string[] MaskDNonCodeRegions(string[] lines) + { + var result = new string[lines.Length]; + var tokenStringDepth = 0; + var nestedCommentDepth = 0; + var inCStyleBlockComment = false; + var inBacktickString = false; + string? tokenStringClosing = null; + var tokenStringClosingRequiresLineBoundary = false; + var tokenStringQuote = '\0'; + var tokenStringQuoteUsesEscapes = false; + var tokenStringInBacktickString = false; + var tokenStringNestedCommentDepth = 0; + var tokenStringInCStyleBlockComment = false; + string? nestedTokenStringClosing = null; + var nestedTokenStringClosingRequiresLineBoundary = false; + var quote = '\0'; + var quoteUsesEscapes = false; + for (var lineIndex = 0; lineIndex < lines.Length; lineIndex++) + { + var line = lines[lineIndex]; + char[]? chars = null; + var firstNonWhitespaceIndex = 0; + while (firstNonWhitespaceIndex < line.Length + && char.IsWhiteSpace(line[firstNonWhitespaceIndex])) + { + firstNonWhitespaceIndex++; + } + + void MaskAt(int index) => + (chars ??= line.ToCharArray())[index] = ' '; + + var cursor = 0; + while (cursor < line.Length) + { + if (inBacktickString) + { + var current = line[cursor]; + MaskAt(cursor++); + if (current == '`') + inBacktickString = false; + continue; + } + + if (tokenStringClosing != null) + { + if ((!tokenStringClosingRequiresLineBoundary + || cursor == firstNonWhitespaceIndex) + && StartsWith(line, cursor, tokenStringClosing)) + { + MaskToken(tokenStringClosing); + tokenStringClosing = null; + tokenStringClosingRequiresLineBoundary = false; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (tokenStringDepth > 0) + { + if (nestedTokenStringClosing != null) + { + if ((!nestedTokenStringClosingRequiresLineBoundary + || cursor == firstNonWhitespaceIndex) + && StartsWith(line, cursor, nestedTokenStringClosing)) + { + MaskToken(nestedTokenStringClosing); + nestedTokenStringClosing = null; + nestedTokenStringClosingRequiresLineBoundary = false; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (tokenStringInBacktickString) + { + var backtickCurrent = line[cursor]; + MaskAt(cursor++); + if (backtickCurrent == '`') + tokenStringInBacktickString = false; + continue; + } + + if (tokenStringNestedCommentDepth > 0) + { + if (StartsWith(line, cursor, "/+")) + { + MaskToken("/+"); + tokenStringNestedCommentDepth++; + continue; + } + + if (StartsWith(line, cursor, "+/")) + { + MaskToken("+/"); + tokenStringNestedCommentDepth--; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (tokenStringInCStyleBlockComment) + { + if (StartsWith(line, cursor, "*/")) + { + MaskToken("*/"); + tokenStringInCStyleBlockComment = false; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (tokenStringQuote != '\0') + { + var quotedCurrent = line[cursor]; + MaskAt(cursor++); + if (tokenStringQuoteUsesEscapes + && quotedCurrent == '\\' + && cursor < line.Length) + { + MaskAt(cursor++); + continue; + } + + if (quotedCurrent == tokenStringQuote) + tokenStringQuote = '\0'; + continue; + } + + if (StartsWith(line, cursor, "//")) + { + while (cursor < line.Length) + MaskAt(cursor++); + break; + } + + if (StartsWith(line, cursor, "/*")) + { + MaskToken("/*"); + tokenStringInCStyleBlockComment = true; + continue; + } + + if (StartsWith(line, cursor, "/+")) + { + MaskToken("/+"); + tokenStringNestedCommentDepth++; + continue; + } + + if (StartsWith(line, cursor, "q\"") + && (cursor == 0 || !IsIdentifierChar(line[cursor - 1])) + && TryGetDTokenStringClosing( + line, + cursor + 2, + out var nestedOpeningLength, + out var nestedClosing, + out var nestedClosingRequiresLineBoundary)) + { + for (var openingIndex = 0; openingIndex < 2 + nestedOpeningLength; openingIndex++) + MaskAt(cursor++); + nestedTokenStringClosing = nestedClosing; + nestedTokenStringClosingRequiresLineBoundary = + nestedClosingRequiresLineBoundary; + continue; + } + + if (line[cursor] == '`') + { + MaskAt(cursor++); + tokenStringInBacktickString = true; + continue; + } + + if (StartsWith(line, cursor, "r\"") + && (cursor == 0 || !IsIdentifierChar(line[cursor - 1]))) + { + MaskAt(cursor++); + MaskAt(cursor++); + tokenStringQuote = '"'; + tokenStringQuoteUsesEscapes = false; + continue; + } + + if (line[cursor] is '"' or '\'') + { + tokenStringQuote = line[cursor]; + tokenStringQuoteUsesEscapes = true; + MaskAt(cursor++); + continue; + } + + var structuralCurrent = line[cursor]; + MaskAt(cursor++); + if (structuralCurrent == '{') + tokenStringDepth++; + else if (structuralCurrent == '}') + tokenStringDepth--; + continue; + } + + if (nestedCommentDepth > 0) + { + if (StartsWith(line, cursor, "/+")) + { + MaskToken("/+"); + nestedCommentDepth++; + continue; + } + + if (StartsWith(line, cursor, "+/")) + { + MaskToken("+/"); + nestedCommentDepth--; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (inCStyleBlockComment) + { + if (StartsWith(line, cursor, "*/")) + { + MaskToken("*/"); + inCStyleBlockComment = false; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (quote != '\0') + { + var current = line[cursor]; + MaskAt(cursor++); + if (quoteUsesEscapes && current == '\\' && cursor < line.Length) + { + MaskAt(cursor++); + continue; + } + + if (current == quote) + quote = '\0'; + continue; + } + + if (StartsWith(line, cursor, "//")) + { + while (cursor < line.Length) + MaskAt(cursor++); + break; + } + + if (StartsWith(line, cursor, "/*")) + { + MaskToken("/*"); + inCStyleBlockComment = true; + continue; + } + + if (StartsWith(line, cursor, "/+")) + { + MaskToken("/+"); + nestedCommentDepth++; + continue; + } + + if (StartsWith(line, cursor, "q{") + && (cursor == 0 || !IsIdentifierChar(line[cursor - 1]))) + { + MaskAt(cursor++); + MaskAt(cursor++); + tokenStringDepth = 1; + continue; + } + + if (StartsWith(line, cursor, "q\"") + && (cursor == 0 || !IsIdentifierChar(line[cursor - 1])) + && TryGetDTokenStringClosing( + line, + cursor + 2, + out var openingLength, + out var closing, + out var closingRequiresLineBoundary)) + { + for (var openingIndex = 0; openingIndex < 2 + openingLength; openingIndex++) + MaskAt(cursor++); + tokenStringClosing = closing; + tokenStringClosingRequiresLineBoundary = closingRequiresLineBoundary; + continue; + } + + if (line[cursor] == '`') + { + MaskAt(cursor++); + inBacktickString = true; + continue; + } + + if (StartsWith(line, cursor, "r\"") + && (cursor == 0 || !IsIdentifierChar(line[cursor - 1]))) + { + MaskAt(cursor++); + MaskAt(cursor++); + quote = '"'; + quoteUsesEscapes = false; + continue; + } + + if (line[cursor] is '"' or '\'') + { + quote = line[cursor]; + quoteUsesEscapes = true; + MaskAt(cursor++); + continue; + } + + cursor++; + } + + result[lineIndex] = chars is null ? line : new string(chars); + + void MaskToken(string token) + { + for (var tokenIndex = 0; tokenIndex < token.Length; tokenIndex++) + MaskAt(cursor++); + } + } + + return result; + } + + private static bool TryGetDTokenStringClosing( + string line, + int delimiterIndex, + out int openingLength, + out string closing, + out bool closingRequiresLineBoundary) + { + openingLength = 0; + closing = string.Empty; + closingRequiresLineBoundary = false; + if (delimiterIndex >= line.Length) + return false; + + closing = line[delimiterIndex] switch + { + '[' => "]\"", + '(' => ")\"", + '{' => "}\"", + '<' => ">\"", + _ => string.Empty, + }; + if (closing.Length != 0) + { + openingLength = 1; + return true; + } + + var end = delimiterIndex; + while (end < line.Length && IsIdentifierChar(line[end])) + end++; + if (end == delimiterIndex + || (end < line.Length && !char.IsWhiteSpace(line[end]))) + { + return false; + } + + openingLength = end - delimiterIndex; + closing = line[delimiterIndex..end] + '"'; + closingRequiresLineBoundary = true; + return true; + } + + private static string[] MaskNestedBlockComments( + string[] lines, + string opening, + string closing, + string lineComment, + bool singleQuoteCanBePostfix = false) + { + if (!MayContain(lines, opening)) + return lines; + + var result = new string[lines.Length]; + var depth = 0; + for (var lineIndex = 0; lineIndex < lines.Length; lineIndex++) + { + var line = lines[lineIndex]; + char[]? chars = null; + var quote = '\0'; + + void MaskAt(int index) => + (chars ??= line.ToCharArray())[index] = ' '; + + var cursor = 0; + while (cursor < line.Length) + { + if (depth > 0) + { + if (StartsWith(line, cursor, opening)) + { + MaskToken(opening); + depth++; + continue; + } + + if (StartsWith(line, cursor, closing)) + { + MaskToken(closing); + depth--; + continue; + } + + MaskAt(cursor++); + continue; + } + + if (quote != '\0') + { + if (line[cursor] == '\\' && cursor + 1 < line.Length) + { + cursor += 2; + continue; + } + + if (line[cursor] == quote) + quote = '\0'; + cursor++; + continue; + } + + if (line[cursor] is '"' or '\'' or '`') + { + if (line[cursor] == '\'' + && singleQuoteCanBePostfix + && IsPostfixSingleQuote(line, cursor)) + { + cursor++; + continue; + } + + quote = line[cursor++]; + continue; + } + + if (StartsWith(line, cursor, opening)) + { + MaskToken(opening); + depth++; + continue; + } + + if (StartsWith(line, cursor, lineComment)) + break; + + cursor++; + } + + result[lineIndex] = chars is null ? line : new string(chars); + + void MaskToken(string token) + { + for (var tokenIndex = 0; tokenIndex < token.Length; tokenIndex++) + MaskAt(cursor++); + } + } + + return result; + } + + private static bool IsPostfixSingleQuote( + string line, + int quoteIndex, + bool skipWhitespace = true) + { + for (var index = quoteIndex - 1; index >= 0; index--) + { + if (skipWhitespace && char.IsWhiteSpace(line[index])) + continue; + + if (char.IsLetterOrDigit(line[index]) || line[index] is '_' or ')' or ']' or '}') + return true; + + return line[index] == '.' + && index > 0 + && (char.IsLetterOrDigit(line[index - 1]) + || line[index - 1] is '_' or ')' or ']' or '}'); + } + + return false; + } + + private static string[] MaskMatlabBlockComments(string[] lines) + { + if (!MayContain(lines, "%{")) + return lines; + + var result = new string[lines.Length]; + var inBlockComment = false; + for (var index = 0; index < lines.Length; index++) + { + var line = lines[index]; + var trimmed = line.Trim(); + var opens = !inBlockComment && trimmed.Equals("%{", StringComparison.Ordinal); + if (opens) + inBlockComment = true; + + if (!inBlockComment) + { + result[index] = line; + continue; + } + + result[index] = new string(' ', line.Length); + if (trimmed.Equals("%}", StringComparison.Ordinal)) + inBlockComment = false; + } + + return result; + } + + private static bool MayContain(IEnumerable lines, string token) + { + foreach (var line in lines) + { + if (line.Contains(token, StringComparison.Ordinal)) + return true; + } + + return false; + } + + private static bool StartsWith(string line, int start, string value) => + start + value.Length <= line.Length + && line.AsSpan(start, value.Length).SequenceEqual(value); + + private static bool IsIdentifierChar(char value) => + char.IsLetterOrDigit(value) || value == '_'; +} diff --git a/src/CodeIndex/Indexer/References/Languages/ScientificNativeReferenceExtractor.cs b/src/CodeIndex/Indexer/References/Languages/ScientificNativeReferenceExtractor.cs new file mode 100644 index 000000000..683d1445a --- /dev/null +++ b/src/CodeIndex/Indexer/References/Languages/ScientificNativeReferenceExtractor.cs @@ -0,0 +1,813 @@ +using System.Text.RegularExpressions; +using CodeIndex.Models; +using Regex = CodeIndex.Indexer.BoundedRegex; + +namespace CodeIndex.Indexer; + +internal static class ScientificNativeReferenceExtractor +{ + internal const string CurrentContainerReceiverMarker = "\u001fcurrent-container"; + private const string JuliaIdentifierPattern = @"[\p{L}_]\w*"; + private const string JuliaCallableIdentifierPattern = JuliaIdentifierPattern + @"!?"; + + internal readonly record struct DTemplateArgumentCallSpan(int Start, int EndExclusive); + private readonly record struct DTemplateInvocation( + string Name, + int NameIndex, + int ArgumentStart, + int EndExclusive); + + private static readonly HashSet SupportedLanguages = + new(StringComparer.Ordinal) { "ada", "cython", "d", "julia", "matlab", "nim", "objc" }; + + private static readonly Regex NimFromImportRegex = new( + @"^\s*from\s+(?[A-Za-z_][\w./]*)\s+import\b", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex NimImportListRegex = new( + @"^\s*(?:import|include)\s+(?[^\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex NimBaseTypeRegex = new( + @"\bobject\s+of\s+(?[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)*)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex NimAnnotatedTypeRegex = new( + @":\s*(?:(?:var|lent|sink)\s+)?(?[A-Z][A-Za-z0-9_]*(?:\.[A-Za-z_]\w*)*)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + private static readonly Regex MatlabImportListRegex = new( + @"^\s*import\s+(?[^\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex MatlabBaseTypeListRegex = new( + @"^\s*classdef\b[^<\r\n]*<\s*(?[^\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + private static readonly Regex JuliaImportListRegex = new( + @"^\s*(?:using|import)\s+(?[^\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex JuliaTypeRegex = new( + @"(?:<:|::)\s*(?[A-Z][A-Za-z0-9_]*(?:\.[A-Za-z_]\w*)*)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex JuliaMacroCallRegex = new( + $@"(?{JuliaIdentifierPattern})", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex JuliaBangCallRegex = new( + $@"(?{JuliaIdentifierPattern}!)\s*\(", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex JuliaBroadcastCallRegex = new( + $@"(?{JuliaCallableIdentifierPattern})\s*\.\s*\(", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + private static readonly Regex DImportListRegex = new( + @"^\s*(?:(?:public|private|protected|package|static|export)\s+)*import\s+(?[^;\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex DBaseTypeListRegex = new( + @"^\s*(?:(?:public|private|protected|package|static|abstract|final|extern)\s+)*(?:class|interface)\s+[A-Za-z_]\w*(?:\s*\([^)]*\))?\s*:\s*(?[^{\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex CythonFromImportRegex = new( + @"^\s*from\s+(?\.*[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)*)\s+(?:cimport|import)\b", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex CythonImportListRegex = new( + @"^\s*(?:cimport|import)\s+(?[^\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex CythonStringDependencyRegex = new( + """^\s*(?:include\s+|cdef\s+extern\s+from\s+)(?:'(?[^']+)'|"(?[^"]+)")""", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex CythonStringDependencyDirectiveRegex = new( + @"^\s*(?:include\b|cdef\s+extern\s+from\b)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex CythonBaseTypeListRegex = new( + @"^\s*(?:cdef\s+)?class\s+[A-Za-z_]\w*\s*\(\s*(?[^)\r\n]+)", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + private static readonly Regex AdaImportListRegex = new( + @"^\s*(?:(?:limited|private)\s+)*with\s+(?[^;\r\n]+)", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaDerivedTypeRegex = new( + @"^\s*type\s+[A-Za-z]\w*\s+is\s+new\s+(?[A-Za-z]\w*(?:\.[A-Za-z]\w*)*)", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaBareCallRegex = new( + @"(?:^|;|\b(?:begin|then|else|loop)\b|=>)\s*(?!(?:end|null|return|exit|raise|goto)\b)(?[A-Za-z]\w*(?:\.[A-Za-z]\w*)*)\s*(?=;)", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex ObjectiveCImportRegex = new( + """^\s*#\s*(?:import|include)\s*[<"](?[^>"]+)[>"]""", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex ObjectiveCImportDirectiveRegex = new( + @"^\s*#\s*(?:import|include)\b", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + internal static bool Supports(string language) => SupportedLanguages.Contains(language); + + internal static string? GetParenthesizedCallTargetQualifier( + string language, + string preparedLine, + int callIndex) + { + var separatorIndex = callIndex - 1; + while (separatorIndex >= 0 && char.IsWhiteSpace(preparedLine[separatorIndex])) + separatorIndex--; + if (separatorIndex >= 0 && preparedLine[separatorIndex] == '@') + { + separatorIndex--; + while (separatorIndex >= 0 && char.IsWhiteSpace(preparedLine[separatorIndex])) + separatorIndex--; + } + if (separatorIndex < 0 || preparedLine[separatorIndex] != '.') + return null; + + var segments = new List(); + while (separatorIndex >= 0 && preparedLine[separatorIndex] == '.') + { + var segmentEnd = separatorIndex; + var segmentStart = segmentEnd - 1; + while (segmentStart >= 0 && char.IsWhiteSpace(preparedLine[segmentStart])) + segmentStart--; + segmentEnd = segmentStart + 1; + while (segmentStart >= 0 && IsQualifierIdentifierPart(preparedLine[segmentStart])) + segmentStart--; + segmentStart++; + if (segmentStart >= segmentEnd + || !IsQualifierIdentifierStart(preparedLine[segmentStart])) + { + return null; + } + + segments.Add(preparedLine[segmentStart..segmentEnd]); + separatorIndex = segmentStart - 1; + while (separatorIndex >= 0 && char.IsWhiteSpace(preparedLine[separatorIndex])) + separatorIndex--; + } + + segments.Reverse(); + if ((language == "cython" && segments[0] is "self" or "cls") + || (language == "d" && segments[0] == "this")) + { + return CurrentContainerReceiverMarker; + } + + return string.Join('.', segments); + } + + internal static bool IsDTemplateArgumentCall( + IReadOnlyList? spans, + ref int spanIndex, + int callIndex) + { + if (spans == null) + return false; + + while (spanIndex < spans.Count && callIndex >= spans[spanIndex].EndExclusive) + spanIndex++; + + return spanIndex < spans.Count && callIndex >= spans[spanIndex].Start; + } + + internal static IReadOnlyList? EmitReferences( + string language, + string preparedLine, + string originalLine, + List references, + ReferenceDedupeSet seen, + long fileId, + string context, + int lineNumber, + Func resolveContainerForColumn, + Action addCallLikeReference, + int maxDependenciesPerDeclaration, + Action? reportDiagnostic) + { + var dependencyLimitReported = false; + List? dTemplateArgumentCallSpans = null; + + switch (language) + { + case "nim": + EmitMatch(NimFromImportRegex, "import"); + EmitNimImportList(); + EmitMatches(NimBaseTypeRegex, "type_reference", normalizeQualifiedTypeName: true); + EmitMatches(NimAnnotatedTypeRegex, "type_reference", normalizeQualifiedTypeName: true); + break; + case "matlab": + EmitNameList(MatlabImportListRegex, "import", ',', splitOnWhitespace: true); + EmitNameList( + MatlabBaseTypeListRegex, + "type_reference", + '&', + normalizeQualifiedTypeName: true); + break; + case "julia": + EmitNameList( + JuliaImportListRegex, + "import", + ',', + stopAtColon: true, + stripLeadingRelativePrefix: true); + EmitMatches(JuliaTypeRegex, "type_reference", normalizeQualifiedTypeName: true); + foreach (Match match in JuliaMacroCallRegex.Matches(preparedLine)) + { + var group = match.Groups["name"]; + addCallLikeReference(group.Value, group.Index); + } + foreach (Match match in JuliaBangCallRegex.Matches(preparedLine)) + { + var group = match.Groups["name"]; + addCallLikeReference(group.Value, group.Index); + } + foreach (Match match in JuliaBroadcastCallRegex.Matches(preparedLine)) + { + var group = match.Groups["name"]; + addCallLikeReference(group.Value, group.Index); + } + break; + case "d": + EmitNameList(DImportListRegex, "import", ',', stopAtColon: true, stripLeadingAlias: true); + EmitNameList( + DBaseTypeListRegex, + "type_reference", + ',', + normalizeQualifiedTypeName: true); + foreach (var invocation in FindDTemplateInvocations(preparedLine)) + { + addCallLikeReference(invocation.Name, invocation.NameIndex); + (dTemplateArgumentCallSpans ??= []).Add( + new DTemplateArgumentCallSpan( + invocation.ArgumentStart, + invocation.EndExclusive)); + } + break; + case "cython": + EmitMatch(CythonFromImportRegex, "import", stripLeadingRelativePrefix: true); + EmitNameList(CythonImportListRegex, "import", ','); + EmitCythonStringDependency(); + EmitNameList( + CythonBaseTypeListRegex, + "type_reference", + ',', + normalizeQualifiedTypeName: true); + break; + case "ada": + EmitNameList(AdaImportListRegex, "import", ','); + EmitMatches(AdaDerivedTypeRegex, "type_reference", normalizeQualifiedTypeName: true); + foreach (Match bareCall in AdaBareCallRegex.Matches(preparedLine)) + { + var group = bareCall.Groups["name"]; + var separatorIndex = group.Value.LastIndexOf('.'); + var leafOffset = separatorIndex + 1; + EmitName( + group.Value[leafOffset..], + group.Index + leafOffset, + "call", + separatorIndex >= 0 ? group.Value[..separatorIndex] : null); + } + break; + case "objc": + EmitObjectiveCImport(); + break; + } + + return dTemplateArgumentCallSpans; + + void EmitMatch( + Regex regex, + string referenceKind, + bool stripLeadingRelativePrefix = false) + { + var match = regex.Match(preparedLine); + if (!match.Success) + return; + + var group = match.Groups["name"]; + if (!stripLeadingRelativePrefix) + { + EmitGroup(group, referenceKind); + return; + } + + var nameStart = 0; + while (nameStart < group.Length && group.Value[nameStart] == '.') + nameStart++; + if (nameStart < group.Length) + EmitName(group.Value[nameStart..], group.Index + nameStart, referenceKind); + } + + void EmitMatches( + Regex regex, + string referenceKind, + bool normalizeQualifiedTypeName = false) + { + foreach (Match match in regex.Matches(preparedLine)) + EmitGroup(match.Groups["name"], referenceKind, normalizeQualifiedTypeName); + } + + void EmitNameList( + Regex regex, + string referenceKind, + char separator, + bool splitOnWhitespace = false, + bool stopAtColon = false, + bool stripLeadingAlias = false, + bool stripLeadingRelativePrefix = false, + bool normalizeQualifiedTypeName = false) + { + var match = regex.Match(preparedLine); + if (!match.Success) + return; + + var group = match.Groups["names"]; + if (!group.Success || group.Length == 0) + return; + + var names = group.Value; + var namesEnd = names.Length; + if (stopAtColon) + { + var colonIndex = names.IndexOf(':'); + if (colonIndex >= 0) + namesEnd = colonIndex; + } + + var dependencyCount = 0; + var segmentStart = 0; + for (var index = 0; index <= namesEnd; index++) + { + var atEnd = index == namesEnd; + var isSeparator = !atEnd + && (names[index] == separator || (splitOnWhitespace && char.IsWhiteSpace(names[index]))); + if (!atEnd && !isSeparator) + continue; + + var canEmit = dependencyCount < maxDependenciesPerDeclaration; + if (TryEmitDependencySegment( + names, + segmentStart, + index, + group.Index, + referenceKind, + stripLeadingAlias, + stripLeadingRelativePrefix, + normalizeQualifiedTypeName, + emit: canEmit)) + { + if (!canEmit) + { + ReportDependencyLimit(); + return; + } + + dependencyCount++; + } + + segmentStart = index + 1; + while (segmentStart < namesEnd + && (names[segmentStart] == separator + || (splitOnWhitespace && char.IsWhiteSpace(names[segmentStart])))) + { + segmentStart++; + index++; + } + } + } + + bool TryEmitDependencySegment( + string names, + int segmentStart, + int segmentEnd, + int absoluteOffset, + string referenceKind, + bool stripLeadingAlias, + bool stripLeadingRelativePrefix, + bool normalizeQualifiedTypeName = false, + bool emit = true) + { + while (segmentStart < segmentEnd && char.IsWhiteSpace(names[segmentStart])) + segmentStart++; + while (segmentEnd > segmentStart && char.IsWhiteSpace(names[segmentEnd - 1])) + segmentEnd--; + if (segmentStart >= segmentEnd) + return false; + + if (stripLeadingAlias) + { + var equalsIndex = names.LastIndexOf('=', segmentEnd - 1, segmentEnd - segmentStart); + if (equalsIndex >= segmentStart) + { + segmentStart = equalsIndex + 1; + while (segmentStart < segmentEnd && char.IsWhiteSpace(names[segmentStart])) + segmentStart++; + } + } + + for (var index = segmentStart; index + 3 < segmentEnd; index++) + { + if (!char.IsWhiteSpace(names[index]) + || !names.AsSpan(index + 1, 2).Equals("as", StringComparison.OrdinalIgnoreCase) + || !char.IsWhiteSpace(names[index + 3])) + { + continue; + } + + segmentEnd = index; + break; + } + + while (segmentEnd > segmentStart && char.IsWhiteSpace(names[segmentEnd - 1])) + segmentEnd--; + + var nameEnd = segmentStart; + while (nameEnd < segmentEnd && IsDependencyNameChar(names[nameEnd])) + nameEnd++; + while (nameEnd > segmentStart && names[nameEnd - 1] is '.' or '/') + nameEnd--; + + var firstIdentifierIndex = segmentStart; + while (firstIdentifierIndex < nameEnd && names[firstIdentifierIndex] == '.') + firstIdentifierIndex++; + if (firstIdentifierIndex >= nameEnd + || !(char.IsLetter(names[firstIdentifierIndex]) || names[firstIdentifierIndex] == '_')) + { + return false; + } + + var emittedNameStart = stripLeadingRelativePrefix + ? firstIdentifierIndex + : segmentStart; + string? targetQualifier = null; + if (normalizeQualifiedTypeName) + { + var lastDotIndex = names.LastIndexOf( + '.', + nameEnd - 1, + nameEnd - emittedNameStart); + if (lastDotIndex >= emittedNameStart) + { + targetQualifier = names[emittedNameStart..lastDotIndex]; + emittedNameStart = lastDotIndex + 1; + } + } + if (emit) + { + EmitName( + names[emittedNameStart..nameEnd], + absoluteOffset + emittedNameStart, + referenceKind, + targetQualifier); + } + + return true; + } + + void EmitNimImportList() + { + var match = NimImportListRegex.Match(preparedLine); + if (!match.Success) + return; + + var group = match.Groups["names"]; + if (!group.Success || group.Length == 0) + return; + + var names = group.Value; + var dependencyCount = 0; + var segmentStart = 0; + var bracketDepth = 0; + for (var index = 0; index <= names.Length; index++) + { + if (index < names.Length) + { + if (names[index] == '[') + bracketDepth++; + else if (names[index] == ']' && bracketDepth > 0) + bracketDepth--; + } + + if (index < names.Length && (names[index] != ',' || bracketDepth != 0)) + continue; + + var (emittedCount, truncated) = EmitNimImportSegment( + names, + segmentStart, + index, + group.Index, + Math.Max(0, maxDependenciesPerDeclaration - dependencyCount)); + dependencyCount += emittedCount; + if (truncated) + { + ReportDependencyLimit(); + return; + } + + segmentStart = index + 1; + } + } + + (int EmittedCount, bool Truncated) EmitNimImportSegment( + string names, + int segmentStart, + int segmentEnd, + int absoluteOffset, + int remainingCapacity) + { + while (segmentStart < segmentEnd && char.IsWhiteSpace(names[segmentStart])) + segmentStart++; + while (segmentEnd > segmentStart && char.IsWhiteSpace(names[segmentEnd - 1])) + segmentEnd--; + if (segmentStart >= segmentEnd) + return (0, false); + + var openingBracket = names.IndexOf('[', segmentStart, segmentEnd - segmentStart); + var closingBracket = openingBracket >= 0 + ? names.IndexOf(']', openingBracket + 1, segmentEnd - openingBracket - 1) + : -1; + if (openingBracket < 0 || closingBracket < 0) + { + var canEmit = remainingCapacity > 0; + var hasDependency = TryEmitDependencySegment( + names, + segmentStart, + segmentEnd, + absoluteOffset, + "import", + stripLeadingAlias: false, + stripLeadingRelativePrefix: false, + emit: canEmit); + return hasDependency + ? (canEmit ? 1 : 0, !canEmit) + : (0, false); + } + + var prefixStart = segmentStart; + var prefixEnd = openingBracket; + while (prefixEnd > prefixStart && char.IsWhiteSpace(names[prefixEnd - 1])) + prefixEnd--; + if (prefixEnd <= prefixStart || names[prefixEnd - 1] != '/') + return (0, false); + + var prefix = names[prefixStart..prefixEnd]; + var emittedCount = 0; + var itemStart = openingBracket + 1; + for (var index = itemStart; index <= closingBracket; index++) + { + if (index < closingBracket && names[index] != ',') + continue; + + var itemEnd = index; + while (itemStart < itemEnd && char.IsWhiteSpace(names[itemStart])) + itemStart++; + while (itemEnd > itemStart && char.IsWhiteSpace(names[itemEnd - 1])) + itemEnd--; + if (itemStart < itemEnd) + { + var nameEnd = itemStart; + while (nameEnd < itemEnd && IsDependencyNameChar(names[nameEnd])) + nameEnd++; + while (nameEnd > itemStart && names[nameEnd - 1] is '.' or '/') + nameEnd--; + if (nameEnd > itemStart) + { + if (emittedCount >= remainingCapacity) + return (emittedCount, true); + + EmitName( + prefix + names[itemStart..nameEnd], + absoluteOffset + itemStart, + "import"); + emittedCount++; + } + } + + itemStart = index + 1; + } + + return (emittedCount, false); + } + + void ReportDependencyLimit() + { + if (dependencyLimitReported) + return; + + dependencyLimitReported = true; + reportDiagnostic?.Invoke(new ReferenceExtractionDiagnostic( + "reference_scientific_native_dependency_name_budget_exceeded", + $"Scientific/native dependency extraction used the first {maxDependenciesPerDeclaration:N0} names on line {lineNumber:N0} and skipped additional names.")); + } + + void EmitGroup( + Group group, + string referenceKind, + bool normalizeQualifiedTypeName = false) + { + if (!group.Success || group.Length == 0) + return; + + if (!normalizeQualifiedTypeName) + { + EmitName(group.Value, group.Index, referenceKind); + return; + } + + var lastDotIndex = group.Value.LastIndexOf('.'); + EmitName( + lastDotIndex >= 0 ? group.Value[(lastDotIndex + 1)..] : group.Value, + group.Index + lastDotIndex + 1, + referenceKind, + lastDotIndex >= 0 ? group.Value[..lastDotIndex] : null); + } + + void EmitName( + string name, + int index, + string referenceKind, + string? targetQualifier = null) + { + ReferenceExtractor.AddReference( + references, + seen, + fileId, + name, + index, + referenceKind, + context, + lineNumber, + resolveContainerForColumn(index), + language, + targetQualifier); + } + + void EmitObjectiveCImport() + { + var directiveLine = ObjectiveCImportRegex.IsMatch(preparedLine) + ? preparedLine + : ObjectiveCImportDirectiveRegex.IsMatch(preparedLine) + ? originalLine + : null; + if (directiveLine == null) + return; + + var match = ObjectiveCImportRegex.Match(directiveLine); + if (match.Success) + EmitGroup(match.Groups["name"], "import"); + } + + void EmitCythonStringDependency() + { + var directiveLine = CythonStringDependencyRegex.IsMatch(preparedLine) + ? preparedLine + : CythonStringDependencyDirectiveRegex.IsMatch(preparedLine) + ? originalLine + : null; + if (directiveLine == null) + return; + + var match = CythonStringDependencyRegex.Match(directiveLine); + if (match.Success) + EmitGroup(match.Groups["name"], "import"); + } + } + + private static bool IsDependencyNameChar(char value) => + char.IsLetterOrDigit(value) || value is '_' or '.' or '/' or '*'; + + private static IReadOnlyList FindDTemplateInvocations(string line) + { + List? invocations = null; + var cursor = 0; + while (cursor < line.Length) + { + if (!IsDIdentifierStart(line[cursor]) + || (cursor > 0 && IsDIdentifierPart(line[cursor - 1]))) + { + cursor++; + continue; + } + + var nameIndex = cursor; + cursor = ScanDIdentifier(line, cursor); + var nameEnd = cursor; + var scan = cursor; + while (true) + { + SkipWhitespace(line, ref scan); + if (scan >= line.Length || line[scan] != '.') + break; + + var nextNameIndex = scan + 1; + SkipWhitespace(line, ref nextNameIndex); + if (nextNameIndex >= line.Length || !IsDIdentifierStart(line[nextNameIndex])) + break; + + nameIndex = nextNameIndex; + nameEnd = ScanDIdentifier(line, nextNameIndex); + scan = nameEnd; + } + + SkipWhitespace(line, ref scan); + if (scan >= line.Length || line[scan] != '!') + { + cursor = scan; + continue; + } + + if (scan + 1 < line.Length && line[scan + 1] == '=') + { + cursor = scan + 2; + continue; + } + + scan++; + SkipWhitespace(line, ref scan); + var argumentStart = nameEnd; + if (scan < line.Length && line[scan] == '(') + { + if (!TryScanBalancedDTemplateArguments(line, scan, out scan)) + { + cursor = line.Length; + continue; + } + } + else + { + var tokenStart = scan; + while (scan < line.Length + && !char.IsWhiteSpace(line[scan]) + && line[scan] is not ('(' or ';' or ',')) + { + scan++; + } + + if (scan == tokenStart) + continue; + } + + SkipWhitespace(line, ref scan); + if (scan >= line.Length || line[scan] != '(') + { + cursor = scan; + continue; + } + + (invocations ??= []).Add(new DTemplateInvocation( + line[nameIndex..nameEnd], + nameIndex, + argumentStart, + scan + 1)); + cursor = scan + 1; + } + + return invocations ?? []; + } + + private static bool TryScanBalancedDTemplateArguments( + string line, + int openingParenthesis, + out int endExclusive) + { + var depth = 0; + for (var cursor = openingParenthesis; cursor < line.Length; cursor++) + { + if (line[cursor] == '(') + { + depth++; + continue; + } + + if (line[cursor] != ')') + continue; + + depth--; + if (depth == 0) + { + endExclusive = cursor + 1; + return true; + } + } + + endExclusive = line.Length; + return false; + } + + private static int ScanDIdentifier(string line, int start) + { + var cursor = start + 1; + while (cursor < line.Length && IsDIdentifierPart(line[cursor])) + cursor++; + + return cursor; + } + + private static void SkipWhitespace(string line, ref int cursor) + { + while (cursor < line.Length && char.IsWhiteSpace(line[cursor])) + cursor++; + } + + private static bool IsDIdentifierStart(char value) => + char.IsLetter(value) || value == '_'; + + private static bool IsDIdentifierPart(char value) => + char.IsLetterOrDigit(value) || value == '_'; + + private static bool IsQualifierIdentifierStart(char value) => + char.IsLetter(value) || value == '_'; + + private static bool IsQualifierIdentifierPart(char value) => + char.IsLetterOrDigit(value) || value == '_'; +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs index 0696544bf..a31ec6c63 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs @@ -20,6 +20,9 @@ internal static List ExtractCore(ReferenceExtractionContext req var isJsxFile = IsJsxFilePath(path); var isRazorFile = IsRazorFilePath(path) || requestedLanguage is "razor" or "blazor" or "cshtml"; + if (language == "ambiguous_m") + return ExtractAmbiguousMReferences(request); + if (language is "clojure" or "erlang" or "ocaml" or "raku") return ExtractFunctionalLanguageReferences(request); @@ -59,6 +62,9 @@ internal static List ExtractCore(ReferenceExtractionContext req var csharpLinesInsideBlockComment = preparedInput.CSharpLinesInsideBlockComment; var referenceStructuralLines = preparedInput.ReferenceStructuralLines; var preparedLines = preparedInput.PreparedLines; + var scientificNativeDependencyLimit = ScientificNativeReferenceExtractor.Supports(language) + ? GetSafetyLimits().MaxNamesPerLine + : 0; var goImportBlockLines = preparedInput.GoImportBlockLines; var luaReferenceLines = preparedInput.LuaReferenceLines; var luaPreparedLines = preparedInput.LuaPreparedLines; @@ -91,6 +97,11 @@ internal static List ExtractCore(ReferenceExtractionContext req var csharpAttrTopLevelRanges = csharpAttrTables.Item2; var definitionNamesComparer = GetDefinitionNamesComparer(language); var definitionNamesByLine = BuildDefinitionNamesByLine(language, symbols, request.ReportDiagnostic); + var scientificDefinitionNameIndicesByLine = BuildScientificDefinitionNameIndicesByLine( + language, + lines, + symbols, + definitionNamesByLine); var allDefinitionNames = language == "stylus" ? BuildAllDefinitionNames(language, symbols, request.ReportDiagnostic) : null; @@ -1354,6 +1365,10 @@ static string NormalizeCSharpBclRegexQualifiedName(string value) var definitionNames = definitionNamesByLine.TryGetValue(lineNumber, out var namesOnLine) ? namesOnLine : null; + Dictionary>? scientificDefinitionNameIndices = null; + scientificDefinitionNameIndicesByLine?.TryGetValue( + lineNumber, + out scientificDefinitionNameIndices); Dictionary? definitionNameIndices = null; List? sqlDefinitionLeafSpans = null; if (language == "sql") @@ -1604,6 +1619,35 @@ bool ShouldSuppressDefinitionCall(string resolvedName, string rawName, int callI } } + if (scientificDefinitionNameIndices != null + && scientificDefinitionNameIndices.TryGetValue( + resolvedName, + out var scientificDefinitionIndices)) + { + return scientificDefinitionIndices.Contains(callIndex); + } + + if (language == "julia") + { + var targetQualifier = + ScientificNativeReferenceExtractor.GetParenthesizedCallTargetQualifier( + language, + preparedLine, + callIndex); + if (targetQualifier != null) + { + var qualifiedName = $"{targetQualifier}.{resolvedName}"; + var qualifiedDefinitionIndex = + preparedLine.IndexOf(qualifiedName, StringComparison.Ordinal); + if (qualifiedDefinitionIndex >= 0 + && callIndex == qualifiedDefinitionIndex + targetQualifier.Length + 1 + && definitionNames.Contains(qualifiedName)) + { + return true; + } + } + } + if (language != "sql") return TryGetDefinitionNameIndex(resolvedName, out var definitionIndex) && callIndex == definitionIndex; @@ -2292,7 +2336,15 @@ bool TryGetDefinitionNameIndex(string resolvedName, out int definitionIndex) } void AddCallLikeReference(string name, int callIndex) => - _ = TryAddCallLikeReference(name, callIndex); + _ = TryAddCallLikeReference( + name, + callIndex, + ScientificNativeReferenceExtractor.Supports(language) + ? ScientificNativeReferenceExtractor.GetParenthesizedCallTargetQualifier( + language, + preparedLine, + callIndex) + : null); void AddPowerShellParameterReference(string name, int callIndex) { @@ -2300,7 +2352,10 @@ void AddPowerShellParameterReference(string name, int callIndex) AddReference(references, seen, fileId, name, callIndex, "parameter", context, lineNumber, callContainer, language); } - bool TryAddCallLikeReference(string name, int callIndex) + bool TryAddCallLikeReference( + string name, + int callIndex, + string? targetQualifier = null) { var normalizedName = language == "fsharp" && FSharpReferenceExtractor.IsOperatorCallName(name) ? $"operator {name}" @@ -2399,7 +2454,18 @@ bool TryAddCallLikeReference(string name, int callIndex) } if (IsConstructorCallName(language, preparedLine, callIndex)) { - AddReference(references, seen, fileId, normalizedName, callIndex, "instantiate", context, lineNumber, callContainer, language); + AddReference( + references, + seen, + fileId, + normalizedName, + callIndex, + "instantiate", + context, + lineNumber, + callContainer, + language, + targetQualifier); return true; } if (language == "rust" @@ -2466,7 +2532,18 @@ bool TryAddCallLikeReference(string name, int callIndex) return true; } - AddReference(references, seen, fileId, normalizedName, callIndex, "call", context, lineNumber, callContainer); + AddReference( + references, + seen, + fileId, + normalizedName, + callIndex, + "call", + context, + lineNumber, + callContainer, + ScientificNativeReferenceExtractor.Supports(language) ? language : null, + targetQualifier: targetQualifier); return true; bool TryGetKnownPythonTypeCall(string candidate, out string canonicalName) @@ -2559,12 +2636,46 @@ bool TryGetKnownPythonTypeCall(string candidate, out string canonicalName) } else { + IReadOnlyList? + dTemplateArgumentCallSpans = null; + if (ScientificNativeReferenceExtractor.Supports(language)) + { + dTemplateArgumentCallSpans = ScientificNativeReferenceExtractor.EmitReferences( + language, + preparedLine, + originalLine, + references, + seen, + fileId, + context, + lineNumber, + ResolveContainerForCall, + AddCallLikeReference, + scientificNativeDependencyLimit, + request.ReportDiagnostic); + } + + var dTemplateArgumentCallSpanIndex = 0; foreach (Match match in CallRegex.Matches(preparedLine)) { var name = match.Groups["name"].Value; var callIndex = match.Groups["name"].Index; if (language == "rust" && RustReferenceExtractor.IsRawIdentifierPrefix(preparedLine, callIndex)) continue; + if (language == "d" + && ScientificNativeReferenceExtractor.IsDTemplateArgumentCall( + dTemplateArgumentCallSpans, + ref dTemplateArgumentCallSpanIndex, + callIndex)) + { + continue; + } + if (language == "ada" + && callIndex > 0 + && preparedLine[callIndex - 1] == '\'') + { + continue; + } if (language == "objc" && IsObjCSelectorLiteralCall(preparedLine, name, callIndex)) continue; if (sqlSuppressedCallIndices != null && sqlSuppressedCallIndices.Contains(callIndex)) @@ -2573,7 +2684,15 @@ bool TryGetKnownPythonTypeCall(string candidate, out string canonicalName) && sqlWindowFunctionCallSiteSuppressions.Contains((lineNumber, callIndex))) continue; GetMatchedCallIndices().Add(callIndex); - if (TryAddCallLikeReference(name, callIndex)) + if (TryAddCallLikeReference( + name, + callIndex, + ScientificNativeReferenceExtractor.Supports(language) + ? ScientificNativeReferenceExtractor.GetParenthesizedCallTargetQualifier( + language, + preparedLine, + callIndex) + : null)) { EmitGenericInvocationTypeArgumentReferences( language, @@ -3441,5 +3560,75 @@ void AddGradleDslReference(string name, int callIndex) return references; } + private static List ExtractAmbiguousMReferences(ReferenceExtractionContext request) + { + if (string.IsNullOrEmpty(request.Content) + || (request.HasOversizeLine ?? ChunkSplitter.HasOversizeLine(request.Content)) + || (request.ConflictMarkerLine ?? FileIndexer.GetConflictMarkerLine(request.Content)) > 0) + { + return []; + } + + var normalizedContent = request.ContentIsNormalized + ? request.Content + : FileIndexer.NormalizeContentForPrepass(request.Content); + var originalLines = SplitContentLines(normalizedContent); + var matlabContent = AmbiguousMContentMasker.MaskComments( + normalizedContent, + maskMatlabComments: true, + maskObjectiveCComments: true); + var objectiveCContent = AmbiguousMContentMasker.MaskComments( + normalizedContent, + maskMatlabComments: true, + maskObjectiveCComments: true, + preserveObjectiveCModuloExpressions: true); + var matlabReferences = ExtractCore(request with + { + Language = "matlab", + Content = matlabContent, + RequestedLanguage = "ambiguous_m", + ContentIsNormalized = true, + HasOversizeLine = false, + ConflictMarkerLine = 0, + }); + var objectiveCReferences = ExtractCore(request with + { + Language = "objc", + Content = objectiveCContent, + RequestedLanguage = "ambiguous_m", + ContentIsNormalized = true, + HasOversizeLine = false, + ConflictMarkerLine = 0, + }); + var merged = CreateReferenceList( + request.MaxReferenceCount, + Math.Min(matlabReferences.Count + objectiveCReferences.Count, ReferenceListInitialCapacityMax)); + var seen = new ReferenceDedupeSet(merged.Capacity); + + AddUnique(matlabReferences); + AddUnique(objectiveCReferences); + return merged; + + void AddUnique(IReadOnlyList candidates) + { + for (var index = 0; index < candidates.Count && !ReferenceLimitReached(merged); index++) + { + var candidate = candidates[index]; + if (candidate.Line > 0 && candidate.Line <= originalLines.Length) + candidate.Context = originalLines[candidate.Line - 1].Trim(); + var key = CreateReferenceDedupeKey( + candidate.FileId, + "ambiguous_m", + candidate.Line, + candidate.Column, + candidate.ReferenceKind, + candidate.SymbolName, + candidate.ContainerKind, + candidate.ContainerName); + if (seen.Add(key)) + TryAddReference(merged, candidate); + } + } + } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.Preparation.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.Preparation.cs index 352835ea1..e93b64bb9 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.Preparation.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.Preparation.cs @@ -80,7 +80,8 @@ private static bool TryPrepareReferenceLines( ? MaskJavaTextBlocks(content) : content; var lines = SplitContentLines(maskedContent); - var structuralLines = StructuralLineMasker.MaskLines(language, lines, out var jsTaggedTemplateHits); + var structuralMaskLanguage = language == "cython" ? "python" : language; + var structuralLines = StructuralLineMasker.MaskLines(structuralMaskLanguage, lines, out var jsTaggedTemplateHits); var csharpLineState = language == "csharp" && MightContainCSharpXmlDocComment(content) ? BuildCSharpLineStateMasks(lines) : (MultilineStringContent: null, BlockComment: null); @@ -99,7 +100,8 @@ [new CppLexicalRange(0, maskedContent.Length)], : UsesCStyleBlockComments(language) ? MaskCStyleBlockCommentLines(language, structuralLines) : structuralLines; - if (language == "python") + referenceStructuralLines = ScientificNativeCommentMasker.MaskBlockComments(language, referenceStructuralLines); + if (language is "python" or "cython") referenceStructuralLines = MaskPythonFStrings(referenceStructuralLines); var linePrepareOptions = CreateReferenceLinePrepareOptions(language); diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.State.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.State.cs index a104be74c..852084ce6 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.State.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.State.cs @@ -37,6 +37,7 @@ internal sealed class CSharpWhereConstraintState "solution", "dependency_manifest", "dependency_lock", "json", "jsonl", "toml", "gitignore", "gitattributes", "editorconfig", "dockerignore", "config", "app_manifest", "zig", "css", "sass", "stylus", "xml", "yaml", "graphql", "html", "markdown", "fortran", "pascal", "objc", "smalltalk", + "ada", "cython", "d", "julia", "matlab", "nim", "ambiguous_m", "verilog", "systemverilog", "vhdl", ]; diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.TypeReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.TypeReferences.cs index a3edf8ea5..5320f86a5 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.TypeReferences.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.TypeReferences.cs @@ -2713,11 +2713,17 @@ private readonly record struct ReferenceLinePrepareOptions( bool UseCSharpTriggerFastPath, bool MaskRustLifetimes, bool MaskStringLiterals, + bool MaskNimRawStrings, bool IncludeBacktickStringDelimiter, + bool PreserveStringLiteralLength, + bool PreservePostfixSingleQuotes, + bool UseMatlabStringRules, + bool ScientificStringUsesBackslashEscapes, bool UsesHashComments, bool UsesRHashComments, bool UsesSlashComments, bool UsesDashDashComments, + bool UsesPercentComments, bool UsesFortranBangComments, bool UsesPascalBlockComments, bool UsesVisualBasicComments); @@ -2727,11 +2733,17 @@ private static ReferenceLinePrepareOptions CreateReferenceLinePrepareOptions(str UseCSharpTriggerFastPath: lang == "csharp", MaskRustLifetimes: lang == "rust", MaskStringLiterals: lang != "cobol", + MaskNimRawStrings: lang == "nim", IncludeBacktickStringDelimiter: lang is not ("kotlin" or "r"), + PreserveStringLiteralLength: ScientificNativeReferenceExtractor.Supports(lang), + PreservePostfixSingleQuotes: lang is "ada" or "julia" or "matlab", + UseMatlabStringRules: lang == "matlab", + ScientificStringUsesBackslashEscapes: lang is "cython" or "d" or "julia" or "nim" or "objc", UsesHashComments: UsesHashComments(lang), UsesRHashComments: lang == "r", UsesSlashComments: UsesSlashComments(lang), UsesDashDashComments: UsesDashDashComments(lang), + UsesPercentComments: lang == "matlab", UsesFortranBangComments: lang == "fortran", UsesPascalBlockComments: lang == "pascal", UsesVisualBasicComments: lang == "vb"); @@ -2750,12 +2762,25 @@ private static string PrepareLine(string line, ReferenceLinePrepareOptions optio var result = line; if (options.MaskRustLifetimes) result = MaskRustLifetimeTokens(result); + if (options.MaskNimRawStrings) + result = ScientificNativeCommentMasker.MaskNimRawStringLiterals(result); if (options.MaskStringLiterals && MayContainStringLiteralDelimiter(result, options.IncludeBacktickStringDelimiter)) { - var stringLiteralRegex = !options.IncludeBacktickStringDelimiter - ? NonBacktickStringLiteralRegex - : StringLiteralRegex; - result = stringLiteralRegex.Replace(result, "\"\""); + if (options.PreserveStringLiteralLength) + { + result = ScientificNativeCommentMasker.MaskLineStringLiteralsPreservingPostfixSingleQuotes( + result, + options.UseMatlabStringRules, + options.ScientificStringUsesBackslashEscapes, + options.PreservePostfixSingleQuotes); + } + else + { + var stringLiteralRegex = !options.IncludeBacktickStringDelimiter + ? NonBacktickStringLiteralRegex + : StringLiteralRegex; + result = stringLiteralRegex.Replace(result, "\"\""); + } } if (result.Contains("/*", StringComparison.Ordinal)) result = InlineBlockCommentRegex.Replace(result, " "); @@ -2784,6 +2809,19 @@ private static string PrepareLine(string line, ReferenceLinePrepareOptions optio result = result[..dashCommentIndex]; } + if (options.UsesPercentComments) + { + // Outside strings, MATLAB treats `...` and the rest of the physical line as a + // continuation comment. MATLAB では文字列外の `...` 以降は継続コメントになる。 + var continuationIndex = result.IndexOf("...", StringComparison.Ordinal); + if (continuationIndex >= 0) + result = result[..continuationIndex]; + + var percentCommentIndex = result.IndexOf('%'); + if (percentCommentIndex >= 0) + result = result[..percentCommentIndex]; + } + if (options.UsesFortranBangComments) { var bangCommentIndex = result.IndexOf('!'); @@ -4782,15 +4820,17 @@ private static bool IsAnnotationContext(string line, int probe) private static bool UsesHashComments(string lang) => lang is "python" or "ruby" or "perl" or "php" or "elixir" or "r" or "powershell" - or "shell" or "makefile" or "terraform" or "dockerfile" or "protobuf"; + or "shell" or "makefile" or "terraform" or "dockerfile" or "protobuf" + or "nim" or "julia" or "cython"; private static bool UsesSlashComments(string lang) => lang is not "python" and not "ruby" and not "r" and not "haskell" and not "makefile" and not "terraform" and not "dockerfile" - and not "css" and not "fortran"; + and not "css" and not "fortran" and not "nim" and not "matlab" + and not "julia" and not "cython" and not "ada"; private static bool UsesDashDashComments(string lang) => - lang is "lua" or "sql" or "haskell"; + lang is "lua" or "sql" or "haskell" or "ada"; private static bool IsPythonStringPrefixChar(char c) => c is 'r' or 'R' or 'u' or 'U' or 'b' or 'B' or 'f' or 'F'; diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.cs index 4d3b26438..31f0550c6 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.cs @@ -64,6 +64,7 @@ public static partial class ReferenceExtractor "reference_definition_lookup_line_name_budget_exceeded", "reference_definition_lookup_symbol_budget_exceeded", "reference_enclosing_type_candidate_budget_exceeded", + "reference_scientific_native_dependency_name_budget_exceeded", ShaderReferenceExtractor.LineNameBudgetDiagnosticKind, ShaderReferenceExtractor.TrackedNameBudgetDiagnosticKind, "reference_swift_property_line_budget_exceeded", @@ -409,6 +410,39 @@ private static bool IsFunctionLikeSymbolKind(string kind) { "false", "nil", "self", "super", "thisContext", "true", }, + ["ada"] = new HashSet(StringComparer.OrdinalIgnoreCase) + { + "accept", "begin", "case", "declare", "delay", "else", "elsif", "end", "entry", + "exception", "exit", "function", "generic", "if", "loop", "package", "pragma", + "procedure", "raise", "record", "renames", "return", "select", "task", "terminate", + "type", "use", "when", "while", "with", + }, + ["cython"] = new HashSet(StringComparer.Ordinal) + { + "cdef", "cpdef", "ctypedef", "cimport", "def", "extern", "gil", "include", + "nogil", "property", + }, + ["d"] = new HashSet(StringComparer.Ordinal) + { + "__traits", "assert", "cast", "debug", "extern", "is", "mixin", "pragma", "scope", + "static", "unittest", "version", + }, + ["julia"] = new HashSet(StringComparer.Ordinal) + { + "abstract", "baremodule", "begin", "do", "export", "finally", "function", "import", + "let", "macro", "module", "mutable", "primitive", "quote", "struct", "using", "where", + }, + ["matlab"] = new HashSet(StringComparer.Ordinal) + { + "arguments", "case", "catch", "classdef", "elseif", "end", "function", "import", + "methods", "otherwise", "parfor", "properties", "spmd", + }, + ["nim"] = new HashSet(StringComparer.Ordinal) + { + "block", "case", "concept", "converter", "defer", "discard", "distinct", "from", + "func", "import", "include", "iterator", "macro", "method", "mixin", "object", + "proc", "template", "type", "when", + }, // Gradle/Groovy keywords / Gradle/Groovy キーワード ["gradle"] = new HashSet(StringComparer.Ordinal) { @@ -1393,6 +1427,112 @@ private static IReadOnlyDictionary> BuildDefinitionNamesByL return namesByLine; } + private static IReadOnlyDictionary>>? + BuildScientificDefinitionNameIndicesByLine( + string language, + IReadOnlyList lines, + IReadOnlyList symbols, + IReadOnlyDictionary> definitionNamesByLine) + { + if (!ScientificNativeReferenceExtractor.Supports(language) || symbols.Count == 0) + return null; + + var limits = GetSafetyLimits(); + var comparer = GetDefinitionNamesComparer(language); + var comparison = comparer == StringComparer.OrdinalIgnoreCase + ? StringComparison.OrdinalIgnoreCase + : StringComparison.Ordinal; + var indicesByLine = new Dictionary>>(); + for (var symbolIndex = 0; + symbolIndex < symbols.Count && symbolIndex < limits.MaxLookupSymbols; + symbolIndex++) + { + var symbol = symbols[symbolIndex]; + if (symbol.Line <= 0 + || symbol.Line > lines.Count + || !definitionNamesByLine.TryGetValue(symbol.Line, out var retainedNames) + || !retainedNames.Contains(symbol.Name)) + { + continue; + } + + var line = lines[symbol.Line - 1]; + var searchStart = Math.Clamp(symbol.StartColumn ?? 0, 0, line.Length); + var definitionIndex = FindScientificDefinitionNameIndex( + line, + symbol.Name, + searchStart, + comparison); + if (definitionIndex < 0) + continue; + + if (!indicesByLine.TryGetValue(symbol.Line, out var indicesByName)) + { + indicesByName = new Dictionary>(comparer); + indicesByLine[symbol.Line] = indicesByName; + } + + AddScientificDefinitionNameIndex( + indicesByName, + symbol.Name, + definitionIndex); + + var leafSeparatorIndex = symbol.Name.LastIndexOf('.'); + if (leafSeparatorIndex >= 0 && leafSeparatorIndex + 1 < symbol.Name.Length) + { + AddScientificDefinitionNameIndex( + indicesByName, + symbol.Name[(leafSeparatorIndex + 1)..], + definitionIndex + leafSeparatorIndex + 1); + } + } + + return indicesByLine; + } + + private static int FindScientificDefinitionNameIndex( + string line, + string name, + int searchStart, + StringComparison comparison) + { + while (searchStart <= line.Length - name.Length) + { + var index = line.IndexOf(name, searchStart, comparison); + if (index < 0) + return -1; + + var beforeIsBoundary = index == 0 + || !IsScientificDefinitionIdentifierChar(line[index - 1]); + var end = index + name.Length; + var afterIsBoundary = end == line.Length + || !IsScientificDefinitionIdentifierChar(line[end]); + if (beforeIsBoundary && afterIsBoundary) + return index; + + searchStart = index + 1; + } + + return -1; + } + + private static bool IsScientificDefinitionIdentifierChar(char value) + => char.IsLetterOrDigit(value) || value is '_' or '!' or '?' or '$'; + + private static void AddScientificDefinitionNameIndex( + Dictionary> indicesByName, + string name, + int index) + { + if (!indicesByName.TryGetValue(name, out var indices)) + { + indices = []; + indicesByName[name] = indices; + } + + indices.Add(index); + } + private static IReadOnlySet BuildAllDefinitionNames( string language, IReadOnlyList symbols, @@ -1508,7 +1648,7 @@ private static int GetRustEnumCandidateSpan(SymbolRecord symbol) => (symbol.BodyEndLine ?? symbol.EndLine) - (symbol.BodyStartLine ?? symbol.StartLine); private static StringComparer GetDefinitionNamesComparer(string language) - => language == "sql" + => language is "sql" or "ada" ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal; @@ -1927,7 +2067,8 @@ internal static void AddReference( string context, int lineNumber, SymbolRecord? container, - string? language = null) + string? language = null, + string? targetQualifier = null) { AddReference( references, @@ -1939,7 +2080,8 @@ internal static void AddReference( context, lineNumber, container, - language); + language, + targetQualifier); } internal static void AddReference( @@ -1952,24 +2094,38 @@ internal static void AddReference( string context, int lineNumber, SymbolRecord? container, - string? language = null) + string? language = null, + string? targetQualifier = null) { var column = nameIndex + 1; var dedupeKey = CreateReferenceDedupeKey(fileId, language, lineNumber, column, referenceKind, name, container); if (!seen.Add(dedupeKey)) return; + var currentContainerReceiver = string.Equals( + targetQualifier, + ScientificNativeReferenceExtractor.CurrentContainerReceiverMarker, + StringComparison.Ordinal); TryAddReference(references, new ReferenceRecord { FileId = fileId, SymbolName = name, + IdentitySymbolNameFolded = language == "nim" + ? NimIdentifierIdentity.Fold(name) + : null, ReferenceKind = referenceKind, Line = lineNumber, Column = column, Context = context, ContainerKind = container?.Kind, ContainerName = container?.Name, - IsSelfReference = IsSameReferenceName(container?.Name, name), + IdentityContainerNameFolded = language == "nim" + ? NimIdentifierIdentity.Fold(container?.Name) + : null, + TargetQualifier = currentContainerReceiver ? null : targetQualifier, + SuppressInferredTargetQualifier = currentContainerReceiver, + IsSelfReference = (targetQualifier == null || currentContainerReceiver) + && IsSameReferenceName(container?.Name, name), }); } diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs index 06bd2543d..a845c4e34 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs @@ -86,6 +86,43 @@ private static List ExtractCore( return DependencyPackageExtractor.ExtractSymbols(fileId, content, SplitContentLines(content), filePath, lang); } + if (lang == "ambiguous_m") + { + var matlabContent = AmbiguousMContentMasker.MaskComments( + content, + maskMatlabComments: true, + maskObjectiveCComments: true); + var objectiveCContent = AmbiguousMContentMasker.MaskComments( + content, + maskMatlabComments: true, + maskObjectiveCComments: true, + preserveObjectiveCModuloExpressions: true); + var matlabSymbols = ExtractCore( + fileId, + "matlab", + matlabContent, + contentIsNormalized: true, + hasOversizeLine: false, + conflictMarkerLine: 0, + filePath, + projectRoot, + patternConfigsAlreadyLoaded: true, + cancellationToken); + var objectiveCSymbols = ExtractCore( + fileId, + "objc", + objectiveCContent, + contentIsNormalized: true, + hasOversizeLine: false, + conflictMarkerLine: 0, + filePath, + projectRoot, + patternConfigsAlreadyLoaded: true, + cancellationToken); + matlabSymbols.AddRange(objectiveCSymbols); + return matlabSymbols; + } + if (lang == "markdown") { var markdownLines = SplitContentLines(content); @@ -148,7 +185,16 @@ private static List ExtractCore( ? GetPythonModulePrefix(filePath) : null; - var structuralLines = StructuralLineMasker.MaskLines(lang, lines); + var structuralMaskLanguage = lang == "cython" ? "python" : lang; + var structuralLines = StructuralLineMasker.MaskLines(structuralMaskLanguage, lines); + if (lang is "d" or "julia" or "matlab" or "nim") + structuralLines = ScientificNativeCommentMasker.MaskBlockComments(lang, structuralLines); + var scientificBodyScannerLines = lang is "julia" or "matlab" + ? PrepareScientificBodyScannerLines(structuralLines, lang) + : null; + var matlabExplicitOuterClosureByLine = lang == "matlab" && scientificBodyScannerLines != null + ? BuildMatlabExplicitOuterClosureMap(scientificBodyScannerLines) + : null; string[]? javaScriptTypeScriptSanitizedLines = null; string[] GetJavaScriptTypeScriptSanitizedLines() => javaScriptTypeScriptSanitizedLines ??= BuildJavaScriptTypeScriptSanitizedLines(lines); @@ -853,7 +899,14 @@ bool[] GetCssQualifiedRuleAncestors() => i, absoluteStartColumn, csharpGateRawStartColumn) - : ResolveRange(rangeLines, i, pattern.BodyStyle, lang, absoluteStartColumn); + : ResolveRange( + rangeLines, + i, + pattern.BodyStyle, + lang, + absoluteStartColumn, + scientificBodyScannerLines, + matlabExplicitOuterClosureByLine); if (fortranContinuationCandidate != null) endLine = Math.Max(endLine, fortranContinuationCandidate.Value.LastConsumedLineIndex + 1); var startLine = i + 1; @@ -1556,8 +1609,10 @@ bool[] GetCssQualifiedRuleAncestors() => Name = name, Line = startLine, StartLine = startLine, - StartColumn = lang == "rust" && pattern.Kind == "function" - ? match.Groups["name"].Index + StartColumn = lang is "ada" or "cython" or "d" or "julia" or "matlab" or "nim" + ? lineOffset + match.Groups["name"].Index + : lang == "rust" && pattern.Kind == "function" + ? match.Groups["name"].Index : (csharpSingleLineCollapsedMatch ? csharpSignatureRawStartColumn : absoluteStartColumn), @@ -2154,6 +2209,11 @@ bool[] GetCssQualifiedRuleAncestors() => if (lang == "shell") ExpandShellAliasSymbols(fileId, lines, symbols, extractionState); PopulateDeclaredContainerQualifiedNames(symbols); + if (lang == "nim") + { + foreach (var symbol in symbols) + symbol.IdentityNameFolded = NimIdentifierIdentity.Fold(symbol.Name); + } return symbols; } diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Pascal.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Pascal.cs index f4a094807..6d113fe5a 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Pascal.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.Pascal.cs @@ -10,6 +10,168 @@ public static partial class SymbolExtractor private static readonly Regex PascalNestedEndBlockStartRegex = new(@"\b(?:case|try)\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); private static readonly Regex PascalRoutineStartRegex = new(@"^\s*(?:(?:class|static)\s+)?(?:procedure|function|constructor|destructor)\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); private static readonly Regex PascalRangeBoundaryRegex = new(@"^\s*(?:interface|implementation|initialization|finalization)\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaRangeDeclarationNameRegex = new( + @"^\s*(?:(?:overriding|not\s+overriding)\s+)?(?:(?:package\s+(?:body\s+)?)|(?:function|procedure)\s+(?:(?:[A-Za-z]\w*)\.)*|(?:task|protected)\s+(?:type\s+)?)(?[A-Za-z]\w*)\b", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaNamedEndRegex = new( + @"\bend\s+(?[A-Za-z]\w*(?:\.[A-Za-z]\w*)*)\s*;", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaBeginRegex = new( + @"\bbegin\b", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaUnnamedOuterEndRegex = new( + @"\bend\s*;", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaBodylessAfterIsRegex = new( + @"^(?:abstract|separate|null|new)\b", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + private static readonly Regex AdaRoutineBodySignatureRegex = new( + @"^\s*(?:(?:overriding|not\s+overriding)\s+)?(?:procedure\b[\s\S]*?\bis\b|function\b[\s\S]*?\breturn\b[\s\S]*?\bis\b)", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + + private static (int EndLine, int? BodyStartLine, int? BodyEndLine) FindAdaRange( + string[] lines, + int startIndex) + { + var declaration = AdaRangeDeclarationNameRegex.Match(lines[startIndex]); + if (!declaration.Success) + return (startIndex + 1, null, null); + + if (TryFindAdaBodylessDeclarationEnd(lines, startIndex, out var declarationEndLine)) + return (declarationEndLine, null, null); + + var declarationName = declaration.Groups["name"].Value; + int? bodyStartLine = null; + for (var i = startIndex; i < lines.Length; i++) + { + var code = MaskAdaRangeStringsAndComments(lines[i]); + if (bodyStartLine == null && AdaBeginRegex.IsMatch(code)) + bodyStartLine = i + 1; + + foreach (Match endMatch in AdaNamedEndRegex.Matches(code)) + { + var endName = endMatch.Groups["name"].Value; + var endLeaf = endName[(endName.LastIndexOf('.') + 1)..]; + if (string.Equals(endLeaf, declarationName, StringComparison.OrdinalIgnoreCase)) + return (i + 1, bodyStartLine, i + 1); + } + } + + if (bodyStartLine == null) + return (startIndex + 1, null, null); + + var beginDepth = 0; + for (var i = bodyStartLine.Value - 1; i < lines.Length; i++) + { + var code = MaskAdaRangeStringsAndComments(lines[i]); + beginDepth += AdaBeginRegex.Matches(code).Count; + foreach (Match _ in AdaUnnamedOuterEndRegex.Matches(code)) + { + if (beginDepth > 0) + beginDepth--; + if (beginDepth == 0) + return (i + 1, bodyStartLine, i + 1); + } + } + + return (lines.Length, bodyStartLine, lines.Length); + } + + private static bool TryFindAdaBodylessDeclarationEnd( + string[] lines, + int startIndex, + out int declarationEndLine) + { + declarationEndLine = startIndex + 1; + var delimiterDepth = 0; + for (var lineIndex = startIndex; lineIndex < lines.Length; lineIndex++) + { + var code = MaskAdaRangeStringsAndComments(lines[lineIndex]); + for (var index = 0; index < code.Length; index++) + { + if (code[index] is '(' or '[') + { + delimiterDepth++; + continue; + } + if (code[index] is ')' or ']') + { + delimiterDepth = Math.Max(0, delimiterDepth - 1); + continue; + } + if (delimiterDepth != 0) + continue; + + if (code[index] == ';') + { + declarationEndLine = lineIndex + 1; + return true; + } + + if (index + 2 > code.Length + || !code.AsSpan(index, 2).Equals("is", StringComparison.OrdinalIgnoreCase) + || (index > 0 && (char.IsLetterOrDigit(code[index - 1]) || code[index - 1] == '_')) + || (index + 2 < code.Length + && (char.IsLetterOrDigit(code[index + 2]) || code[index + 2] == '_'))) + { + continue; + } + + var tail = code[(index + 2)..].TrimStart(); + for (var tailLineIndex = lineIndex + 1; + tail.Length == 0 && tailLineIndex < lines.Length; + tailLineIndex++) + { + tail = MaskAdaRangeStringsAndComments(lines[tailLineIndex]).TrimStart(); + } + + return tail.StartsWith('(') || AdaBodylessAfterIsRegex.IsMatch(tail); + } + } + + return false; + } + + private static string MaskAdaRangeStringsAndComments(string line) + { + char[]? chars = null; + var inString = false; + for (var i = 0; i < line.Length; i++) + { + if (inString) + { + chars![i] = ' '; + if (line[i] != '"') + continue; + if (i + 1 < line.Length && line[i + 1] == '"') + { + chars[++i] = ' '; + continue; + } + + inString = false; + continue; + } + + if (line[i] == '"') + { + chars ??= line.ToCharArray(); + chars[i] = ' '; + inString = true; + continue; + } + + if (line[i] == '-' && i + 1 < line.Length && line[i + 1] == '-') + { + chars ??= line.ToCharArray(); + for (; i < line.Length; i++) + chars[i] = ' '; + break; + } + } + + return chars == null ? line : new string(chars); + } private static (int EndLine, int? BodyStartLine, int? BodyEndLine) FindPascalRange(string[] lines, int startIndex) { diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ScientificNative.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ScientificNative.cs new file mode 100644 index 000000000..d75148b88 --- /dev/null +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ScientificNative.cs @@ -0,0 +1,641 @@ +using System.Text.RegularExpressions; +using Regex = CodeIndex.Indexer.BoundedRegex; + +namespace CodeIndex.Indexer; + +public static partial class SymbolExtractor +{ + private static readonly Regex JuliaScientificBlockTokenRegex = new( + @"\b(?baremodule|module|mutable\s+struct|struct|abstract\s+type|primitive\s+type|function|macro|if|for|while|try|begin|let|quote|do|end)\b", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + private static readonly Regex MatlabScientificBlockTokenRegex = new( + @"\b(?function|classdef|methods|properties|events|enumeration|arguments|if|for|parfor|while|switch|try|spmd|end)\b", + RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + + private static (int EndLine, int? BodyStartLine, int? BodyEndLine) FindScientificEndRange( + string[] scannerLines, + int startIndex, + string language, + int? openingTokenLineIndex = null, + bool[]? matlabExplicitOuterClosureByLine = null) + { + var tokenRegex = language == "julia" + ? JuliaScientificBlockTokenRegex + : MatlabScientificBlockTokenRegex; + var depth = 0; + int? bodyStartLine = null; + var delimiterFrames = new Stack<(char ClosingDelimiter, int BlockDepth)>(); + + for (var lineIndex = startIndex; lineIndex < scannerLines.Length; lineIndex++) + { + var code = scannerLines[lineIndex]; + if (string.IsNullOrWhiteSpace(code)) + continue; + + var skipDeclarationToken = lineIndex == (openingTokenLineIndex ?? startIndex); + var matches = tokenRegex.Matches(code); + if (language == "matlab" + && lineIndex > startIndex + && depth == 1 + && delimiterFrames.Count == 0 + && IsMatlabPeerDeclaration( + lineIndex, + code, + matches, + matlabExplicitOuterClosureByLine)) + { + return bodyStartLine == null + ? (lineIndex, null, null) + : (lineIndex, bodyStartLine, lineIndex); + } + + if (!skipDeclarationToken) + bodyStartLine ??= lineIndex + 1; + + var delimiterScanIndex = 0; + foreach (Match match in matches) + { + ScanScientificDelimiterFrames( + code, + delimiterScanIndex, + match.Index, + depth, + delimiterFrames); + delimiterScanIndex = match.Index + match.Length; + + if (skipDeclarationToken) + { + skipDeclarationToken = false; + depth++; + continue; + } + + var keyword = match.Groups["keyword"].Value; + if (!IsScientificBlockTokenAtStatementBoundary(code, match.Index, keyword, language)) + continue; + + // In Julia, `begin` inside square-bracket indexing is a first-index sentinel, + // not a `begin ... end` block opener (`A[begin]`). + // Julia の角括弧 index 内の `begin` は先頭 index sentinel であり、 + // `begin ... end` block の開始ではない。 + if (language == "julia" + && keyword == "begin" + && delimiterFrames.TryPeek(out var indexingFrame) + && indexingFrame.ClosingDelimiter == ']') + { + continue; + } + + if (keyword.Equals("end", StringComparison.OrdinalIgnoreCase)) + { + if (delimiterFrames.TryPeek(out var delimiterFrame) + && depth <= delimiterFrame.BlockDepth) + { + continue; + } + + depth--; + if (depth == 0) + return (lineIndex + 1, bodyStartLine ?? lineIndex + 1, lineIndex + 1); + continue; + } + + // Julia comprehension clauses inside delimiters do not open `end`-terminated + // blocks. Julia の区切り記号内にある内包表記句は `end` 終端ブロックを開かない。 + if (language == "julia" + && delimiterFrames.Count > 0 + && keyword is "for" or "if") + { + continue; + } + + depth++; + } + + ScanScientificDelimiterFrames( + code, + delimiterScanIndex, + code.Length, + depth, + delimiterFrames); + } + + return bodyStartLine == null + ? (startIndex + 1, null, null) + : (scannerLines.Length, bodyStartLine, scannerLines.Length); + } + + private static (int EndLine, int? BodyStartLine, int? BodyEndLine) FindJuliaShortFunctionRange( + string[] scannerLines, + int startIndex) + { + if (!TryGetJuliaShortBlockExpressionStartLine(scannerLines, startIndex, out var blockStartLine)) + { + if (TryFindJuliaShortDelimitedExpressionEnd(scannerLines, startIndex, out var expressionEndLine)) + { + return (expressionEndLine, startIndex + 1, expressionEndLine); + } + + if (TryFindJuliaShortContinuationEnd(scannerLines, startIndex, out expressionEndLine)) + { + return (expressionEndLine, startIndex + 1, expressionEndLine); + } + + return (startIndex + 1, startIndex + 1, startIndex + 1); + } + + var blockRange = FindScientificEndRange( + scannerLines, + startIndex, + "julia", + openingTokenLineIndex: blockStartLine); + return blockRange.BodyStartLine == null + ? (startIndex + 1, startIndex + 1, startIndex + 1) + : (blockRange.EndLine, startIndex + 1, blockRange.BodyEndLine); + } + + private static bool TryGetJuliaShortBlockExpressionStartLine( + string[] scannerLines, + int startIndex, + out int blockStartLine) + { + blockStartLine = startIndex; + var startLine = scannerLines[startIndex]; + var parameterEnd = startLine.IndexOf(')'); + var assignmentIndex = parameterEnd >= 0 + ? startLine.IndexOf('=', parameterEnd + 1) + : -1; + if (assignmentIndex < 0) + return false; + + var expression = startLine[(assignmentIndex + 1)..]; + if (string.IsNullOrWhiteSpace(expression)) + { + for (var lineIndex = startIndex + 1; lineIndex < scannerLines.Length; lineIndex++) + { + expression = scannerLines[lineIndex]; + if (!string.IsNullOrWhiteSpace(expression)) + { + blockStartLine = lineIndex; + break; + } + } + } + + foreach (Match match in JuliaScientificBlockTokenRegex.Matches(expression)) + { + var keyword = match.Groups["keyword"].Value; + if (keyword == "begin" && IsJuliaSquareBracketIndexToken(expression, match.Index)) + continue; + if (IsJuliaExpressionPositionBlockOpener(expression, match.Index, keyword) + || (keyword == "for" && IsJuliaShortForBlockStart(expression, match.Index))) + return true; + } + + return false; + } + + private static bool IsJuliaSquareBracketIndexToken(string expression, int tokenIndex) + { + var delimiters = new Stack(); + for (var index = 0; index < tokenIndex; index++) + { + var closingDelimiter = expression[index] switch + { + '(' => ')', + '[' => ']', + '{' => '}', + _ => '\0', + }; + if (closingDelimiter != '\0') + { + delimiters.Push(closingDelimiter); + continue; + } + + if (delimiters.TryPeek(out var expectedClosingDelimiter) + && expression[index] == expectedClosingDelimiter) + { + delimiters.Pop(); + } + } + + return delimiters.TryPeek(out var enclosingDelimiter) + && enclosingDelimiter == ']'; + } + + private static bool TryFindJuliaShortDelimitedExpressionEnd( + string[] scannerLines, + int startIndex, + out int expressionEndLine) + { + expressionEndLine = startIndex + 1; + var startLine = scannerLines[startIndex]; + var parameterEnd = startLine.IndexOf(')'); + var assignmentIndex = parameterEnd >= 0 + ? startLine.IndexOf('=', parameterEnd + 1) + : -1; + if (assignmentIndex < 0) + return false; + + var delimiters = new Stack(); + var sawExpressionContent = false; + var sawUnclosedDelimiter = false; + for (var lineIndex = startIndex; lineIndex < scannerLines.Length; lineIndex++) + { + var line = scannerLines[lineIndex]; + var cursor = lineIndex == startIndex ? assignmentIndex + 1 : 0; + for (; cursor < line.Length; cursor++) + { + var value = line[cursor]; + if (!char.IsWhiteSpace(value)) + sawExpressionContent = true; + var closingDelimiter = value switch + { + '(' => ')', + '[' => ']', + '{' => '}', + _ => '\0', + }; + if (closingDelimiter != '\0') + { + delimiters.Push(closingDelimiter); + continue; + } + + if (delimiters.TryPeek(out var expectedClosingDelimiter) + && value == expectedClosingDelimiter) + { + delimiters.Pop(); + } + } + + if (delimiters.Count > 0) + { + sawUnclosedDelimiter = true; + continue; + } + + if (sawUnclosedDelimiter) + { + expressionEndLine = lineIndex + 1; + return true; + } + + if (sawExpressionContent) + return false; + } + + return false; + } + + private static bool TryFindJuliaShortContinuationEnd( + string[] scannerLines, + int startIndex, + out int expressionEndLine) + { + expressionEndLine = startIndex + 1; + var startLine = scannerLines[startIndex]; + var parameterEnd = startLine.IndexOf(')'); + var assignmentIndex = parameterEnd >= 0 + ? startLine.IndexOf('=', parameterEnd + 1) + : -1; + if (assignmentIndex < 0) + { + return false; + } + + var initialExpression = startLine[(assignmentIndex + 1)..]; + if (!string.IsNullOrWhiteSpace(initialExpression) + && !EndsWithJuliaContinuationOperator(initialExpression)) + { + return false; + } + + var declarationColumn = GetFirstNonWhitespaceColumn(startLine); + for (var lineIndex = startIndex + 1; lineIndex < scannerLines.Length; lineIndex++) + { + var line = scannerLines[lineIndex]; + if (string.IsNullOrWhiteSpace(line)) + continue; + + if (GetFirstNonWhitespaceColumn(line) <= declarationColumn) + return expressionEndLine > startIndex + 1; + + expressionEndLine = lineIndex + 1; + if (!EndsWithJuliaContinuationOperator(line)) + return true; + } + + return expressionEndLine > startIndex + 1; + } + + private static bool EndsWithJuliaContinuationOperator(string expression) + { + var trimmed = expression.AsSpan().TrimEnd(); + if (trimmed.IsEmpty) + return false; + + return trimmed[^1] is '+' + or '-' + or '*' + or '/' + or '\\' + or '^' + or '%' + or '&' + or '|' + or '<' + or '>' + or '=' + or '?' + or ':' + or ',' + or '÷' + or '⊻'; + } + + private static bool IsMatlabPeerDeclaration( + int lineIndex, + string code, + MatchCollection matches, + bool[]? matlabExplicitOuterClosureByLine) + { + foreach (Match match in matches) + { + var keyword = match.Groups["keyword"].Value; + if (!keyword.Equals("function", StringComparison.OrdinalIgnoreCase) + && !keyword.Equals("classdef", StringComparison.OrdinalIgnoreCase)) + { + return false; + } + + return matlabExplicitOuterClosureByLine == null + || lineIndex >= matlabExplicitOuterClosureByLine.Length + || !matlabExplicitOuterClosureByLine[lineIndex]; + } + + return false; + } + + private static bool[] BuildMatlabExplicitOuterClosureMap(string[] scannerLines) + { + var result = new bool[scannerLines.Length]; + var tokens = new List<(int LineIndex, int BalanceAfter, bool IsDeclaration)>(); + var balance = 0; + var delimiterFrames = new Stack<(char ClosingDelimiter, int BlockDepth)>(); + for (var lineIndex = 0; lineIndex < scannerLines.Length; lineIndex++) + { + var code = scannerLines[lineIndex]; + var delimiterScanIndex = 0; + foreach (Match match in MatlabScientificBlockTokenRegex.Matches(code)) + { + ScanScientificDelimiterFrames( + code, + delimiterScanIndex, + match.Index, + balance, + delimiterFrames); + delimiterScanIndex = match.Index + match.Length; + + var keyword = match.Groups["keyword"].Value; + if (!IsScientificBlockTokenAtStatementBoundary(code, match.Index, keyword, "matlab")) + continue; + + if (keyword.Equals("end", StringComparison.OrdinalIgnoreCase)) + { + if (delimiterFrames.TryPeek(out var delimiterFrame) + && balance <= delimiterFrame.BlockDepth) + { + continue; + } + + balance--; + } + else + { + balance++; + } + + tokens.Add(( + lineIndex, + balance, + keyword.Equals("function", StringComparison.OrdinalIgnoreCase) + || keyword.Equals("classdef", StringComparison.OrdinalIgnoreCase))); + } + + ScanScientificDelimiterFrames( + code, + delimiterScanIndex, + code.Length, + balance, + delimiterFrames); + } + + var suffixMinimumBalance = int.MaxValue; + for (var tokenIndex = tokens.Count - 1; tokenIndex >= 0; tokenIndex--) + { + var token = tokens[tokenIndex]; + if (token.IsDeclaration) + { + result[token.LineIndex] = + suffixMinimumBalance <= token.BalanceAfter - 2; + } + + suffixMinimumBalance = Math.Min(suffixMinimumBalance, token.BalanceAfter); + } + + return result; + } + + private static void ScanScientificDelimiterFrames( + string line, + int startIndex, + int endIndex, + int blockDepth, + Stack<(char ClosingDelimiter, int BlockDepth)> delimiterFrames) + { + for (var index = startIndex; index < endIndex; index++) + { + var closingDelimiter = line[index] switch + { + '(' => ')', + '[' => ']', + '{' => '}', + _ => '\0', + }; + if (closingDelimiter != '\0') + { + delimiterFrames.Push((closingDelimiter, blockDepth)); + continue; + } + + if (delimiterFrames.TryPeek(out var delimiterFrame) + && line[index] == delimiterFrame.ClosingDelimiter) + { + delimiterFrames.Pop(); + } + } + } + + private static int GetFirstNonWhitespaceColumn(string line) + { + for (var index = 0; index < line.Length; index++) + { + if (!char.IsWhiteSpace(line[index])) + return index; + } + + return 0; + } + + private static string[] PrepareScientificBodyScannerLines( + string[] lines, + string language) + { + var blockMaskedLines = ScientificNativeCommentMasker.MaskBlockComments(language, lines); + var scannerLines = new string[blockMaskedLines.Length]; + for (var lineIndex = 0; lineIndex < blockMaskedLines.Length; lineIndex++) + scannerLines[lineIndex] = MaskScientificBodyScanLine(blockMaskedLines[lineIndex], language); + + return scannerLines; + } + + private static string MaskScientificBodyScanLine(string line, string language) + { + var masked = ScientificNativeCommentMasker.MaskLineStringLiteralsPreservingPostfixSingleQuotes( + line, + useMatlabStringRules: language == "matlab", + useBackslashEscapes: language == "julia"); + var commentMarker = language == "julia" ? '#' : '%'; + var commentIndex = masked.IndexOf(commentMarker); + return commentIndex >= 0 ? masked[..commentIndex] : masked; + } + + private static bool IsScientificBlockTokenAtStatementBoundary( + string line, + int tokenIndex, + string keyword, + string language) + { + if (language == "julia" && IsJuliaExpressionPositionBlockOpener(line, tokenIndex, keyword)) + return true; + + for (var index = tokenIndex - 1; index >= 0; index--) + { + if (char.IsWhiteSpace(line[index])) + continue; + + if (line[index] == ';' + || (language == "matlab" && line[index] == ',')) + { + return IsScientificStatementSeparatorAtTopLevel(line, index); + } + + return language == "julia" + && (line[index] == '=' || IsJuliaMacroBlockPrefix(line, tokenIndex)); + } + + return true; + } + + private static bool IsJuliaExpressionPositionBlockOpener( + string line, + int tokenIndex, + string keyword) + { + if (keyword is "begin" or "do" or "function" or "let" or "quote" or "try" or "while") + return true; + if (keyword != "if") + return false; + + var index = tokenIndex - 1; + while (index >= 0 && char.IsWhiteSpace(line[index])) + index--; + if (index < 0 || line[index] is '(' or ',' or '=' or ';') + return true; + + var tokenEnd = index + 1; + while (index >= 0 && (char.IsLetterOrDigit(line[index]) || line[index] == '_')) + index--; + var precedingToken = line.AsSpan(index + 1, tokenEnd - index - 1); + return precedingToken.Equals("else", StringComparison.Ordinal) + || precedingToken.Equals("return", StringComparison.Ordinal); + } + + private static bool IsJuliaShortForBlockStart(string expression, int tokenIndex) + { + for (var index = 0; index < tokenIndex; index++) + { + if (!char.IsWhiteSpace(expression[index]) && expression[index] != '(') + return false; + } + + return true; + } + + private static bool IsScientificStatementSeparatorAtTopLevel(string line, int separatorIndex) + { + var parentheses = 0; + var brackets = 0; + var braces = 0; + for (var index = 0; index < separatorIndex; index++) + { + switch (line[index]) + { + case '(': + parentheses++; + break; + case ')': + parentheses = Math.Max(0, parentheses - 1); + break; + case '[': + brackets++; + break; + case ']': + brackets = Math.Max(0, brackets - 1); + break; + case '{': + braces++; + break; + case '}': + braces = Math.Max(0, braces - 1); + break; + } + } + + return parentheses == 0 && brackets == 0 && braces == 0; + } + + private static bool IsJuliaMacroBlockPrefix(string line, int tokenIndex) + { + var index = tokenIndex - 1; + while (index >= 0 && char.IsWhiteSpace(line[index])) + index--; + if (index < 0) + return false; + + var tokenEnd = index + 1; + while (index >= 0 + && (char.IsLetterOrDigit(line[index]) || line[index] is '_' or '.' or '@')) + { + index--; + } + + var token = line.AsSpan(index + 1, tokenEnd - index - 1); + var atIndex = token.LastIndexOf('@'); + if (atIndex < 0 || atIndex == token.Length - 1) + return false; + + for (var tokenIndexOffset = 0; tokenIndexOffset < token.Length; tokenIndexOffset++) + { + if (!(char.IsLetterOrDigit(token[tokenIndexOffset]) + || token[tokenIndexOffset] is '_' or '.' or '@')) + { + return false; + } + } + + return true; + } +} diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs index b7e79e470..1b0c79857 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs @@ -24,6 +24,7 @@ public static partial class SymbolExtractor public const int FunctionalLanguageContractVersion = 3; public const int DynamicLanguageContractVersion = 2; public const int SystemsLanguageContractVersion = 2; + public const int ScientificNativeGraphContractVersion = 4; public const int RepositoryMetadataContractVersion = 2; public const int ApplicationManifestContractVersion = 3; private static readonly string[] AdditionalSymbolLanguages = @@ -43,6 +44,7 @@ public static partial class SymbolExtractor "solidity", "solution", "cuda", + "ambiguous_m", "dependency_manifest", "dependency_lock", "jsonl", @@ -56,6 +58,9 @@ public static partial class SymbolExtractor private const int SymbolListInitialCapacityLineThreshold = 128; private const int SymbolListInitialCapacityMax = 1024; + private const string JuliaIdentifierPattern = @"[\p{L}_]\w*"; + private const string JuliaQualifiedCallableIdentifierPattern = + JuliaIdentifierPattern + @"(?:\." + JuliaIdentifierPattern + @")*!?"; private static string[] SplitContentLines(string content) => content.IndexOf('\n', StringComparison.Ordinal) < 0 ? [content] : content.Split('\n'); @@ -88,8 +93,8 @@ public static int GetContractVersion(string? lang) "sass" or "stylus" => StyleAndXamlContractVersion, "xml" => XmlContractVersion, "clojure" or "erlang" or "ocaml" or "raku" => FunctionalLanguageContractVersion, - "crystal" or "groovy" or "julia" or "tcl" => DynamicLanguageContractVersion, - "ada" or "d" or "nim" => SystemsLanguageContractVersion, + "crystal" or "groovy" or "tcl" => DynamicLanguageContractVersion, + "ada" or "ambiguous_m" or "cython" or "d" or "julia" or "matlab" or "nim" or "objc" => ScientificNativeGraphContractVersion, "config" or "dockerignore" or "editorconfig" or "gitattributes" or "gitignore" or "jsonl" or "toml" => RepositoryMetadataContractVersion, "app_manifest" => ApplicationManifestContractVersion, "cmake" or "dependency_lock" or "dependency_manifest" or "graphql" or "html" or "json" or "justfile" or "markdown" or "msbuild" or "solution" or "yaml" => ExpandedLanguageContractVersion, @@ -490,8 +495,11 @@ private enum BodyStyle RubyEnd, FortranEnd, ElixirEnd, + ScientificEnd, + JuliaShortFunction, VisualBasicEnd, PascalEnd, + AdaEnd, SmalltalkMethod, SqlProcBody, } @@ -1642,12 +1650,12 @@ private enum JavaScriptTypeScriptFunctionHeaderConsumeResult ], ["julia"] = [ - new("namespace", new Regex(@"^\s*(?:baremodule|module)\s+(?[A-Za-z_]\w*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ElixirEnd), - new("struct", new Regex(@"^\s*(?:mutable\s+)?struct\s+(?[A-Za-z_]\w*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ElixirEnd), - new("type", new Regex(@"^\s*(?:abstract|primitive)\s+type\s+(?[A-Za-z_]\w*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ElixirEnd), - new("function", new Regex(@"^\s*function\s+(?[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)?)\s*(?:\(|\{)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ElixirEnd), - new("function", new Regex(@"^\s*macro\s+(?[A-Za-z_]\w*)\s*(?:\(|$)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ElixirEnd), - new("function", new Regex(@"^\s*(?[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)?)\s*\([^)\r\n]*\)\s*=", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.None), + new("namespace", new Regex(@"^\s*(?:baremodule|module)\s+(?[A-Za-z_]\w*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ScientificEnd), + new("struct", new Regex(@"^\s*(?:mutable\s+)?struct\s+(?[A-Za-z_]\w*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ScientificEnd), + new("type", new Regex(@"^\s*(?:abstract|primitive)\s+type\s+(?[A-Za-z_]\w*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ScientificEnd), + new("function", new Regex(@"^\s*function\s+(?" + JuliaQualifiedCallableIdentifierPattern + @")\s*(?:\(|\{)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ScientificEnd), + new("function", new Regex(@"^\s*macro\s+(?[A-Za-z_]\w*)\s*(?:\(|$)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ScientificEnd), + new("function", new Regex(@"^\s*(?" + JuliaQualifiedCallableIdentifierPattern + @")\s*\([^)\r\n]*\)\s*(?:where\s*(?:\{[^}\r\n]*\}|[A-Za-z_]\w*)\s*)?=", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.JuliaShortFunction), new("property", new Regex(@"^\s*const\s+(?[A-Z_]\w*)\s*=", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.None), new("import", new Regex(@"^\s*(?:using|import)\s+(?[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.None), ], @@ -1661,10 +1669,10 @@ private enum JavaScriptTypeScriptFunctionHeaderConsumeResult ], ["ada"] = [ - new("namespace", new Regex(@"^\s*package\s+(?:body\s+)?(?[A-Za-z]\w*(?:\.[A-Za-z]\w*)*)\s+is\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.PascalEnd), + new("namespace", new Regex(@"^\s*package\s+(?:body\s+)?(?[A-Za-z]\w*(?:\.[A-Za-z]\w*)*)\s+is\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.AdaEnd), new("type", new Regex(@"^\s*(?:subtype|type)\s+(?[A-Za-z]\w*)\s+is\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.None), - new("type", new Regex(@"^\s*(?:task|protected)\s+(?:type\s+)?(?[A-Za-z]\w*)\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.PascalEnd), - new("function", new Regex(@"^\s*(?:(?:overriding|not\s+overriding)\s+)?(?:function|procedure)\s+(?:(?:[A-Za-z]\w*)\.)*(?[A-Za-z]\w*)\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.PascalEnd), + new("type", new Regex(@"^\s*(?:task|protected)\s+(?:type\s+)?(?[A-Za-z]\w*)\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.AdaEnd), + new("function", new Regex(@"^\s*(?:(?:overriding|not\s+overriding)\s+)?(?:function|procedure)\s+(?:(?:[A-Za-z]\w*)\.)*(?[A-Za-z]\w*)\b", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.AdaEnd), new("import", new Regex(@"^\s*with\s+(?[A-Za-z]\w*(?:\.[A-Za-z]\w*)*)\s*;", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant), BodyStyle.None), ], ["d"] = @@ -1714,8 +1722,8 @@ private enum JavaScriptTypeScriptFunctionHeaderConsumeResult ], ["matlab"] = [ - new("class", new Regex(@"^\s*classdef\s*(?:\([^)]*\)\s*)?(?[A-Za-z]\w*)\b", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.None), - new("function", new Regex(@"^\s*function\s+(?:(?:\[[^\]]+\]|[A-Za-z]\w*)\s*=\s*)?(?[A-Za-z]\w*)\s*(?:\(|$)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.None), + new("class", new Regex(@"^\s*classdef\s*(?:\([^)]*\)\s*)?(?[A-Za-z]\w*)\b", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ScientificEnd), + new("function", new Regex(@"^\s*function\s+(?:(?:\[[^\]]+\]|[A-Za-z]\w*)\s*=\s*)?(?[A-Za-z]\w*)\s*(?:\(|$)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.ScientificEnd), new("import", new Regex(@"^\s*import\s+(?[A-Za-z]\w*(?:\.[A-Za-z*]\w*)*)", RegexOptions.Compiled | RegexOptions.CultureInvariant), BodyStyle.None), ], ["prolog"] = @@ -4763,7 +4771,8 @@ private static void RemoveTrailingSameNameDeclarationOnlyFunctions( var signature = prior.Signature?.TrimStart(); if (signature != null && (signature.StartsWith("declare ", StringComparison.Ordinal) - || CSharpPartialFunctionDeclarationSignatureRegex.IsMatch(signature))) + || CSharpPartialFunctionDeclarationSignatureRegex.IsMatch(signature) + || IsAdaForwardDeclarationPair(signature, symbol.Signature))) { break; } @@ -4773,6 +4782,28 @@ private static void RemoveTrailingSameNameDeclarationOnlyFunctions( } } + private static bool IsAdaForwardDeclarationPair( + string declarationSignature, + string? implementationSignature) + { + if (implementationSignature == null + || !AdaRoutineBodySignatureRegex.IsMatch(implementationSignature)) + { + return false; + } + + var trimmed = declarationSignature.Trim(); + if (!trimmed.EndsWith(';')) + return false; + + return trimmed.StartsWith("procedure ", StringComparison.OrdinalIgnoreCase) + || trimmed.StartsWith("function ", StringComparison.OrdinalIgnoreCase) + || trimmed.StartsWith("overriding procedure ", StringComparison.OrdinalIgnoreCase) + || trimmed.StartsWith("overriding function ", StringComparison.OrdinalIgnoreCase) + || trimmed.StartsWith("not overriding procedure ", StringComparison.OrdinalIgnoreCase) + || trimmed.StartsWith("not overriding function ", StringComparison.OrdinalIgnoreCase); + } + // Some compact same-line C# fixtures can legitimately contain two distinct siblings with // the same short signature on the same physical line // (`Child { } } public partial class Child { }`). Allow as many identical rows as the raw @@ -4795,9 +4826,16 @@ private static bool HasRemainingSameLineSignatureOccurrence(SymbolRecord symbol, private static (int EndLine, int? BodyStartLine, int? BodyEndLine) ResolveRange(string[] lines, int startIndex, BodyStyle bodyStyle) => - ResolveRange(lines, startIndex, bodyStyle, null, 0); + ResolveRange(lines, startIndex, bodyStyle, null, 0, null, null); - private static (int EndLine, int? BodyStartLine, int? BodyEndLine) ResolveRange(string[] lines, int startIndex, BodyStyle bodyStyle, string? lang = null, int startColumn = 0) + private static (int EndLine, int? BodyStartLine, int? BodyEndLine) ResolveRange( + string[] lines, + int startIndex, + BodyStyle bodyStyle, + string? lang = null, + int startColumn = 0, + string[]? scientificBodyScannerLines = null, + bool[]? matlabExplicitOuterClosureByLine = null) { return bodyStyle switch { @@ -4810,8 +4848,17 @@ private static (int EndLine, int? BodyStartLine, int? BodyEndLine) ResolveRange( BodyStyle.RubyEnd => FindRubyRange(lines, startIndex), BodyStyle.FortranEnd => FindFortranRange(lines, startIndex), BodyStyle.ElixirEnd => FindElixirRange(lines, startIndex), + BodyStyle.ScientificEnd when lang is "julia" or "matlab" => FindScientificEndRange( + scientificBodyScannerLines ?? PrepareScientificBodyScannerLines(lines, lang), + startIndex, + lang, + matlabExplicitOuterClosureByLine: matlabExplicitOuterClosureByLine), + BodyStyle.JuliaShortFunction when lang == "julia" => FindJuliaShortFunctionRange( + scientificBodyScannerLines ?? PrepareScientificBodyScannerLines(lines, lang), + startIndex), BodyStyle.VisualBasicEnd => FindVisualBasicRange(lines, startIndex), BodyStyle.PascalEnd => FindPascalRange(lines, startIndex), + BodyStyle.AdaEnd => FindAdaRange(lines, startIndex), BodyStyle.SmalltalkMethod => FindSmalltalkMethodRange(lines, startIndex), BodyStyle.SqlProcBody => FindSqlProcBodyRange(lines, startIndex), _ => (startIndex + 1, null, null), diff --git a/src/CodeIndex/Models/ReferenceRecord.cs b/src/CodeIndex/Models/ReferenceRecord.cs index 872598609..f89e34dae 100644 --- a/src/CodeIndex/Models/ReferenceRecord.cs +++ b/src/CodeIndex/Models/ReferenceRecord.cs @@ -1,3 +1,5 @@ +using System.Text.Json.Serialization; + namespace CodeIndex.Models; /// @@ -12,6 +14,10 @@ public class ReferenceRecord /// Referenced symbol name / 参照先シンボル名 public string SymbolName { get; set; } = string.Empty; + /// Language-specific persisted identity key when generic folding is insufficient / 一般的なfoldでは不十分な場合の言語固有identity key + [JsonInclude] + internal string? IdentitySymbolNameFolded { get; set; } + /// Reference kind such as call or instantiate / 参照種別 public string ReferenceKind { get; set; } = string.Empty; @@ -30,6 +36,10 @@ public class ReferenceRecord /// Enclosing symbol name when known / 親シンボル名 public string? ContainerName { get; set; } + /// Language-specific persisted container identity key / 言語固有の永続化container identity key + [JsonInclude] + internal string? IdentityContainerNameFolded { get; set; } + /// /// Receiver/type qualifier immediately before the referenced name when it is a stable /// type-like identifier (for example FileShare in FileShare.ReadWrite). @@ -37,6 +47,10 @@ public class ReferenceRecord /// public string? TargetQualifier { get; set; } + /// True when a language-specific receiver explicitly denotes the current container / 言語固有receiverが現在のcontainerを明示する場合はtrue + [JsonInclude] + internal bool SuppressInferredTargetQualifier { get; set; } + /// True when the enclosing symbol references itself / 親シンボル自身への参照なら true public bool IsSelfReference { get; set; } diff --git a/src/CodeIndex/Models/SymbolRecord.cs b/src/CodeIndex/Models/SymbolRecord.cs index 3dd1e6d6f..0f7847854 100644 --- a/src/CodeIndex/Models/SymbolRecord.cs +++ b/src/CodeIndex/Models/SymbolRecord.cs @@ -1,3 +1,5 @@ +using System.Text.Json.Serialization; + namespace CodeIndex.Models; /// @@ -21,6 +23,10 @@ public class SymbolRecord /// Symbol name / シンボル名 public string Name { get; set; } = string.Empty; + /// Language-specific persisted identity key when generic folding is insufficient / 一般的なfoldでは不十分な場合の言語固有identity key + [JsonInclude] + internal string? IdentityNameFolded { get; set; } + /// Line number (1-based) / 行番号（1始まり） public int Line { get; set; } diff --git a/tests/CodeIndex.Tests/DatabaseTests.cs b/tests/CodeIndex.Tests/DatabaseTests.cs index d39240509..b609d9151 100644 --- a/tests/CodeIndex.Tests/DatabaseTests.cs +++ b/tests/CodeIndex.Tests/DatabaseTests.cs @@ -637,6 +637,779 @@ WHERE reference.id IS NULL OR symbol.id IS NULL } } + [Fact] + public void ReferenceGraph_AmbiguousMResolvesAgainstDefinitiveDialects_Issue4738() + { + var callerId = UpsertTestFileWithLanguage("src/caller.m", "ambiguous_m", "ambiguous-caller"); + var unresolvedTargetId = UpsertTestFileWithLanguage( + "src/unresolved-target.m", + "ambiguous_m", + "ambiguous-target"); + _writer.InsertSymbols([ + new SymbolRecord { FileId = unresolvedTargetId, Kind = "function", Name = "MatlabTarget", Line = 1 }, + new SymbolRecord { FileId = unresolvedTargetId, Kind = "function", Name = "ObjectiveCTarget", Line = 2 }, + ]); + _writer.InsertReferences([ + new ReferenceRecord + { + FileId = callerId, + SymbolName = "MatlabTarget", + ReferenceKind = "call", + Line = 1, + Column = 1, + Context = "MatlabTarget();", + }, + new ReferenceRecord + { + FileId = callerId, + SymbolName = "ObjectiveCTarget", + ReferenceKind = "call", + Line = 2, + Column = 1, + Context = "ObjectiveCTarget();", + }, + ], refreshMutualRecursionFlags: false); + _writer.RefreshMutualRecursionFlags(); + Assert.Equal(2, ExecuteScalarLong($""" + SELECT COUNT(*) + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + AND resolution_state = 'unresolved' + """)); + + using (var scope = _writer.BeginReferenceGraphRefreshScope()) + { + using var transaction = _writer.BeginTransaction(); + var matlabTargetId = _writer.InsertNewFile(new FileRecord + { + Path = "src/matlab-target.m", + Lang = "matlab", + Size = 32, + Lines = 2, + Modified = new DateTime(2026, 7, 24, 0, 0, 0, DateTimeKind.Utc), + Checksum = "matlab-target", + }); + var objectiveCTargetId = _writer.InsertNewFile(new FileRecord + { + Path = "src/objective-c-target.m", + Lang = "objc", + Size = 32, + Lines = 2, + Modified = new DateTime(2026, 7, 24, 0, 0, 0, DateTimeKind.Utc), + Checksum = "objective-c-target", + }); + _writer.InsertSymbols([ + new SymbolRecord { FileId = matlabTargetId, Kind = "function", Name = "MatlabTarget", Line = 1 }, + new SymbolRecord { FileId = objectiveCTargetId, Kind = "function", Name = "ObjectiveCTarget", Line = 1 }, + ]); + transaction.Commit(); + _writer.RefreshMutualRecursionFlags(); + } + + Assert.Equal(2, ExecuteScalarLong($""" + SELECT COUNT(*) + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + AND resolution_state = 'resolved' + """)); + + _writer.RefreshMutualRecursionFlags(); + Assert.Equal(2, ExecuteScalarLong($""" + SELECT COUNT(*) + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + AND resolution_state = 'resolved' + """)); + + _writer.MarkGraphReady(); + _writer.MarkReferenceIdentityContractReady(); + var reader = new DbReader(_db.Connection); + foreach (var (name, language, path) in new[] + { + ("MatlabTarget", "matlab", "src/matlab-target.m"), + ("ObjectiveCTarget", "objc", "src/objective-c-target.m"), + }) + { + var symbolId = ExecuteScalarLong($""" + SELECT symbol.id + FROM symbols AS symbol + JOIN files AS file ON file.id = symbol.file_id + WHERE symbol.name = '{name}' + AND file.path = '{path}' + """); + var definition = new DefinitionResult + { + SymbolId = symbolId, + Path = path, + Lang = language, + Kind = "function", + Name = name, + Line = 1, + StartLine = 1, + EndLine = 1, + }; + Assert.Equal(1, ExecuteScalarLong($""" + SELECT COUNT(*) + FROM symbol_reference_candidates + WHERE symbol_id = {symbolId.ToString(CultureInfo.InvariantCulture)} + """)); + + var identityScopedReference = Assert.Single( + reader.GetReferencesForDefinition(definition, limit: 20)); + Assert.Equal("src/caller.m", identityScopedReference.Path); + Assert.Equal("ambiguous_m", identityScopedReference.Lang); + Assert.Equal(name, identityScopedReference.SymbolName); + } + } + + [Fact] + public void ReferenceGraph_AmbiguousMUniqueFallbackRequiresUnionWideUniqueness_Issue4738() + { + var callerId = UpsertTestFileWithLanguage( + "src/caller.m", + "ambiguous_m", + "ambiguous-union-caller"); + var objectiveCTargetId = UpsertTestFileWithLanguage( + "src/objective-c-target.m", + "objc", + "ambiguous-union-objc"); + _writer.InsertSymbols([ + new SymbolRecord { FileId = objectiveCTargetId, Kind = "function", Name = "Foo", Line = 1 }, + ]); + _writer.InsertReferences([ + new ReferenceRecord + { + FileId = callerId, + SymbolName = "Foo", + ReferenceKind = "call", + Line = 1, + Column = 1, + Context = "Foo();", + }, + ], refreshMutualRecursionFlags: false); + + _writer.RefreshMutualRecursionFlags(); + + Assert.Equal("resolved", ReadReferenceResolutionState(callerId)); + + using (var scope = _writer.BeginReferenceGraphRefreshScope()) + { + using var transaction = _writer.BeginTransaction(); + var firstMatlabTargetId = _writer.InsertNewFile(new FileRecord + { + Path = "src/first-matlab-target.m", + Lang = "matlab", + Size = 32, + Lines = 1, + Modified = new DateTime(2026, 7, 24, 0, 0, 0, DateTimeKind.Utc), + Checksum = "ambiguous-union-matlab-first", + }); + var secondMatlabTargetId = _writer.InsertNewFile(new FileRecord + { + Path = "src/second-matlab-target.m", + Lang = "matlab", + Size = 32, + Lines = 1, + Modified = new DateTime(2026, 7, 24, 0, 0, 0, DateTimeKind.Utc), + Checksum = "ambiguous-union-matlab-second", + }); + _writer.InsertSymbols([ + new SymbolRecord { FileId = firstMatlabTargetId, Kind = "function", Name = "Foo", Line = 1 }, + new SymbolRecord { FileId = secondMatlabTargetId, Kind = "function", Name = "Foo", Line = 1 }, + ]); + transaction.Commit(); + _writer.RefreshMutualRecursionFlags(); + } + + Assert.Equal("unresolved", ReadReferenceResolutionState(callerId)); + Assert.Equal(0, ExecuteScalarLong($""" + SELECT COUNT(*) + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + AND target_symbol_id IS NOT NULL + """)); + Assert.Equal(0, ExecuteScalarLong($""" + SELECT COUNT(*) + FROM symbol_reference_candidates AS candidate + JOIN symbol_references AS reference ON reference.id = candidate.reference_id + WHERE reference.file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + """)); + } + + [Fact] + public void ReferenceGraph_ScientificQualifiersUseModuleEvidenceOrSafeUniqueFallback_Issue4738() + { + var dCallerId = UpsertTestFileWithLanguage("src/child.d", "d", "qualified-d-caller"); + var dTargetId = UpsertTestFileWithLanguage("src/pkg/base.d", "d", "qualified-d-target"); + var cythonCallerId = UpsertTestFileWithLanguage( + "src/child.pyx", + "cython", + "qualified-cython-caller"); + var cythonTargetId = UpsertTestFileWithLanguage( + "src/pkg/base.pyx", + "cython", + "qualified-cython-target"); + var adaCallerId = UpsertTestFileWithLanguage("src/main.adb", "ada", "qualified-ada-caller"); + var adaP1Id = UpsertTestFileWithLanguage("src/p1.adb", "ada", "qualified-ada-p1"); + var adaP2Id = UpsertTestFileWithLanguage("src/p2.adb", "ada", "qualified-ada-p2"); + _writer.InsertSymbols([ + new SymbolRecord { FileId = dTargetId, Kind = "namespace", Name = "pkg.base", Line = 1 }, + new SymbolRecord { FileId = dTargetId, Kind = "class", Name = "Base", Line = 2 }, + new SymbolRecord { FileId = cythonTargetId, Kind = "class", Name = "NativeBase", Line = 1 }, + new SymbolRecord { FileId = adaP1Id, Kind = "namespace", Name = "P1", Line = 1 }, + new SymbolRecord { FileId = adaP1Id, Kind = "function", Name = "Flush", Line = 2 }, + new SymbolRecord { FileId = adaP2Id, Kind = "namespace", Name = "P2", Line = 1 }, + new SymbolRecord { FileId = adaP2Id, Kind = "function", Name = "Flush", Line = 2 }, + ]); + _writer.InsertReferences([ + new ReferenceRecord + { + FileId = dCallerId, + SymbolName = "Base", + TargetQualifier = "pkg", + ReferenceKind = "type_reference", + Line = 1, + Column = 1, + Context = "class Child : pkg.Base {}", + }, + new ReferenceRecord + { + FileId = cythonCallerId, + SymbolName = "NativeBase", + TargetQualifier = "pkg", + ReferenceKind = "type_reference", + Line = 1, + Column = 1, + Context = "cdef class Child(pkg.NativeBase):", + }, + new ReferenceRecord + { + FileId = adaCallerId, + SymbolName = "Flush", + TargetQualifier = "P1", + ReferenceKind = "call", + Line = 1, + Column = 1, + Context = "P1.Flush;", + }, + ], refreshMutualRecursionFlags: false); + + _writer.RefreshMutualRecursionFlags(); + + AssertResolvedTo(dCallerId, dTargetId, "Base"); + AssertResolvedTo(cythonCallerId, cythonTargetId, "NativeBase"); + AssertResolvedTo(adaCallerId, adaP1Id, "Flush"); + + void AssertResolvedTo(long callerId, long targetFileId, string symbolName) + { + var expectedTargetId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {targetFileId.ToString(CultureInfo.InvariantCulture)} + AND name = '{symbolName}' + """); + Assert.Equal(expectedTargetId, ExecuteScalarLong($""" + SELECT target_symbol_id + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + """)); + Assert.Equal("resolved", ReadReferenceResolutionState(callerId)); + } + } + + [Fact] + public void ReferenceGraph_ModuleEvidenceDoesNotBroadenCSharpQualifiedCandidates_Issue4738() + { + var callerId = UpsertTestFileWithLanguage( + "src/caller.cs", + "csharp", + "qualified-csharp-caller"); + var targetId = UpsertTestFileWithLanguage( + "src/targets.cs", + "csharp", + "qualified-csharp-target"); + _writer.InsertSymbols([ + new SymbolRecord + { + FileId = targetId, + Kind = "namespace", + Name = "A", + Line = 1, + }, + new SymbolRecord + { + FileId = targetId, + Kind = "namespace", + Name = "B", + Line = 2, + }, + new SymbolRecord + { + FileId = targetId, + Kind = "class", + Name = "Thing", + ContainerName = "A", + Line = 3, + }, + new SymbolRecord + { + FileId = targetId, + Kind = "class", + Name = "Thing", + ContainerName = "B", + Line = 4, + }, + ]); + _writer.InsertReferences([ + new ReferenceRecord + { + FileId = callerId, + SymbolName = "Thing", + TargetQualifier = "A", + ReferenceKind = "type_reference", + Line = 1, + Column = 1, + Context = "A.Thing value;", + }, + ], refreshMutualRecursionFlags: false); + + _writer.RefreshMutualRecursionFlags(); + + var expectedTargetId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {targetId.ToString(CultureInfo.InvariantCulture)} + AND name = 'Thing' + AND container_name = 'A' + """); + Assert.Equal(expectedTargetId, ExecuteScalarLong($""" + SELECT target_symbol_id + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + """)); + Assert.Equal("resolved", ReadReferenceResolutionState(callerId)); + } + + [Fact] + public void ReferenceGraph_ExtractedScientificQualifierResolvesDuplicateLeaf_Issue4738() + { + const string callerContent = "void Run() { p1.Flush(); }\n"; + var callerId = UpsertTestFileWithLanguage( + "src/main.d", + "d", + "qualified-d-call-caller"); + var p1Id = UpsertTestFileWithLanguage( + "src/p1.d", + "d", + "qualified-d-call-p1"); + var p2Id = UpsertTestFileWithLanguage( + "src/p2.d", + "d", + "qualified-d-call-p2"); + var callerSymbols = SymbolExtractor.Extract(callerId, "d", callerContent); + _writer.InsertSymbols([ + .. callerSymbols, + new SymbolRecord { FileId = p1Id, Kind = "namespace", Name = "p1", Line = 1 }, + new SymbolRecord { FileId = p1Id, Kind = "function", Name = "Flush", Line = 2 }, + new SymbolRecord { FileId = p2Id, Kind = "namespace", Name = "p2", Line = 1 }, + new SymbolRecord { FileId = p2Id, Kind = "function", Name = "Flush", Line = 2 }, + ]); + var callerReferences = ReferenceExtractor.Extract( + callerId, + "d", + callerContent, + callerSymbols); + _writer.InsertReferences(callerReferences, refreshMutualRecursionFlags: false); + + _writer.RefreshMutualRecursionFlags(); + + var call = Assert.Single(callerReferences, reference => + reference.SymbolName == "Flush" && reference.ReferenceKind == "call"); + Assert.Equal("p1", call.TargetQualifier); + var expectedTargetId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {p1Id.ToString(CultureInfo.InvariantCulture)} + AND name = 'Flush' + """); + Assert.Equal(expectedTargetId, ExecuteScalarLong($""" + SELECT target_symbol_id + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + AND symbol_name = 'Flush' + """)); + Assert.Equal("resolved", ReadReferenceResolutionState(callerId)); + } + + [Fact] + public void ReferenceGraph_JuliaQualifierSelectsContainerWithinSharedModuleFile_Issue4738() + { + const string targetContent = """ + module A + function foo() + end + macro trace(value) + value + end + end + module B + function foo() + end + macro trace(value) + value + end + end + """; + const string callerContent = """ + function run() + A.foo() + A.@trace 1 + end + """; + var targetId = UpsertTestFileWithLanguage( + "src/modules.jl", + "julia", + "qualified-julia-shared-module-target"); + var callerId = UpsertTestFileWithLanguage( + "src/caller.jl", + "julia", + "qualified-julia-shared-module-caller"); + var targetSymbols = SymbolExtractor.Extract(targetId, "julia", targetContent); + var callerSymbols = SymbolExtractor.Extract(callerId, "julia", callerContent); + _writer.InsertSymbols([.. targetSymbols, .. callerSymbols]); + var callerReferences = ReferenceExtractor.Extract( + callerId, + "julia", + callerContent, + callerSymbols); + _writer.InsertReferences(callerReferences, refreshMutualRecursionFlags: false); + + _writer.RefreshMutualRecursionFlags(); + + foreach (var name in new[] { "foo", "trace" }) + { + var reference = Assert.Single(callerReferences, candidate => + candidate.SymbolName == name && candidate.ReferenceKind == "call"); + Assert.Equal("A", reference.TargetQualifier); + var expectedTargetId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {targetId.ToString(CultureInfo.InvariantCulture)} + AND name = '{name}' + AND container_name = 'A' + """); + Assert.Equal(expectedTargetId, ExecuteScalarLong($""" + SELECT target_symbol_id + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + AND symbol_name = '{name}' + """)); + Assert.Equal("resolved", ExecuteScalarString($""" + SELECT resolution_state + FROM symbol_references + WHERE file_id = {callerId.ToString(CultureInfo.InvariantCulture)} + AND symbol_name = '{name}' + """)); + } + } + + [Fact] + public void ReferenceGraph_CythonSelfReceiverResolvesSameClassDuplicateLeaf_Issue4738() + { + const string content = """ + cdef class A: + def helper(self): + pass + def run(self): + self.helper() + cdef class B: + def helper(self): + pass + """; + var fileId = UpsertTestFileWithLanguage( + "src/workers.pyx", + "cython", + "cython-self-receiver"); + var symbols = SymbolExtractor.Extract(fileId, "cython", content); + _writer.InsertSymbols(symbols); + var references = ReferenceExtractor.Extract(fileId, "cython", content, symbols); + _writer.InsertReferences(references, refreshMutualRecursionFlags: false); + + _writer.RefreshMutualRecursionFlags(); + + var helperReference = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Null(helperReference.TargetQualifier); + var expectedTargetId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND name = 'helper' + AND container_name = 'A' + """); + Assert.Equal(expectedTargetId, ExecuteScalarLong($""" + SELECT target_symbol_id + FROM symbol_references + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND symbol_name = 'helper' + """)); + Assert.Equal("resolved", ReadReferenceResolutionState(fileId)); + } + + [Fact] + public void ReferenceGraph_NimStyleInsensitiveIdentityResolvesAndSearches_Issue4738() + { + const string content = """ + proc myProc() = discard + proc RunGraph() = + my_proc() + """; + var fileId = UpsertTestFileWithLanguage( + "src/style.nim", + "nim", + "nim-style-insensitive"); + _writer.InsertChunks([ + new() + { + FileId = fileId, + ChunkIndex = 0, + StartLine = 1, + EndLine = 3, + Content = content, + }, + ]); + var symbols = SymbolExtractor.Extract(fileId, "nim", content); + _writer.InsertSymbols(symbols); + var references = ReferenceExtractor.Extract(fileId, "nim", content, symbols); + _writer.InsertReferences(references, refreshMutualRecursionFlags: false); + + _writer.RefreshMutualRecursionFlags(); + Assert.True(_writer.MarkFoldReady( + stampCurrentSymbolExtractorVersions: true, + symbolExtractorLanguagesToStamp: ["nim"])); + _writer.MarkGraphReady(); + + var expectedTargetId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND name = 'myProc' + """); + Assert.Equal(expectedTargetId, ExecuteScalarLong($""" + SELECT target_symbol_id + FROM symbol_references + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND symbol_name = 'my_proc' + """)); + Assert.Equal("resolved", ReadReferenceResolutionState(fileId)); + var expectedSourceId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND name = 'RunGraph' + """); + Assert.Equal(expectedSourceId, ExecuteScalarLong($""" + SELECT source_symbol_id + FROM symbol_references + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND symbol_name = 'my_proc' + """)); + Assert.Equal("myproc", ExecuteScalarString($""" + SELECT name_folded + FROM symbols + WHERE id = {expectedTargetId.ToString(CultureInfo.InvariantCulture)} + """)); + Assert.Equal("myproc", ExecuteScalarString($""" + SELECT symbol_name_folded + FROM symbol_references + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND symbol_name = 'my_proc' + """)); + Assert.Equal("Myproc", DbReader.FoldNameForLanguage("My_proc", "nim")); + Assert.Equal(0, ExecuteScalarLong(""" + SELECT COUNT(*) + FROM symbols + WHERE name_folded = 'Myproc' + """)); + + var reader = new DbReader(_db.Connection); + Assert.Single(reader.SearchSymbols("my_proc", lang: "nim", exact: true)); + Assert.Single(reader.SearchReferences("myProc", lang: "nim", exact: true)); + Assert.Empty(reader.SearchSymbols("My_proc", lang: "nim", exact: true)); + Assert.Empty(reader.SearchReferences("MyProc", lang: "nim", exact: true)); + + Assert.Single(reader.SearchSymbols("my_proc", exact: true)); + Assert.Equal(1, reader.CountSearchSymbols("my_proc", exact: true)); + Assert.Equal(1, reader.CountDefinitionsTotal("my_proc", exact: true).Count); + Assert.Single(reader.SearchReferences("my_proc", exact: true)); + Assert.Equal(1, reader.CountSearchReferences("my_proc", exact: true)); + Assert.Equal(1, reader.CountSearchReferencesTotal("my_proc", exact: true).Count); + Assert.Single(reader.GetCallers("my_proc", exact: true)); + Assert.Equal(1, reader.CountCallers("my_proc", exact: true)); + Assert.Equal(1, reader.CountCallersTotal("my_proc", exact: true).Count); + Assert.Single(reader.GetCallees("Run_Graph", exact: true)); + Assert.Equal(1, reader.CountCallees("Run_Graph", exact: true)); + Assert.Equal(1, reader.CountCalleesTotal("Run_Graph", exact: true).Count); + Assert.Empty(reader.SearchSymbols("My_proc", exact: true)); + Assert.Empty(reader.SearchReferences("My_proc", exact: true)); + Assert.Empty(reader.GetCallers("My_proc", exact: true)); + Assert.Empty(reader.GetCallees("run_graph", exact: true)); + } + + [Fact] + public void ReferenceGraph_NimBackfillRewritesLegacyKeysAndRefreshesCandidates_Issue4738() + { + var fileId = UpsertTestFileWithLanguage( + "src/legacy-style.nim", + "nim", + "nim-legacy-style"); + _writer.InsertSymbols([ + new SymbolRecord + { + FileId = fileId, + Kind = "function", + Name = "my_proc", + Line = 1, + StartLine = 1, + EndLine = 1, + }, + new SymbolRecord + { + FileId = fileId, + Kind = "function", + Name = "RunGraph", + Line = 2, + StartLine = 2, + EndLine = 3, + BodyStartLine = 3, + BodyEndLine = 3, + }, + ]); + _writer.InsertReferences([ + new ReferenceRecord + { + FileId = fileId, + SymbolName = "myProc", + ReferenceKind = "call", + Line = 3, + Column = 3, + ContainerKind = "function", + ContainerName = "RunGraph", + Context = "myProc()", + }, + ], refreshMutualRecursionFlags: false); + _writer.RefreshMutualRecursionFlags(); + + Assert.Equal("unresolved", ReadReferenceResolutionState(fileId)); + + var rewritten = _writer.BackfillFoldedColumns(rewriteAll: true); + + Assert.Equal((2, 1), rewritten); + var expectedTargetId = ExecuteScalarLong($""" + SELECT id + FROM symbols + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + AND name = 'my_proc' + """); + Assert.Equal(expectedTargetId, ExecuteScalarLong($""" + SELECT target_symbol_id + FROM symbol_references + WHERE file_id = {fileId.ToString(CultureInfo.InvariantCulture)} + """)); + Assert.Equal("resolved", ReadReferenceResolutionState(fileId)); + } + + [Fact] + public void ReferenceGraph_NimBackfillRetriesInterruptedGraphRefresh_Issue4738() + { + var fileId = UpsertTestFileWithLanguage( + "src/interrupted-style.nim", + "nim", + "nim-interrupted-style"); + _writer.InsertSymbols([ + new SymbolRecord + { + FileId = fileId, + Kind = "function", + Name = "my_proc", + Line = 1, + StartLine = 1, + EndLine = 1, + }, + ]); + _writer.InsertReferences([ + new ReferenceRecord + { + FileId = fileId, + SymbolName = "myProc", + ReferenceKind = "call", + Line = 2, + Column = 1, + Context = "myProc()", + }, + ], refreshMutualRecursionFlags: false); + + var previousRefreshHook = DbWriter.MutualRecursionRefreshForTesting; + var interruptRefresh = true; + try + { + DbWriter.MutualRecursionRefreshForTesting = () => + { + previousRefreshHook?.Invoke(); + if (interruptRefresh) + throw new OperationCanceledException("interrupt fold graph refresh"); + }; + + Assert.Throws( + () => _writer.BackfillFoldedColumns(rewriteAll: true)); + + interruptRefresh = false; + Assert.Equal((0, 0), _writer.BackfillFoldedColumns(rewriteAll: true)); + } + finally + { + DbWriter.MutualRecursionRefreshForTesting = previousRefreshHook; + } + + Assert.Equal("resolved", ReadReferenceResolutionState(fileId)); + } + + [Fact] + public void SearchSymbols_NimExactDegradedPathPreservesRawUnderscores_Issue4738() + { + var fileId = UpsertTestFileWithLanguage( + "src/degraded-style.nim", + "nim", + "nim-degraded-style"); + _writer.InsertSymbols([ + new SymbolRecord + { + FileId = fileId, + Kind = "function", + Name = "my_proc", + Line = 1, + StartLine = 1, + EndLine = 1, + }, + ]); + _writer.SetMeta( + "fold_key_version", + (NameFold.Version - 1).ToString(CultureInfo.InvariantCulture)); + _writer.SetMeta("fold_key_fingerprint", NameFold.Fingerprint()); + using (var ready = _db.Connection.CreateCommand()) + { + ready.CommandText = + $"PRAGMA user_version = {_db.GetUserVersion() | DbContext.FoldReadyFlag}"; + ready.ExecuteNonQuery(); + } + + var reader = new DbReader(_db.Connection); + + Assert.False(reader._foldReady); + Assert.Single(reader.SearchSymbols("my_proc", lang: "nim", exact: true)); + Assert.Empty(reader.SearchSymbols("myProc", lang: "nim", exact: true)); + } + [Fact] public void ReferenceGraphDirtyScope_RollbackAndCancellationPreserveRetryState() { diff --git a/tests/CodeIndex.Tests/DbReaderImpactTests.cs b/tests/CodeIndex.Tests/DbReaderImpactTests.cs index 4af9ddb49..aa2ade755 100644 --- a/tests/CodeIndex.Tests/DbReaderImpactTests.cs +++ b/tests/CodeIndex.Tests/DbReaderImpactTests.cs @@ -40,6 +40,52 @@ public void Leaf() { } Assert.Contains("--max-hops 1", analysis.Suggestion, StringComparison.Ordinal); } + [Fact] + public void AnalyzeImpact_DefinitiveMatlabTargetTraversesAmbiguousMCallers_Issue4738() + { + InsertIndexedFile( + "src/target.m", + "matlab", + """ + function Target() + end + """); + InsertIndexedFile( + "src/caller.m", + "ambiguous_m", + """ + function Bridge() + Target(); + end + function Outer() + Bridge(); + end + """); + + var analysis = _reader.AnalyzeImpact( + "Target", + maxDepth: 2, + limit: 10, + lang: "matlab", + pathPatterns: ["src/*.m"]); + + Assert.Equal("callers", analysis.ImpactMode); + Assert.Collection( + analysis.Callers.OrderBy(caller => caller.Depth), + caller => + { + Assert.Equal("ambiguous_m", caller.Lang); + Assert.Equal("Bridge", caller.CallerName); + Assert.Equal(1, caller.Depth); + }, + caller => + { + Assert.Equal("ambiguous_m", caller.Lang); + Assert.Equal("Outer", caller.CallerName); + Assert.Equal(2, caller.Depth); + }); + } + [Fact] public void AnalyzeImpact_CSharpVerbatimQueryKeepsOriginalInputOnMiss() { diff --git a/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs b/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs index 34b2ee511..2df866e91 100644 --- a/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs +++ b/tests/CodeIndex.Tests/IndexCommandRunnerTests.cs @@ -498,6 +498,32 @@ public void SymbolExtractionWorker_Utf8RequestsPreserveUnicodeAcrossLanguages() } } + [Fact] + public void SymbolExtractionWorker_NimIdentityKeySurvivesProtocolRoundTrip_Issue4738() + { + var projectRoot = CreateTempProject(); + try + { + using var worker = new SymbolExtractionWorkerClient(); + var result = worker.Invoke( + 0, + "nim", + "proc my_proc() = discard\n", + Path.Combine(projectRoot, "sample.nim"), + projectRoot, + TimeSpan.FromSeconds(5)); + + Assert.True(result.Success, result.WorkerError); + var symbol = Assert.Single(result.Symbols!); + Assert.Equal("my_proc", symbol.Name); + Assert.Equal("myproc", symbol.IdentityNameFolded); + } + finally + { + DeleteDirectory(projectRoot); + } + } + [Fact] public void SymbolExtractionWorker_StreamResponseWritesBomlessUtf8Frame() { diff --git a/tests/CodeIndex.Tests/NameFoldTests.cs b/tests/CodeIndex.Tests/NameFoldTests.cs index 35310cfc4..2e6efcacf 100644 --- a/tests/CodeIndex.Tests/NameFoldTests.cs +++ b/tests/CodeIndex.Tests/NameFoldTests.cs @@ -42,4 +42,11 @@ public void Fingerprint_ReturnsLowercaseHex() Assert.Equal(fingerprint.ToLowerInvariant(), fingerprint); Assert.DoesNotContain(fingerprint, c => c is >= 'A' and <= 'F'); } + + [Fact] + public void PersistedKeyContract_VersionsNimStyleInsensitiveIdentity_Issue4738() + { + Assert.Equal(3, NameFold.Version); + Assert.Equal("myproc", DbReader.FoldNameForLanguage("my_proc", "nim")); + } } diff --git a/tests/CodeIndex.Tests/PostExtractionHookContractTests.cs b/tests/CodeIndex.Tests/PostExtractionHookContractTests.cs index 5c9dc6990..314626ec3 100644 --- a/tests/CodeIndex.Tests/PostExtractionHookContractTests.cs +++ b/tests/CodeIndex.Tests/PostExtractionHookContractTests.cs @@ -45,12 +45,64 @@ public void CallbackProtocol_RoundTripsRequestAndResponseContracts_Issue4185() Assert.Equal("TrimmedApp", Assert.Single(roundTrippedResponse.Symbols!).Name); } + [Fact] + public void CallbackProtocol_RoundTripsLanguageIdentityState_Issue4738() + { + var request = new PostExtractionHookCallbackProtocol.WorkerRequest( + nameof(IPostExtractionHook.OnReferencesExtracted), + new FileContext("project", "src/style.nim", "/project/src/style.nim", "nim"), + [ + new SymbolRecord + { + FileId = 5, + Kind = "function", + Name = "my_proc", + IdentityNameFolded = "myproc", + Line = 1, + StartLine = 1, + EndLine = 1, + }, + ], + [ + new ReferenceRecord + { + FileId = 5, + SymbolName = "myProc", + IdentitySymbolNameFolded = "myproc", + ReferenceKind = "call", + Line = 2, + Column = 5, + ContainerName = "RunGraph", + IdentityContainerNameFolded = "Rungraph", + SuppressInferredTargetQualifier = true, + }, + ]); + + var roundTripped = PostExtractionHookCallbackProtocol.DeserializeRequest( + PostExtractionHookCallbackProtocol.SerializeRequest(request)); + + Assert.Equal("myproc", Assert.Single(roundTripped.Symbols!).IdentityNameFolded); + var reference = Assert.Single(roundTripped.References!); + Assert.Equal("myproc", reference.IdentitySymbolNameFolded); + Assert.Equal("Rungraph", reference.IdentityContainerNameFolded); + Assert.True(reference.SuppressInferredTargetQualifier); + } + [Fact] public void MutationMaterializer_ClonesAndTrimsRecordsWithinContracts_Issue4185() { var symbols = new List { - new() { FileId = 7, Kind = "class", Name = "Original", Line = 1, StartLine = 1, EndLine = 1 }, + new() + { + FileId = 7, + Kind = "class", + Name = "Original", + IdentityNameFolded = "original-key", + Line = 1, + StartLine = 1, + EndLine = 1, + }, new() { FileId = 7, Kind = "method", Name = "Extra", Line = 2, StartLine = 2, EndLine = 2 }, }; @@ -58,6 +110,7 @@ public void MutationMaterializer_ClonesAndTrimsRecordsWithinContracts_Issue4185( Assert.True(inputTruncated); var clonedSymbol = Assert.Single(cloned); + Assert.Equal("original-key", clonedSymbol.IdentityNameFolded); clonedSymbol.Name = "ChangedByHook"; Assert.Equal("Original", symbols[0].Name); @@ -71,4 +124,57 @@ public void MutationMaterializer_ClonesAndTrimsRecordsWithinContracts_Issue4185( Assert.True(PostExtractionHookMutationMaterializer.TrimToLimit(references, maxCount: 2)); Assert.Equal(["Original", "Extra"], references.Select(reference => reference.SymbolName)); } + + [Fact] + public void MutationMaterializer_RecomputesNimIdentityAfterHookMutation_Issue4738() + { + var symbols = new List + { + new() + { + FileId = 7, + Kind = "function", + Name = "renamed_proc", + IdentityNameFolded = "stale", + Line = 1, + StartLine = 1, + EndLine = 1, + }, + }; + var references = new List + { + new() + { + FileId = 7, + SymbolName = "renamedProc", + IdentitySymbolNameFolded = "stale", + ReferenceKind = "call", + Line = 2, + Column = 1, + ContainerName = "Run_Graph", + IdentityContainerNameFolded = "stale", + TargetQualifier = "pkg", + SuppressInferredTargetQualifier = true, + }, + }; + + var clonedReferences = PostExtractionHookMutationMaterializer.CloneReferences( + references, + maxCount: null, + out var referencesTruncated); + Assert.False(referencesTruncated); + var clonedReference = Assert.Single(clonedReferences); + Assert.Equal("pkg", clonedReference.TargetQualifier); + Assert.True(clonedReference.SuppressInferredTargetQualifier); + + PostExtractionHookMutationMaterializer.RefreshLanguageIdentity("nim", symbols); + PostExtractionHookMutationMaterializer.RefreshLanguageIdentity("nim", clonedReferences); + + Assert.Equal("renamedproc", Assert.Single(symbols).IdentityNameFolded); + var reference = Assert.Single(clonedReferences); + Assert.Equal("renamedproc", reference.IdentitySymbolNameFolded); + Assert.Equal("Rungraph", reference.IdentityContainerNameFolded); + Assert.Equal("pkg", reference.TargetQualifier); + Assert.True(reference.SuppressInferredTargetQualifier); + } } diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerImpactTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerImpactTests.cs index ed0ab5a9b..0ece2adb3 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerImpactTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerImpactTests.cs @@ -105,6 +105,7 @@ public void GetTransitiveCallers_MaxDepthBoundaryProbeBudgetTerminatesStably_Iss null, null, false, + false, ]); Assert.NotNull(inspection); var type = inspection!.GetType(); diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs index b50d7e074..8610a5b15 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerSymbolTests.cs @@ -1515,14 +1515,14 @@ public void RunSymbols_UnsupportedExtractorLanguageExplainsSearchOnlyFallback() var projectRoot = TestProjectHelper.CreateTempProject("cdidx_symbols_unsupported_extractor"); try { - File.WriteAllText(Path.Combine(projectRoot, "settings.m"), "enabled = true\n"); + File.WriteAllText(Path.Combine(projectRoot, "settings.pl"), "enabled = true\n"); var dbPath = Path.Combine(projectRoot, ".cdidx", "codeindex.db"); var (indexExitCode, _, indexStderr) = CaptureConsole(() => IndexCommandRunner.Run( [projectRoot, "--json", "--quiet"], _jsonOptions)); var (exitCode, stdout, stderr) = CaptureConsole(() => QueryCommandRunner.RunSymbols( - ["enabled", "--db", dbPath, "--lang", "ambiguous_m"], + ["enabled", "--db", dbPath, "--lang", "ambiguous_pl"], _jsonOptions)); Assert.Equal(CommandExitCodes.Success, indexExitCode); @@ -1530,7 +1530,7 @@ public void RunSymbols_UnsupportedExtractorLanguageExplainsSearchOnlyFallback() Assert.Equal(CommandExitCodes.Success, exitCode); Assert.Equal(string.Empty, stdout); Assert.Contains("symbol extraction is not available", stderr); - Assert.Contains("cdidx search --lang ambiguous_m", stderr); + Assert.Contains("cdidx search --lang ambiguous_pl", stderr); Assert.Contains("missing-symbols", stderr); } finally diff --git a/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs b/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs index ead43e28b..5a0fc91f9 100644 --- a/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs +++ b/tests/CodeIndex.Tests/QueryCommandRunnerTests.cs @@ -2710,7 +2710,7 @@ public void RunLanguages_FormatCountReturnsCapabilitySummary_Issue4316() using var project = TestProjectHelper.CreateTempProjectScope("cdidx_languages_format_count_issue4316"); var dbPath = TestProjectHelper.CreateProjectDb(project.Root); TestProjectHelper.InsertIndexedFile(dbPath, "src/App.cs", "csharp", "class App { }\n"); - TestProjectHelper.InsertIndexedFile(dbPath, "src/main.adb", "ada", "procedure Main is begin null; end Main;\n"); + TestProjectHelper.InsertIndexedFile(dbPath, "src/main.pl", "prolog", "main :- true.\n"); var (exitCode, stdout, stderr) = CaptureConsole(() => QueryCommandRunner.RunLanguages(["--db", dbPath, "--indexed-only", "--capability", "missing-any", "--format", "count"], _jsonOptions)); @@ -2762,15 +2762,15 @@ public void RunLanguages_JsonIncludesUnsupportedCapabilityGuidance_Issue4122() using var document = ParseJsonOutput(stdout); var languages = document.RootElement.GetProperty("languages").EnumerateArray() .ToDictionary(entry => entry.GetProperty("lang").GetString()!, entry => entry); - var adaGuidance = languages["ada"].GetProperty("unsupported_guidance").EnumerateArray().ToList(); + var prologGuidance = languages["prolog"].GetProperty("unsupported_guidance").EnumerateArray().ToList(); - var referenceGuidance = adaGuidance.Single(guidance => guidance.GetProperty("capability").GetString() == "references"); - Assert.Contains("Reference extraction is not advertised for 'ada'", referenceGuidance.GetProperty("message").GetString()); + var referenceGuidance = prologGuidance.Single(guidance => guidance.GetProperty("capability").GetString() == "references"); + Assert.Contains("Reference extraction is not advertised for 'prolog'", referenceGuidance.GetProperty("message").GetString()); var referenceCommands = referenceGuidance.GetProperty("recommended_commands").EnumerateArray().Select(command => command.GetString()).ToList(); Assert.Contains("search", referenceCommands); Assert.Contains("definition", referenceCommands); - var graphGuidance = adaGuidance.Single(guidance => guidance.GetProperty("capability").GetString() == "graph"); + var graphGuidance = prologGuidance.Single(guidance => guidance.GetProperty("capability").GetString() == "graph"); Assert.Contains("empty callers, callees, or impact results are not authoritative", graphGuidance.GetProperty("message").GetString()); var graphCommands = graphGuidance.GetProperty("recommended_commands").EnumerateArray().Select(command => command.GetString()).ToList(); Assert.Contains("search", graphCommands); @@ -2856,7 +2856,7 @@ public void RunLanguages_JsonListsModernNodeModuleExtensions() } [Fact] - public void RunLanguages_JsonReportsCythonSymbolsAndCudaReferences_Issue4737() + public void RunLanguages_JsonReportsCythonAndCudaReferences_Issues4737And4738() { var (exitCode, stdout, stderr) = CaptureConsole(() => QueryCommandRunner.RunLanguages(["--json"], _jsonOptions)); @@ -2869,8 +2869,8 @@ public void RunLanguages_JsonReportsCythonSymbolsAndCudaReferences_Issue4737() var cuda = languages.EnumerateArray().Single(lang => lang.GetProperty("lang").GetString() == "cuda"); Assert.True(cython.GetProperty("symbol_extraction").GetBoolean()); - Assert.False(cython.GetProperty("reference_extraction").GetBoolean()); - Assert.False(cython.GetProperty("graph_queries").GetBoolean()); + Assert.True(cython.GetProperty("reference_extraction").GetBoolean()); + Assert.True(cython.GetProperty("graph_queries").GetBoolean()); Assert.True(cuda.GetProperty("symbol_extraction").GetBoolean()); Assert.True(cuda.GetProperty("reference_extraction").GetBoolean()); Assert.True(cuda.GetProperty("graph_queries").GetBoolean()); @@ -3150,7 +3150,7 @@ public void RunLanguages_JsonReportsLanguageMapOverrideProvenance_Issue4617() } [Fact] - public void RunLanguages_JsonReportsMatlabAndPrologSymbolOnlyCapabilities_Issue4612() + public void RunLanguages_JsonReportsScientificNativeReferenceCapabilities_Issue4738() { var (exitCode, stdout, stderr) = CaptureConsole(() => QueryCommandRunner.RunLanguages(["--json"], _jsonOptions)); @@ -3160,13 +3160,16 @@ public void RunLanguages_JsonReportsMatlabAndPrologSymbolOnlyCapabilities_Issue4 using var document = ParseJsonOutput(stdout); var languages = document.RootElement.GetProperty("languages").EnumerateArray() .ToDictionary(entry => entry.GetProperty("lang").GetString()!, entry => entry); - foreach (var language in new[] { "matlab", "prolog" }) + foreach (var language in new[] { "ada", "ambiguous_m", "cython", "d", "julia", "matlab", "nim" }) { Assert.True(languages[language].GetProperty("symbol_extraction").GetBoolean()); - Assert.False(languages[language].GetProperty("reference_extraction").GetBoolean()); - Assert.False(languages[language].GetProperty("graph_queries").GetBoolean()); + Assert.True(languages[language].GetProperty("reference_extraction").GetBoolean()); + Assert.True(languages[language].GetProperty("graph_queries").GetBoolean()); } + Assert.True(languages["prolog"].GetProperty("symbol_extraction").GetBoolean()); + Assert.False(languages["prolog"].GetProperty("reference_extraction").GetBoolean()); + Assert.False(languages["prolog"].GetProperty("graph_queries").GetBoolean()); Assert.Contains(".m", languages["ambiguous_m"].GetProperty("extensions").EnumerateArray().Select(value => value.GetString())); Assert.Contains(".pl", languages["ambiguous_pl"].GetProperty("extensions").EnumerateArray().Select(value => value.GetString())); } @@ -3301,7 +3304,7 @@ public void RunLanguages_Json_ExtractorBucketsAdvertiseAccurateGraphSupport_Issu var languages = document.RootElement.GetProperty("languages").EnumerateArray() .ToDictionary(entry => entry.GetProperty("lang").GetString()!, entry => entry); - foreach (var symbolOnly in new[] { "ada", "crystal", "d", "groovy", "julia", "nim", "tcl" }) + foreach (var symbolOnly in new[] { "crystal", "groovy", "tcl" }) { Assert.True(languages.ContainsKey(symbolOnly), $"expected '{symbolOnly}' to be listed"); var entry = languages[symbolOnly]; diff --git a/tests/CodeIndex.Tests/ReferenceExtractorTests.ScientificNative.cs b/tests/CodeIndex.Tests/ReferenceExtractorTests.ScientificNative.cs new file mode 100644 index 000000000..3ac27aae4 --- /dev/null +++ b/tests/CodeIndex.Tests/ReferenceExtractorTests.ScientificNative.cs @@ -0,0 +1,1895 @@ +using CodeIndex.Indexer; +using CodeIndex.Models; + +namespace CodeIndex.Tests; + +public partial class ReferenceExtractorTests +{ + public static TheoryData ScientificNativeReferenceCases => new() + { + { + "nim", + """ + import std/math, strutils + type Child = object of Base + proc run() = + # ignoredCall() + #[ + ignoredBlockCall() + ]# + helper() + """, + "std/math", + "strutils", + "Base" + }, + { + "matlab", + """ + classdef Child < Base + methods + function run(obj) + import pkg.Tools pkg.Other + % ignoredCall() + %{ + ignoredBlockCall() + %} + helper(); + end + end + end + """, + "pkg.Tools", + "pkg.Other", + "Base" + }, + { + "julia", + """ + module Sample + using LinearAlgebra, Statistics + struct Child <: Base + end + function run() + # ignoredCall() + value' #= + ignoredBlockCall() + =# + helper() + end + end + """, + "LinearAlgebra", + "Statistics", + "Base" + }, + { + "d", + """ + module sample; + import std.stdio, std.algorithm; + class Child : Base, IFace { + void run() { + /* ignoredCall(); */ + /+ + ignoredBlockCall(); + +/ + helper(); + } + } + """, + "std.stdio", + "std.algorithm", + "Base" + }, + { + "cython", + """" + cimport numpy, cython + cdef class Child(Base): + def run(self): + # ignoredCall() + """ + ignoredBlockCall() + """ + helper() + """", + "numpy", + "cython", + "Base" + }, + { + "ada", + """ + with Ada.Text_IO, Ada.Command_Line; + package body Demo is + type Child is new Base; + procedure Run is + begin + -- IgnoredCall; + Helper; + end Run; + end Demo; + """, + "Ada.Text_IO", + "Ada.Command_Line", + "Base" + }, + }; + + [Theory] + [MemberData(nameof(ScientificNativeReferenceCases))] + public void Extract_ScientificNativeLanguagesEmitBoundedGraphReferences_Issue4738( + string language, + string content, + string importedName, + string secondImportedName, + string baseTypeName) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == importedName && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == secondImportedName && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == baseTypeName && reference.ReferenceKind == "type_reference"); + var helperReference = Assert.Single(references, reference => + reference.SymbolName.Equals("helper", StringComparison.OrdinalIgnoreCase) + && reference.ReferenceKind == "call"); + Assert.Equal("run", helperReference.ContainerName, ignoreCase: true); + Assert.DoesNotContain(references, reference => + reference.SymbolName.Equals("ignoredCall", StringComparison.OrdinalIgnoreCase)); + Assert.DoesNotContain(references, reference => + reference.SymbolName.Equals("ignoredBlockCall", StringComparison.OrdinalIgnoreCase)); + } + + public static TheoryData ScientificNativePhantomSymbolCases => new() + { + { + "d", + """ + /+ + void Phantom() {} + +/ + void real() { Phantom(); } + """ + }, + { + "nim", + """ + #[ + proc Phantom() = discard + ]# + proc real() = Phantom() + """ + }, + { + "julia", + """ + #= + function Phantom() + end + =# + function real() + Phantom() + end + """ + }, + { + "matlab", + """ + %{ + function Phantom() + end + %} + function real() + Phantom(); + end + """ + }, + { + "cython", + """" + """ + def Phantom(): + pass + """ + def real(): + Phantom() + """" + }, + }; + + [Theory] + [MemberData(nameof(ScientificNativePhantomSymbolCases))] + public void Extract_ScientificNativeNonCodeDeclarationsDoNotBecomeResolutionTargets_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + Assert.DoesNotContain(symbols, symbol => symbol.Name == "Phantom"); + Assert.Single(references, reference => + reference.SymbolName == "Phantom" && reference.ReferenceKind == "call"); + } + + [Theory] + [InlineData( + "matlab", + """ + function result = run(a, b) + text = 'stringCall()'; + result = a' * helper() * b'; + end + """)] + [InlineData( + "julia", + """ + function run(a, b) + text = "stringCall()" + a' * helper() * b' + end + """)] + public void Extract_MatlabAndJuliaPreservePostfixTransposeCalls_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + var helperReference = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Equal("run", helperReference.ContainerName); + Assert.DoesNotContain(references, reference => reference.SymbolName == "stringCall"); + } + + [Fact] + public void Extract_MatlabAppliesCharacterVectorQuoteRules_Issue4738() + { + const string content = """ + function run() + path = 'C:\'; helper(); + items = [prefix 'fake()']; + end + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + var helperReference = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Equal("run", helperReference.ContainerName); + Assert.DoesNotContain(references, reference => reference.SymbolName == "fake"); + } + + [Fact] + public void Extract_MatlabPreservesCallsAfterDotTranspose_Issue4738() + { + const string content = """ + function run(A) + value = A.'; helper(); + end + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + var helperReference = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Equal("run", helperReference.ContainerName); + } + + [Fact] + public void Extract_NimExpandsGroupedImports_Issue4738() + { + const string content = """ + import std/[strutils, sequtils], os + proc run() = + helper() + """; + var symbols = SymbolExtractor.Extract(1, "nim", content); + + var references = ReferenceExtractor.Extract(1, "nim", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "std/strutils" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "std/sequtils" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "os" && reference.ReferenceKind == "import"); + Assert.DoesNotContain(references, reference => reference.SymbolName is "std" or "sequtils"); + } + + [Fact] + public void Extract_JuliaNormalizesRelativeImportsAndBroadcastCalls_Issue4738() + { + const string content = """ + module Main + using .Utils, ..Parent + function run(xs) + helper.(xs) + end + end + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "Utils" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "Parent" && reference.ReferenceKind == "import"); + Assert.Equal(8, Assert.Single(references, reference => + reference.SymbolName == "Utils" && reference.ReferenceKind == "import").Column); + Assert.Equal(17, Assert.Single(references, reference => + reference.SymbolName == "Parent" && reference.ReferenceKind == "import").Column); + var helperReference = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Equal("run", helperReference.ContainerName); + Assert.DoesNotContain(references, reference => + reference.SymbolName is ".Utils" or "..Parent"); + } + + [Fact] + public void Extract_DTemplateInvocationsEmitCalleeCalls_Issue4738() + { + const string content = """ + void run() { + helper!int(); + other!(string, int)(42); + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Contains(references, reference => + reference.SymbolName == "other" && reference.ReferenceKind == "call"); + Assert.DoesNotContain(references, reference => + reference.SymbolName is "int" or "string" && reference.ReferenceKind == "call"); + } + + [Fact] + public void Extract_DTemplateInvocationsAcceptLiteralAndNestedArguments_Issue4738() + { + const string content = """ + void run() { + literal!42(); + nested!(Tuple!(int))(); + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "literal" && reference.ReferenceKind == "call"); + Assert.Contains(references, reference => + reference.SymbolName == "nested" && reference.ReferenceKind == "call"); + Assert.DoesNotContain(references, reference => + reference.SymbolName is "Tuple" or "int" && reference.ReferenceKind == "call"); + } + + [Fact] + public void Extract_DManyTemplateInvocationsSuppressArgumentCallsInOnePass_Issue4738() + { + var content = "void run() { " + string.Join(' ', Enumerable.Repeat("helper!Type();", 512)) + " }"; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + Assert.Equal(512, references.Count(reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call")); + Assert.DoesNotContain(references, reference => + reference.SymbolName == "Type" && reference.ReferenceKind == "call"); + } + + [Fact] + public void Extract_AdaQualifiedBareCallUsesResolvableLeafName_Issue4738() + { + const string content = """ + package body Demo is + procedure Run is + begin + Helpers.Flush; + end Run; + end Demo; + """; + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var references = ReferenceExtractor.Extract(1, "ada", content, symbols); + + var flushReference = Assert.Single(references, reference => + reference.SymbolName == "Flush" && reference.ReferenceKind == "call"); + Assert.Equal("Run", flushReference.ContainerName); + Assert.Equal("Helpers", flushReference.TargetQualifier); + Assert.DoesNotContain(references, reference => reference.SymbolName == "Helpers.Flush"); + } + + [Fact] + public void Extract_AdaConsecutiveBareCallsDoNotConsumeNextSeparator_Issue4738() + { + const string content = """ + procedure Run is + begin + First; Second; Third; + end Run; + """; + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var references = ReferenceExtractor.Extract(1, "ada", content, symbols); + + foreach (var name in new[] { "First", "Second", "Third" }) + { + Assert.Equal("Run", Assert.Single(references, reference => + reference.SymbolName == name && reference.ReferenceKind == "call").ContainerName); + } + } + + [Fact] + public void Extract_AdaInlineStatementBoundariesPreserveBareCalls_Issue4738() + { + const string content = """ + procedure Run is begin First; end Run; + procedure Check is begin if Ready then Second; else Third; end if; end Check; + """; + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var references = ReferenceExtractor.Extract(1, "ada", content, symbols); + + foreach (var name in new[] { "First", "Second", "Third" }) + { + Assert.Single(references, reference => + reference.SymbolName == name && reference.ReferenceKind == "call"); + } + } + + [Fact] + public void Extract_MatlabCommaSeparatedEndKeepsFollowingCallTopLevel_Issue4738() + { + const string content = """ + function first() + if true, helper(), end + end + toplevel(); + function second() + other(); + end + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + var first = Assert.Single(symbols, symbol => symbol.Name == "first"); + Assert.Equal(3, first.EndLine); + var topLevelReference = Assert.Single(references, reference => + reference.SymbolName == "toplevel" && reference.ReferenceKind == "call"); + Assert.Null(topLevelReference.ContainerName); + } + + [Theory] + [InlineData("matlab", "function run(), helper(), end\noutside();\n")] + [InlineData("julia", "function run(); helper(); end\noutside()\n")] + public void Extract_CompactScientificFunctionEndsOnDeclarationLine_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + var function = Assert.Single(symbols, symbol => symbol.Name == "run"); + Assert.Equal(1, function.EndLine); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_MatlabIndexEndDoesNotCloseFunction_Issue4738() + { + const string content = """ + function run(A) + value = A(:, end); + helper(); + end + outside(); + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + Assert.Equal(4, Assert.Single(symbols, symbol => symbol.Name == "run").EndLine); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaMacroBlockDoesNotCloseFunctionEarly_Issue4738() + { + const string content = """ + function run() + @async begin + helper() + end + after() + end + outside() + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Equal(6, Assert.Single(symbols, symbol => symbol.Name == "run").EndLine); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "after" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Theory] + [InlineData("run(x) = helper(x)\nhelper(x) = x\n", 1)] + [InlineData("run(x) = begin\n helper(x)\nend\nhelper(x) = x\n", 3)] + [InlineData("run(x) =\nbegin\n helper(x)\nend\nhelper(x) = x\n", 4)] + [InlineData("run(x) = map(x) do item\n helper(item)\nend\nhelper(x) = x\n", 3)] + [InlineData("run(x) = (if x\n helper(x)\nend)\nhelper(x) = x\n", 3)] + public void Extract_JuliaShortFunctionsOwnTheirCallReferences_Issue4738( + string content, + int expectedEndLine) + { + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + var run = Assert.Single(symbols, symbol => symbol.Name == "run"); + Assert.Equal(expectedEndLine, run.EndLine); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaExpressionPositionBlocksKeepOuterFunctionRange_Issue4738() + { + const string content = """ + function outer(xs) + map(function (item) + inner(item) + end, xs) + push!(xs, begin + nested() + end) + values = [item for item in xs if item > 0] + after() + end + outside() + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Equal(10, Assert.Single(symbols, symbol => symbol.Name == "outer").EndLine); + foreach (var name in new[] { "inner", "nested", "after" }) + { + Assert.Equal("outer", Assert.Single(references, reference => + reference.SymbolName == name && reference.ReferenceKind == "call").ContainerName); + } + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_MatlabPeerFunctionsWithoutClosingEndHaveSeparateRanges_Issue4738() + { + const string content = """ + function first() + helper1(); + function second() + helper2(); + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + var first = Assert.Single(symbols, symbol => symbol.Name == "first"); + var second = Assert.Single(symbols, symbol => symbol.Name == "second"); + Assert.Equal(2, first.EndLine); + Assert.Equal(4, second.EndLine); + Assert.Equal("first", Assert.Single(references, reference => + reference.SymbolName == "helper1" && reference.ReferenceKind == "call").ContainerName); + Assert.Equal("second", Assert.Single(references, reference => + reference.SymbolName == "helper2" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_MatlabSameIndentNestedFunctionKeepsExplicitOuterRange_Issue4738() + { + const string content = """ + function outer() + helper1(); + function nested() + inner(); + end + helper2(); + end + outside(); + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + Assert.Equal(7, Assert.Single(symbols, symbol => symbol.Name == "outer").EndLine); + Assert.Equal(5, Assert.Single(symbols, symbol => symbol.Name == "nested").EndLine); + Assert.Equal("outer", Assert.Single(references, reference => + reference.SymbolName == "helper2" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaTransposeBeforeBlockCommentDoesNotMaskFollowingCalls_Issue4738() + { + const string content = """" + function run(value) + value' #= + """ + =# + helper() + end + """"; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Theory] + [InlineData( + "matlab", + """ + function run(A) + value = A(1, ... + end); + helper(); + end + outside(); + """)] + [InlineData( + "julia", + """ + function run(A) + value = A[ + end] + helper() + end + outside() + """)] + public void Extract_ScientificIndexEndOnContinuationLinePreservesFunctionScope_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + Assert.Equal(5, Assert.Single(symbols, symbol => symbol.Name == "run").EndLine); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaMultilineDelimitedShortFunctionKeepsItsCallScope_Issue4738() + { + const string content = """ + run(value) = ( + helper(value) + ) + outside() + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Equal(3, Assert.Single(symbols, symbol => symbol.Name == "run").EndLine); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaEscapedTripleQuoteStaysInsideMultilineString_Issue4738() + { + const string content = """" + function run() + text = """ + escaped \""" fake() + still literal + """ + helper() + end + """"; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.DoesNotContain(references, reference => reference.SymbolName == "fake"); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Theory] + [InlineData("from .helpers cimport thing\n", "helpers")] + [InlineData("from ..pkg.helpers cimport thing\n", "pkg.helpers")] + public void Extract_CythonRelativeCimportsNormalizeTheirModuleName_Issue4738( + string content, + string expectedModule) + { + var references = ReferenceExtractor.Extract(1, "cython", content, []); + + Assert.Single(references, reference => + reference.SymbolName == expectedModule && reference.ReferenceKind == "import"); + } + + [Fact] + public void Extract_ScientificDependencyNameLimitReportsOnlyAfterTheSharedBoundary_Issue4738() + { + var previousLimits = ReferenceExtractor.SafetyLimitsForTesting; + ReferenceExtractor.SafetyLimitsForTesting = new ReferenceExtractionSafetyLimits + { + MaxLookupSymbols = 100, + MaxLookupLines = 100, + MaxNamesPerLine = 2, + MaxContainerCandidates = 100, + }; + + try + { + var exact = ReferenceExtractor.ExtractDetailed( + 1, + "ada", + "with Alpha, Beta;\n", + []); + var exceeded = ReferenceExtractor.ExtractDetailed( + 1, + "ada", + "with Alpha, Beta, Gamma;\n", + []); + + Assert.Equal(2, exact.References.Count(reference => reference.ReferenceKind == "import")); + Assert.Equal(2, exceeded.References.Count(reference => reference.ReferenceKind == "import")); + Assert.DoesNotContain(exact.Diagnostics, diagnostic => + diagnostic.Kind == "reference_scientific_native_dependency_name_budget_exceeded"); + Assert.Contains(exceeded.Diagnostics, diagnostic => + diagnostic.Kind == "reference_scientific_native_dependency_name_budget_exceeded"); + } + finally + { + ReferenceExtractor.SafetyLimitsForTesting = previousLimits; + } + } + + public static TheoryData ScientificNativeMultilineLiteralCases => new() + { + { + "julia", + """" + function run() + text = """ + #= tokenOnly() + """ + helper() + end + """" + }, + { + "nim", + """" + proc run() = + let text = """ + #[ tokenOnly() + """ + helper() + """" + }, + { + "d", + """ + void run() { + auto text = q{ + /* tokenOnly(); */ + }; + helper(); + } + """ + }, + { + "d", + """ + void run() { + auto text = `literal + tokenOnly() + literal`; + helper(); + } + """ + }, + { + "d", + """ + void run() { + auto text = q"EOS + tokenOnly() + EOS"; + helper(); + } + """ + }, + }; + + [Theory] + [MemberData(nameof(ScientificNativeMultilineLiteralCases))] + public void Extract_ScientificNativeLiteralTokensDoNotSuppressFollowingCalls_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "tokenOnly"); + } + + [Fact] + public void Extract_JuliaMultilineCommandLiteralDoesNotEmitOrRescopePhantomCode_Issue4738() + { + const string content = """ + function real() + command = `echo + phantomCall() + function Phantom() + ` + helper() + end + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.DoesNotContain(symbols, symbol => symbol.Name == "Phantom"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "phantomCall"); + Assert.Equal("real", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaShortFunctionOperatorContinuationKeepsReferenceContainer_Issue4738() + { + const string content = """ + f(x) = first(x) + + second(x) + """; + var symbol = Assert.Single(SymbolExtractor.Extract(1, "julia", content)); + + var references = ReferenceExtractor.Extract(1, "julia", content, [symbol]); + + Assert.Equal(2, symbol.EndLine); + Assert.Equal("f", Assert.Single(references, reference => + reference.SymbolName == "second" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaShortFunctionTrailingAssignmentKeepsIndentedRhsContainer_Issue4738() + { + const string content = """ + f(x) = + helper(x) + outside() + """; + var symbol = Assert.Single(SymbolExtractor.Extract(1, "julia", content)); + + var references = ReferenceExtractor.Extract(1, "julia", content, [symbol]); + + Assert.Equal(2, symbol.EndLine); + Assert.Equal("f", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_DTokenStringNestedLiteralsAndCommentsDoNotChangeBraceDepth_Issue4738() + { + const string content = """ + void run() { + enum code = q{ + auto first = "}"; + auto second = q"[}]"; + /* } */ + /+ { } +/ + phantomCall(); + }; + helper(); + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + Assert.DoesNotContain(references, reference => reference.SymbolName == "phantomCall"); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_DIdentifierTokenStringRequiresDelimiterLineBoundary_Issue4738() + { + const string content = """ + void run() { + enum text = q"EOS + inside text contains EOS"; phantomCall() + still inside + EOS"; + helper(); + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + Assert.DoesNotContain(references, reference => reference.SymbolName == "phantomCall"); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_DCommentTextCannotOpenTokenStrings_Issue4738() + { + const string content = """ + /* documentation mentions q{ without a closing brace */ + /* documentation mentions an unmatched ` delimiter */ + void run() { + helper(); + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Theory] + [InlineData( + "d", + """ + void run() { + auto text = r"fake()\"; + helper(); + } + """)] + [InlineData( + "nim", + """ + proc run() = + let text = r"fake()\" + helper() + """)] + public void Extract_DAndNimRawStringsUseLiteralBackslashRules_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.DoesNotContain(references, reference => reference.SymbolName == "fake"); + } + + [Fact] + public void Extract_NimGeneralizedRawStringsUseLiteralBackslashRules_Issue4738() + { + const string content = """ + proc run() = + let text = foo"notACall()\" + helper() + """; + var symbols = SymbolExtractor.Extract(1, "nim", content); + + var references = ReferenceExtractor.Extract(1, "nim", content, symbols); + + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.DoesNotContain(references, reference => reference.SymbolName == "notACall"); + } + + [Fact] + public void Extract_NimRawStringBeforeBlockCommentDoesNotExposeCommentCode_Issue4738() + { + const string content = """ + let text = r"literal\" #[ + proc Phantom() = discard + phantomCall() + ]# + proc real() = + helper() + """; + var symbols = SymbolExtractor.Extract(1, "nim", content); + + var references = ReferenceExtractor.Extract(1, "nim", content, symbols); + + Assert.DoesNotContain(symbols, symbol => symbol.Name == "Phantom"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "phantomCall"); + Assert.Equal("real", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_NimRawTripleStringUsesMultilineMask_Issue4738() + { + const string content = """" + let text = r""" + proc Phantom() = discard + phantomCall() + """ + proc real() = + helper() + """"; + var symbols = SymbolExtractor.Extract(1, "nim", content); + + var references = ReferenceExtractor.Extract(1, "nim", content, symbols); + + Assert.DoesNotContain(symbols, symbol => symbol.Name == "Phantom"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "phantomCall"); + Assert.Equal("real", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_CythonStringDependenciesPreserveQuotedNames_Issue4738() + { + const string content = """" + include "helpers.pxi" + cdef extern from 'native.h': + void native_call() + """ + include "phantom.pxi" + """ + """"; + var symbols = SymbolExtractor.Extract(1, "cython", content); + + var references = ReferenceExtractor.Extract(1, "cython", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "helpers.pxi" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "native.h" && reference.ReferenceKind == "import"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "phantom.pxi"); + } + + [Fact] + public void Extract_CythonOrdinaryFromImportsEmitModuleDependencies_Issue4738() + { + const string content = """ + from helpers import thing + from .local import other + """; + var symbols = SymbolExtractor.Extract(1, "cython", content); + + var references = ReferenceExtractor.Extract(1, "cython", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "helpers" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "local" && reference.ReferenceKind == "import"); + } + + [Fact] + public void Extract_AdaAttributesPreserveNestedCallsWithoutPhantomAttributeCalls_Issue4738() + { + const string content = """ + procedure Run is + begin + First(Integer'Image(helper()) & Float'Image(other())); + end Run; + """; + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var references = ReferenceExtractor.Extract(1, "ada", content, symbols); + + foreach (var name in new[] { "First", "helper", "other" }) + { + Assert.Equal("Run", Assert.Single(references, reference => + reference.SymbolName == name && reference.ReferenceKind == "call").ContainerName); + } + Assert.DoesNotContain(references, reference => + reference.SymbolName == "Image" && reference.ReferenceKind == "call"); + } + + [Fact] + public void Extract_DMultipleBaseTypesAndCastSyntaxStayGraphAccurate_Issue4738() + { + const string content = """ + private import pkg.mod; + public abstract class Child : Base, IFace { + void run() { + auto value = cast(int)(helper()); + } + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "Base" && reference.ReferenceKind == "type_reference"); + Assert.Contains(references, reference => + reference.SymbolName == "IFace" && reference.ReferenceKind == "type_reference"); + Assert.Contains(references, reference => + reference.SymbolName == "pkg.mod" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.DoesNotContain(references, reference => + reference.SymbolName == "cast" && reference.ReferenceKind == "call"); + } + + [Fact] + public void Extract_CythonRegularClassEmitsBaseTypeReference_Issue4738() + { + const string content = """ + class Child(Base): + def run(self): + helper() + """; + var symbols = SymbolExtractor.Extract(1, "cython", content); + + var references = ReferenceExtractor.Extract(1, "cython", content, symbols); + + Assert.Contains(references, reference => + reference.SymbolName == "Base" && reference.ReferenceKind == "type_reference"); + } + + [Fact] + public void Extract_AmbiguousMCombinesMatlabAndObjectiveCSymbolsAndReferences_Issue4738() + { + const string content = """ + #import + @interface Widget : NSObject + @end + + function result = run() + import pkg.Tools + % ignoredMatlabCall() + %{ + ignoredMatlabBlockCall() + %} + helper(); + end + """; + + var symbols = SymbolExtractor.Extract(1, "ambiguous_m", content, "mixed.m"); + var references = ReferenceExtractor.Extract(1, "ambiguous_m", content, symbols, "mixed.m"); + + Assert.Contains(symbols, symbol => symbol.Name == "Widget" && symbol.Kind == "class"); + Assert.Contains(symbols, symbol => symbol.Name == "run" && symbol.Kind == "function"); + Assert.Contains(references, reference => + reference.SymbolName == "Foundation/Foundation.h" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "pkg.Tools" && reference.ReferenceKind == "import"); + Assert.Contains(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "ignoredMatlabCall"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "ignoredMatlabBlockCall"); + } + + [Fact] + public void Extract_AmbiguousMRespectsObjectiveCCommentsDuringMatlabFallback_Issue4738() + { + const string content = """ + // ignoredObjectiveCCall() + void run(void) { + helper(); + } + """; + + var symbols = SymbolExtractor.Extract(1, "ambiguous_m", content, "unknown.m"); + var references = ReferenceExtractor.Extract(1, "ambiguous_m", content, symbols, "unknown.m"); + + Assert.Contains(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "ignoredObjectiveCCall"); + } + + [Fact] + public void Extract_AmbiguousMPreservesObjectiveCModuloExpressions_Issue4738() + { + const string content = """ + @implementation Widget + - (void)run { + int first = left % helper(); + int second = left % -other(); + int third = left % *pointer(); + value %= divisor; + afterAssignment(); + int fourth = index++ % postIncrement(); + int fifth = index-- % postDecrement(); + int sixth = left % [self computeValue]; + int seventh = 'x' % charRhs(); + int eighth = left % @(boxed()); + } + @end + """; + + var symbols = SymbolExtractor.Extract(1, "ambiguous_m", content, "unknown.m"); + var references = ReferenceExtractor.Extract(1, "ambiguous_m", content, symbols, "unknown.m"); + + foreach (var name in new[] + { + "helper", + "other", + "pointer", + "afterAssignment", + "postIncrement", + "postDecrement", + "computeValue", + "charRhs", + "boxed", + }) + { + Assert.Single(references, reference => + reference.SymbolName == name && reference.ReferenceKind == "call"); + } + } + + [Fact] + public void Extract_AmbiguousMMatlabCommentsDoNotMasqueradeAsModuloExpressions_Issue4738() + { + const string content = "x = left % helper();"; + + var symbols = SymbolExtractor.Extract(1, "ambiguous_m", content, "unknown.m"); + var references = ReferenceExtractor.Extract(1, "ambiguous_m", content, symbols, "unknown.m"); + + Assert.DoesNotContain(references, reference => reference.SymbolName == "helper"); + } + + [Theory] + [InlineData( + """ + %{ + @interface Fake + %} + function result = run(left) + result = left % helper(); + end + """)] + [InlineData( + """ + const char *text = "\ + @interface Fake"; + function result = run(left) + result = left % helper(); + end + """)] + public void Extract_AmbiguousMCommentedOrQuotedObjectiveCMarkersDoNotEnableModulo_Issue4738( + string content) + { + var symbols = SymbolExtractor.Extract(1, "ambiguous_m", content, "unknown.m"); + + var references = ReferenceExtractor.Extract( + 1, + "ambiguous_m", + content, + symbols, + "unknown.m"); + + Assert.DoesNotContain(references, reference => reference.SymbolName == "helper"); + } + + [Fact] + public void Extract_AmbiguousMRetainsSharedSafetyGuards_Issue4738() + { + var oversizeContent = new string('x', ChunkSplitter.MaxLineLength + 1) + " helper();"; + const string conflictContent = """ + function result = run() + <<<<<<< ours + helper(); + ======= + alternate(); + >>>>>>> theirs + end + """; + + var oversizeReferences = ReferenceExtractor.Extract( + 1, + "ambiguous_m", + oversizeContent, + [], + "oversize.m"); + var conflictReferences = ReferenceExtractor.Extract( + 1, + "ambiguous_m", + conflictContent, + [], + "conflict.m"); + + Assert.Empty(oversizeReferences); + Assert.Empty(conflictReferences); + } + + [Theory] + [InlineData("nim", "type Child = object of pkg.Base\n", "pkg")] + [InlineData("matlab", "classdef Child < pkg.Base\nend\n", "pkg")] + [InlineData("julia", "struct Child <: Pkg.Base\nend\n", "Pkg")] + [InlineData("d", "class Child : pkg.Base {}\n", "pkg")] + [InlineData("cython", "cdef class Child(pkg.Base):\n pass\n", "pkg")] + [InlineData("ada", "type Child is new Pkg.Base;\n", "Pkg")] + public void Extract_QualifiedScientificBaseTypesUseResolvableLeafNames_Issue4738( + string language, + string content, + string expectedQualifier) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var reference = Assert.Single( + ReferenceExtractor.Extract(1, language, content, symbols), + candidate => candidate.ReferenceKind == "type_reference"); + + Assert.Equal("Base", reference.SymbolName); + Assert.Equal(expectedQualifier, reference.TargetQualifier); + } + + [Theory] + [InlineData("d", "void run() { pkg.tools.flush(); }\n", "flush", "pkg.tools")] + [InlineData("ada", "procedure Run is begin Pkg.Tools.Flush(); end Run;\n", "Flush", "Pkg.Tools")] + [InlineData("nim", "proc run() = pkg.tools.flush()\n", "flush", "pkg.tools")] + [InlineData("julia", "function run()\n Pkg.Tools.flush()\nend\n", "flush", "Pkg.Tools")] + [InlineData("cython", "def run():\n pkg.tools.flush()\n", "flush", "pkg.tools")] + [InlineData("matlab", "function run()\n pkg.tools.flush();\nend\n", "flush", "pkg.tools")] + public void Extract_QualifiedScientificCallsPreserveTargetQualifiers_Issue4738( + string language, + string content, + string expectedName, + string expectedQualifier) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var reference = Assert.Single( + ReferenceExtractor.Extract(1, language, content, symbols), + candidate => candidate.SymbolName == expectedName + && candidate.ReferenceKind == "call"); + + Assert.Equal(expectedQualifier, reference.TargetQualifier); + } + + [Fact] + public void Extract_QualifiedJuliaDefinitionsDoNotEmitPhantomCalls_Issue4738() + { + const string content = """ + module Base + function Base.foo(x) + helper() + end + Base.bar(x) = helper2() + end + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.DoesNotContain(references, reference => + reference.SymbolName is "foo" or "bar" + && reference.ReferenceKind == "call"); + Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Single(references, reference => + reference.SymbolName == "helper2" && reference.ReferenceKind == "call"); + } + + [Theory] + [InlineData("objc")] + [InlineData("ambiguous_m")] + public void Extract_ObjectiveCQuotedImportsSurviveStringMasking_Issue4738(string language) + { + const string content = """ + /* + #import "Ignored.h" + */ + #import "Local/Header.h" + #include + """; + var symbols = SymbolExtractor.Extract(1, language, content, "sample.m"); + + var references = ReferenceExtractor.Extract(1, language, content, symbols, "sample.m"); + + Assert.Single(references, reference => + reference.SymbolName == "Local/Header.h" && reference.ReferenceKind == "import"); + Assert.Single(references, reference => + reference.SymbolName == "Framework/Header.h" && reference.ReferenceKind == "import"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "Ignored.h"); + } + + [Fact] + public void Extract_AdaBackslashesDoNotEscapeClosingStringQuotes_Issue4738() + { + const string content = """ + procedure Run is + begin + Put("C:\"); Helper; + end Run; + """; + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var references = ReferenceExtractor.Extract(1, "ada", content, symbols); + + Assert.Single(references, reference => + reference.SymbolName == "Helper" + && reference.ReferenceKind == "call" + && reference.ContainerName == "Run"); + } + + [Theory] + [InlineData( + """ + f(x) = helper( + nested(x) + ) + outside() + """)] + [InlineData( + """ + f(x) = + helper( + nested(x) + ) + outside() + """)] + public void Extract_JuliaShortFunctionMultilineCallsKeepNestedReferencesScoped_Issue4738( + string content) + { + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Equal("f", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.Equal("f", Assert.Single(references, reference => + reference.SymbolName == "nested" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_MatlabManyEndlessPeerFunctionsKeepIndependentLinearRanges_Issue4738() + { + const int functionCount = 2_048; + var content = new System.Text.StringBuilder(functionCount * 40); + for (var index = 0; index < functionCount; index++) + { + content.Append("function f") + .Append(index) + .Append("()\n helper") + .Append(index) + .Append("();\n"); + } + + var symbols = SymbolExtractor.Extract(1, "matlab", content.ToString()); + + Assert.Equal(functionCount, symbols.Count(symbol => symbol.Kind == "function")); + Assert.Equal(2, Assert.Single(symbols, symbol => symbol.Name == "f0").EndLine); + Assert.Equal(functionCount * 2 + 1, Assert.Single( + symbols, + symbol => symbol.Name == $"f{functionCount - 1}").EndLine); + } + + [Theory] + [InlineData( + "julia", + """ + function f() + f() + end + """, + "f")] + [InlineData( + "matlab", + """ + function f() + f(); + end + """, + "f")] + [InlineData( + "cython", + """ + def f(): + f() + """, + "f")] + [InlineData( + "nim", + """ + proc p() = + p() + """, + "p")] + [InlineData( + "d", + """ + id id() { + id(); + } + """, + "id")] + public void Extract_ScientificDeclarationIdentifierSpanDoesNotHideRecursiveCall_Issue4738( + string language, + string content, + string functionName) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + Assert.Single(references, reference => + reference.SymbolName == functionName && reference.ReferenceKind == "call"); + } + + [Fact] + public void Extract_JuliaMultilineComprehensionDoesNotExtendFunctionRange_Issue4738() + { + const string content = """ + function run(xs) + values = [ + helper(x) + for x in xs + if x > 0 + ] + after() + end + outside() + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Equal(8, Assert.Single(symbols, symbol => symbol.Name == "run").EndLine); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "after" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_JuliaBangCallablesAndQualifiedMacrosPreserveGraphIdentity_Issue4738() + { + const string content = """ + function mutate!(value) + value + end + push!(values, value) = values + function run(values) + A.mutate!(values) + A.push!.(values) + A.@trace values + end + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Contains(symbols, symbol => symbol.Name == "mutate!"); + Assert.Contains(symbols, symbol => symbol.Name == "push!"); + foreach (var name in new[] { "mutate!", "push!", "trace" }) + { + var reference = Assert.Single(references, candidate => + candidate.SymbolName == name && candidate.ReferenceKind == "call"); + Assert.Equal("A", reference.TargetQualifier); + Assert.Equal("run", reference.ContainerName); + } + } + + public static TheoryData ScientificNativeFollowingStringReferenceCases => new() + { + { + "cython", + """def run(): consume("a very long string", pkg.helper())""" + }, + { + "d", + """void run() { consume("a very long string", pkg.helper()); }""" + }, + { + "nim", + """proc run() = consume("a very long string", pkg.helper())""" + }, + { + "objc", + """void run() { consume(@"a very long string", pkg.helper()); }""" + }, + }; + + [Theory] + [MemberData(nameof(ScientificNativeFollowingStringReferenceCases))] + public void Extract_ScientificNativeStringsPreserveFollowingReferenceColumns_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + var helper = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Equal("pkg", helper.TargetQualifier); + Assert.Equal(content.IndexOf("helper", StringComparison.Ordinal) + 1, helper.Column); + } + + [Fact] + public void Extract_MatlabContinuationTailDoesNotEmitCalls_Issue4738() + { + const string content = """ + function run(value) + first(value, ... ignoredCall() + value); + realCall(); + end + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + Assert.Single(references, reference => + reference.SymbolName == "first" && reference.ReferenceKind == "call"); + Assert.Single(references, reference => + reference.SymbolName == "realCall" && reference.ReferenceKind == "call"); + Assert.DoesNotContain(references, reference => reference.SymbolName == "ignoredCall"); + } + + [Fact] + public void Extract_DCompileTimeConstructsDoNotEmitCalls_Issue4738() + { + const string content = """ + extern(C) void exported() { + static if (is(typeof(value) == int)) { + pragma(msg, __traits(compiles, helper())); + } + realCall(); + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + foreach (var name in new[] { "extern", "is", "pragma", "__traits" }) + { + Assert.DoesNotContain(references, reference => + reference.SymbolName == name && reference.ReferenceKind == "call"); + } + Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Single(references, reference => + reference.SymbolName == "realCall" && reference.ReferenceKind == "call"); + } + + [Fact] + public void Extract_MatlabArgumentsBlockHeaderDoesNotEmitCall_Issue4738() + { + const string content = """ + function run(values) + arguments (Repeating) + values + end + helper(); + end + """; + var symbols = SymbolExtractor.Extract(1, "matlab", content); + + var references = ReferenceExtractor.Extract(1, "matlab", content, symbols); + + Assert.DoesNotContain(references, reference => + reference.SymbolName == "arguments" && reference.ReferenceKind == "call"); + Assert.Equal("run", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + } + + [Theory] + [InlineData( + "cython", + """ + cdef class Worker: + def run(self): + self.helper() + """)] + [InlineData( + "d", + """ + class Worker { + void run() { + this.helper(); + } + } + """)] + public void Extract_InstanceReceiversUseCurrentContainerResolution_Issue4738( + string language, + string content) + { + var symbols = SymbolExtractor.Extract(1, language, content); + + var references = ReferenceExtractor.Extract(1, language, content, symbols); + + var helper = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Equal("run", helper.ContainerName); + Assert.Null(helper.TargetQualifier); + } + + [Fact] + public void Extract_AdaNestedEndsPreserveOuterProcedureScope_Issue4738() + { + const string content = """ + procedure Run is + begin + if Ready then + First; + end if; + Later; + end Run; + """; + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var references = ReferenceExtractor.Extract(1, "ada", content, symbols); + + Assert.Equal(7, Assert.Single(symbols, symbol => symbol.Name == "Run").EndLine); + foreach (var name in new[] { "First", "Later" }) + { + Assert.Equal("Run", Assert.Single(references, reference => + reference.SymbolName == name && reference.ReferenceKind == "call").ContainerName); + } + } + + [Fact] + public void Extract_CythonFStringExpressionsEmitCalls_Issue4738() + { + const string content = """ + def run(): + value = f"{pkg.helper()}" + """; + var symbols = SymbolExtractor.Extract(1, "cython", content); + + var references = ReferenceExtractor.Extract(1, "cython", content, symbols); + + var helper = Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call"); + Assert.Equal("run", helper.ContainerName); + Assert.Equal("pkg", helper.TargetQualifier); + Assert.Equal(20, helper.Column); + } + + [Fact] + public void Extract_JuliaWhereShortFunctionOwnsRhsCalls_Issue4738() + { + const string content = """ + f(x::T) where {T} = helper(x) + outside() + """; + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Equal(1, Assert.Single(symbols, symbol => symbol.Name == "f").EndLine); + Assert.Equal("f", Assert.Single(references, reference => + reference.SymbolName == "helper" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + Assert.DoesNotContain(references, reference => + reference.SymbolName == "f" && reference.ReferenceKind == "call"); + } + + [Theory] + [InlineData( + """ + function f(A) + value = A[begin] + end + outside() + """)] + [InlineData( + """ + f(A) = A[begin] + outside() + """)] + public void Extract_JuliaIndexBeginDoesNotExtendFunctionRange_Issue4738(string content) + { + var symbols = SymbolExtractor.Extract(1, "julia", content); + + var references = ReferenceExtractor.Extract(1, "julia", content, symbols); + + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_DSuperReceiverDoesNotBecomeCurrentContainerSelfCall_Issue4738() + { + const string content = """ + class Base { + void run() { } + } + class Child : Base { + override void run() { super.run(); } + } + """; + var symbols = SymbolExtractor.Extract(1, "d", content); + + var references = ReferenceExtractor.Extract(1, "d", content, symbols); + + var call = Assert.Single(references, reference => + reference.SymbolName == "run" && reference.ReferenceKind == "call"); + Assert.Equal("super", call.TargetQualifier); + Assert.False(call.IsSelfReference); + } + + [Fact] + public void Extract_AdaUnnamedOuterEndTracksNestedDeclareBlock_Issue4738() + { + const string content = """ + procedure Outer is + begin + declare + begin + Inner; + end; + Later; + end; + outside; + """; + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var references = ReferenceExtractor.Extract(1, "ada", content, symbols); + + Assert.Equal(8, Assert.Single(symbols, symbol => symbol.Name == "Outer").EndLine); + Assert.Equal("Outer", Assert.Single(references, reference => + reference.SymbolName == "Later" && reference.ReferenceKind == "call").ContainerName); + Assert.Null(Assert.Single(references, reference => + reference.SymbolName == "outside" && reference.ReferenceKind == "call").ContainerName); + } + + [Fact] + public void Extract_AdaForwardDeclarationDoesNotBorrowLaterBodyRange_Issue4738() + { + const string content = """ + procedure Inner; + procedure Inner is + begin + Helper; + end Inner; + """; + + var symbols = SymbolExtractor.Extract(1, "ada", content); + + var declarations = symbols.Where(symbol => symbol.Name == "Inner").ToList(); + Assert.Equal(2, declarations.Count); + Assert.Equal(1, declarations[0].EndLine); + Assert.Null(declarations[0].BodyStartLine); + Assert.Equal(5, declarations[1].EndLine); + Assert.Equal(3, declarations[1].BodyStartLine); + } +} diff --git a/tests/CodeIndex.Tests/SymbolExtractorTests.cs b/tests/CodeIndex.Tests/SymbolExtractorTests.cs index 4054e6a9e..3777c609b 100644 --- a/tests/CodeIndex.Tests/SymbolExtractorTests.cs +++ b/tests/CodeIndex.Tests/SymbolExtractorTests.cs @@ -12522,6 +12522,14 @@ public void GetContractVersion_LanguageSpecificKinds_UseDedicatedVersions() Assert.True(SymbolExtractor.StyleAndXamlContractVersion > SymbolExtractor.DefaultContractVersion); Assert.Equal(SymbolExtractor.XmlContractVersion, SymbolExtractor.GetContractVersion("xml")); Assert.True(SymbolExtractor.XmlContractVersion > SymbolExtractor.StyleAndXamlContractVersion); + foreach (var language in new[] { "ada", "ambiguous_m", "cython", "d", "julia", "matlab", "nim", "objc" }) + { + Assert.Equal( + SymbolExtractor.ScientificNativeGraphContractVersion, + SymbolExtractor.GetContractVersion(language)); + } + Assert.True(SymbolExtractor.ScientificNativeGraphContractVersion > SymbolExtractor.DynamicLanguageContractVersion); + Assert.True(SymbolExtractor.ScientificNativeGraphContractVersion > SymbolExtractor.SystemsLanguageContractVersion); } [Fact] diff --git a/tests/CodeIndex.Tests/golden/status.json b/tests/CodeIndex.Tests/golden/status.json index 32f07b18f..902c009bf 100644 --- a/tests/CodeIndex.Tests/golden/status.json +++ b/tests/CodeIndex.Tests/golden/status.json @@ -44,6 +44,8 @@ } }, "graph_supported_languages": [ + "ada", + "ambiguous_m", "app_manifest", "assembly", "batch", @@ -59,6 +61,8 @@ "cshtml", "css", "cuda", + "cython", + "d", "dart", "dependency_lock", "dependency_manifest", @@ -82,13 +86,16 @@ "javascript", "json", "jsonl", + "julia", "justfile", "kotlin", "lua", "makefile", "markdown", + "matlab", "metal", "msbuild", + "nim", "objc", "ocaml", "pascal",